agourlay / cornichon Goto Github PK
View Code? Open in Web Editor NEWTesting tool in Scala for HTTP JSON API
Home Page: http://agourlay.github.io/cornichon
License: Apache License 2.0
Testing tool in Scala for HTTP JSON API
Home Page: http://agourlay.github.io/cornichon
License: Apache License 2.0
Can be used to fail faster.
And I create_user
Then assert status_is(201)
The status assertion can be removed by adding the expects
arg. to the POST.
def create_user = effectful(
title = s"create a user",
effect =
s ⇒
http.Post(
url = "/users",
payload = ???,
params = Seq.empty,
headers = Seq.empty,
expects = "201"
)(s)
)
No one wants to see placeholders in steps logs.
Propose a DSL block that generates a standard CRUD test from:
It could hold previous values and offer an API to retrieve them.
<name>
takes the last element<name[0]>
takes the head<name[1]>
takes the second element in the listNew values would be appended to the tail.
Use json4s everywhere.
Having all the doc inside the readme file is becoming cumbersome.
Investigate sbt-microsite https://github.com/47deg/sbt-microsites
The engine should expose a clear API enabling users to definer their own steps (regular and wrapper).
Offer better typed JsonPath using Dynamic Trait
root.Episodes(0).Released
instead of "Episodes[0].Released"
Equivalent of a nested filter on JArray.
"remove field in each element of an array" in {
val p1 = ("name" → "bob") ~ ("age", 50)
val p2 = ("name" → "jim") ~ ("age", 40)
val p3 = ("name" → "john") ~ ("age", 30)
val input = JArray(List(p1, p2, p3))
val expected = JArray(List(JObject(JField("name", "bob"), JField("name", "jim"), JField("name", "john"))))
removeFieldsByPath(input, Seq(JsonPath.root.age)) should be(expected)
}
Really usefull to finally ignore the imdbRating of all episodes in the OpenMovieDatabase
integration test.
Make sure that all error messages contain a formatted JSON output.
For instance status_is
shows the JSON inlined
When using a json path like this:
Then assert body.path("masterData / current / masterVariant / availability")
the error message is not very helpful:
[info] DeferredAbortedSuite:
[info] Exception encountered when attempting to run a suite with class name: org.scalatest.DeferredAbortedSuite *** ABORTED ***
[info] com.github.agourlay.cornichon.json.JsonPathParsingError:
[info] ...
[info] ScalaCheck
[info] Passed: Total 0, Failed 0, Errors 0, Passed 0
Will test the JSON equality for the two bodies and display a nice diff in the error.
When I get("/superheroes/Batman")
When I get("/superheroes/batman")
Then assert lastBodies(2).ignoring(root.name).areEquals
It could potentially accept 'n' requests.
It is quite noisy, easy to forget and scary for non Scala dev
The repeat block does not appear in the execution log.
The current implementation is a bit hacky and just repeats a seq of Steps.
Ideally the Repeat block should be clearly visible in the logs with a proper identation.
Currently Gql JSON can only be used within the body assertion.
It should be possible to use also in the payload of the HTTP effects steps.
Currently it is necessary to provide a full mapping for a custom mapper.
view-key
+ key-in-session
+ extract-fct
It creates a unique binding.
What if most of the object in session have the same shape, you would still have to create all bindings.
The goal of this issue is to design a new custom mapper that accepts generic/regex like binding.
*-id -> * -> v \ id
Debug steps should not require a specific keyword.
It is already possible to ignore Features and Scenarios by passing ignore = true
to the DSL builder.
It would be nice to do it just with an annotation as well.
class CornichonExamplesSpec extends CornichonFeature {
lazy val feature =
@Ignore
Feature("Cornichon feature example") {
@Ignore
Scenario("demonstrate CRUD features") {
...
}
Each step should log it's execution time
When I Get /superheroes/random ( executed in 233 ms )
Current implem. should be improved...
https://github.com/travisbrown/circe
Maybe it has everything necessary to replace the mix of Json4s and SprayJson
When two scenarios have the same name, the error is not very helpful:
[info] DeferredAbortedSuite:
[info] Exception encountered when attempting to run a suite with class name: org.scalatest.DeferredAbortedSuite *** ABORTED ***
[info] Exception encountered when attempting to run a suite with class name: org.scalatest.DeferredAbortedSuite (ScalatestIntegration.scala:28)
Say we have a endpoint to create a user with an email respecting the constraint that all email must be unique.
Given I Post("/user", "{ "email" : "<random-string>"}
<random-string>
is a builtin markup in the resolver machanism.
Now what happens if we write
Concurrently(3, timeout){
Given I Post("/user", "{ "email" : "<random-string>"}
}
A user could expect different behaviours according to the semantic of the random values.
Used to check that only one call should pass, in our case testing that the the email constraint is applied concurrently.
Used to verify that an endpoint behaves correctly under load.
The current implementation uses identical random values accross concurrent executions:
In this case a proper CornichonError
should be returned to the user and not a technical exception Future Timeout
.
exception thrown 'java.util.concurrent.TimeoutException: Futures timed out after [2000 milliseconds]
[info] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
[info] at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
[info] at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
[info] at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread$$anon$3.block(ThreadPoolBuilder.scala:169)
[info] at scala.concurrent.forkjoin.ForkJoinPool.managedBlock(ForkJoinPool.java:3640)
[info] at akka.dispatch.MonitorableThreadFactory$AkkaForkJoinWorkerThread.blockOn(ThreadPoolBuilder.scala:167)
[info] at scala.concurrent.Await$.result(package.scala:190)
[info] at com.github.agourlay.cornichon.http.HttpService$$anonfun$withPayload$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(HttpService.scala:34)
[info] at com.github.agourlay.cornichon.http.HttpService$$anonfun$withPayload$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(HttpService.scala:33)
[info] at cats.data.Xor.flatMap(Xor.scala:98)
[info] at com.github.agourlay.cornichon.http.HttpService$$anonfun$withPayload$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3$$anonfun$apply$4$$anonfun$apply$5.apply(HttpService.scala:33)
[info] at com.github.agourlay.cornichon.http.HttpService$$anonfun$withPayload$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3$$anonfun$apply$4$$anonfun$apply$5.apply(HttpService.scala:32)
The number of spaces in hardcoded in the log statement.
It should be generated depending of the level of nesting.
def log(input: String, depth: Int)
Make it possible to feed a Scalacheck Generator to a custom mapper.
"user-name" → GenMapper(Gen.alphaStr)
The usage of specific keyword between Given/And/Then and the step definition is not clear.
Currently cornichon offers I
, a
, assert
.
Those can't be combined and have side effects on the steps title.
The main usage for those keywords is actually to make a clean "infix notation" possible.
A better alternative would be to have as a minimal step definition Given/And/Then directly followed by a step definition.
Linking words could be added in between if desired.
For instance
And assert response body with transformation is '<film-title>'
The transformation should be explained in terms of JsonPath for more clarity.
Fix this asap, or you will have no chance
It should print as well the inital array on which the transformation was applied.
The current reporting is not really helpful while debugging
Then assert response body 'results' array size is '1' *** FAILED ***
expected result was:
'true'
but actual result is:
'false'
beforeEachScenario
and afterEachScenario
are treated as regular steps, this means if it will simply exit at the first failed step without trying to run the cleanup steps in afterEachScenario
.
It should be possible to run a single scenario within a feature from the CLI.
Split the RunnableStep
concept into assertStep
and effectStep
@agourlay, thank's for putting this together! You have asked for feedback, so here we go:
FeatureSpec
) in combination with the akka-http-testkitThis test fails
"do not trip on duplicate" in {
val input: JValue =
("name" → "bob") ~
("age", 50) ~
("brother" →
(("name" → "john") ~ ("age", 40))) ~
("friend" →
(("name" → "john") ~ ("age", 30)))
val expected = ("name" → "bob") ~ ("age", 50) ~ ("brother" → ("age", 40)) ~ ("friend" → (("name" → "john") ~ ("age", 30)))
removeFieldsByPath(input, Seq("brother.name")) should be(expected)```
Duplicate nested fields confuse removeFieldsByPath
as it is not really exploring the jsonPath but merely relying on content equality.
Making sure those examples compile is cumbersome.
Integrate with Tut https://github.com/tpolecat/tut
The error reporting of the HttpService is not proper.
When the underlying call times out an expection is thrown and bubble up in the Engine.
exception thrown 'java.util.concurrent.TimeoutException: Futures timed out after [4 seconds]
[info] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
[info] at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
[info] at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
[info] at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
[info] at scala.concurrent.Await$.result(package.scala:190)
[info] at com.github.agourlay.cornichon.http.HttpService$$anonfun$withoutPayload$1$$anonfun$apply$9$$anonfun$apply$10$$anonfun$apply$11$$anonfun$apply$12.apply(HttpService.scala:47)
[info] at com.github.agourlay.cornichon.http.HttpService$$anonfun$withoutPayload$1$$anonfun$apply$9$$anonfun$apply$10$$anonfun$apply$11$$anonfun$apply$12.apply(HttpService.scala:46)
[info] at cats.data.Xor.flatMap(Xor.scala:98)
Use proper HttpCornichonError
instead with explicit error message.
Provide hooks offering a subscription to Feature's
results to enable users to build specific post processing action.
save_body_key("addresses[0].id" -> "customer-address-id")
does not work, as the JsonPath "addresses[0].id"
is not properly evaluated.
(FYI: body.path("addresses[0].id").is("test")
does work)
Offer a RepeatDuring bloc that accepts a duration as an argument and repeat the execution of the inner steps during the specified time.
RepeatDuring(3.seconds) {
When I get("/superheroes/Batman")
Then assert status.is(200)
}
The key must be there but the value can be discarded.
Parse String input to JSON at compile time using a macro to fail faster.
The input String is likely to contain placeholders so it can not be a simple JSON parse
.
exemple of invalid JSON prior placeholder resolution:
{
"a number" : <number-placeholder>
}
The ignoring param in the DSL should allow to ignore nested key using .
e.g. ignoring = "company", "employee.adress"
JSON is cumbersome to write, it would be interesting to also support the GraphQL query AST as input.
{
id: 1
name: "door"
items: [
# pretty broken door
{state: Open, durability: 0.1465645654675762354763254763343243242}
null
{state: Open, durability: 0.5, foo: null}
]
}
ref sangria-graphql/sangria@1998e0f#diff-cf4a54c5a5ef49edc6332de99a35377aR34
Currently the wrapper steps like Eventually
or RetryMax
causing series of steps being replayed upon errors are displaying all the logs from failed run which makes debugging difficult.
Ideally only the first error, the last error and the number of errors in between should be reported.
Metaspace is never garbage collected.
With Scalatest 2.2.6 the scenarios seem to be executed in // within a feature.
It breaks CornichonExamplesSpec
as the state on the server is globally shared.
Then assert
is redundant.
The usage of Then
should be enough to infer that an assertion follows.
It should look more natural, something like:
Then X is Y
where X is a generic extraction and Y a value.
e.g Then status is 200
Requires chaining of infix notation : obj method arg method arg method arg ...
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.