I wrote about this feature: https://medium.com/@matryer/new-variables-feature-for-silk-4c84ac6d3d83#.icqbqdqts

from silk.

Wanted to add my thoughts to this as I would like to use this in the future, but without auth and variables, it's probably won't suffice for any real world apis.

I've used other api testing solutions, and most of them seem to not make a distinction between auth and using returned data. I think this is a mistake, indeed you can solve two features with the same implementation but from a consumer standpoint it leaves the user usually having to duplicate and convolute each test with auth setup.

I think a good approach would be to treat them as two different things. Seeing as auth is pretty given functionality for an api, tackling it in isolation would allow a more appropriate solution I believe.

What about making auth a first class concept, perhaps a new section, tag or separate markdown file entirely, with configuration to be either per test or per "test group" as @timchunght suggested. That way in the generated documentation it would be possible to output auth info once for all tests, at the start of a "test group" or per test. That way, providing a means to prevent duplication.

Looking forward to seeing Silk gain some traction!

from silk.

Guys, please test the new variables feature: https://github.com/matryer/silk#capturing-data-beta

from silk.

Did you guys get chance to test this feature? @Dashue @timchunght @grounded042 @mattdotmatt @butsjoh @neocortical

from silk.

Is there a way to name groups in Regex?

from silk.

or else, it could be done with comments:

* Data.id = /.?/ // this is the {id}

• By mentioning {id} in a comment, we could store the actual value in that variable

from silk.

The only open issue is the issue I want. That makes me happy. :)

matryer's use case is a good one. Mine is slightly different -- I'd like to bind variables from an authentication request (i.e. an auth token) to use in subsequent authed-only requests.

I like the last idea of using comments to bind variables.

from silk.

Maybe consider using some sort of environment file as well (like postman does) which allows you to define global variables which you could use. I think there is a big difference in making a tool that checks data on an existing API (mostly GET and format testing) and one that is able to create an verify that created information (POST, PUT). Somehow you always need to setup an environment where you can run your tests against. I don't think this tool needs to be made so complex to support all of it, but that is up to the maintainer of course.

from silk.

Yes, environment vars etc. as well as some way to capture data within tests is definitely needed. The hard part is figuring out how to make that look nice for documentation, as well as functional for the test itself.

from silk.

Does this mean that the ordering of the tests become important? If test 3 relies on a value returned by test 1 then test 1 would need to be guaranteed to execute before test 3.

If so then the individual tests have become part of a bigger workflow which might need to be managed.

from silk.

I also have use cases where this is needed. My initial thought is declaring the variable using something along the lines of response.jwttoken from GET /login or something of that sort. The first part is like nested json and could potentially mention more data such as header, request, status, etc. However, that is just the initial thought. There could be some problems with that such as what if you declare more than one GET /login?

from silk.

Same, my use case is pretty much what @matryer Mat described, the POST creates an object and returns the id and I want to use that id to test another GET endpoint. In addition, I think this feature will be useful to test an API with OAuth features or any other means of authentication like JWT. Hope this will be added soon. Thanks and good work!

from silk.

Ok this is the next thing we should solve.

Let's all brainstorm implementation ideas.

The only rule of Silk club: It must look good as documentation and be readable to human.

I've definitely used {id} in docs before.

Mat

On 1 Mar 2016, at 12:58, timchunght [email protected] wrote:

Same, my use case is pretty much what @matryer Mat described, the POST creates an object and returns the id and I want to use that id to test another GET endpoint. In addition, I think this feature will be useful to test an API with OAuth features or any other means of authentication like JWT. Hope this will be added soon. Thanks and good work!

—
Reply to this email directly or view it on GitHub.

from silk.

@matryer I think it will also be great to have the ability to group tests into a context since you are testing related features. In regards to markdown syntax, I propose the following three options:

Option 1:

* id = Data.id: /.?/

Option 2:

* var id = Data.id: /.?/

Option 3:

* {id} = Data.id: /.?/

and the variable name must match the variable name used in the following test

GET /objects/{id}

These are some of the most ideal syntax I am thinking about. Of course, the context is not completely necessary but I think it might be a good feature for v2. For v1, it will be nice to achieve just the variable reuse. Please feel free to tell me what you think. Thank you.

PS. I really hope we can get this feature done soon since I have some API tests to write and they all rely on each other. I am quite reluctant to write them in ginkgo but would really love to use Silk as both tests and documentation for the API.

Tim

from silk.

Starting work on the data capture piece of this, I'm going with this:

* Data.id: /.?/ // The unique {id} 

Mention of the {id} in the comment on the line will 'capture' the value as a variable. Then future mentions (outside of comments) will insert the value in place of the placeholder. Mentions of non-set variables will be an error.

Started in issue-2 branch.

from silk.

I'm using it to document/test an api I have written for tech tests. It's working great so far - only used {id} Will carry on tonight and push up my specs and share the repo.

from silk.

Like it.

Works well for me in https://github.com/mattdotmatt/moodicle/tree/master/specs.

2-planets-post.md gets the id and checks the POST succeeded with a GET.
3-planets-delete.md then uses the same id and deletes it.

(NB: This implies a workflow e.g. POST needs to occur before DELETE. To ensure this ordering I've numbered the filenames so they start 1-, 2-, 3-)

from silk.

@matryer I REALLY REALLY LOVE IT SO FAR! I recently implemented the Parse.com API in Golang (I didn't like the nodejs version they open sourced as a result of their shutdown in 2017). https://parse.com/docs/rest/guide#objects-creating-objects.

I was actually going to go through the pain of writing all the code in ginkgo again but am really happy you updated the package.

Create Objects /classes/:objectName

POST /classes/HelloWorld

• Content-Type: "application/json"
• X-Another-Header: "value"

{"name":"Silk","status":"awesome","a_bool":true,"nothing":null,"release_year":2016}

Response

• Status: 201
• Data.objectId: /(\S){10}/ // The object's unique {objectId}.

GET /users

GET /classes/HelloWorld/{objectId}

• Content-Type: "application/json"
• X-Another-Header: "value"

Response

• Status: 200
• Data.objectId: {objectId}

Sample test md file. It is incredibly easy to write and read. Thanks for the good work!

from silk.

Excellent. Thanks for the feedback. Let me know anything else you run into.

Sent from my iPhone

On 23 Apr 2016, at 08:18, Timothy Chung [email protected] wrote:

@matryer I REALLY REALLY LOVE IT SO FAR! I recently implemented the Parse.com API in Golang (I didn't like the nodejs version they open sourced as a result of their shutdown in 2017). https://parse.com/docs/rest/guide#objects-creating-objects.

I was actually going to go through the pain of writing all the code in ginkgo again but am really happy you updated the package.

Create Objects /classes/:objectName

POST /classes/HelloWorld

Content-Type: "application/json"
X-Another-Header: "value"
{"name":"Silk","status":"awesome","a_bool":true,"nothing":null,"release_year":2016}
Response

Status: 201
Data.objectId: /(\S){10}/ // The object's unique {objectId}.
GET /users

GET /classes/HelloWorld/{objectId}

Content-Type: "application/json"
X-Another-Header: "value"
Response

Status: 200
Data.objectId: {objectId}
Sample test md file. It is incredibly easy to write and read.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

from silk.

@matryer I'm trying to use the environment variables feature
Does it work only from code (including it as os.Setenv("$AppNameFromEnv", "Silk")) or can we use it with the exec silk?

from silk.

It should work with normal environment variables too. Haven’t test it, but let me know.

On 9 May 2016, at 14:48, rafavisa [email protected] wrote:

@matryer https://github.com/matryer I'm trying to use the environment variables feature
Does it work only from code (including it as os.Setenv("$AppNameFromEnv", "Silk")) or can we use it with the exec silk?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub #2 (comment)

from silk.

@matryer I’ve might done something wrong then

• I’ve build last master change with build.sh script (Version 0.4.1 - 5ddd898)
• Created hello.silk.md from example https://medium.com/@matryer/introducing-silk-markdown-driven-api-tests-1f8cfb0ef99a#.yqweysf3u
• Run it successfully from terminal with ./silk -test.v -silk.url="http://outlearn-hello.appspot.com" hello.silk.md
• Define env var NAME=Mat
• Changed "name" param value from Mat to {$NAME} in hello.silk.md
• And running ./silk -test.v -silk.url="http://outlearn-hello.appspot.com" hello.silk.md

GET http://outlearn-hello.appspot.com/hello
  name: "{$NAME}"
Hello {$NAME}.

body expected:  
Hello Mat.  

actual:  
Hello {$NAME}.  
 hello.silk.md:9 - body doesn't match  
FAIL

Hello API

GET /hello

Gets a personalised greeting.

* ?name={$NAME} // The name of the person to greet

• Status: 200
• Content-Type: text/html; charset=utf-8

Hello Mat.

from silk.

@rafavisa can you open a new issue as a bug with that exact issue? It seems like you've found a place where the {...} format isn't implemented.

Or better still, can you add a PR with that as a failing test.

from silk.

Could the problem be running the test using the silk exe file instead of running it with run_test.go ?

It was my first guess as I've also tried using the environment variable within the request body instead of the request parameters (as you are doing in captured-vars.silk.md) and neither that worked for me.

from silk.

Shouldn't matter re run_test.go vs the command. The {...} thing was implemented 'manually', so some places may have been missed.

from silk.

@matryer I created a PR using environment vars in path, body and parameters #42

PR passed checks so I don't really know if the issue is where the env var is used.

from silk.