matryer / silk Goto Github PK

Hey found silk quite interesting! I'm looking for to give a try in my next microservice project.

By the way while reading i was thinking in postman as far i know several people using for QA and test (not automated) purpose.

So i was think if make sense to add support for that in this project, if is that something that you are willing for to have into core, or might be some a part, a command that we run pointing the markdown that generates that?

I'm asking because i was thinking in give a try and implement that. 🍻

Silk shouldn't necessarily follow redirects - after all, it's an API testing framework and redirects are part of the API.

Possibly make it false by default, but with an option:

runner.FollowRedirects = true

• see http://stackoverflow.com/questions/23297520/how-can-i-make-the-go-http-client-not-follow-redirects-automatically

Request bodies for next request sometimes interpreted as response assertions for previous request.

For a large bodies it'd be much user-friendlier to support formatted JSON in the body assert field. Currently if I have an assertion:

===

'''
{
  "resource": "api_status",
  "body": {
    "blah": "blah"
  }
}
'''

and my API returns {"resource":"api_status","body":{"blah":"blah"}}, I get "body doesn't match" error.

https://github.com/oren/mdd/blob/master/api.md

the above server sends 'ok' in the body but the test fails. any idea why?

It would be nice to have the ability to assert response bodies based on embedded JSONSchema definitions in the same Markdown document or linking to the file system file definition.

Hey!

Silk looks interesting! How does it handle dependent api requests, and cookie based authentication?

Eg a user logs in, then CRUDs a todo?

Hi,
First of all I'd like to thank you for such a useful library. It's both simple to use and pretty powerful.
I want to report an issue I've been facing using Silk. I've discovered it adding a property to the JSON I'm expecting in the response.
Here is the JSON which works:

{
    "is_complete": true,
    "user": {
        "id": 1,
        "name": "Juan Carlos",
        "email": "[email protected]",
        "phone": "541122334455",
        "gender": "male",
        "gender_of_interest": "women",
        "birthday": "1990-02-15T00:00:00Z",
        "height": 170,
        "job_title": "Software Engineer",
        "company": "Whim",
        "school": "UADE",
        "degree": {
            "id": 3,
            "description": "Bachelor's"
        },
        "descriptors": [
            {
                "id": 5,
                "description": "Sarcastic"
            },
            {
                "id": 10,
                "description": "Night owl"
            }
        ],
        "created_at": "2016-02-26T14:00:00Z",
        "updated_at": "2016-03-16T12:00:00Z"
    }
}

And here is the JSON which breaks my tests (notice the ethnicities array was added):

{
    "is_complete": true,
    "user": {
        "id": 1,
        "name": "Juan Carlos",
        "email": "[email protected]",
        "phone": "541122334455",
        "gender": "male",
        "gender_of_interest": "women",
        "birthday": "1990-02-15T00:00:00Z",
        "height": 170,
        "job_title": "Software Engineer",
        "company": "Whim",
        "school": "UADE",
        "degree": {
            "id": 3,
            "description": "Bachelor's"
        },
        "descriptors": [
            {
                "id": 5,
                "description": "Sarcastic"
            },
            {
                "id": 10,
                "description": "Night owl"
            }
        ],
        "ethnicities": [
            {
                "id": 1,
                "description": "Asian"
            },
            {
                "id": 3,
                "description": "Latino"
            }
        ],
        "created_at": "2016-02-26T14:00:00Z",
        "updated_at": "2016-03-16T12:00:00Z"
    }
}

The error that I get is the following:

[GIN] 2016/05/04 - 15:46:38 | 404 |       2.599µs | 127.0.0.1 |   iJS     /
body expected:  

{
    "is_complete": true,
    "user": {
        "id": 1,
        "name": "Juan Carlos",
        "email": "[email protected]",
        "phone": "541122334455",
        "gender": "male",
        "gender_of_interest": "women",
        "birthday": "1990-02-15T00:00:00Z",
        "height": 170,
        "job_title": "Software Engineer",
        "company": "Whim",
        "school": "UADE",
        "degree": {
            "id": 3,
            "description": "Bachelor's"
        },
        "descriptors": [
            {
                "id": 5,
                "description": "Sarcastic"
            },
            {
                "id": 10,
                "description": "Night owl"
            }
        ],
        "ethnicities": [
            {
                "id": 1,
                "description": "Asian"
            },
            {
                "id": 3,
                "description": "Latino"
            }
        ],
        "created_at": "2016-02-26T14:00:00Z",
        "updated_at": "2016-03-16T12:00:00Z"
    }
}  


actual:  

404 page not found  

--- FAIL: iJS zUxMiIsInR5c 
 users.md:22 - body doesn't match  
--- FAIL: TestUsers (0.00s)
FAIL
coverage: 0.0% of statements

Thanks in advance.
JC

Is there any length limit for the .md file? Or any limitations in the number of asserts?

I am having some test failures due to .md parsing.
invalid request: net/http: invalid method "a@st"
** This is not what the .md files contains.

I can reproduce it by replicating a simple test that pass in the same file for several times. When I try to run exactly the same test for 6-7 times.

Any known issue?

Thanks

Back tics should be ignored so they can be used to format things nicely in docs.

Just curious to what extent silk's syntax and conventions are similar and/or compatible with blueprint? I ask, because blueprint is a pretty established standard for documenting web APIs (along w/ Swagger and RAML, all of which are interconvertible) and there's already a lot of nice tooling around it (e.g., test2doc).

If different, is it possible to convert silk-based api docs into blueprints or other api-spec formats?

Comments (anything after // should be ignored)

When I'm running silk against certificate signed by untrusted root, I got the following error.

> silk -silk.url="https://untrusted-root.badssl.com/" ./**/*.silk.md
silk: running 1 file(s)...
x509: certificate signed by unknown authority
--- FAIL: silk (0.67s)
FAIL

I tried to find a way to skip certification checking in help but couldn't find it. (like option -k in curl)

 -k, --insecure      Allow connections to SSL sites without certs (H)

I think the error was caused by http.DefaultTransport used in Runner.DoRequest. It seems to me that DefaultTransport does not allow a connection to insecure SSL sites.

I found a way to skip it here https://groups.google.com/d/msg/golang-nuts/v5ShM8R7Tdc/I2wyTy1o118J

I'm thinking of adding another option in silk to allow connections to SSL sites without certs. Any idea what the option should be? Is -k ok?

Hello.

Couldn't find any information about this feature. Is it possible to define dynamic variables (i.e. random generated ones) in a request?

Thanks.

As a Tester
I want to Exclude variable data (timestamps, id fields etc.)
So that I can validate things that matter

Proposed feature

Designate a pattern that will be used to "erase" values from both the expected data, and the response.

Possible solutions

Use a special character field / operator to indicate the exclusion, in this example the prefix ! designates a type of expression being provided.

Text Patterns
Substitute using sed style replace, mostly for text.
!s: s/ABCDE/XXXXX/g

Remove data matching regex, in this case a valid email address. Any value matching the regex would be replaced with nothing.

!r: (?:[a-z0-9!#$%&'+/=?^{|}~-]+(?:.[a-z0-9!#$%&'_+/=?^_{|}~-]+)|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|[\x01-\x09\x0b\x0c\x0e-\x7f])")@(?:(?:a-z0-9?.)+a-z0-9?|[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|[\x01-\x09\x0b\x0c\x0e-\x7f])+)])

JSON Patterns
Replace a field's contents with a dummy value.
!_id: 123

Exclude field from json, the field "_id" field and value would be removed prior to compare.
!x: _id

Support multipart data and uploading file as a form parameter in POST and PUT methods.
There can be a folder that contains all the files that will be uploaded while testing, this path can be set as an argument while running silk.
The names of the file mentioned in the markdown could be relative to this folder. We can add a comment after the field that the filed is a file field and it's value is a file path.

name: Mat
CV: CV/mat-cv.pdf // file

If I create a thing:

   ## POST /item

and it creates an ID for me:

     * Data.id = /.?/

I might need to be able to use that ID in another request later:

   ## GET /item/{id}

Support for lines in the way of --- instead of === to split request/response

Original example is

# Forgot your password

## POST /users

Create a new user.


{
    "name": "Test user",
    "username": "testreset",
    "email": "[email protected]",
    "password": "abc123",
    "password_confirm": "abc123"
}

===

Standard MD will be:

# Forgot your password

## POST /users

Create a new user.


{
    "name": "Test user",
    "username": "testreset",
    "email": "[email protected]",
    "password": "abc123",
    "password_confirm": "abc123"
}

---

As a Developer
I want to Record tests to silk files
As I manually test my application.

Concept
Provide a middleware handler(s) with flags to enable a record mode and options. The middleware would generate a silk compatible file including the HTTP method and related request parameters, headers, status and response. Start with the standard library http, then adapt to specific frameworks that have more advanced routers.

Proof of Concept
Handle POST request and response for a form. Produce an acceptable set of silk test document(s) for a series of requests.

When you have actual data strings with facing and/or trailing spaces, error messages are confusing.

For example an actual data is " expected" then error message is:

Data[9].title expected string: "expected"  actual string: "expected"  

In this case actual string should be shown with a facing space.

clean() function causes this.

Are these Trim callings really necessary?

Having Body as an assertible field will mean people can make regex assertions against the body.

Would be nice if you can test a Graphql api backend using silk. Technically it would be possible but might look ugly most of the time. I was using cucumber for the tests but the idea of converting tests to documentation seems really inspiring.

Would be cool to embed environment variables like this:

## POST /something

* Authorization: Bearer {$TOKEN}

To reduce noise, only show the field types if the fields are different.

Parameters are hard to self-encode in the path - support them like this:

## GET /path

* ?q=query // comment about q
* ?another=123 // comment about another
* ?something=1 // something about something

Hi Mat,

i'm new to go so this may certainly be a problem between the chair and the keyboard, but when i try to go install github.com/matryer/silk from the freshly cloned repo, i get this error :

silk/runner/run.go:17:2: cannot find package "github.com/matryer/m" in any of:
    /usr/local/Cellar/go/1.7.1/libexec/src/github.com/matryer/m (from $GOROOT)
    /Users/sgaide/dev/go/src/github.com/matryer/m (from $GOPATH)

i can find no m package in your github repos either, am i missing something ?

thanks for any help,

Sebastien

http://silktest.org/ isn't a good URL reference for this project.

whois:

Tech Name: Domain Manager
Tech Organization: samirnet -domain names for sale
Tech Street: Flat No. 48 Cunningham Apts Edward Road
Tech City: Bangalore
Tech State/Province: Karnataka
Tech Postal Code: 560052
Tech Country: IL
Tech Phone: +91.802260640
Tech Email: [email protected]

When I send a request with a body option I can't receive form values on server side.

Request

## POST /api
* Content-type: "application/x-www-form-urlencoded"
{
  id: 1
}
===
* Status: 200

Server

r.ParseForm()
fmt.Printf("%v", r.Form)

yields:
map[{ id: "1"}:[] ]

@matryer I'm happy to put up a PR, but I won't be able to release a new version.

This would be awesome if you could get this done soon, as I'm finding silk very useful in integrated e2e API testing 🙇‍♂️

Hello, I just cloned the repo. But it is impossible to run anything at all from the files in the repo:

silk -silk.url="{endpoint}" {testfiles...}

as suggested in the README produces "command not found"

Executing build.sh results in "main.go:8:2: cannot find package "github.com/matryer/silk/runner""

What is the correct procedure for running the application? Could you adjust the documentation accordingly?

I try to get unavailable index

* Data.result[2].id: /^[0-9a-f]{24}$/

and get global panic instead test FAIL.

As an advanced user, I would like to be able to replace parts of the request-body with parameters from the go-code. For example: todays date, some random id etc.

POST /users

{
    "name": "Mat"
}

Proposal: better bodies in Silk

Bodies in Silk could make use of the 'formatting' tag thing where not only the data type is specified, but also the kind of assertions that could take place:

``json(mode=exact)
{
  "name": "Silk",
  "released": 2016
}
``

• Two back tics are used so as not to confuse renderer
• This assumes that the text after the opening code block line is never rendered (needs validating)

The types of bodies could be:

• exact - the body must match verbatim
• same - the JSON must essentially be equal but fields may be in different orders and formatting (whitespace) is ignored
• subset - each mentioned field is asserted, additional fields in the body is ignored

The syntax highlight language specified could also drive which ParseBody function is called.

Cons

• When rendered, you can't see the kind of assertion, but that might not be too much of a problem and could always be explained in the plain text surrounding the block.

Alternatives

Other options to consider:

• the json language specifier might be enough to indicate the datatype, and we could use flags (when running the file) to indicate what type of assertion to make

Hey, not sure if I'm missing something but is there a way to do numerical assertions like:

• Data.hits.total: > 0