refs #97 (comment)

Proposal

If the request and test are executed in the same step, we want easy access to the response results.

Define the current variable so that it can be accessed by the same variable name in the named and unnamed steps.

steps:
  -
    req:
      /get?var=foo:
        get:
          body: null
    test: |
      current.res.status == 200

steps:
  reqget:
    req:
      /get?var=foo:
        get:
          body: null
    test: |
      current.res.status == 200

Proposal

Duplicate runner settings in each Runbook when creating multiple scenarios.
It would be good to define a runn configuration file and summarize in it the options that can be specified for runners and CLI execution.
Enables scenarios to be written in a reduced amount of detail and allows for separation of settings for each environment.

Thanks for the great tool.

Is it possible to use SSL certificate with http protocol?
In GRPC, cert and key can be specified in runners, but is it possible to specify the same in HTTP?

I checked the ReadMe and couldn't find it, so I asked a question. I am sorry if I have overlooked it.

Items to consider

• How do you want the file to be specified?
• How to determine content type?
• Is multipart/form-data support only required?

Preparation

Use the yaml

desc: Exec test
steps:
  -
    exec:
      command: echo -n ‘{“field”:“hello world!!“}’
  -
    exec:
      command: cat
      stdin: ‘{{ steps[0].stdout }}’

Result

The stdin is failed to parse.

F

1) test.yaml
 Failure/Error: invalid ‘Exec test’.steps[1]: map[command:cat stdin:map[field:hello world!!]]

1 scenario, 0 skipped, 1 failure
exit status 1

It should be parsed.

Envionment

I found the issue on debian,
And it reproduces on runn docker.

#copy senario
docker pull ghcr.io/k1low/runn:latest
docker run --rm --name runn -v /path/to/your/senario/dir:/books ghcr.io/k1low/runn:latest run /books/test.yaml

I would like to run command runn run towards a GRPC server that does not implement reflection.

How to read and execute a proto file on the command line?
Or is there a set key to read the proto file into the GRPC client on the runbook?

As a side note, the method of setting of the option to read the proto file with the grpcurl command did not work.

$ runn new --and-run --grpc-no-tls -- grpcurl -plaintext -d '{ "name":"test"}' \
-import-path ./src/api -proto hello.proto  localhost:8080 myapp.GreetingService/Hello

Error: failed to run C:\Users\userName\AppData\Local\Temp\runn1431535509\new.yml: gRPC request failed on 'Generated by `runn new`'.steps[0]: failed to list services from reflection enabled gRPC server: rpc error: code = Unimplemented desc = unknown service grpc.reflection.v1alpha.ServerReflection

What I want to do is

vars:
  itemId: ${TEST_ITEM_ID}
steps:
  hoge:
    req:
      /api1:
       get:
    test: |
      ## sorry like go
      for _, v := range hoge.res.body.items {
        if v.id == {{ vars.itemId }} {
          set_var("itemName", v.name)
        }
      }
  fuga:
    req:
      /items/{{ vars.itemName }}:
        get:

Can I do the case by current features?

We want to remove the difference in behavior between runn as CLI and runn as test helper.

Proposal

It is difficult to identify errors when the scenario becomes large and multiple includes are made.
I want to be able to display the number of lines as well as the file that had the error.
I would like to display the desc defined in step as well.

We would like to display the number of lines in the normal error output if we can achieve the following
#121

Motivation

We want to be able to use actual responses to ensure accurate validation.
While it is possible to prepare response data manually, scenario creation is more efficient when captured data can be used quickly.

Ideas

• Added file output option to dump syntax.
• Make the response of each step of the Runn command output to a file as an option when the command is executed.

I want to send a request to the url obtained in the previous step, but runners do not allow setting variables and the following error is reported.

failed to run /books/jobs/xxx/xxx.yml: http request failed on 'xxxxxxxxx'.steps.xxxxxxxxxx: Get "%7B%7B%20previous.res.body.xxxxxxxx%5B0%5D%20%7D%7D": unsupported protocol scheme ""

The runners are set as follows.

desc: xxxxxxxx
if: included
runners:
  myapi:
    endpoint: "{{ previous.res.body.xxxxxx[0] }}"
steps:
  xxxxxxxx:
    myapi:
      /:
        get:
          headers:
            Cookie: xxxxxxxxx
    test: |
      vars.skip_test == true || current.res.status == 200

What should I do if I want to use the url obtained from the previous step?

Executing the following command will result in an error.

• command

go run cmd/runn/main.go loadt --concurrent 2  testdata/book/include_main.yml

• result

.. snip ..
fatal error: concurrent map writes

goroutine 3757 [running]:
github.com/k1LoW/runn.(*store).recordAsMapped(...)
        /workspaces/runn/store.go:31
github.com/k1LoW/runn.(*operator).recordAsMapped(...)
        /workspaces/runn/operator.go:128
github.com/k1LoW/runn.(*operator).record(0xc0003f16c0, 0xc09b6dfd10?)
        /workspaces/runn/operator.go:108 +0x13e
github.com/k1LoW/runn.(*includeRunner).Run(0xc000014760, {0x1487eb8, 0xc09b755280}, 0xc0003918c0)
        /workspaces/runn/include.go:74 +0xa5f
github.com/k1LoW/runn.(*operator).runInternal.func2.1(0xc09b75e540?)
        /workspaces/runn/operator.go:784 +0xfee
github.com/k1LoW/runn.(*operator).runInternal.func2(0xc08f6cd938, 0xc0003f16c0, 0xc08f6cd908, {0x1487eb8, 0xc09b755280})
        /workspaces/runn/operator.go:898 +0x52d
github.com/k1LoW/runn.(*operator).runInternal(0xc0003f16c0, {0x1487eb8, 0xc09b755280})
        /workspaces/runn/operator.go:903 +0x5e5
github.com/k1LoW/runn.(*operator).run(0xc0003f16c0, {0x1487eb8?, 0xc09b755280})
        /workspaces/runn/operator.go:541 +0x189
github.com/k1LoW/runn.(*operators).RunN.func1()
        /workspaces/runn/operator.go:1089 +0x14f
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
        /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0xa5

.. snip ..

or

.. snip ..
panic: runtime error: index out of range [1] with length 1

goroutine 50 [running]:
github.com/k1LoW/runn.(*operator).recordAsMapped(...)
        /Users/runner/work/runn/runn/operator.go:127
github.com/k1LoW/runn.(*operator).record(0x14090533a40, 0x1409173f200?)
        /Users/runner/work/runn/runn/operator.go:108 +0x338
github.com/k1LoW/runn.(*httpRunner).Run(0x140000a6f60, {0x1038bee18, 0x14000daae40}, 0x14091181650)
        /Users/runner/work/runn/runn/http.go:313 +0x938
github.com/k1LoW/runn.(*operator).runInternal.func2.1(0x14090cda180?)
        /Users/runner/work/runn/runn/operator.go:718 +0xe0
github.com/k1LoW/runn.(*operator).runInternal.func2(0x140907dab48, 0x14090533a40, 0x140907dab18, {0x1038bee18, 0x14000daae40})
        /Users/runner/work/runn/runn/operator.go:898 +0x3c0
github.com/k1LoW/runn.(*operator).runInternal(0x14090533a40, {0x1038bee18, 0x14000daae40})
.. snip ..

Although not known to be reproducible, the following error occurred in some cases.

$ runn loadt --load-concurrent 3 --duration 20s tests/**/*.yml
panic: runtime error: index out of range [7] with length 7

goroutine 26 [running]:
github.com/k1LoW/runn.(*operator).recordAsMapped(...)
        /Users/k2tzumi/go/pkg/mod/github.com/k1!lo!w/[email protected]/operator.go:374
github.com/k1LoW/runn.(*operator).recordNotRun(0x140aa154780, 0x103f8fee0?)
        /Users/k2tzumi/go/pkg/mod/github.com/k1!lo!w/[email protected]/operator.go:343 +0x370
github.com/k1LoW/runn.(*operator).runInternal(0x140aa154780, {0x103fb0778, 0x14002caa000})
        /Users/k2tzumi/go/pkg/mod/github.com/k1!lo!w/[email protected]/operator.go:962 +0x53c
github.com/k1LoW/runn.(*operator).run(0x140aa154780, {0x103fb0778?, 0x14002caa000})
        /Users/k2tzumi/go/pkg/mod/github.com/k1!lo!w/[email protected]/operator.go:808 +0x144
github.com/k1LoW/runn.(*operators).runN.func1()
        /Users/k2tzumi/go/pkg/mod/github.com/k1!lo!w/[email protected]/operator.go:1283 +0x138
github.com/k1LoW/concgroup.(*Group).Go.func1()
        /Users/k2tzumi/go/pkg/mod/github.com/k1!lo!w/[email protected]/concgroup.go:37 +0xc8
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /Users/k2tzumi/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x5c
created by golang.org/x/sync/errgroup.(*Group).Go
        /Users/k2tzumi/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0xa4

Thanks for great tool!

Is there a skip or only functionality for step execution? I found skipTest option, but it is skipping only test section, right? I want an option to skip entire steps or execute certain steps, because such functionality helps us to focus on a small subset of test cases and speed up writing tests.

I'm imaging something like this(https://playwright.dev/docs/test-annotations#focus-a-test).

I apologize, if I missed something.

Display the number of loop times when an error occurs during loop

when you define vars with double quote, it seems to ignore yml config? The result is the same when single quote.

desc: basicauth
vars:
  user: "hoge"
  password: "pass@?$"
steps:
  basicAuth:
    bind:
      basicToken: base64encode(vars.user + ':' + vars.password)
  output:
    dump: basicToken

$ runn run ./book.yml --debug
Generated by `runn new` ... ok

1 scenario, 0 skipped, 0 failures

if you remove double quote, it works well

desc: basicauth
vars:
  user: hoge
  password: pass@?$
steps:
  basicAuth:
    bind:
      basicToken: base64encode(vars.user + ':' + vars.password)
  output:
    dump: basicToken

$ runn run ./book.yml --debug
Run 'bind' on 'basicauth'.steps.basicAuth

Run 'dump' on 'basicauth'.steps.output
aG9nZTpwYXNzQD8k
basicauth ... ok

1 scenario, 0 skipped, 0 failures

For example, email verification code step or MFA step.
book design image. Wait for users' stdin input at stdin runner step.

desc: Exec test
steps:
  -
    stdin:
      comment: 'Get MFA code: '
      (optional)timeout:
  -
    req:
      /hoge/{{ steps[0].stdin.value }}

If exec can solve what I want, please let me know!

When we have the concensus, I'll contribute👍

For example, if you try to compare the response json, the elements will be sorted and changed from the original response

I would like to be able to configure from httpRunnerOption.

The logic to evaluate an expression against a store is duplicated in several programs.

We are dealing directly with the antonmedv/expr library, which we believe is problematic for unified expression expansion.

I think we need to organize the use cases for the process of evaluating expressions, clarify who is in charge of the logic, and do some refactoring.

Problem

Inability to successfully take over vars when included at multiple levels

• a.yml

vars:
   foo: 123
steps:
  -
   include: b.yml
   vars:
     foo: '{{ vars.foo }}'

.. snip ..

• b.yml

vars:
   foo: 123
steps:
  -
   include: c.yml
   vars:
     foo: '{{ vars.foo }}'

.. snip ..

• c.yml

vars:
   foo: 123
steps:
  -
    test: vars.foo == 123

Error when executing a.yml
The value of foo is the string '123'.

Question

If I want to specify overwriting the vars in c.yml from a.yml in the above example, is the above description the only way?
Is there any problem to make the specification that vars defined in the parent yaml are inherited?
When specifying vars at include time, I feel that numerical values are inevitably evaluated as strings.

This is the proposal issue on supporting GraphQL runner (like gRPC runner)

I know the current implementation already supports GraphQL request, but I have an idea to improve the usability.

Example

runners:
  gqlreq: https://gql.example.com/query
steps:
  - desc: GraphQL request
    gqlreq:
       headers:
         authorization: bearer xxxxxx
       query: |
         mutation createUser(
           $name: String!, 
           $age: Int!,
           $enabled: Bool
         ) {
           createUser(name: $name, age: $age, enabled: $enabled) {
             id
           }
         }
       variables:
         name: alice
         age: 30
         enabled: true

Advantage

Simpler request definition

In GraphQL with HTTP request, requesting URL (/query), method (POST), Content-Type (application/json), requesting body (query key) are almost always constant.

Currently, in the case using HTTP request form , we have to have several nesting keys like as follows:

steps:
  - desc: GraphQL request
    req:
      /query
        post:
          headers:
            authorization: "bearer xxxxxx"
          body:
            application/json:
              query: |
                mutation {
                  createUser(
                    name: "alice", 
                    age: 30, 
                    enabled: true
                  ) {
                    id
                  }
                }

Constant keys (req."/query".post.body."application/json".query") are redundunt.

I think they can be omitted if GraphQL runner is supported.

Null variable support

Currently, we have no way to set null on GraphQL request.

vars:
  enabled: null
steps:
  - desc: GraphQL request
    req:
      /query
        post:
          headers:
            authorization: "bearer xxxxxx"
          body:
            application/json:
              query: |
                mutation {
                  createUser(
                    name: "alice", 
                    age: 30, 

                    # Oops... this will just be empty, but here should be "enabled: null"
                    # "{{ vars.enabled }}" will be stringified null (eg. "null")
                    enabled: {{ vars.enabled }} 
                  ) {
                    id
                  }
                }

So I would like to support variable at the same level of headers and query which does not omit null as an empty value.

steps:
  - desc: GraphQL request
    gqlreq:
       headers:
         authorization: bearer xxxxxx
       query: |
         mutation createUser(
           $name: String!, 
           $age: Int!,
           $enabled: Bool
         ) {
           createUser(name: $name, age: $age, enabled: $enabled) {
             id
           }
         }
       variables:
         name: alice
         age: 30
         enabled: null # this will send `null` in `enabled` key.

If this proposal sounds good to you, let me work on implementation on GraphQL runner.

Supporting GraphQL runner will make better experience for GraphQL users (like me!)

Any thought? @k1LoW

go net/http client performs redirect by default.
I had a problem when I wanted to test before redirect.

When using runn as a golang helper, I would use HTTPRunnerWithHandler. But it can't be change from the CLI.

For example, in the case of the curl -L .. can be used to redirect. (default is no redirect).

I suggest adding a runner option as follows:

# Example
runners:
  req:
    endpoint: ...
    openapi3: ...
    skip_check_redirect: true

$  runn loadt --concurrent 10 --duration 5s path/to/**/*.yml
... snip ..
Total.........................: 11
Succeeded.....................: 6
Failed........................: 5
Error rate....................: 45.4%
RunN per seconds..............: 2.2
Latency ......................: max=5,931.0ms min=817.5ms avg=4,575.3ms med=4,656.2ms p(90)=5,774.1ms p(99)=5,841.6ms

$ echo $?
0

I want to make the CLI return value non-zero for easy handling at runtime with CI.

Proposal

I want to display an error message on the code if there is an error when incorporating runn into Github actions.

Allow errors to be output in the following format
https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#setting-an-error-message

As the development cycle of the product is to create a scenario that will end successfully locally and then push to git, the priority seems to be quite low.
The idea is to have a simple error output that can be used in various ways, not limited to Github actions.

It might be a good idea to always show the overwritten vars if the debug option was enabled

Proposal

time.Time object can now be handled by the following PR
#129

I want to define a built-in function for comparison.

• After
• Before
• Equal
  May not be necessary as it can be replaced by Compare

Things to consider

• Should it be a generic comparison function that is supposed to handle objects other than the time.Time object?
• Is it possible to call a function of struct with antonmedv/expr?
  For example
  current.rows[0].created_at.After(current.rows[0].updated_at)
  The above syntax does not evaluate correctly, but we think it would be best if we could address this by revising the syntax.

I would like to use the template function added here to dynamically enter the DB value.
#117

When I specified as follows, an error occurred at the "[" part.

{
  "id": {{.steps.getIdFromDB.rows[0].id}}
}

How can I put the DB value dynamically? Sorry to trouble you, but please confirm.

Reproduction

test.yaml

desc: intersect diff
vars:
  v1: [1, 2, 3, 4]
  v2: [2, 3]
  v3: [2, 3]
steps:
  a:
    test: diff(intersect(vars.v1, vars.v2), vars.v3) == false

Result

$ runn run intersect_diff.yaml
intersect diff ... failed to run intersect_diff.yaml: test failed on 'intersect diff'.steps.a: (diff(intersect(vars.v1, vars.v2), vars.v3) == false) is not true
diff(intersect(vars.v1, vars.v2), vars.v3) == false
├── diff(, vars.v3) => ?
├──  => ?
├── vars.v3 => [2,3]
└── false => false


1 scenario, 0 skipped, 1 failure

Expection: it should succeed, but looks like the compiler cannot understand the type returned by intersect function.

What is this?

Fail section that always fails when described.

Use Cases

This is when you want to make a scenario fail after some operation in a later step when you already know that the previous step failed.

A more concrete example would be an API call that results in a 500 error and you want to display the application log.

• example

  steps:
    -
      req:
        desc: Some kind of error occurs. Errors are not evaluated here.
        /fail-end-point:
          get:
            body: null
    -
      desc: Handling errors
      if: previous.res.status == 500
      exec:
        command: tail -10 /var/log/error.log
      # Whenever there is a FAIL section, it fails and does not evaluate the subsequent steps.
      fail: 'Fail messages.'

If you try to send a request for a json file that contains a newline code in the json value, the json itself will be sent as a string.

• json files containing line feed codes
  https://github.com/k2tzumi/runn/blob/5cbb9f68afb2bc3344ec5aec094755ddeaa4f3dd/testdata/newline.json

• runbook that calls the json file
  https://github.com/k2tzumi/runn/pull/20/files#diff-529bf6e17bec3f0713aa61973fdc8ff6ad93c059f1ba1650a90e498e582ab934

• result
  https://github.com/k2tzumi/runn/actions/runs/4901420664/jobs/8752680939#step:7:132

• debug
  json body is not json!

  $ go run cmd/runn/main.go run testdata/book/httpbin_include.yml --debug
  .. snip ..
  
  Run 'req' on 'testing include'.steps[1]
  -----START HTTP REQUEST-----
  POST /post?count=0 HTTP/1.1
  Host: httpbin.org
  Content-Type: application/json
  X-Test: default
  
  "{\"bar\":\"abc\\r\\ndef\",\"foo\":\"abc\\ndef\"}"
  -----END HTTP REQUEST-----
  -----START HTTP RESPONSE-----
  HTTP/2.0 200 OK
  Content-Length: 581
  Access-Control-Allow-Credentials: true
  Access-Control-Allow-Origin: *
  Content-Type: application/json
  Date: Sat, 06 May 2023 12:17:56 GMT
  Server: gunicorn/19.9.0
  
  {
    "args": {
      "count": "0"
    }, 
    "data": "\"{\\\"bar\\\":\\\"abc\\\\r\\\\ndef\\\",\\\"foo\\\":\\\"abc\\\\ndef\\\"}\"", 
    "files": {}, 
    "form": {}, 
    "headers": {
      "Accept-Encoding": "gzip", 
      "Content-Length": "50", 
      "Content-Type": "application/json", 
      "Host": "httpbin.org", 
      "User-Agent": "Go-http-client/2.0", 
      "X-Amzn-Trace-Id": "Root=1-64564571-525c7cd93ad52c044831da9c", 
      "X-Test": "default"
    }, 
    "json": "{\"bar\":\"abc\\r\\ndef\",\"foo\":\"abc\\ndef\"}", 
    "origin": "20.205.207.201", 
    "url": "https://httpbin.org/post?count=0"
  }
  
  -----END HTTP RESPONSE-----
  Run 'test' on 'testing include'.steps[1]
  F
  
  1) t/b/httpbin_include.yml
    Failure/Error: test failed on 'testing include'.steps[1]: (current.res.status == 200
    && diff(current.res.body.json, vars.externalJsonRequestBody) == "") is not true
    current.res.status == 200
    && diff(current.res.body.json, vars.externalJsonRequestBody) == ""
    ├── current.res.status => 200
    ├── 200 => 200
    ├── diff(current.res.body.json, vars.externalJsonRequestBody) => "  any(
    │   -         string(`{"bar":"abc\r\ndef","foo":"abc\ndef"}`),
    │   +         map[string]any{"bar": string("abc\r\ndef"), "foo": string("abc\ndef")},
    │     )
    │   "
    ├── current.res.body.json => "{"bar":"abc\r\ndef","foo":"abc\ndef"}"
    ├── vars.externalJsonRequestBody => {"bar":"abc\r\ndef","foo":"abc\ndef"}
    └── "" => ""
  
  1 scenario, 0 skipped, 1 failure

• For json without newline code
  Essentially, it's correct to leave the json

  $ go run cmd/runn/main.go run testdata/book/httpbin_include.yml --debug
  .. snip ..
  
  Run 'req' on 'testing include'.steps[1]
  -----START HTTP REQUEST-----
  POST /post?count=0 HTTP/1.1
  Host: httpbin.org
  Content-Type: application/json
  X-Test: default
  
  {"bar":1,"foo":"test"}
  -----END HTTP REQUEST-----
  -----START HTTP RESPONSE-----
  HTTP/2.0 200 OK
  Content-Length: 523
  
  .. snip ..

ref: https://docs.github.com/en/actions/using-jobs/using-concurrency

I want to control concurrency in RunN runs on a scenario-by-scenario basis.

It would be nice to have a list of definitions of variable types available for expressions from the structure of store

Proposal

To allow variables to be embedded in json files for more flexibility as test data.

{
    "code": "{{ vars.code }}",
    "data": {
       "foo": "test",
      "bar": 1
   }
}

Use vars syntax to expand variables when reading json.

Allow previous to be used in the same way as current

If build-in function base64encode is prepared, we can write like below?

vars:
  user: "hoge"
  password: ${TEST_PASS}
  basicauth: {{ base64encode("hoge:${TEST_PASS}") }}  # work well?
steps:
  auth:
    req:
      /hoge:
       post:
          headers:
            Authorization: "Basic {{ base64encode({{ vars.user }}:{{ vars.password }}) }}" # work well?
            #or
            Authorization: "Basic {{ vars.basicauth }}" # work well?

When reading the json file with the following correspondence, it has been changed to read with the relative path from the scenario file.
#146

The above response itself is very convenient and helpful. thank you.

However, as shown below, when the commonly used setting values ​​are put in another folder, if the path from the executable file

json://../../Common/SettingParam.json

Is there a way to describe the relative path from the execution path of runn?

runn
  - Common
    - SettingParam.json
  - TestCase
    - TestCaseA
      - TestCase1.yaml
      - TestCase2.yaml
    - TestCaseB
      - TestCase1.yaml

For example, the following statement

vars:
   path: "path/to"
step:
   dump:
     expr: "dump"
     out: {{ vars.path }}

If this is already possible, I may be mistaken.

I got multipart error at server side.

code=400, message=multipart: NextPart: EOF, internal=multipart: NextPart: EOF

Proposal

Debug output in the test syntax is the following output when using functions

• As-Is

  -----START TEST CONDITION-----
  compare(steps[8].res.body, vars.wantBody, "Content-Length")
  
  
  ├── compare(steps[8].res.body => ?
  ├── vars.wantBody => {"Content-Type":"application/json","freeform":"foo"}
  ├── "Content-Length" => "Content-Length"
  └── ) => ?
  -----END TEST CONDITION-----
  

I want to optimize the output as follows

• To-be

  -----START TEST CONDITION-----
  compare(steps[8].res.body, vars.wantBody, "Content-Length")
  
  
  └── compare(steps[8].res.body, vars.wantBody, "Content-Length") => true
              ├── steps[8].res.body => {"Content-Type":"application/json","freeform":"foo", "Content-Length": 81}
              ├── vars.wantBody => {"Content-Type":"application/json","freeform":"foo"}
              └── "Content-Length" => "Content-Length"
  -----END TEST CONDITION-----
  

multipart body should accept array as well😢 I'll fix without breaking change

requestBody:
  content:
    multipart/form-data:
      schema:
        properties:
          # The property name 'file' will be used for all files.
          file:
            type: array
            items:
              type: string
              format: binary

https://github.com/OAI/OpenAPI-Specification/blob/main/versions/3.0.3.md#considerations-for-file-uploads

Currently it is loaded relative to the runn execution path, but I would like to load it relative to the scenario file.
We believe that the above specification is preferable when trying to manage multiple scenario files distributed in multiple directories.

TestShard is flaky

ref: #95 (comment)

Proposal

We would like to improve the conditionals as they become more complex and the intent becomes more difficult to understand.

We believe that the current yaml syntax makes it difficult to include comments.

    test: |
      current.res.status == 200
      &&  current.res.body.foo == vars.expectFoo
      &&  current.res.body.bar == vars.expectBar
      # Can't write a comment.
      &&  current.res.body.xxx[0].zzz == vars.expectXXXZZZ

It might be good to write verification content using labels such as context and it, as in rspec
Maybe add a desc while allowing conditions to be specified in an array?

We would like to number the requests with an ID that can identify the scenario in order to trace from which scenario the request was made during regression testing.

What are the possible methods?
I would like to consider extending runn if necessary.

Examples of the use of unique IDs may include the following.

• Set in the x-header of the request
• Used in code value when creating data

Problem

The following error occurred.

strconv.Atoi: parsing “2022-08-30 15:07:26”: invalid syntax

Verified DB was MySQL

Reproduction

test.yaml

desc: object map binding
vars:
  value: [
    { "value": 10 },
    { "value": 20 },
    { "value": 30 }
  ]
steps:
  a:
    bind:
      extracted: map(vars.value, {#.value})
  b:
    test: len(steps.a.extracted) == 3

Result

$ runn run test.yaml --debug
Run 'bind' on 'object map binding'.steps.a

Run 'test' on 'object map binding'.steps.b
object map binding ... failed to run test.yaml: test failed on 'object map binding'.steps.b: eval error: invalid argument for len (type <nil>) (1:1)
 | len(steps.a.extracted) == 3
 | ^

1 scenario, 0 skipped, 1 failure

Expected: It should succeed (or my syntax is wrong?)

Preparation

test.yaml

desc: object array mapping
vars:
  expected: [10, 20]
runners:
  req: http://localhost:8000
steps:
  a:
    req:
      /resp.json:
        get:
          body: null
  b:
    test: vars.expected in map(steps.a.res.body, {#.value})

resp.json

[
  { "value": 10 },
  { "value": 20 },
  { "value": 30 }
]

run a mock server responding with the resp.json

$ python3 -m http.server

Result

$ runn run test.yaml --debug
Run 'req' on 'object array mapping'.steps.a
-----START HTTP REQUEST-----
GET /resp.json HTTP/1.1
Host: localhost:8000


-----END HTTP REQUEST-----
-----START HTTP RESPONSE-----
HTTP/1.0 200 OK
Connection: close
Content-Length: 60
Content-Type: application/json
Date: Thu, 09 Feb 2023 00:41:33 GMT
Last-Modified: Wed, 08 Feb 2023 08:14:49 GMT
Server: SimpleHTTP/0.6 Python/3.10.8

[
  { "value": 10 },
  { "value": 20 },
  { "value": 30 }
]

-----END HTTP RESPONSE-----

Run 'test' on 'object array mapping'.steps.b
object array mapping ... failed to run test.yaml: test failed on 'object array mapping'.steps.b: (vars.expected in map(steps.a.res.body, {#.value})) is not true
vars.expected in map(steps.a.res.body, {#.value})
├── vars.expected => [10,20]
├── map(steps.a.res.body, ) => ?
├── steps.a.res.body => [{"value":10},{"value":20},{"value":30}]
└──  => ?


1 scenario, 0 skipped, 1 failure

Expected: it should succeed, but looks like a compiler cannot understand types when using map function.

I don't understand why mapping says map(steps.a.res.body, ) => ?. It just can extract values by keys from an object array.

The another thing I have observed is that it works as expected when I don't use response coming from web servers but put the same response in vars instead. I don't know why but it looks typed.

testdata/book plays a role as example codes. But they are accosiated with test so it would be nice to have a directory containing example runbooks to encourage utilization for users' themselves.

like #287 and what you taught me at a lot of PRs

https://github.com/k1LoW/runn#load-test-using-runbooks

$ runn loadt --concurrent 2 --threshold 'error_rate < 10' path/to/*.yml

Number of runbooks per RunN...: 15
Warm up time (--warm-up)......: 5s
Duration (--duration).........: 10s
Concurrent (--concurrent).....: 2

Total.........................: 13
Succeeded.....................: 12
Failed........................: 1
Error rate....................: 7.6%
RunN per seconds..............: 1.3
Latency ......................: max=1,790.2ms min=95.0ms avg=1,541.4ms med=1,640.4ms p(90)=1,749.7ms p(99)=1,786.5ms

Error: (error_rate < 10) is not true
error_rate < 10
├── error_rate => 14.285714285714285
└── 10 => 10

Error rate....................: 7.6%

Even though the

error_rate => 14.285714285714285

is evaluated, resulting in an error.

The following pattern occurred locally as well

$ runn loadt --concurrent 3 --duration 3s --threshold 'failed == 0' path/to/*.yml

Number of runbooks per RunN...: 1
Warm up time (--warm-up)......: 5s
Duration (--duration).........: 3s
Concurrent (--concurrent).....: 3

Total.........................: 3
Succeeded.....................: 3
Failed........................: 0
Error rate....................: 0%
RunN per seconds..............: 1
Latency ......................: max=2,771.0ms min=2,694.0ms avg=2,726.1ms med=2,694.0ms p(90)=2,713.4ms p(99)=2,713.4ms

Error: (failed == 0) is not true
failed == 0
├── failed => 2
└── 0 => 0

Are those that have not returned a response at the end of the execution time treated as Fail?