matt-ball / newman-action Goto Github PK

Hi,
What is the syntax to pass a global variable to collection json in newman action?
I am looking for action syntax for below nodejs snippet.
envVar: [{
"key": "target",
"value": "stage"
}]

Each of echo error return null or empty string. How get error output from apiTestInstance?

  api-test-instance:
    runs-on: self-hosted
    steps:
      - uses: rlespinasse/github-slug-action@v4
        with:
          short-length: 5
      - uses: actions/checkout@master
      - name: Test api postman
        id: apiTestInstance
        uses: matt-ball/newman-action@master
        with:
          apiKey: ${{ secrets.POSTMAN_API_KEY }}
          collection: "postman/collections/collection.json"
          environment: "xxx"
      - name: Echo Error
        if: failure()
        run: |
          echo "apiTestInstance failed with the following error message:"
          echo "${{ steps.apiTestInstance.log }}"
      - name: Echo Error 2
        if: failure()
        run: |
          echo "apiTestInstance failed with the following error message:"
          echo "${{ steps.apiTestInstance.outputs['newman-error'] }}"
      - name: Echo Error 3
        if: failure()
        run: |
          echo "apiTestInstance failed with the following error message:"
          echo "${{ steps.apiTestInstance.outputs.stdout }}"
      - name: Echo Error 4
        if: failure()
        run: |
          echo "apiTestInstance failed with the following error message:"
          echo "${{ steps.apiTestInstance.outputs.newman-error }}"
      - name: Echo Error 5
        if: failure()
        run: |
          echo "apiTestInstance failed with the following error message:"
          echo "${{ steps.apiTestInstance.outputs }}"
      - name: Echo Error 6
        if: failure()
        run: |
          echo "apiTestInstance failed with the following error message:"
          echo "${{ toJson(steps.apiTestInstance.outputs) }}"
      - name: Send Slack Message
        if: failure()
        uses: slackapi/[email protected]
        with:
          payload: |
            {
              "text": "Errors from tests api postman",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "Newman test failed: ${{ steps.apiTestInstance.outputs['newman-error'] }}"
                  }
                }
              ]
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_POSTMAN_NOTIFICATION_URL }}
          SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK

If the collection or the environment listed does not exist, the test should fail, not pass on silently in 1 second.
I believe #12 ran into this as well, but that is just a guess based on the description.

Hey guys, i am using your lib to test my collection in github actions. But i am facing a problem. I need to upload a file in a endpoint of the collection but I get this error:

BInary file load error: "path-to-file/myfile.txt", no such file

I tryed to use upload-artifact but it does not recognize any directories.
What do you guys suggest i should do?

this uses 4.5.4 and I'm running into the following issue: https://community.postman.com/t/pre-request-error-on-newman-cannot-read-property-set-of-undefined/9364/4

The fix requires 4.5.6 or newer.

I have env variables that contain username and password that are not committed to the environment or global files, is it possible to set them in the workflow file and use GitHub secrets?

This took us quite awhile to track down. It seems that whatever version of node this action is using might not support ?. object accessor chaining.

For example:

questions[0]?.rules?.required

I tried ensuring node 16 was used within the job with an action we use elsewhere for our webpack builds:

    - name: Setup Node 16.x
      uses: actions/setup-node@v3
      with:
        node-version: 16.x

But that didn't seem to fix the issue so it must be within the scope of this action. We ended up changing the test scripts for now but this might warrant an update on your end.

This is how we're using your action:

    - name: Run questionnaire_completions collection
      uses: matt-ball/newman-action@master
      with:
        collection: https://api.getpostman.com/collections/{{redacted}}?apikey=${{ secrets.POSTMAN_API_KEY }}
        environment: ${{ secrets.POSTMAN_ENVIRONMENT}}
        workingDir: ./storage/tests/files/

FYI.

Our integration tests used to specify matt-ball/newman-action@master, but the tests started to fail yesterday, no changes were made to the test collections. I noticed the new releases and assumed that was the cause of the failures.

Specifying 0.0.28 instead made the tests pass again.

Hi,

I am using this github action and getting below error when i use envVar. I want to overwrite token variable which is already defined in a environment.json.
CODE:
name: Run API Test Suite
id: run-newman
uses: matt-ball/newman-action@master
with:
collection: test/src/collection/cicd-testsuite.json
environment: test/src/collection/LocalDev.postman_environment.json
reporters: cli
delayRequest: 250
envVar: '[{ "key": "Token", "value":"XYZ123@"}]'

Error:
Warning: Bad object passed in config!
Error: Newman run failed!

Thanks!!

The action will generate a warning like the following one:

##[warning]Unexpected input(s) 'postmanApiKey', valid inputs are ['apiKey', 'collection', 'environment', 'globals', 'iterationCount', 'iterationData', 'folder', 'workingDir', 'insecureFileRead', 'timeout', 'timeoutRequest', 'timeoutScript', 'delayRequest', 'ignoreRedirects', 'insecure', 'bail', 'suppressExitCode', 'reporters', 'reporter', 'color', 'sslClientCert', 'sslClientKey', 'sslClientPassphrase']

Wouldn't it be better to use the apiKey param directly instead of the postmanApiKey which will be passed to the apiKey anyway?

Could be related to this Pull Request

I have 2 problems with running Action

1. it not display result ( any of possible --reporters options `
2. because of point 1 tests not failing job or step

Run matt-ball/newman-action@master
  with:
    collection: ./tests/Reference_Service.postman_collection.json
    environment: ./tests/integration.postman_environment.json
    iterationCount: 4
    reporters: cli
    bail: true
    apiKey: postmanApiKey
    insecureFileRead: true
    timeout: Infinity
    timeoutRequest: Infinity
    timeoutScript: Infinity
    delayRequest: 0
    ignoreRedirects: false
    insecure: false
    suppressExitCode: false
    color: auto

I'm trying to create a workflow like this:

- uses: actions/checkout@master
- uses: matt-ball/newman-action@master
  with:
    apiKey: ${{ secrets.postmanApiKey }}
    collection: 5922408-c22ef764-b464-424c-8702-750343478723
    environment: 5922408-228c7edd-fc15-4f68-9665-a35d7df6945b

but the service is failing processing the collection or the environment.

hi,

i do this in my workflow

      - name: Install newman
        run: |
          npm install -g newman newman-reporter-htmlextra

and this

      - name: Newman
        id: newman
        uses: matt-ball/newman-action@master
        with:
          apiKey: ${{ secrets.POSTMAN_API_KEY }}
          collection: ${{ env.PM_COLL_ID }} # or path to local collection.json
          environment: ${{ env.PM_ENV_ID }} # or path to local environment.json
          folder: "[CI]"
          reporters: cli,junit,htmlextra
          reporter: "{ \"junit\": { \"export\": \"/tmp/newman/TEST-newman-report.xml\" }, \"htmlextra\": { \"export\": \"/tmp/newman/newman-report-extra.html\" }}"
          bail: true

i do get the junit-report, but it's complaining about the htmlextra:

newman: could not find "htmlextra" reporter
|   ensure that the reporter is installed in the same directory as newman
|   please install reporter using npm```


thank you for support

I don't understand how to specify reporter options in yaml
for example:
how to use argument reporter : { junit : { export : './xmlResults.xml' } }
Please tell me

The action finishes in 1 second as success returning

Run matt-ball/newman-action@master
  with:
    collection: e2e-views-backend.json
    environment: Development.postman_environment.json
    apiKey: postmanApiKey
    iterationCount: 1
    insecureFileRead: true
    timeout: Infinity
    timeoutRequest: Infinity
    timeoutScript: Infinity
    delayRequest: 0
    ignoreRedirects: false
    insecure: false
    bail: false
    suppressExitCode: false
    reporters: cli
    color: auto
  env:
    GOROOT: /opt/hostedtoolcache/go/1.15.0/x64

However no tests are done.

I am using:

    - name: Performing E2E tests
      uses: matt-ball/newman-action@master
      with:
        collection: e2e-views-backend.json
        environment: Development.postman_environment.json

Hi matt,
I have forked your repo and tried using it as I wanted to add "export-environment " and it works fine. However if I use iterationData option it is not working

After upgrading to latest version the postman action fails, even though all tests passed:

    - name: Run postman tests
      uses: matt-ball/newman-action@master
      with:
        collection: 'Search Service Tests.postman_collection.json'
        environment: 'Local.postman_environment.json'

Reverting back to 0.3.7 and the tests pass:

    - name: Run postman tests
      uses: matt-ball/[email protected]
      with:
        collection: 'Search Service Tests.postman_collection.json'
        environment: 'Local.postman_environment.json'

Node.js 12 actions are deprecated, this should be updated to run on 16+

When running the following job:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: matt-ball/[email protected]
        with:
          apiKey: ${{ secrets.POSTMAN_API_KEY }}
          collection: 2d8b8a29-29a7-4a70-8db7-615ad8e44874
          environment: 3d1532d7-22bd-4d50-99ea-4e66144530fd

I'm seeing the following error:

Newman run failed! Error: collection could not be loaded
  unable to read data from file "2d8b8a29-29a7-4a70-8db7-615ad8e44874"
  ENOENT: no such file or directory, open '2d8b8a29-29a7-4a70-8db7-615ad8e44874'

Any thoughts on why Newman's interpreting the GUIDs as file paths?

In my yml file. I used like this:

  - name: Run newman CLI - negative test periods
    uses: matt-ball/[email protected]
    with:
      apiKey: ${{ secrets.POSTMAN_API_KEY }}
      collection: ***
      environment: ***
      globalVar: '[{"CATALYST_ACCESS_TOKEN": "${{ env.token }}" }]'

And collection and environment both are UUID.
After run github action, it does not seem to recognize the globalVar option.
How to use this globalVar? I think should be same with envVar.

The envVar use like this:
envVar: '[{ "key": "url", "value": "http://localhost:3000" }]'
Thanks to have a check.

Hi matt-ball,

I would like to use the option "export-globals" (https://github.com/postmanlabs/newman#command-line-options)

It's possible to adding this option to your GitHub Action ?

Thanks

Gérôme

Hi,

Many thanks for putting this together!
I am trying out this action for running our collections via API and it works without an issue. We would like however to have the possibility to generate the html reports, e.g. using https://www.npmjs.com/package/newman-reporter-htmlextra .
It would be great if you could also provide the possibility, to just specify the reporter with the corresponding option and be able to get the html/htmlextra report as well.

Many thanks,
Elena

I am trying to run Newman in a GH Action using the same process as an example in the README:

- uses: actions/checkout@master
- uses: matt-ball/newman-action@master
  with:
    apiKey: ${{ secrets.postmanApiKey }}
    collection: ${{ env.COLLECTION }}
    environment: ${{ env.DEV }}

When I run the test in GH Actions I get the message:

Warning: Bad object passed in config!
Error: Newman run failed! Error: could not load environment
  ENOENT: no such file or directory, open '<my UUID>'

Am I missing something?

Using the following steps in the yaml:
steps: - name: Checkout uses: actions/checkout@v2 - name: Run Postman test scripts with Newman - Test Environment uses: matt-ball/newman-action@master with: apiKey: ${{ secrets.postmanApiKey }} collection: 15647561-02887baa-4609-40d1-a02b-3d5a2c943953 environment: 3069900-2bdc0b8d-c45t-4d41-99fa-31c0007e2a98 folder: 15647561-b5653c42-3157-5103-ab6c-d1ce39f05952

Newman runs the entire collection and fails to restrict the run to the selected folder. Prior to the test run, a "Bad object passed in config!" warning appears. The same error occurs whether using UID or folder name, and the same for folders and sub-folders.

2022-02-25T21:26:28.4480643Z ##[group]Run matt-ball/newman-action@master 2022-02-25T21:26:28.4480875Z with: 2022-02-25T21:26:28.4481318Z apiKey: *** 2022-02-25T21:26:28.4481628Z collection: 15647561-02887baa-4609-40d1-a02b-3d5a2c943953 2022-02-25T21:26:28.4481880Z environment: 3069900-2bdc0b8d-c45t-4d41-99fa-31c0007e2a98 2022-02-25T21:26:28.4482123Z folder: 15647561-b5653c42-3157-5103-ab6c-d1ce39f05952 2022-02-25T21:26:28.4482306Z insecure: true 2022-02-25T21:26:28.4482470Z color: on 2022-02-25T21:26:28.4482621Z reporters: ["cli"] 2022-02-25T21:26:28.4482785Z env: 2022-02-25T21:26:28.4482963Z APP_NAME: *** 2022-02-25T21:26:28.4483138Z NAMESPACE: *** 2022-02-25T21:26:28.4483360Z DOCKER_REGISTRY: *** 2022-02-25T21:26:28.4483787Z DOCKER_REPOSITORY: *** 2022-02-25T21:26:28.4484000Z HARNESS_APP_NAME: *** 2022-02-25T21:26:28.4484223Z ARTIFACT_SOURCE: *** 2022-02-25T21:26:28.4484487Z HARNESS_APPLICATION_NAME: *** 2022-02-25T21:26:28.4484746Z HARNESS_PIPELINE_DEV: *** 2022-02-25T21:26:28.4485013Z HARNESS_PIPELINE_TEST: *** 2022-02-25T21:26:28.4485264Z HARNESS_PIPELINE_STAGE: *** 2022-02-25T21:26:28.4485487Z PERFORM_AGAINST: *** 2022-02-25T21:26:28.4485660Z ##[endgroup] 2022-02-25T21:26:28.9440648Z ##[warning]Bad object passed in config! 2022-02-25T21:26:34.5299782Z newman