matt-ball / newman-action Goto Github PK
View Code? Open in Web Editor NEWUse Postman's headless collection runner, Newman, via a GitHub Action.
License: Apache License 2.0
Use Postman's headless collection runner, Newman, via a GitHub Action.
License: Apache License 2.0
Hi,
What is the syntax to pass a global variable to collection json in newman action?
I am looking for action syntax for below nodejs snippet.
envVar: [{
"key": "target",
"value": "stage"
}]
Each of echo error return null or empty string. How get error output from apiTestInstance?
api-test-instance:
runs-on: self-hosted
steps:
- uses: rlespinasse/github-slug-action@v4
with:
short-length: 5
- uses: actions/checkout@master
- name: Test api postman
id: apiTestInstance
uses: matt-ball/newman-action@master
with:
apiKey: ${{ secrets.POSTMAN_API_KEY }}
collection: "postman/collections/collection.json"
environment: "xxx"
- name: Echo Error
if: failure()
run: |
echo "apiTestInstance failed with the following error message:"
echo "${{ steps.apiTestInstance.log }}"
- name: Echo Error 2
if: failure()
run: |
echo "apiTestInstance failed with the following error message:"
echo "${{ steps.apiTestInstance.outputs['newman-error'] }}"
- name: Echo Error 3
if: failure()
run: |
echo "apiTestInstance failed with the following error message:"
echo "${{ steps.apiTestInstance.outputs.stdout }}"
- name: Echo Error 4
if: failure()
run: |
echo "apiTestInstance failed with the following error message:"
echo "${{ steps.apiTestInstance.outputs.newman-error }}"
- name: Echo Error 5
if: failure()
run: |
echo "apiTestInstance failed with the following error message:"
echo "${{ steps.apiTestInstance.outputs }}"
- name: Echo Error 6
if: failure()
run: |
echo "apiTestInstance failed with the following error message:"
echo "${{ toJson(steps.apiTestInstance.outputs) }}"
- name: Send Slack Message
if: failure()
uses: slackapi/[email protected]
with:
payload: |
{
"text": "Errors from tests api postman",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "Newman test failed: ${{ steps.apiTestInstance.outputs['newman-error'] }}"
}
}
]
}
env:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_POSTMAN_NOTIFICATION_URL }}
SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK
If the collection or the environment listed does not exist, the test should fail, not pass on silently in 1 second.
I believe #12 ran into this as well, but that is just a guess based on the description.
Hey guys, i am using your lib to test my collection in github actions. But i am facing a problem. I need to upload a file in a endpoint of the collection but I get this error:
BInary file load error: "path-to-file/myfile.txt", no such file
I tryed to use upload-artifact but it does not recognize any directories.
What do you guys suggest i should do?
this uses 4.5.4 and I'm running into the following issue: https://community.postman.com/t/pre-request-error-on-newman-cannot-read-property-set-of-undefined/9364/4
The fix requires 4.5.6 or newer.
I have env variables that contain username and password that are not committed to the environment or global files, is it possible to set them in the workflow file and use GitHub secrets?
This took us quite awhile to track down. It seems that whatever version of node this action is using might not support ?.
object accessor chaining.
For example:
questions[0]?.rules?.required
I tried ensuring node 16 was used within the job with an action we use elsewhere for our webpack builds:
- name: Setup Node 16.x
uses: actions/setup-node@v3
with:
node-version: 16.x
But that didn't seem to fix the issue so it must be within the scope of this action. We ended up changing the test scripts for now but this might warrant an update on your end.
This is how we're using your action:
- name: Run questionnaire_completions collection
uses: matt-ball/newman-action@master
with:
collection: https://api.getpostman.com/collections/{{redacted}}?apikey=${{ secrets.POSTMAN_API_KEY }}
environment: ${{ secrets.POSTMAN_ENVIRONMENT}}
workingDir: ./storage/tests/files/
FYI.
Our integration tests used to specify matt-ball/newman-action@master
, but the tests started to fail yesterday, no changes were made to the test collections. I noticed the new releases and assumed that was the cause of the failures.
Specifying 0.0.28
instead made the tests pass again.
Hi,
I am using this github action and getting below error when i use envVar. I want to overwrite token variable which is already defined in a environment.json.
CODE:
name: Run API Test Suite
id: run-newman
uses: matt-ball/newman-action@master
with:
collection: test/src/collection/cicd-testsuite.json
environment: test/src/collection/LocalDev.postman_environment.json
reporters: cli
delayRequest: 250
envVar: '[{ "key": "Token", "value":"XYZ123@"}]'
Error:
Warning: Bad object passed in config!
Error: Newman run failed!
Thanks!!
The action will generate a warning like the following one:
##[warning]Unexpected input(s) 'postmanApiKey', valid inputs are ['apiKey', 'collection', 'environment', 'globals', 'iterationCount', 'iterationData', 'folder', 'workingDir', 'insecureFileRead', 'timeout', 'timeoutRequest', 'timeoutScript', 'delayRequest', 'ignoreRedirects', 'insecure', 'bail', 'suppressExitCode', 'reporters', 'reporter', 'color', 'sslClientCert', 'sslClientKey', 'sslClientPassphrase']
Wouldn't it be better to use the apiKey param directly instead of the postmanApiKey which will be passed to the apiKey anyway?
Could be related to this Pull Request
I have 2 problems with running Action
--reporters
options `Run matt-ball/newman-action@master
with:
collection: ./tests/Reference_Service.postman_collection.json
environment: ./tests/integration.postman_environment.json
iterationCount: 4
reporters: cli
bail: true
apiKey: postmanApiKey
insecureFileRead: true
timeout: Infinity
timeoutRequest: Infinity
timeoutScript: Infinity
delayRequest: 0
ignoreRedirects: false
insecure: false
suppressExitCode: false
color: auto
I'm trying to create a workflow like this:
- uses: actions/checkout@master
- uses: matt-ball/newman-action@master
with:
apiKey: ${{ secrets.postmanApiKey }}
collection: 5922408-c22ef764-b464-424c-8702-750343478723
environment: 5922408-228c7edd-fc15-4f68-9665-a35d7df6945b
but the service is failing processing the collection or the environment.
hi,
i do this in my workflow
- name: Install newman
run: |
npm install -g newman newman-reporter-htmlextra
and this
- name: Newman
id: newman
uses: matt-ball/newman-action@master
with:
apiKey: ${{ secrets.POSTMAN_API_KEY }}
collection: ${{ env.PM_COLL_ID }} # or path to local collection.json
environment: ${{ env.PM_ENV_ID }} # or path to local environment.json
folder: "[CI]"
reporters: cli,junit,htmlextra
reporter: "{ \"junit\": { \"export\": \"/tmp/newman/TEST-newman-report.xml\" }, \"htmlextra\": { \"export\": \"/tmp/newman/newman-report-extra.html\" }}"
bail: true
i do get the junit-report, but it's complaining about the htmlextra:
newman: could not find "htmlextra" reporter
| ensure that the reporter is installed in the same directory as newman
| please install reporter using npm```
thank you for support
I don't understand how to specify reporter options in yaml
for example:
how to use argument reporter : { junit : { export : './xmlResults.xml' } }
Please tell me
The action finishes in 1 second as success returning
Run matt-ball/newman-action@master
with:
collection: e2e-views-backend.json
environment: Development.postman_environment.json
apiKey: postmanApiKey
iterationCount: 1
insecureFileRead: true
timeout: Infinity
timeoutRequest: Infinity
timeoutScript: Infinity
delayRequest: 0
ignoreRedirects: false
insecure: false
bail: false
suppressExitCode: false
reporters: cli
color: auto
env:
GOROOT: /opt/hostedtoolcache/go/1.15.0/x64
However no tests are done.
I am using:
- name: Performing E2E tests
uses: matt-ball/newman-action@master
with:
collection: e2e-views-backend.json
environment: Development.postman_environment.json
Hi matt,
I have forked your repo and tried using it as I wanted to add "export-environment " and it works fine. However if I use iterationData option it is not working
After upgrading to latest version the postman action fails, even though all tests passed:
- name: Run postman tests
uses: matt-ball/newman-action@master
with:
collection: 'Search Service Tests.postman_collection.json'
environment: 'Local.postman_environment.json'
Reverting back to 0.3.7 and the tests pass:
- name: Run postman tests
uses: matt-ball/[email protected]
with:
collection: 'Search Service Tests.postman_collection.json'
environment: 'Local.postman_environment.json'
Node.js 12 actions are deprecated, this should be updated to run on 16+
When running the following job:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: matt-ball/[email protected]
with:
apiKey: ${{ secrets.POSTMAN_API_KEY }}
collection: 2d8b8a29-29a7-4a70-8db7-615ad8e44874
environment: 3d1532d7-22bd-4d50-99ea-4e66144530fd
I'm seeing the following error:
Newman run failed! Error: collection could not be loaded
unable to read data from file "2d8b8a29-29a7-4a70-8db7-615ad8e44874"
ENOENT: no such file or directory, open '2d8b8a29-29a7-4a70-8db7-615ad8e44874'
Any thoughts on why Newman's interpreting the GUIDs as file paths?
In my yml file. I used like this:
- name: Run newman CLI - negative test periods
uses: matt-ball/[email protected]
with:
apiKey: ${{ secrets.POSTMAN_API_KEY }}
collection: ***
environment: ***
globalVar: '[{"CATALYST_ACCESS_TOKEN": "${{ env.token }}" }]'
And collection and environment both are UUID.
After run github action, it does not seem to recognize the globalVar option.
How to use this globalVar? I think should be same with envVar.
The envVar use like this:
envVar: '[{ "key": "url", "value": "http://localhost:3000" }]'
Thanks to have a check.
Hi matt-ball,
I would like to use the option "export-globals" (https://github.com/postmanlabs/newman#command-line-options)
It's possible to adding this option to your GitHub Action ?
Thanks
Gérôme
Hi,
Many thanks for putting this together!
I am trying out this action for running our collections via API and it works without an issue. We would like however to have the possibility to generate the html reports, e.g. using https://www.npmjs.com/package/newman-reporter-htmlextra .
It would be great if you could also provide the possibility, to just specify the reporter with the corresponding option and be able to get the html/htmlextra report as well.
Many thanks,
Elena
I am trying to run Newman in a GH Action using the same process as an example in the README:
- uses: actions/checkout@master
- uses: matt-ball/newman-action@master
with:
apiKey: ${{ secrets.postmanApiKey }}
collection: ${{ env.COLLECTION }}
environment: ${{ env.DEV }}
When I run the test in GH Actions I get the message:
Warning: Bad object passed in config!
Error: Newman run failed! Error: could not load environment
ENOENT: no such file or directory, open '<my UUID>'
Am I missing something?
Using the following steps in the yaml:
steps: - name: Checkout uses: actions/checkout@v2 - name: Run Postman test scripts with Newman - Test Environment uses: matt-ball/newman-action@master with: apiKey: ${{ secrets.postmanApiKey }} collection: 15647561-02887baa-4609-40d1-a02b-3d5a2c943953 environment: 3069900-2bdc0b8d-c45t-4d41-99fa-31c0007e2a98 folder: 15647561-b5653c42-3157-5103-ab6c-d1ce39f05952
Newman runs the entire collection and fails to restrict the run to the selected folder. Prior to the test run, a "Bad object passed in config!" warning appears. The same error occurs whether using UID or folder name, and the same for folders and sub-folders.
2022-02-25T21:26:28.4480643Z ##[group]Run matt-ball/newman-action@master 2022-02-25T21:26:28.4480875Z with: 2022-02-25T21:26:28.4481318Z apiKey: *** 2022-02-25T21:26:28.4481628Z collection: 15647561-02887baa-4609-40d1-a02b-3d5a2c943953 2022-02-25T21:26:28.4481880Z environment: 3069900-2bdc0b8d-c45t-4d41-99fa-31c0007e2a98 2022-02-25T21:26:28.4482123Z folder: 15647561-b5653c42-3157-5103-ab6c-d1ce39f05952 2022-02-25T21:26:28.4482306Z insecure: true 2022-02-25T21:26:28.4482470Z color: on 2022-02-25T21:26:28.4482621Z reporters: ["cli"] 2022-02-25T21:26:28.4482785Z env: 2022-02-25T21:26:28.4482963Z APP_NAME: *** 2022-02-25T21:26:28.4483138Z NAMESPACE: *** 2022-02-25T21:26:28.4483360Z DOCKER_REGISTRY: *** 2022-02-25T21:26:28.4483787Z DOCKER_REPOSITORY: *** 2022-02-25T21:26:28.4484000Z HARNESS_APP_NAME: *** 2022-02-25T21:26:28.4484223Z ARTIFACT_SOURCE: *** 2022-02-25T21:26:28.4484487Z HARNESS_APPLICATION_NAME: *** 2022-02-25T21:26:28.4484746Z HARNESS_PIPELINE_DEV: *** 2022-02-25T21:26:28.4485013Z HARNESS_PIPELINE_TEST: *** 2022-02-25T21:26:28.4485264Z HARNESS_PIPELINE_STAGE: *** 2022-02-25T21:26:28.4485487Z PERFORM_AGAINST: *** 2022-02-25T21:26:28.4485660Z ##[endgroup] 2022-02-25T21:26:28.9440648Z ##[warning]Bad object passed in config! 2022-02-25T21:26:34.5299782Z newman
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.