jackcoded / newman-reporter-slackmsg Goto Github PK
View Code? Open in Web Editor NEWCustom Newman reporter for slack
Custom Newman reporter for slack
The Slack webhook provides the ability to override the default channel. We are setting up newman to run a dynamic set of folders based on the service running the tests and we want to alert different team channels with the results. It would be great to be able to override the default.
https://getweave.slack.com/services/B01C28M8EKA?added=1
Channel Override
Incoming webhooks have a default channel, but it can be overridden in your JSON payload. A public channel can be specified with "channel": "#other-channel", and a Direct Message with "channel": "@username".
I tried using the new channel option added but can't seem to get it working. Nothing is posting to the channel. The app is installed in our workspace and added to the channel I'm trying to post to. This is the command that I am running:
newman run https://api.getpostman.com/collections/10839446-8485b6f7-f984-4889-bc27-cc60eb218c0b\?apikey=<postman-apikey> --environment https://api.getpostman.com/environments/10839446-3cf10fb8-b07b-4703-ad19-1a07069aaa45\?apikey=<postman-apikey> --folder <foldername> --suppress-exit-code -r slackmsg --reporter-slackmsg-webhookurl https://slack.com/api/chat.postMessage --reporter-slackmsg-token <oAuth token> --reporter-slackmsg-channel #postman-workflow-notifications
I've tried variations on the channel argument - with and without the # sign, wrapping in single quotes and double quotes. Nothing is appearing in slack.
Hi @jackcoded
Is it possible to add to add below information to the slack channel when newman test completed?
Thanks
creating PR #14
Thanks @jackcoded for creating this reporter. It's been super helpful for us at @moov-io to include Newman test results in our workflows.
I was wondering what the license is on this code? It doesn't seem to be apparent to us.
Edit: Does the ISC declaration cover the source code and released artifacts?
When I sending this command:
newman run /Users/young/Desktop/TestMock.postman_collection.json --suppress-exit-code -r slackmsg --reporter-slackmsg-webhookurl 'https://hooks.slack.com/services/xxxxxxxx/B01HU2PQKCK/ayUEz4pumSbXuilJfdsfdsg2z1fX'
The error message will show:
Error in sending message to slack Error: Request failed with status code 400
There are 2 issues in telegram, the first is that it's no longer sending, and the next one is that when your chat_id is negative, most of the newly created group is a negative value.
error: unknown option '-6250XXXXX'
I am having a problem with connecting to Slack. The same issue as #7
I would like to be able to debug this, but there's nothing more that I can see other than there was a 400. It would be great if I got any messages from Slack and any information from the reporter on how to fix that.
Hi,
I have a Github Actions workflow using a newman action. I would like to use this reporter to provide automated feedback. I realise I need to change the reporters to include slackmsg, but am not sure how to include the relevant options for token, channel, etc. Any help is very much appreciated!
- name: Run API Tests
id: run-newman
uses: anthonyvscode/newman-action@v1
with:
delayRequest: 100
collection: ./test/newman/postman_collection.json
environment: ./test/newman/postman_environment.json
reporters: cli
folder: specifc tests
@jackcoded following on from #7 (comment) I modified slackUtils
to print out the payload object as follows.
This produced the output below, which when validated shows that the only validation errors are trailing commas.
{
method: 'POST',
url: '<SNIP>',
headers: { 'content-type': 'application/json', Authorization: 'Bearer ' },
data: '{"channel":"","blocks":[{"type":"divider"},{"type":"section","text":{"type":"mrkdwn","text":"*Test Summary*"}},{"type":"section","fields":[{"type":"mrkdwn","text":"Total Tests:"},{"type":"mrkdwn","text":"116"},{"type":"mrkdwn","text":"Test Passed:"},{"type":"mrkdwn","text":"96"},{"type":"mrkdwn","text":"Test Failed:"},{"type":"mrkdwn","text":"20"},{"type":"mrkdwn","text":"Test Skipped:"},{"type":"mrkdwn","text":"0"},{"type":"mrkdwn","text":"Test Duration:"},{"type":"mrkdwn","text":"1m 35.9s"},],},{"type":"section","fields":[{"type":"mrkdwn","text":"Assertions:"},{"type":"mrkdwn","text":"Total: 397 Failed: 40"},]},{"type":"divider"},],"attachments":[{"mrkdwn_in":["text"],"color":"#FF0000","author_name":"Newman Tests","title":":fire: Failures :fire:","fields":[{"title":"Create RealCard","short":false},{"value":"*`1. AssertionError - Status code is 201`*","short":false},{"value":"• Expected - response to have status code 201 but got 400","short":false,},{"title":"Update RealCard","short":false},{"value":"*`1. AssertionError - Status code is 200`*","short":false},{"value":"• Expected - response to have status code 200 but got 400","short":false,},{"title":"Query RealCard","short":false},{"value":"*`1. AssertionError - Status code is 200`*","short":false},{"value":"• Expected - response to have status code 200 but got 400","short":false,},{"title":"Create RealCard","short":false},{"value":"*`1. AssertionError - Status code is 201`*","short":false},{"value":"• Expected - response to have status code 201 but got 400","short":false,},{"title":"WIP - Update Account - null Id","short":false},{"value":"*`1. AssertionError - Error Code Check`*","short":false},{"value":"• Expected - 0000 to deeply equal 1002","short":false,},{"value":"*`2. AssertionError - Error Description Check`*","short":false},{"value":"• Expected - The value \\null\\ is not valid. to deeply equal Account \\Id\\ is required.","short":false,},{"title":"Delete Account - Null Id","short":false},{"value":"*`1. AssertionError - Error Code Check`*","short":false},{"value":"• Expected - 0000 to deeply equal 1002","short":false,},{"value":"*`2. AssertionError - Error Description Check`*","short":false},{"value":"• Expected - The value \\null\\ is not valid. to deeply equal Account \\Id\\ required.","short":false,},{"title":"Query Accounts - offset is -1","short":false},{"value":"*`1. AssertionError - Error Description Check`*","short":false},{"value":"• Expected - PagerInfo \\Offset\\ must be greater than or equal to 0. to deeply equal ","short":false,},{"title":"WIP - Query Accounts - parentAccountId is non matching","short":false},{"value":"*`1. AssertionError - Status code is 404`*","short":false},{"value":"• Expected - response to have status code 404 but got 200","short":false,},{"value":"*`2. TypeError - Error Code Check`*","short":false},{"value":"• Cannot read property 0 of undefined","short":false,},{"value":"*`3. TypeError - Error Description Check`*","short":false},{"value":"• Cannot read property 0 of undefined","short":false,},{"title":"Create Real Card Account - accountId does not match","short":false},{"value":"*`1. AssertionError - Error Code Check`*","short":false},{"value":"• Expected - 4001 to deeply equal 2003","short":false,},{"value":"*`2. AssertionError - Error Description Check`*","short":false},{"value":"• Expected - AccountRealCardAccount \\Account. Id\\ is not an Account that exists, is in scope, or is an...","short":false,},{"title":"Enable Real Card Account - Null Id","short":false},{"value":"*`1. AssertionError - Status code is 404`*","short":false},{"value":"• Expected - response to have status code 404 but got 400","short":false,},{"title":"Query Real Card Accounts - offset Value is -1","short":false},{"value":"*`1. AssertionError - Error Description Check`*","short":false},{"value":"• Expected - PagerInfo \\Offset\\ must be greater than or equal to 0. to deeply equal ","short":false,},{"title":"API Bug - Create Real Card - No Required Values Provided","short":false},{"value":"*`1. AssertionError - Error Code Check`*","short":false},{"value":"• Expected - 0000 to deeply equal 3004","short":false,},{"value":"*`2. AssertionError - Error Description Check`*","short":false},{"value":"• Expected - Error converting value {null} to type \\System.Boolean\\. Path \\current\\, line 15, position...","short":false,},{"value":"*`3. AssertionError - Error Code Check`*","short":false},{"value":"• Expected - 0000 to deeply equal 3029","short":false,},{"value":"*`4. AssertionError - Error Description Check`*","short":false},{"value":"• Expected - The RealCard field is required. to deeply equal RealCard \\Name\\ is required.","short":false,},{"value":"*`5. AssertionError - Error Code Check`*","short":false},{"value":"• Expected - 0000 to deeply equal 3008","short":false,},{"value":"*`6. AssertionError - Error Description Check`*","short":false},{"value":"• Expected - Error converting value {null} to type \\System.Int32\\. Path \\validFromYear\\, line 11, posi...","short":false,},{"value":"*`7. AssertionError - Error Code Check`*","short":false},{"value":"• Expected - 0000 to deeply equal 3031","short":false,},{"value":"*`8. AssertionError - Error Description Check`*","short":false},{"value":"• Expected - Error converting value {null} to type \\System.Int32\\. Path \\expiresEndYear\\, line 9, posi...","short":false,},{"value":"*`9. AssertionError - Error Code Check`*","short":false},{"value":"• Expected - 0000 to deeply equal 3013","short":false,},{"value":"*`10. AssertionError - Error Description Check`*","short":false},{"value":"• Expected - Error converting value {null} to type \\System.Int32\\. Path \\validFromMonth\\, line 10, pos...","short":false,},{"title":"WIP - Update Real Card - Id Null","short":false},{"value":"*`1. AssertionError - Error Code Check`*","short":false},{"value":"• Expected - 0000 to deeply equal 3025","short":false,},{"value":"*`2. AssertionError - Error Description Check`*","short":false},{"value":"• Expected - The value \\null\\ is not valid. to deeply equal \\Id\\ is required.","short":false,},{"title":"Update Real Card - Id is non matching","short":false},{"value":"*`1. JSONError - Error Code Check`*","short":false},{"value":"• UnExpected - token Q at 1:1Query returned zero rows for model RealCard.^","short":false,},{"value":"*`2. JSONError - Error Description Check`*","short":false},{"value":"• UnExpected - token Q at 1:1Query returned zero rows for model RealCard.^","short":false,},{"title":"Update Real Card - Card holder name is over 128 characters","short":false},{"value":"*`1. AssertionError - Error Code Check`*","short":false},{"value":"• Expected - 0000 to deeply equal 3003","short":false,},{"value":"*`2. AssertionError - Error Description Check`*","short":false},{"value":"• Expected - The value \\null\\ is not valid. to deeply equal RealCard \\Card Holder. Name\\ must be 128 c...","short":false,},{"title":"Update Real Card - Card holder name is null","short":false},{"value":"*`1. AssertionError - Error Code Check`*","short":false},{"value":"• Expected - 0000 to deeply equal 3004","short":false,},{"value":"*`2. AssertionError - Error Description Check`*","short":false},{"value":"• Expected - The value \\null\\ is not valid. to deeply equal RealCard \\Card Holder. Name\\ is required.","short":false,},{"title":"Update Real Card - Name is over 128 characters","short":false},{"value":"*`1. AssertionError - Error Code Check`*","short":false},{"value":"• Expected - 0000 to deeply equal 3028","short":false,},{"value":"*`2. AssertionError - Error Description Check`*","short":false},{"value":"• Expected - The value \\null\\ is not valid. to deeply equal RealCard \\Name\\ must be 128 characters or ...","short":false,},{"title":"Update Real Card - Name is null","short":false},{"value":"*`1. AssertionError - Error Code Check`*","short":false},{"value":"• Expected - 0000 to deeply equal 3029","short":false,},{"value":"*`2. AssertionError - Error Description Check`*","short":false},{"value":"• Expected - The value \\null\\ is not valid. to deeply equal RealCard \\Name\\ is required.","short":false,},{"title":"Query Real Cards - offset value is -1","short":false},{"value":"*`1. AssertionError - Error Description Check`*","short":false},{"value":"• Expected - PagerInfo \\Offset\\ must be greater than or equal to 0. to deeply equal RealCard \\Name\\ is...","short":false,},{"title":"WIP - Query Real Cards - real card account id value is non matching","short":false},{"value":"*`1. AssertionError - Status code is 404`*","short":false},{"value":"• Expected - response to have status code 404 but got 200","short":false,},],"footer":"Newman Test","footer_icon":"https://platform.slack-edge.com/img/default_application_icon.png",}] }'
}
Hi there!
I really like this slack-reporter to newman.
So much better than the others.
But there is one thing I'm missing.
And the thing that I'm missing is that you cant see what collection or environment file that's been used.
It would be really nice if you could add this feature.
Maybe place this in top of everything.
Like...
Collection:
Environment:
Test Summary
Totalt Tests
etc
etc
And maybe also add number of assertions to the test summary?
Best regards Niclas
First of all, thanks for providing such a great feature! It is really useful.
I have an issue when testing a large collection of tests where they occasionally fail due to network issues / misconfigured servers. This causes a lot of assertion failures, which completely spams slack with the failure messages.
Endpoint 1
1. AssertionError - Status code return 200
• Expected - response to have status code 200 but got 404
2. AssertionError - Status response is JSON
• Expected - response body to be a valid json but got error Unexpected token < at 1:1
3. JSONError - connection error
• UnExpected - token < at 1:1<!DOCTYPE html>^
Could you please implement an optional parameter that would limit failure message lines to any desired number. i.e display only 10 assertion failures.
--reporter-slackmsg-limitFailures '<50>'
This would help us a lot!
Thanks.
The pass % would be nice at a glance to see if we are getting better or worse with our tests
Hello,
Is it possible to have an improvement to customize the "Newman Tests" on slack messages ?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.