Add travis or any other CI for tests about mailgun-go HOT 7 CLOSED

I would like to make this my top priority as I wish to use this library to implement a new suite of integration tests here @mailgun

from mailgun-go.

I'm certainly open to the idea. The main problem I see is the need for API keys for testing - do you have any idea how to solve this?

from mailgun-go.

@mbanzon We have an account specifically for testing -- we can set up encrypted secrets with Travis to expose the API keys that are needed for tests :)

from mailgun-go.

Good idea @pirogoeth. It's probably the easiest option when using wercker or travis.
A second option is to run tests separately. I will look for some solutions this week.

But, to be honest, in my opinion it's not practical to not run tests against the real mailgun api. I know that it feels safer to test against the real endpoint, but in fact, you cannot change the behavior of this endpoint within this package. Because in this package you can only control the request itself and the handling of the possible responses.
What happens if mailgun changes the api willingly?
With the tests against the api, the upcoming tests will fail. But only if there are any at this time. Most likely you, as the library devs, should/will know of the problem before.
What happens if mailgun is offline or mailgun response (accidentally) changed?
It's not on the library to check these possibilities, because there no chance to "fix" it. The only thing that can be tested is the handling for these cases.

Sorry for the edit and long post. Just wanted to publish my thoughts.

from mailgun-go.

@sbani you are correct.

Using Travis eg. would give the advantage to show PR's breaking changes - at least breaking with the Mailgun API - as you mention it is not really the concern of the package - it is trivial to test (locally) before merging and somewhat difficult to setup - failing doesn't even mean anything was changed to brake compatibility with the API, it might be the API causing the brake. Furthermore the changes I'm most concerned about at the moment are changes in the library that may cause braking builds by users - a well crafted PR should change tests accordingly so these changes shouldn't ever be an issue with CI - they'll be missed completely.

Personally I don't know of any Mailgun changes before they hit me in the face.

I've tried to set something up for now but as I am not the administrator of the repository Travis will not let me setup the webhook - this complicates matters further. I realized that if I set up the Travis integration I need to have an extra Mailgun account for these testing purposes - and the owners/contributers of the repo should rely on me being around (for ever-ish). This isn't a problem - for now - but it seems somewhat limiting.

There more I think about it, there more I lean towards not having Travis eg. setup for this library for these reasons (test-account upkeep etc.). One small addition could be to have a test.bash with environment variables etc. specified (but not set) and then have local_test.bash (or something similar) added to .gitignore - contributers could then simply copy the included test.bash and add their keys etc. and have easy access to testing.

Let me know what you guys think - I'd love to se an elegant solution.

from mailgun-go.

@thrawn01 what is your thoughts on this regarding API keys etc?

from mailgun-go.

We are using a mailgun testing account which is used for travis-ci, the API keys are in travis and are restricted to members of the mailgun repo. In addition the tests will not run against Pull Requests as this could expose the API Key to malicious code. IE fmt.Println(os.Getenv("MG_API_KEY")) then users could look at the travis logs.

I'm going to close this as this is the easiest path to testing. Eventually I would like to add more unit/mocked tests but that will have to be another day. =(

from mailgun-go.