Use Travis CI to test examples about angr-doc HOT 18 CLOSED

I have a test branch on claripy that does things fast enough and in
parallel, using prebuilt wheels to speed up the build. That approach should
work pretty well!

On Aug 13, 2016 11:57 PM, "David Manouchehri" [email protected]
wrote:

Alright, back to this!

Building angr-dev takes forever (just tried it on my machine), so I don't
think we can do every repo on every pass.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#51 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADSzl4c4Q_yDRbo5xg4JOj_moYzQ0J8zks5qfrxugaJpZM4IZqRO
.

from angr-doc.

If the angr team is willing, I can set this up myself if I'm added to angr's GitHub organization group.

from angr-doc.

So there is a script that runs everything -- all of the angr repos have a tests directory with tests in them that are run on every push to our internal gitlab setup. That's the Continuous Integration that we refer to on IRC all the time (i.e., "the CI is broken"). The tests for angr-doc run all the examples that run in a reasonable amount of time (i.e., not things like simplehash, that take about half an hour).

There are a few considerations that we need to think about for travis:

1. The basic requirements for CI are that every time a push happens to any repo, every repo that depends on it will be tested to make sure everything still works. For example, when we push to claripy, we need to immediately test claripy, simuvex, angr, and angr-doc.
2. We'd like to test every push, so having Travis pull the latest docker build isn't really sufficient (as the docker builds will be an hour or so behind, because of how long it takes docker to build).
3. Travis has a level of integration that Docker does not. For example, Travis will test PRs, but Docker won't get a build trigger on a PR opening, so there's nothing for Travis to test.

I think we'd need to have the build happen fully in Travis. The biggest timesink is building Z3, so we should look into having a prebuilt version of it (I think Travis allows us to save some sort of cache, or we could just push a prebuilt angr-z3 wheel to pypi). After that, the install should be mostly instantaneous with the exception of VEX itself, and that's not too slow. As I see it, the testing could work like this:

1. Every repo has a simple travis yaml that basically pulls angr-dev and runs its test.sh.
2. test.sh checks the environment variables that Travis sets up to understand what repo was pushed to, which remote the repo is on, and what commit we should test.
3. test.sh then needs to check if the same branch exists on other angr repos, since many features require changes across different parts of angr and these changes need to be tested together. In our internal CI, we use branches on the same repo, but github complicates this further with forks, so in the case of PRs, we'd need to check the other repos that the user who sent the PR has forked, and check them for the branch being tested.
4. test.sh should then run the tests of all repositories. We could optimize this by only running tests on the repositories that could be affected (i.e., a push to angr/angr can't break angr/cle, but could break the examples in angr/angr-doc), but honestly, the "lower-level" repos have pretty quick tests anyways, so this might not be worth the implementation time.

Of course, we have our internal CI that has scripts that do some of this stuff. Maybe we can push those scripts out to angr/angr-dev as a base for building the travis scripts (this is a @rhelmot and @nebirhos question).

In terms of adding you to the angr github organization, things are made a bit complicated by the complex interplay between github and our internal gitlab setup. For example, we synchronize github and gitlab regularly, and code ends up running on our CI in gitlab. Before we start expanding the org, we gotta make sure our professors are ok with this sort of stuff. Generally speaking, we trust everyone, but if weird stuff happens, we're the ones that the professors will pummel. :-)

from angr-doc.

Lots of good points. One key thing is that we can't share cache between repos on Travis CI, so here's my suggested plan to work around that.

1. Put a .travis.yml in each repo, which will pull angr/angr from the Docker Hub.
2. Write an update.sh in angr-dev that works basically like git_all.sh pull, but will only rebuild repos if there's new commits.
3. Run test.sh inside the newly updated Docker instance on Travis CI.
4. (Optional) If test.sh passes, then use Travis CI's after_success to push the newly updated Docker image to Docker Hub.

This way each repo will test against the latest commit of other repos too, and the Docker Hub image will be reasonably up to date since each repo will be constantly updating it (my guess is a new build 30-60 minutes?). A brand new clean Docker image will still be created every 4 hours or so too.

I'm not sure if I'm explaining it that well, so please ask questions.

I think you guys should keep your GitLab CI the way it is right now and we should be trying this out only as an experiment.

from angr-doc.

An idea just came up in my mind: how hard is it for our internal CI to automatically pull PRs from GitHub, run CI on them, and reply the PR with the build result? There might be existing work in the wild for this.

from angr-doc.

from angr-doc.

Good call, security is definitely something that we should bear in mind. Securing our CI environment should be doable.

from angr-doc.

I think it should be easy to get GitLab CI to post the build status to GitHub.

https://developer.github.com/guides/building-a-ci-server/#working-with-statuses
https://developer.github.com/v3/repos/statuses/#create-a-status

from angr-doc.

https://github.com/angr/angr-dev/blob/master/angr-test

This is the script we use for testing internally. It handles syncronization, testing, linting, and pretty much anything you could care about. I'd recommend using it as a base for any other testing operation you could care about.

from angr-doc.

I think @Manouchehri Travis solution is doable, but with a small variant.

Instead of pushing the updated image from Travis (which will generate an additional layer) I'd let Docker do a fresh build with the latest version of the code.

We can easily trigger builds with a curl call https://docs.docker.com/docker-hub/builds/#remote-build-triggers

from angr-doc.

Docker already builds on any push to master from any angr repo, though. I'm
more concerned about using Travis to make sure that PRs don't break
anything.

On Mon, May 9, 2016 at 3:55 PM, nebirhos [email protected] wrote:

I think @Manouchehri https://github.com/Manouchehri Travis solution is
doable, but with a small variant.

Instead of pushing the updated image from Travis (which will generate an
additional layer) I'd let Docker do a fresh build with the latest version
of the code.

We can easily trigger builds with a curl call
https://docs.docker.com/docker-hub/builds/#remote-build-triggers

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#51 (comment)

from angr-doc.

from angr-doc.

Alright, back to this!

Building angr-dev takes forever (just tried it on my machine), so I don't think we can do every repo on every pass.

from angr-doc.

Implemented in 5442653.

from angr-doc.

I'm gonna reopen this because as far as I know there's still the major problem that CI for pull requests has the huge problem that it tests the current master branch instead of the branch for the pull request.

from angr-doc.

So I spent a lot of time looking into this and it's 100% impossible to do branch synchronization for pull requests on multiple repos at once. The way appveyor/travis implement this is that they never even touch the remote for the person who submits the pull request, they just check out a specific ref on the upstream remote.

I did a quick hack to enable testing of PRs correctly, but at the cost of breaking tests on any branch other than master. this shouldn't be too hard to fix, it'll just be a couple carefully placed if statements.

from angr-doc.

So in travis' already checked-out version of the repo, there's only the upstream remote? How does the ref make it into the git manifest?

from angr-doc.

https://ci.appveyor.com/project/rhelmot/claripy/build/1.0.36

git fetch -q origin +refs/pull/31/merge:

Apparently opening a merge request on github creates a ref on the target repo where the branches have already been merged. That's why github doesn't run tests unless "checking if branches can merge" succeeds, I'm guessing.

from angr-doc.