idaholab / civet Goto Github PK

Currently with GitLab, if the user creating a MR hasn't granted the correct permissions on their branch, civet can't fill out certain database information it needs to initiate testing. This means that no GitLab CI is initiated and the merge button is green and ready to go. We do post a comment on the MR when this happens but it would probably also be a good idea to set a dummy CI status that gets reflected in the merge button.

Had a case where all the jobs on an event were cancelled before some of them even started.
The GitHub status for the jobs that had started got set to failed but the ones that hadn't started were still pending.

Currently the client status is only set when the client communicates in some fashion to the server.
Since it listens on multiple servers, when it is running a job on one server, the other servers don't know
what happened to it.
It would be nice if it periodically pinged other servers to let them know it is still alive and running a job for another server.

Currently the client just runs the scripts the server tells it to and reports back the output and status.
We already parse the output for several things so would could also search the output for special commands.
For example, the output could contain:
CIVET_POST_COMMENT: "This would be posted to the PR"
and the server would post the comment.
This might be used to add a comment when there is a libmesh submodule update.
Or for the documentation system, it could post the URL for where the temporary documentation site is.

A job displayed random characters ( seemingly from reading from initialized memory ).
The client couldn't convert it:

Error: Traceback (most recent call last):
  File "/home/moosetest/civet/civet/client/ServerUpdater.py", line 199, in post_json
    in_json = json.dumps(data, separators=(",", ": "))
  File "/usr/lib/python2.7/json/__init__.py", line 250, in dumps
    sort_keys=sort_keys, **kw).encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe0 in position 12390: invalid continuation byte

This caused the client to hang as it kept trying to post it.

Currently if there is a very long word with no spaces it can turn horizontal scrolling on in the event table. This can happen, for example, if the user has a very long branch name.
It might be better to manually break up such words. Or figure out the proper CSS magic (probably some combination of overflow and word-wrap settings).

This has come up often enough that this would be useful.
Often we want to try something different on a single build machine but we don't have an easy way
to get a specific job to run there.
Having a pulldown of clients next to the invalidate button on the job page would set the job to run on a specific client.

@permcody suggested that we could use the GitHub API to remove the "Please Address Comments" label when a PR is updated. Maybe removing the label and adding another label to indicate that another review should happen?

I noticed last night (and still this morning) that the branch color for MOOSE-devel is currently yellow even though there are no jobs running anywhere that I can see. John just pushed a libMesh PR so that PR is yellow but so is the branch color of MOOSE-devel.

More people have access to invalidating jobs so we might want to keep a better log of invalidations.
We could keep track of only the last person who invalidated or keep a record for each invalidation.
Also the time in which it was invalidated might be useful as well.

There are multiple cases where the main CIVET server has to talk to the git server (like GitHub). If internet is slow or GitHub is down then it causes severe slowness in the CIVET server showing pages. We mitigate this somewhat by having timeouts when talking to the git server but it would be better to have a task queue like Celery to handle this. It could also handle failures and retry again in a certain amount of time. This is important in that when CIVET gets a pull request event it needs to do additional API requests to the git server. If these timeout then the event is not processed and is dropped.

When a new user on GitLab submits a merge request but has not given the moosetest account access to their repo then Civet can't process the event and just raises an Exception. A more user friendly approach would be to add a comment to the merge request informing people of the problem.

@dschwen suggested it would be nice to put a comment on a PR if it has been invalidated to let other people know what is happening.

If a merge request changes then GitLab will send a new event on the webhook with "action" = "update". We should try to be a little smarter about handling this.

The current issue is that once a MR is in we start testing the current upstream branch with the MR.
Then if the MR changes (even if the assigned status changes, or other non commit related changes) then if the upstream branch commit has changed we cancel tests and start a new round of testing.

This is problematic since several apps (like BISON) get updated frequently due to MOOSE submodule updates. So if someone assigns someone to a MR, or adds labels, etc, then it could trigger a new round of testing.

Looks like GitLab 8.1 allows GitHub style "statuses" to be posted for commits:

http://doc.gitlab.com/ce/api/commits.html#post-the-status-to-commit

Should be easy to integrate

The sequence is

1. Someone creates a PR.
2. They push a new commit while CIVET is building. So CIVET cancels the current jobs and starts testing the new commit.
3. The user then force pushes again back to the original commit. This cancels the current jobs. It doesn't create a new event because one with the same commit hash already exists. It would be nice though if it automatically uncancelled the jobs on the original commit and started testing (leaving previously tested jobs that finished alone).

Occasionally the client will stop suddenly (for example someone just killed it on the machine) or the machine might get rebooted while the client is running.
Since the client didn't have time to tell the server what is happening the server just keeps thinking
the client is still running even though there will no longer be any updates for the job the client was running.

It would be nice if the server could detect that the client is active again and check if there are previously running jobs by that client, and if so, cancel or invalidate them.

In the discussion with the assessors one place we may have a hole is recording "system support software" changes. This may not have anything to do with CIVET but I want to start the discussion here. When we make a change to the test system can we come up with a way to figure out that it changed if we needed to pull that information during an audit?

e.g. If Jason updates the compiler, that gets reported in the environment step of every build, it would be really cool if we could figure out a way to report the changes so that we could verify the software works as the environment changes.

1. Come up with a standard way or or ticket reference when the environment changes
2. Have CIVET monitor for changes and record them in the database? (maybe, maybe not)

Currently we specify dependencies by specifying just the recipe name. If the recipe has multiple build configs then all configs must pass to move on to the next stage.
For example, for chigger we have a single recipe with build configs for linux and mac and a merge that depends on that recipe. Both linux and mac tests must pass for the merge to happen.

For the package builder we would like a different dependency chain. We would have a "Package build" recipe that has multiple build configs, one for each platform that we build the package on. Then have a "Package create" recipe (with the same build configs) that depends on "Package build". It would be cool to have the dependency only on the same build config. For example, once the build is done for ubuntu 14, then we could immediately move on to creating it without waiting for all the other builds to finish and pass.

The current way around this would be to have different recipes for each platform (ie build-ubuntu14.cfg, create-ubuntu14.cfg, etc).

@milljm If you want this functionality right now you can create all the different recipes.

Currently only a set number of recipes are run on a PR.

It would be nice to be able to run any of the other recipes that usually run on another event.
For example, being able to run the Intel tests on a PR to be able to verify a PR fixes an Intel problem before it actually gets on master.
We currently have the "Test Heavy" option that needs to be activated but that should go away and allow a user to select any optional recipes to run.

Since there's not a good way to echo all commands without making a mess out of your output, we should get in the habbit of adding explicit echo commands before important command like update_and_rebuild_libmesh.sh, make and run_tests

People really like having a presentation, so we should probably make sure it works. This probably only needs to happen nightly.

Currently we set the branch status based only on jobs that are running on that branch.
When multiple events are running on the branch, we would probably want only the latest event to update the branch status.
For example, if there are multiple events running on master and the latest one finishes first, then the other events will currently update the branch status. It would probably make more sense if those earlier events didn't update the branch status at all.

Not sure whether this is a feature request or just a documentation request, but is there any way we can get Civet to spit out gcov output after it's run a test suite? I just saw http://codecov.io/ output for the first time (at libqueso/queso#527), and getting "current coverage is X%; merging will increase/decrease coverage by Y%" on every PR is pretty sweet.

We currently have AUTHORIZED_OWNERS in civet/settings.py. We check if the user is a collaborator on one of its repositories. This isn't a great way to do this and we need to figure out a better way.

The new version (8.0.4) of gitlab has a feature to mark WIP merge requests. People add [WIP] or WIP: to the subject of their MR to mark them so.

It would be nice not to trigger the recipes for these MRs when somebody pushes new commits, changes stuff, etc.

Not sure if this WIP flag is somehow propagated via their API.

We have the data that shows how many tests were run but it would be nice to have a nice graph of it on the site.

I'm happy to be a guinea pig and contribute to the README for setting up a private CIVET server and corresponding client. It's embarrassing how few web dev skills I have given how much C++ I do. As such, I'd appreciate some hand holding in setting up this scenario, if possible. I already have a node up and running for which I intend to use CIVET. I'd like that node to run the CIVET server and client and interact with GitHub PRs for several software repositories (and ideally also kick off nightly tests for more scenarios than what I would run in a PR). So, a few questions as I try to get started.

1. If I follow the server setup instructions, will that also setup the webpage environment, similar to Moosebuild.org?
2. Does the internal.json file mentioned in the server instructions get built during the server installation? I can't seem to find it in the repo.
3. An example recipe (see #174)?

Thanks much. I'm very excited to be able to use CIVET with my own projects - it's been immensely helpful in libMesh development.

Currently if a "precheck" fails, all the other jobs that depend on it are left in a "pending" state and the pipeline status is left as "running" even though none of those jobs are ever going to run.
We should see about setting the pipeline state directly to failed if at least one job is marked as failed. This would update the status of the merge button properly.

This ties in with #198

We are currently updating GitLab frequently and we have no good way to actually test that the new server will behave in a good way.
Currently we would have to manually set up a test civet server to point at the test GitLab server and manually create merge requests and run jobs etc, then manually make sure everything works as expected.

This isn't as big an issue for GitHub since they don't change things as frequently as GitLab.

Currently, the Civet tests uses mock so that we don't actually hit the git server so that we can run tests offline and also have bad values, etc.

This would probably require using quite a bit more of the API than we currently use.

I notice some updates/fixes coming into CIVET. What steps should I take to integrate them on my running server and client? Do I just need to git pull? Do I need to additionally restart the server and client? Other additional steps? Thanks for any clarification. (This one isn't burning, so no rush on the response is needed.)

Currently when adding an additional recipe to a PR we only see the basic name. It is not always clear what exactly each recipe tests so it would be nice to have some additional help there.

Trying to follow the link to the civet_recipes in the README gives me a 404; I'm guessing that the repo is not public yet. Any idea when it will be made public? Failing that, could an example of a recipe be added to the civet repo?

It looks like the way we are using the commit status api is causing weird behavior on GitLab.
Lots of different builds for the same civet target with different statuses (passed, pending, running), many reporting retried. For a single civet job, many of these get reported.

It would be nice to not have all these. This also seems to indicate that we are no longer using the API correctly.

So, the first time I started my client, I had a typo in the BUILD_ROOT variable. I fixed it, killed the current running client, and restarted a new client with the correct value of BUILD_ROOT exported. The first time the client runs my job, it's fine. When I invalidate it to try again (I'm currently debugging my setup, so I'm just invalidating a job on a PR), it's reverting back to the old, mistyped value of BUILD_ROOT.

Any suggestions? Any other information I can provide? Thanks!

I would like to know all the steps that a job is going to take.

It doesn't look like a branch page (like https://www.moosebuild.org/branch/1298/ ) autoupdates. It would be nice to be able to leave my browser there and have new events come in automatically.

Currently when we get a PR event we create jobs no matter what.
There have been several cases where people submit PRs to a non-default branch and then the tests fail. For example, BISON has a 1.3 branch that has a static old MOOSE which causes the MAMMOTH tests to fail.

After looking at the example recipes, I think I follow what's going on, but I did have a couple of doubts and wanted to clarify my current understanding.

1. Each of the "steps" in the recipe is executed in the order that it is in the .cfg file? The reference one mentions that each name must be unique, but doesn't confirm the order of execution. (I'm inferring from the example moose test and looking at moosebuild.org that they are executed in order, but I wanted to be sure.)

2. I see each step of the recipe basically is just calling a shell script in the scripts directory. I see from the instructions where the civet_recipes directory should sit, but where should the scripts directory sit? As in the in example repo, i.e. at the same level as the recipes directory, or should there be a recipes/scripts directory?

3. I see on the moosebuild.org page that some of the repos, e.g. moose, show build tracking against devel branchs and master branches, where as the libmesh one does not. How does that work?

4. Is it possible to control when a recipe is executed when it is being simultaneously executed with other recipes? For example, if I have 4 recipes that will be triggered by a PR and I'm fine with the server load that 3 of them will put on, but the 4th one I know is going to be killer (e.g. I execute all the tests on an increasing number of processors), is there a way for me to set "don't execute this recipe until all others in this batch are finished"? Or setting a max load in make kind of my only option?

Thanks much!

This is really minor but maybe we can clean it up. Right now you can cancel any job even if it has completed. Maybe consider disabling that button for completed jobs?

Currently we have endpoints based on internal database ids. This makes things quick and easy internally but is not user friendly.
For example, a repository URL might be https://moosebuild.org/repo/19/ where it would be friendlier to have https://moosebuild.org/repo/idaholab/civet/.
Similarly for branches, prs, etc.

Currently all the actual scripts of a recipe live in a repository. However, there is quite a bit recipe information that is stored in the Civet database.
It would be good if all of the recipe information was stored in the repository. This would allow for full transparency of changes in recipes, easier to revert, easier to copy, easier to bulk manipulate, have a review process for changes, and allow the possibility to test a recipe before committing it.

We could still possibly support editing of recipes through a web page and just do git commands behind the scenes for those people who don't want to submit PRs to make changes.

Currently when viewing a repository we only show branches that have had the status set. We should probably show all branches that have had jobs run against it, regardless of the status.

For example, the repo page for libmesh doesn't have any branches listed since we only do jobs for PRs on libmesh. So the status for a branch never gets set. But it currently isn't easily possible to see all the events on libmesh. If there was a "master" branch button (which is what PRs are against) then we could get a listing.

For repeat offenders (like myself!) it would be nice for the Valgrind target to remain "yellow" on GitHub for pull requests until someone activates and it runs and passes.

Since I have such a problem with it... I always make sure to go activate the Valgrind target manually on any of my pull requests... but it would be nice if it didn't show "green" on my pull-requests ever...

I can think of a few other users that should get this same treatment...

Just a suggestion... feel free to ignore!

@dschwen put this on the moose_build repo which isn't active anymore.

As a first simple step the list of affected files could be subjected to a regex test, where the recipe or step is only executed if at least one file matches the regexp.

We could have these:

Docs /\.(md|yml)$/
Build/Test code /\.(C|h|py|i)$/

(possibly include subdirectory names (e.g. do not run framework tests if changes are made only in moose/modules/ etc.)

It has been suggested that it would be nice to have persistent settings to allow control of what repos are seen on the home page. As we start having more repos the home page just gets more cluttered with apps that some people might not care about.

We you download all the output of a job in a tarball the text can often be hard to read since it has all the color codes from the terminal in there.

If somebody adds too many additional jobs to a PR then bootstrap adds a horizontal scroll to the event table and you can't necessarily see all the jobs for that event.
It might be a good idea to manually break up the row into multiple rows.
At the beginning I did try to use bootstrap's grid system for the events but I never could get it to behave and look like I wanted.

Currently there is a vertical "..." that starts a new line. It has been suggested that just having an empty white block (with no left border) would be better.

Currently when multiple merges happen on MOOSE master, there can be problems with some of the jobs doing an update.
For example, the update submodule recipe can actually do the wrong thing if two are running at the same time. Since some of our build boxes are faster than others, the second job (with the latest version of MOOSE) could complete but the first job (with the older version of MOOSE) keeps on running and updates apps with a version of MOOSE that isn't the current.

A similar thing can happen with doxygen, docs, etc. Although with these we would probably want to run everything to make sure that they are working for that commit but just not update mooseframework.