cburgmer / buildviz Goto Github PK

When storing job information, fail early for unknown parameters as to provide early feedback.

Currently dots seem to trigger a 400 from buildviz server.

Let the sync jobs be identifiable, to give API providers some way of understanding their traffic.

Showing older test names (or even a temporary rename which will result in a single datapoint and so be prone to outliers) does not provide value. Filtering those out might be a trivial and effective solution.

Sync should stop for first ongoing build so we can resume later.

transform_tests.clj:7 disallows colons in test names. We should be more lenient, considering that we are guessing anyways.

The buildviz.jenkins.sync job fails with a String format exception if a percentage sign shows up in the base URL (say due to basic auth).

• wait time graph should show which build triggered the waiting build in tooltip
• pipeline runtime should show build ids in tooltip
• pipeline runtime hover over line should show tooltip (not just circles)
• wait time graph has not UI test

Although http://llg.cubic.org/docs/junit/ describes classname as required, the JSON schema does not require a classname.

For several inputs the ordering of the revision with source_id is important. Two same inputs in different order will not me recognised as the overall same build input.

http://cburgmer.github.io/buildviz/ci.jenkins-ci.org/#graph_averageTestRuntime has one burst per test class, and we can't shine with the hierarchical structure that we normally offer.

Null values are not caught for required start field. Fixed upstream probably, but not released: bigmlcom/closchema#35

Pipeline triggers are modelled on stage level. Modelling this on job level doesn't seem to be straight forward.

Working with outdated sync can be tedious and unintuitive as the build data slowly vanishes from the graphs, as newer dates are just empty if no new sync has been undertaken. In the case of the examples inside the repo, they just plainly don't show anything for historical builds, if they are e.g. a year back.

Rather let's just show the last x days/months offset from the latest build synched.

Gosync currently will ask buildviz for the newest job and start from there. However it will look only on a stage level, and compare the stage scheduled-time, which is a few seconds earlier than the actual job building-time.

Given gosync was started at time T, it would pick up all stages scheduled until that time. If this included only jobs building at T+2, then stages scheduled at T+1 would not be included on a subsequent sync. Instead gosync would start at T+2.

See e.g. hack in 4eeccf3

.. in Chrome

Workaround for 1575 will leave the job with a nil value for start, and will probably fail at the schema validation, now that start is required.

Only solution is probably to wholly ignore past builds from renamed items.

Given package com.example.a and class a in package com.example, both accumulated test runtimes will collide when rendered in the 'Average test runtime' graph.

Grouping by package structure will make it easier to find out where time is spent.

Similar to the TeamCity sync process, don't print the password inside the URL to stdout/logs.

Currently a 400 error fails the script

Go.cd (and other's maybe as well) first schedule a job after an external event (e.g. source change), and when a build node gets available assigns a job.

Including the delay between both in the job runtime in statistics is misleading, as an increased timing there indicates that build system might be overloaded, rather than the actual step taking more time. However monitoring the delay is important when planning to reduce cycle time.

When builds are stacking up, the following job might be triggered by multiple builds:

"causes" : [
        {
          "shortDescription" : "Started by upstream project \"Deploy\" build number 161",
          "upstreamBuild" : 161,
          "upstreamProject" : "Deploy",
          "upstreamUrl" : "job/Deploy/"
        },
        {
          "shortDescription" : "Started by upstream project \"Deploy\" build number 162",
          "upstreamBuild" : 162,
          "upstreamProject" : "Deploy",
          "upstreamUrl" : "job/Deploy/"
        }
      ]

Currently the first cause is selected, any further ignored. This then will show shorter, partial pipeline runs in the pipeline graph.
The upcoming wait time graph will also not calculate the consequently longer wait times.

Show

• newest slowest test: Create awareness of what changes had negative impact.
• projection of test runtime in 1 year: Allow evaluation of current development, help re-think current choices.
• build runtime by hour of day: Infrastructure load throughout day.
• top n slowest test runtime over time: Feedback on improvements around slow tests.
• Feedback time from check-in: Difference between time of source control check-in and execution of build.
• Meantime to pipeline recovery: How long does the team take to get a pipeline (not all builds) green again (thanks Duda)
• Meantime to feedback: how long until some build fails in the pipeline with a real error (not flaky)
• Build runtime variance: find builds which vary greatly in runtime (possibly due to external deps)
• Maybe we can cluster test runtimes into up to 4-5 buckets, to get a quick overview on how fast the system's tests are. This would provide more transparency into where most of the time is spent.
• Amount of manual re-runs (per day)
• Bottleneck/throughput (following lean principles, what's keeping my pipeline from being faster)

Offer CSV as alternate output so the statistics can be processed in other tools.

• Remove source_id ?
• job (in responses) vs. job-name in triggered-by information

The XML generated by JUnit 5 can be rejected by buildviz with a 400. The offending content looks like this:

<testcase name="my_test" classname="com.example.something" time="14,029.255">
</testcase>

Investigate how flaky tests are currently found.

Technically the algorithm does not need to rely on the build outcome, but just needs to look at all build pairs with same input, and find tests with different outcome.

This would make sure to find flaky tests even if the build still fails for other reasons.

On going phase (whether red/green) is not shown. As the end of the phase is unknown, we could take the timestamp of the last synced build as a current value.

Provide less items in an overview, then only provide all for a "zooming in" (what ever that will mean).

There are two fields we can extract that information from:

    "snapshot-dependencies": {
        "build": [
            {
                "buildTypeId": "SimpleSetup_Test",
                "href": "/httpAuth/app/rest/builds/id:46",
                "id": 46,
                "number": "14",
                "state": "finished",
                "status": "SUCCESS",
                "webUrl": "http://localhost:8111/viewLog.html?buildId=46&buildTypeId=SimpleSetup_Test"
            }
        ],
        "count": 1
    }

and

    "triggered": {
        "date": "20160508T071736+0000",
        "details": "##triggeredByBuildType='bt3' triggeredByBuild='14'",
        "type": "unknown"
    }

snapshot-dependencies seems to have the data in the format we need, but seems to have a bunch of problems:

1. This points to the previous build, even if there is a temporal disconnect due to separate user actions (running the latter job when there's a successful build of the job in the chain before seems to omit a complete chain invocation). As such not well suited for temporal analysis of the pipeline.
2. Requires the pipeline to use the snapshot dependencies configuration for TeamCity, which might make setup difficult as buildviz will not sync out-of-the-box for a setup without. (Unsure how widely this config is used across teams.)

triggered seems to have its own set of issues:

1. The necessary data is encoded in an internal representation, and might change in the future. Resolving though is possible as the alternate id resolves against /app/rest/buildTypes/bt3 for example.
2. In some cases if the latter job of a pipeline chain is triggered, this could lead to the previous chain to be invoked. However the triggered value will only call out the user action (type: user).

Current example has a flaky build with count 1 and a test with > 100 flaky failures.

List of user feedback

• "Top X diagrams": use classic bar charts, much easier to grasp
• Flaky tests: show rings of flaky tests by day, younger inside
• Top 5 failures: Size should show amount of runs, easier to grasp, shows same info as other graphs
• Slowest 5 tests: show average as text, not as graph
• Testcase vs. testclass graphs, difficult to relate
• Simplify descriptions for somebody less focused on build pipeline issues

In Go:
Given two failing jobs in a stage, when rerunning one of the failed one, subsequently turning into green, then the result of the stage will be incorrectly reported as passed.

At least that might be the culprit why curl http://localhost:3000/testsuites | sort -rnt, -k5 returns a ghost entry.

The GoCD sync should not fail on build artifact files that are not JUnit XML. The current heuristic of checking the filename is not good enough.

A simple magic check on top could suffice to weed out other XML files.

Wiremock supports recording and replaying previous request patterns. This might enable end2end testing of the sync implementations without starting up the Vagrant boxes. This could also close the gap left when removing the previous sync through Wiremock in 9e70d51.

cctray.xml for Go.cd exports a "build" like pipeline :: stage :: job, we should use the same for consistency. However that would be a breaking change.

If a job is removed or just renamed (id changes) and the last build failed, the fail phase goes on indefinitely as that job is never turned back into green.

Possible solution:
Fuzzy logic by which a certain interval without new builds means the job has been decommissioned.

Subproject names in TeamCity include colons (e.g. Project :: Subproject) which is invalid for file names under Windows.

Right now multiple ones don't accumulate runtime but actually outcome is lower due to avg.

Although http://llg.cubic.org/docs/junit/ mentions that testcase time is optional:

1. Some tests graphs cannot handle missing runtime.
2. The test JSON schema requires runtime.

As teams will likely end up with multiple independent sets of jobs we should probably support some kind of separation of those sets. This could be a complex setup of multiple pipelines or many simple pipelines of microservices.

Color coding is one important visual aspect, as jobs of one group should by identifiable.

The "fail phases" graph follows a pipeline model, aggregating multiple pipelines in one overview will conflate what's happening below.

Ideas:

1. Provide namespaces when posting job/build information /builds/:ns/:job/:build
2. Trick D3 into providing similar colors for jobs of the same namespace

What we don't want:
buildviz does not want to hold information for multiple teams, rather it should be easy to deploy multiple instances of it.