This is a Concourse pipeline that scans sites for vulnerabilities using OWASP ZAP. This is part of 18F's Compliance Toolkit project, and is essentially the back end of Compliance Viewer.

The config/targets.json is a list of the projects to be scanned. Since ZAP can inject junk data if it's successful in finding certain vulnerabilities, we suggest using a staging URL. To get a new project added:

1. Submit a pull request to this repository to add an entry in config/targets.json like this:

   {
     // Needs to be all lower-case.
     "name": "NAME",
     // (optional) Channel in the 18F Slack to get notifications in.
     "slack_channel": "CHANNEL",
     // Links to scan.
     "links": [
       {
         "url": "URL"
       }
     ]
   }

2. After the pull request is merged, ask someone in #cloud-gov-highbar to run

   TARGET=<fly_target> rake init_targets
   TARGET=<fly_target> rake deploy

• name - This should be all lowercase.
• slack_channel (optional) - This should be the channel where you'd like to get alerts for completed scans. If left out, the alerts will be sent to the default channel, currently #ct-bot-attack.
• links - An array of links that should be scanned with ZAP. The results will be concatenated together.

The running pipeline depends on this repository for the tasks to be performed and targets to scan. By default, the pipeline pulls the master branch for these tasks, but it can be pointed at a different branch for testing.

Normal users of Compliance Toolkit do not need access to the Concourse CI. The pipeline publishes output in a few different modes.

Primarily, the pipeline publishes the ZAP scan results as a JSON file to S3. This is the information that is consumed by the user via Compliance Viewer.

The pipeline also published two types of Slack notifications. The first is a heartbeat notification; it is published to a central channel (currently #ct-bot-attack, but configurable in the pipeline) after every run to confirm that the run happened. This is for the Compliance Toolkit team to monitor that the process is functioning.

The second is for the project teams. It is published to the channel defined in targets.json, or the central channel (as the above notifications) if no channel is defined. It is only published if there is a change in the results. It also includes a link to the results in Compliance Viewer.

For each project, there are two jobs defined, a scheduled job, and an on-demand job. This is due to an oddity in the way Concourse jobs are triggered. If there is a time-based trigger defined, you can not run it at another time. The scheduled job runs every day at midnight. All the project scans are triggered simultaneously, but there are a limited number of workers available. The scans will be queued until a worker becomes available.

Each scan is a multi-step process:

1. Triggered at 12:00 AM.
2. Retrieves scripts to run from the GitHub repository.
3. Retrieves the prior scan results from S3.
4. Performs some filtering/scrubbing of the prior scan results.
5. Run the ZAP scan via zap-cli. The ZAP scan has several sub steps of its own:
   1. Run the spider the current target.
   2. Run the AJAX spider for the current target.
   3. Scan the target.
   4. Output the detected alerts.
6. Repeat i-iv for every target defined for the project in targets.json.
7. Concatenate the results files into a single file.
8. Upload the results file to S3.
9. Summarize the results and the difference between the prior and current scan.
10. Post the two slack messages (heartbeat & notification, described above)
11. Upload the summary results to S3.

These steps are performed for each project in a parallelized fashion.

Give us your feedback! We'd love to hear it. Open an issue and tell us what you think.

This project is in the worldwide public domain. As stated in CONTRIBUTING:

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.