A web dashboard surfacing the state of github security advisories across our estate

License: MIT License

Dockerfile 0.21% Makefile 0.81% Python 49.06% HTML 28.90% Shell 1.39% HCL 19.63%

cyber-security-security-advisory-dashboard's Introduction

Security Advisory Dashboard

Prerequisites

Check you have docker and docker-compose installed

docker --version
docker-compose --version

Set a GITHUB_ORG env var containing the organisation login short name from the GitHub URL.
Set a TOKEN env var containing a read-only personal access token for a GitHub org admin user. The permissions needed for the token are:

repo, read:org, read:public_key, read:repo_hook, read:user, user:email

(Alternatively the token can be retrieved from SSM if the make command is run via aws-vault or similar to set AWS credentials.)

Create or change the settings.[env].json file

Run an audit

make audit

Gets the paged repository data with vulnerability alerts

The audit process runs the api calls to collect vulnerability and activity data from github as well as the dependabot config API to determine which repositories have dependabot enabled.

Run audit component tasks

You can run individual tasks from the audit process for testing.

For example to rebuild the interface route template data files you can call the following:

make task TASK=routes

You can call the tasks separately because the full audit takes a long time to run.

Run a local dev server

The run task currently runs the npm install and then runs the gulp tasks to build the static assets, js and css.

make run

Run the tests

Run the unit tests by running

make test

Terraform

Build

Before you can run the terraform you need to create a zipped lambda deployment.

You can do that by running

make zip

TODO We can probably make the terraform run the zip command

Init

The terraform is in build/terraform

To init you need a backend.tfvars

bucket  = "<bucket name>"
key     = "<state file path>"
region  = "eu-west-2"
encrypt = true

Then you can run

terraform init -reconfigure -backend-config=path/to/backend.tfvars

Plan or apply

You need an apply.tfvars

region              = "eu-west-2"
bucket_prefix       = "cyber-security"
runtime             = "python3.7"

github_org          = "<github organisation shortname>"

Service             = "github-audit"
SvcOwner            = "<who to email>"
Environment         = "<should match a setting file env>"
DeployedUsing       = "Terraform"
SvcCodeURL          = "https://github.com/alphagov/cyber-security-security-advisory-dashboard"

Then you can run

terraform apply -var-file=path/to/apply.tfvars

Find vars at

https://github.com/alphagov/cyber-security-terraform/tree/master/service/github_audit/account/103495720024

cyber-security-security-advisory-dashboard's People

Contributors

Stargazers

Watchers

Forkers

alphagov-mirror denomas tubbz-alt uk-gov-mirror lgtm-migrator pritchyspritch

cyber-security-security-advisory-dashboard's Issues

concourse pipeline deploys even if our tests fail

When we push codes that causes our tests to fail to master, the tests will run on concourse and fail as expected, but then continue onto the deploy stage and deploy the broken code.... which is obviously not what we want.

Fix the redundant/repeated step in the analyse_vulnerability_patch_recommendations function

The analyse_vulnerability_patch_recommendations() function has a repeated step as a hacky fix to ensure we add the maxSeverity, patches and vulnerabiltyCounts fields to the vulnerable_by_severity.json file.

Add Liscence

Routes task assumes you ran an audit today

The routes task in app.py assumes that you have run an audit "today", meaning it won't run properly unless you ran an audit on the same day.

contract tests on concourse notify archived channel

when the contract tests fail they post a slack notification to a channel that has now been archived - we should change this

Add a self update job into the pipeline

requirements-dev.txt does not refer to requirements.txt

Hi guys, I ran test.sh to check with some Dependabot PRs, and I got errors telling me that I was missing requests and addict. I figured out that this was because these packages were only referenced in requirements.txt, and requirements-dev.txt does not reference it.

Adding a

-r requirements.txt

to the top of the file would be appreciated - or an explanation for the separation between these two files.

Thanks!

github advisories lambda not sending events to splunk

The lambda is currently not sending anything through to splunk - this should be fixed as a priority as defence uses this data in reports.

The last time the data was coming through on splunk was 26th April

Refactor repositories.json from lists of dicts into dict of dicts

Index by repository name.

No events in `github_security` index since 27/05/2021 23:09:19

Splunk alert has been sending notifications to Slack health monitoring channel that there are no recent events in the github_security index since 27/05/2021 23:09:19

Requires further investigation.

Change bot user to have a google group email address that sends to engineering ^^^^

Currently the bot user account has Sam P's email address.
We need to transfer that to our ownership
Then we need to generate a new PAT and update in SSM.

Switch email to cyber-security-engineering+advisory-bot@
Verify email
Store creds into shared place
Generate new PAT
store in /ssd/github/pat for staging service
store in /ssd/github/pat for production service
validate new PAT works

Repeated Splunk HEC connections

Running the audit_lambda.py from the CLI it was observed that code makes repeated connections to the Splunk Cloud HEC endpoint.

Fix should be to parse the events and send events in batches, this would make the Lambda run more efficiently -:

https://docs.splunk.com/Documentation/Splunk/8.2.4/Data/FormateventsforHTTPEventCollector

Ref Example 3: Batched data

Do requirements install in pipeline ^

At present the requirements install is done in the concourse-base-image pipeline after the base image is rebuilt. That means that the pipeline is not always run with the current version of the dependencies. This is why the recent dependabot graphql-core breaking-change merge passed the tests.

Repeated calls to SSM

Running the audit_lambda.py on the CLI, the debug output shows it makes repeated calls to SSM. The code should make a single SSM get_parameter/s request.

[DO NOT MERGE] Upgrade version of `graphql-core` to >3.0.0 without breaking `github_security` audit events ^

This shouldn't be merged until we can also upgrade to gql:3 currently in alpha

Dependabot alert to bump the version of graphql-core broke the github audit events.

We reverted back to <3.0.0 but we should look into upgrading to resolve the following issues -:

#572

graphql-python/graphql-core#125
graphql-python/graphql-core#126

Dependabot couldn't authenticate with https://pypi.python.org/simple/

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

SKIPPED [1] tests/test_storage.py:33: Breaks when run out of order
SKIPPED [1] tests/test_storage.py:62: Breaks when run out of order
SKIPPED [1] tests/test_storage.py:43: Breaks when run out of order
FAILED tests/test_app.py::FLAKE8 - AttributeError: 'Application' object has no attribute 'parse_preliminary_options_and_args'
FAILED tests/__init__.py::FLAKE8 - AttributeError: 'Application' object has no attribute 'parse_preliminary_options_and_args'
FAILED tests/test_config.py::FLAKE8 - AttributeError: 'Application' object has no attribute 'parse_preliminary_options_and_args'
FAILED tests/test_splunk.py::FLAKE8 - AttributeError: 'Application' object has no attribute 'parse_preliminary_options_and_args'
FAILED tests/test_storage.py::FLAKE8 - AttributeError: 'Application' object has no attribute 'parse_preliminary_options_and_args'
FAILED tests/test_vulnerable_by_severity_splunk.py::FLAKE8 - AttributeError: 'Application' object has no attribute 'parse_preliminary_options_and_args'
FAILED tests/test_cyber_dependabot.py::FLAKE8 - AttributeError: 'Application' object has no attribute 'parse_preliminary_options_and_args'
FAILED tests/test_audit_lambda.py::FLAKE8 - AttributeError: 'Application' object has no attribute 'parse_preliminary_options_and_args'
FAILED tests/test_vulnerability_summarizer.py::FLAKE8 - AttributeError: 'Application' object has no attribute 'parse_preliminary_options_and_args'```

alphagov / cyber-security-security-advisory-dashboard Goto Github PK