Would be nice to be able to run this as a GitHub Action.

My idea here is creating a .chain-benchrc | chain-bench.config.json file that the repository will hold
And upon running the cli in the context of that repository it will read that config file for any configuration for chain-bench

Leveraging that config file, add rules key into the json with sub keys pass and fail those will hold assertions over the json output of chain-bench that will decide what will be the exit code.

For now when I want to assert over the chain bench output (JSON format), I am doing it with jq or rego.

Description

When using chain-bench to audit repositories which do not belong to an organization, the process failed.

What did you expect to happen?

As a user I expected the tool to work in a similar way for repositories that do not belong to an organization.

What happened instead?

The audit did not start due to 404 errors being returned since the repository was not part of an organization.

./chain-bench scan --repository-url https://github.com/rgreinho/trauma --access-token $GITHUB_TOKEN -o ./results
2022-06-17 09:08:59 INF 🚩 Fetch Starting
2022-06-17 09:09:00 ERR error in fetching organization error="GET https://api.github.com/orgs/rgreinho: 404 Not Found []"
2022-06-17 09:09:00 INF 🏢 Fetching Organization Settings Finished
2022-06-17 09:09:01 INF 🛢️ Fetching Repository Settings Finished
2022-06-17 09:09:01 INF 🌱 Fetching Branch Protection Settings Finished
2022-06-17 09:09:01 ERR error in fetching members error="GET https://api.github.com/orgs/rgreinho/members: 404 Not Found []"
2022-06-17 09:09:01 ERR Failed to fetch client data error="GET https://api.github.com/orgs/rgreinho/members: 404 Not Found []"
Error: GET https://api.github.com/orgs/rgreinho/members: 404 Not Found []

Additional details (base image name, container registry info...)

Implementation ideas

The type of repository (e.g.: User or Organization) could be determined by querying the repo with the
Get A Repository endpoint and used to adjust next requests:

$ curl -sL https://api.github.com/repos/rgreinho/trauma| jq .owner.type
"User"

Description

Executing this simple command: docker run aquasec/chain-bench scan --repository-url [git repo] --access-token [git token]

Produce these errors:
2022-07-28 03:44:50 INF 🚩 Fetch Starting
2022-07-28 03:44:50 ERR error in authenticated user data
2022-07-28 03:44:50 ERR error in fetching repository data
2022-07-28 03:44:50 INF 🛢️ Fetching Repository Settings Finished
2022-07-28 03:44:50 ERR error in fetching branch protection
2022-07-28 03:44:50 INF 🌱 Fetching Branch Protection Settings Finished
2022-07-28 03:44:50 ERR error in fetching workflows
2022-07-28 03:44:50 INF 🔧 Fetching Pipelines Finished
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0xb3f265]

goroutine 1 [running]:
github.com/aquasecurity/chain-bench/internal/scm-clients/clients.FetchClientData({0x0, 0x0}, {0x7fffb403df52?, 0x1?})
/home/runner/work/chain-bench/chain-bench/internal/scm-clients/clients/clients.go:48 +0x3c5
github.com/aquasecurity/chain-bench/internal/commands.NewScanCommand.func1(0xc000242280?, {0xcfc484?, 0x2?, 0x2?})
/home/runner/work/chain-bench/chain-bench/internal/commands/scan.go:22 +0xac
github.com/spf13/cobra.(*Command).execute(0xc000242280, {0xc00024c040, 0x2, 0x2})
/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:872 +0x694
github.com/spf13/cobra.(*Command).ExecuteC(0xc000242000)
/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:990 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:918
github.com/aquasecurity/chain-bench/internal/commands.Execute({0xe4c9a8?, 0xc0000021a0?})
/home/runner/work/chain-bench/chain-bench/internal/commands/root.go:21 +0x32
main.main()
/home/runner/work/chain-bench/chain-bench/cmd/chain-bench/main.go:12 +0x27

OPA version 0.41.0 supports schema enforcement - https://www.openpolicyagent.org/docs/v0.41.0/schemas.
Please add schema for the object that passes to the Rego files and define the schema of the returned value.

Get the branch name as parameter in case the repository has another branch needed to be checked.

Hi! I'm using the SaaS Community Gitlab version, I have the following error.

chain-bench -v scan --repository-url https://gitlab.com/krol1/go-cowsay --access-token glpat-xxxxxx
2022-11-22 18:26:07 INF 🚩	Fetch Starting
2022-11-22 18:26:10 INF 🛢️	Fetching Repository Settings Finished
2022-11-22 18:26:10 INF 🌱	Fetching Branch Protection Settings Finished
2022-11-22 18:26:11 WRN failed to fetch approval configuration
2022-11-22 18:26:11 DBG failed to fetch approval configuration error="GET https://gitlab.com/api/v4/projects/41277134/push_rule: 404 {message: 404 Not Found}"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x30 pc=0x10151aa04]

It seems that the push_rule it's a premium feature: https://docs.gitlab.com/ee/user/project/repository/push_rules.html

Chain-bench only provides the number of tests that passed e.g.: Total Passed Rules: 5 out of 36, more like unit test frameworks do.

While this provides a good overview, not all the checks are equal and should be weighted accordingly to generate a score for the repository. For example, using MFA should be worth more points, than defining a SECURITY.md file.

As a user I would like to get a score (e.g. 87%) associated to my repositories instead of simply displaying the number of tests that passed.

When a check fails, it is most of the time possible to fix it using the GitHub API.

For example, the number of reviewers required can be updated with one REST request:

curl -s \
  -X PATCH \
  -H "Accept: application/vnd.github.v3+json" \
  -H "Authorization: token $GITHUB_TOKEN" \
  https://api.github.com/repos/buildsec/frsca/branches/main/protection/required_pull_request_reviews \
  -d '{"required_approving_review_count":2}'

Attaching this snippet to the remediation explanation (https://avd.aquasec.com/compliance/softwaresupplychain/cis-1.0/cis-1.0-sourcecode/1.1) would help the end users to resolve the issue.

Current options in the chain-bench version 0.1.3

Flags:
  -c, --config-file string   the path to a local configuration file
  -h, --help                 help for chain-bench
  -l, --log-file string      set to print logs into a file
      --log-format string    sets the format of the logs (normal, json)
      --no-color             disables output color
  -o, --output-file string   the path to a file that will contain the results of the scanning
  -q, --quiet                silence logs, prints only error messages
  -v, --verbose count        set the verbosity level (-v: debug, -vv: trace), default: info
      --version              version for chain-bench

The message could be improved to explain that only support json. Similar like to

-o, --output-file string Export in json the results of the scanning

On the aquasec website listing the issues and their remediation, it is currently not possible to give a link pointing to the exact subsection.

For example, it is only possible to link the "Code Changes" (https://avd.aquasec.com/compliance/softwaresupplychain/cis-1.0/cis-1.0-sourcecode/1.1) page, but not to point directly to the "1.1.3 Ensure any change to code receives approval of two strongly authenticated users" item.

Then this permalink should be used in the "Url" value of the report file generated by chain-bench.

For example:

$ chain-bench scan .

I think currently passing --repository-url is a must.

By default the CLI shows a bunch of detailed errors:

./chain-bench scan --repository-url https://github.com/buildsec/frsca --access-token $GITHUB_TOKEN -o ./results
2022-06-17 09:10:07 INF 🚩 Fetch Starting
2022-06-17 09:10:08 ERR error in fetching organization hooks error="GET https://api.github.com/orgs/buildsec/hooks: 404 Not Found []"
2022-06-17 09:10:08 INF 🏢 Fetching Organization Settings Finished
2022-06-17 09:10:08 ERR error in fetching org packages error="GET https://api.github.com/orgs/buildsec/packages?package_type=npm&state=active: 403 You need at least read:packages scope to list packages. []"
2022-06-17 09:10:13 ERR error in fetching hooks data error="GET https://api.github.com/repos/buildsec/frsca/hooks: 404 Not Found []"
2022-06-17 09:10:13 INF 🛢️ Fetching Repository Settings Finished
2022-06-17 09:10:13 ERR error in fetching branch protection error="GET https://api.github.com/repos/buildsec/frsca/branches/main/protection: 404 Not Found []"
2022-06-17 09:10:13 INF 🌱 Fetching Branch Protection Settings Finished
2022-06-17 09:10:13 INF 👫 Fetching Members Finished
2022-06-17 09:10:13 WRN file .github/workflows/ci.yaml not found
2022-06-17 09:10:14 WRN file dynamic/pages/pages-build-deployment not found
2022-06-17 09:10:14 INF 🔧 Fetching Pipelines Finished
2022-06-17 09:10:14 INF 🏁 Fetch succeeded

But these errors just clutter the output, and are not very useful unless debugging information is needed. These types of details should be displayed for debug or tracing log level.

As a user I should be able to adjust the log level from the CLI, for example by supplying -v flags (1 for info, 2 for debug, 3 for trace).

Description

Hey team!
I saw that you added support for GitLab (beta). I tried to run scan against dummy repo hosted on GitLab, but unfortunately it failed. I created a token with the appropriate role and permissions.

What did you expect to happen?

The scan has run successfully

What happened instead?

The scan immediately failed

Out from gitlab ci/cd runner:

$ chain-bench scan --repository-url $CI_PROJECT_URL --access-token $CHAIN_BENCH_TOKEN -o results.json --template @/templates/gitlab_security_scanner.tpl
2022-10-26 11:31:13 INF 🚩	Fetch Starting
2022-10-26 11:31:14 ERR error in fetching repository data
2022-10-26 11:[31](https://gitlab.com/XYZ/security/dummy-repo/-/jobs/3229947555#L31):14 INF 🛢️	Fetching Repository Settings Finished
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xbd97a5]
goroutine 1 [running]:
github.com/aquasecurity/chain-bench/internal/scm-clients/clients.FetchClientData({0x7ffd195cb03b, 0x1a}, {0x7ffd195caff6?, 0x1?}, {0x0, 0x0})
	/home/runner/work/chain-bench/chain-bench/internal/scm-clients/clients/clients.go:40 +0x1a5
github.com/aquasecurity/chain-bench/internal/commands.NewScanCommand.func1(0xc000252a00?, {0xe3962a?, 0x8?, 0x8?})
	/home/runner/work/chain-bench/chain-bench/internal/commands/scan.go:22 +0xcc
github.com/spf13/cobra.(*Command).execute(0xc000252a00, {0xc00024e800, 0x8, 0x8})
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:872 +0x694
github.com/spf13/cobra.(*Command).ExecuteC(0xc000252780)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:990 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:918
github.com/aquasecurity/chain-bench/internal/commands.Execute({0xf93cc8?, 0xc0000021a0?})
	/home/runner/work/chain-bench/chain-bench/internal/commands/root.go:21 +0x[32](https://gitlab.com/tidio/security/automated-aws-audit/-/jobs/3229947555#L32)
main.main()
	/home/runner/work/chain-bench/chain-bench/cmd/chain-bench/main.go:12 +0x27
Uploading artifacts for failed job

Hi! could be possible the output of chain-bench result deliver a sarif report to integrate with Github?

https://docs.github.com/en/code-security/code-scanning/integrating-with-code-scanning/uploading-a-sarif-file-to-github

Description

scan throws segmentation fault.

What did you expect to happen?

$ chain-bench scan --repository-url github.com/Dentrax/cocert --access-token $TOKEN

What happened instead?

Error line:

2022-07-07 17:12:19 INF 🚩      Fetch Starting
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x17370bb]

goroutine 1 [running]:
github.com/aquasecurity/chain-bench/internal/scm-clients/clients.FetchClientData({0x7ff7bfeff380, 0x1}, {0x7ff7bfeff357?, 0x1?})
        /home/runner/work/chain-bench/chain-bench/internal/scm-clients/clients/clients.go:32 +0x9b
github.com/aquasecurity/chain-bench/internal/commands.NewScanCommand.func1(0xc00020e280?, {0x18f4738?, 0x4?, 0x4?})
        /home/runner/work/chain-bench/chain-bench/internal/commands/scan.go:22 +0xac
github.com/spf13/cobra.(*Command).execute(0xc00020e280, {0xc000175f80, 0x4, 0x4})
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:856 +0x67c
github.com/spf13/cobra.(*Command).ExecuteC(0xc00020e000)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
github.com/aquasecurity/chain-bench/internal/commands.Execute({0x1a41c1c?, 0xc0000021a0?})
        /home/runner/work/chain-bench/chain-bench/internal/commands/root.go:21 +0x32
main.main()
        /home/runner/work/chain-bench/chain-bench/cmd/chain-bench/main.go:12 +0x27

Additional details (base image name, container registry info...):

Hi! I can see in the results that it containers the total of controls with status passed/failed and unknown.
How can I know what of the controls represent a critical risk ?

Thank you

Description

When saving the results as JSON, the keys are formatted in a non-conventional nor consistent manner. This makes deserializing the file with other tools unnecessary complicated.

What did you expect to happen?

I expected all the keys to use a conventional encoding, like lowercase or camelCase.

For instance, looking at the JSON API recommendations (https://jsonapi.org/recommendations/), they recommend the keys to use camelCase (which seems to be the most commonly accepted one):

Member names SHOULD be camel-cased (i.e., wordWordWord)

Looking at several libraries, the more common encodings are:

• "lowercase"
• "UPPERCASE"
• "PascalCase"
• "camelCase"
• "snake_case"
• "SCREAMING_SNAKE_CASE"
• "kebab-case"
• "SCREAMING-KEBAB-CASE"

What happened instead?

Instead the id key is UPPERCASE, and the other ones are Capitalized.

[
  {
    "ID": "1.1.3",
    "Name": "Ensure any change to code receives approval of two strongly authenticated users",
    "Descrition": "Ensure that every code change is reviewed and approved by two authorized contributors who are strongly authenticated.",
    "Remediation": "An organization can protect specific code branches — for example, the \"main\" branch which often is the version deployed to production — by setting protection rules. These rules secure your code repository from unwanted or unauthorized changes. You may set requirements for any code change to that branch, and thus specify a minimum number of reviewers required to approve a change.",
    "Result": "Failed",
    "Reason": "",
    "Url": "https://avd.aquasec.com/compliance/softwaresupplychain/cis-1.0/cis-1.0-sourcecode/1.1"
  }
]

Additional info

Tools like gomodifytags can simplify the process of mass editing the struct tags.

Description

For controls 1.2.3 and 1.2.4 it always shows PASSED irrespective of the setting in Github

 1.2.3    Ensure repository deletion is limited to specific members                                       Passed                                                                           
 1.2.4    Ensure issue deletion is limited to specific members                                            Passed                                                                           

Additional details in case it does not expose sensitive data (scanned pipeline files, PR, etc):

Here's the snippet from mapper.go which denotes the value has been hard-coded for the setting.

Reference GitHub setting snapshot which allows users to restrict/allow repository deletion and issue deletion -

Is there any plan to cut a new release? We would need for PR #71 to be available to move forward with our work.

From the PDF:

The hope with the publication of this Guide is to elicit feedback from the global community that will help ensure the future platform-specific guidance (CIS Benchmarks) is even more accurate and relevant.

To facilitate feedback (comments, issues, PRs, etc), it would be great if the recommendations were available in a format like Markdown.

Using the json output: chain-bench scan -r $repo --access-token ${GITHUB_AUTH_TOKEN} -o my.json

The result don't have any information about the repository URL.

{
  "metadata": {
    "date": "2022-09-14T22:49:49-03:00",
    "statistics": {
      "passed": 5,
      "failed": 3,
      "unknown": 28,
      "total": 36
    }
  },
  "results": [
    {
      "id": "1.1.3",
      "name": "Ensure any change to code receives approval of two strongly authenticated users",
      "description": "Ensure that every code change is reviewed and approved by 

A GitHub Action would be very nice to be able to run it.

Originally posted by @hazcod in #26

GitHub provides a list of community standard checks that help improving the quality of a repository.

For instance for the frsca project:

As a user I would like chain-bench to report this missing community standards so that I can improve the overall quality of my repositories.

Description

Support Bitbucket server SCM in addition of Github and Gitlab.

Description

Issue when the sub group has the same name as the repository.
Example: https://my-gitlab-instance.com/top-group/sub-group/repo-name/repo-name

getRepoInfo returns:

host: my-gitlab-instance.com
namespace: top-group/sub-group
repo: repo-name
err: %!s(<nil>)

What did you expect to happen?

should return:

host: my-gitlab-instance.com
namespace: top-group/sub-group/repo-name
repo: repo-name
err: %!s(<nil>)

What happened instead?

getRepoInfo returns:

host: my-gitlab-instance.com
namespace: top-group/sub-group
repo: repo-name
err: %!s(<nil>)

Made a PR for this but due to the contribution guidelines I created this issue as well.

The CLI format table has static columns, but the JSON results file has more data that is worth showing in the table, particularly "Remediation". Presumably the goal is for someone to see the results and try to improve them, and it's easier to see the remediation suggestions in table form than in JSON.

Description

I was reviewing the pdf of checks: https://github.com/aquasecurity/chain-bench/blob/main/docs/CIS-Software-Supply-Chain-Security-Guide-v1.0.pdf and noticed that for section 1.2, there's a duplication of the 1.2.2 value, so it reads as 1.2.2: 1.2.2 Name Of Check

What did you expect to happen?

n/a

What happened instead?

n/a

Additional details (base image name, container registry info...):

This is just for your information if you want to fix it, it causes no actual problems that I can see.

Description

In JSON output, links to compliance details are broken because they're missing a trailing slash

What did you expect to happen?

Links work

What happened instead?

Links are broken

(paste your output here)

Additional details (base image name, container registry info...):

Output from github action:
"url":"https://avd.aquasec.com/compliance/softwaresupplychain/cis-1.0/cis-1.0-sourcecode/1.1"

working link: https://avd.aquasec.com/compliance/softwaresupplychain/cis-1.0/cis-1.0-sourcecode/1.1/

Description

The check "3.2.3: Ensure packages are automatically scanned for license implications" does not seem to be implemented. At https://github.com/aquasecurity/chain-bench/blob/main/internal/checks/dependencies/validate_packages/rules.rego#L16, it appears to be checking the same thing as 3.2.2, whether there are vulnerability scan tasks.

What did you expect to happen?

It checks for license scan tasks.

What happened instead?

It checks for vuln scan tasks

are_pipelines_dependencies_scanned_for_licenses {
	count({job | job := input.Pipelines[_].jobs[_]; does_job_contain_one_of_tasks(job, constsLib.pipeline_vulnerability_scan_tasks)}) == 0
}

Additional details (base image name, container registry info...):

The remediation does not really explain how to fix this. It would be more helpful if it specifically said which types of pipeline tasks it is looking for, in both the vuln scan and license scan checks.

Similar to go report, add the option to add the Chain Bench score to a repository's README.md

It will be nice to use the chain-bench in local mode, for example

chain-bench --local .

Chain-bench currently only supports the SaaS version of GitHub and Gitlab, it would be useful to have support for other self-hosted or private SCMs.

At present the documentation states this needs full repo access, it would be advantageous to make this only require read permission scopes.

If this isn't possible - it should use the new Github fine grained tokens which provide improved permission scopes.

https://github.blog/2022-10-18-introducing-fine-grained-personal-access-tokens-for-github/

"It is required to provide an access token with permission to these scopes: repo(all), read:repo_hook, admin:org_hook, read:org"

Does chain-bench recognize code signing tools like sigstore (cosign, fulcio, rekor)?

It's OK with github.com for instance. But "panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xb3ef3b]" with corporative gitlab repository even for admin with all token permissions.

As a user, I would like to be able to compare scan results to ensure we've improved and make sure we did not regress over time. This could also help capturing human mistakes (e.g. an admin changed a setting by mistake).

Add metadata

An idea could be to add a metadata section at the top of the file, which would contain at least the following information:

• Date
• Result score (related to #37)

Description

Running ./chain-bench scan --repository-url https://github.com/buildsec/frsca --access-token $GITHUB_TOKEN -o ./results, some checks where marked as failed due to not having enough permissions associated with the GitHub token.

For instance:

1.1.3 Ensure any change to code receives approval of two strongly authenticated
users Failed

The problem was that the GITHUB_TOKEN did not have permissions to read the branch protection settings, therefore marking this check as failed.

However it should in this case show another status, like Unknown or not evaluated for instance, as the check was not able to read the results (the endpoint returned a 404).

The failed status implies that the requirement was not met, and should be reserved for cases where the number of required reviewers was strictly less than 2.

The same problem applies to all the checks that returned a 404.

Additional details in case it does not expose sensitive data (scanned pipeline files, PR, etc):

Description

1.1.16 states that for each repository in use, we must validate that no one can “force push” code.

1.1.17 states that for each repository that is being used, we must verify that protected branches cannot be deleted.

The rule logic for these two benchmarks appears to be written in such a way that it produces false positives. When Allow force pushes and Allow deletions are checked, thus permitting the ability to force pushes and/or delete branches, Chain-Bench outputs a Passed where a Failed would be expected.

The opposite will happen if you have them unchecked - you'll get a Failed result.

Looking at the rule logic in question

#Looking for default branch protection that restrict force push to branch
CbPolicy[msg] {
	not is_no_branch_protection
	is_branch_protection_restrict_force_push
	msg := {"ids": ["1.1.16"], "status": constsLib.status.Failed}
}

#Looking for default branch protection that restrict who can delete protected branch
CbPolicy[msg] {
	not is_no_branch_protection
	is_branch_protection_restrict_delete_branch
	msg := {"ids": ["1.1.17"], "status": constsLib.status.Failed}
}

this reads to say "when the branch is protected and disallows force pushes or deletions (in other words, if AllowForcePushes and AllowDeletions == false), produce a Failed result. In my mind, this should read as "when the branch is protected and allows force pushes or deletes, produce a Failed result.

Prepending not to both L226 and L233 causes Chain-Bench to produce an expected result.

Hello!
First of all I would like to say that I really appreciate your work to add support for GitLab. I know that currently chain-bench GitLab are in beta but how many checks are in GitLab scan now?

Description

Is there a reason for the non-release tags like 0.1.4+1? Do they serve any purpose or can they be removed?

Description

Clicking on https://github.com/aquasecurity/chain-bench/blob/main/docs/CIS-Software-Supply-Chain-Security-Guide-v1.0.pdf provide the following error

Description

I tried to use chain-bench with our private GitLab instance.
Unfortunately, it fails with error in authenticated user data.
Starting in with -v reveals that the cause is:
error in authenticated user data error="GET https://myinstance/api/v4/1: 404 {message: 404 User not Found}"

This makes sense because we deleted the default root user with the id 1.
The lowest User ID in our Instance is 2.
I'm unsure why chain-bench requires the default user to work.
Imho it is quite common to delete it for security reasons after you created different admin accounts.

What did you expect to happen?

I expected chain-bench to scan my repository

What happened instead?

error in authenticated user data error="GET https://myinstance/api/v4/1: 404 {message: 404 User not Found}"

Additional details (base image name, container registry info...):

Tested with Version 0.1.7

the chain-bench could support SLSA requirements. https://slsa.dev/spec/v0.1/requirements