cyclonedx / sbom-utility Goto Github PK

TODO: figure out how to support both the v1.5 "Identity" type (a singleton of an anonymous type) vs. the v1.6 "identity" which is an array of named type (i.e., componentIdentityEvidence):

type CDXComponentEvidence struct {
	Licenses  *[]CDXLicense   `json:"licenses,omitempty" cdx:"added:1.3"`
	Copyright *[]CDXCopyright `json:"copyright,omitempty" cdx:"added:1.3"`
	Identity  interface{}     `json:"identity,omitempty" cdx:"added:1.5,changed:1.6"`
...
}

where these struct types would be referenced:

type CDXComponentIdentityEvidence struct {
	Field      string       `json:"field,omitempty" cdx:"added:1.5"`
	Confidence float64      `json:"confidence,omitempty" cdx:"added:1.5"`
	Methods    *[]CDXMethod `json:"methods,omitempty" cdx:"added:1.5"`
	Tools      *[]string    `json:"tools,omitempty" cdx:"added:1.5"`
}

// v1.5: added
type CDXMethod struct {
	Technique  string  `json:"technique,omitempty" cdx:"added:1.5"`
	Confidence float64 `json:"confidence,omitempty" cdx:"added:1.5"`
	Value      string  `json:"value,omitempty" cdx:"added:1.5"`
}

celek sbom-utility-v0.15.0-darwin-arm64 >> ./sbom-utility license list --input-file acdx.json --format csv
Welcome to the sbom-utility! Version `v0.15.0` (sbom-utility) (darwin/arm64)
============================================================================
[INFO] Loading (embedded) default schema config file: `config.json`...
[INFO] Loading (embedded) default license policy file: `license.json`...
[INFO] Attempting to load and unmarshal data from: `acdx.json`...
[INFO] Successfully unmarshalled data from: `acdx.json`
[INFO] Determining file's BOM format and version...
[INFO] Determined BOM format, version (variant): `CycloneDX`, `1.4` (latest)
[INFO] Matching BOM schema (for validation): schema/cyclonedx/1.4/bom-1.4.schema.json
[INFO] Scanning document for licenses...
[WARN] invalid SBOM: licenses not found (metadata.licenses) (acdx.json)
[WARN] invalid SBOM: licenses not found (metadata.component) (acdx.json)
[INFO] Outputting listing (`csv` format)...
License.Id,License.Name,License.Url,Expression,License.Text.ContentType,License.Text.Encoding,License.Text.Content
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x104bfa024]

goroutine 1 [running]:
github.com/CycloneDX/sbom-utility/cmd.DisplayLicenseListCSV(0x140001d4380, {0x104ec1f28?, 0x14000110028?})
	/github/workspace/cmd/license_list.go:331 +0x514
github.com/CycloneDX/sbom-utility/cmd.ListLicenses({0x104ec1f28, 0x14000110028}, 0x0?, {0x0, 0x0, 0x0, {0x16b56f629, 0x9}, {0x0, 0x0}, ...}, ...)
	/github/workspace/cmd/license_list.go:248 +0x430
github.com/CycloneDX/sbom-utility/cmd.listCmdImpl(0x1400017ad00?, {0x1400012f580, 0x0, 0x4})
	/github/workspace/cmd/license_list.go:186 +0x1f0
github.com/spf13/cobra.(*Command).execute(0x140001e6000, {0x1400012f540, 0x4, 0x4})
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:940 +0x66c
github.com/spf13/cobra.(*Command).ExecuteC(0x1051686e0)
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:1068 +0x320
github.com/spf13/cobra.(*Command).Execute(...)
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:992
github.com/CycloneDX/sbom-utility/cmd.Execute()
	/github/workspace/cmd/root.go:279 +0x68
main.main()
	/github/workspace/main.go:96 +0x6c

acdx.json.txt

See if we can take LicenseChoice objects produced in the license list command (JSON format):

{
      "license": {
          "id": "BSD-3-Clause",
          "name": "",
          "text": {
              "contentType": "",
              "encoding": "",
              "content": ""
          },
          "url": ""
      },
      "expression": ""
},

and reduce them to:

{
        "license": {
            "id": "BSD-3-Clause"
        }
},

Thanks for this great software.

I have a software with a component type-fest, this component has got a composite SPDX license expression (MIT OR CC0-1.0).
I have a license configuration file allowing both licenses.

When listing the licenses .\sbom-utility.exe license list ... it does not behave as I expect it. It evaluates the policy to UNDEFINED.

Version: v0.9.3

If more information is needed, I am happy to provide it.

Expected behavior

The policy for the component is allowed.

Actual behavior

The policy for the component is UNDEFINED.

Result:

Policy     Type        ID/Name/Expression  Component(s)                       BOM ref.                               Document location
------     ----        ------------------  ------------                       --------                               -----------------
UNDEFINED  expression  (MIT OR CC0-1.0)    type-fest                          [email protected]                        components
UNDEFINED  invalid     NOASSERTION         ProductWithNpmAndSomeDependencies  -/ProductWithNpmAndSomeDependencies@-  metadata.component

Minimal reproducible example

{
  "$schema": "http://cyclonedx.org/schema/bom-1.4.schema.json",
  "bomFormat": "CycloneDX",
  "specVersion": "1.4",
  "version": 1,
  "serialNumber": "urn:uuid:50a6176e-bc14-4420-afd6-4b5142bec584",
  "metadata": {
    "timestamp": "2023-02-24T11:31:49.124Z",
    "tools": [
      {
        "vendor": "@cyclonedx",
        "name": "cyclonedx-npm",
        "version": "1.8.0",
        "externalReferences": [
          {
            "url": "git+https://github.com/CycloneDX/cyclonedx-node-npm.git",
            "type": "vcs",
            "comment": "as detected from PackageJson property \"repository.url\""
          },
          {
            "url": "https://github.com/CycloneDX/cyclonedx-node-npm#readme",
            "type": "website",
            "comment": "as detected from PackageJson property \"homepage\""
          },
          {
            "url": "https://github.com/CycloneDX/cyclonedx-node-npm/issues",
            "type": "issue-tracker",
            "comment": "as detected from PackageJson property \"bugs.url\""
          }
        ]
      },
      {
        "vendor": "@cyclonedx",
        "name": "cyclonedx-library",
        "version": "1.11.0",
        "externalReferences": [
          {
            "url": "git+https://github.com/CycloneDX/cyclonedx-javascript-library.git",
            "type": "vcs",
            "comment": "as detected from PackageJson property \"repository.url\""
          },
          {
            "url": "https://github.com/CycloneDX/cyclonedx-javascript-library#readme",
            "type": "website",
            "comment": "as detected from PackageJson property \"homepage\""
          },
          {
            "url": "https://github.com/CycloneDX/cyclonedx-javascript-library/issues",
            "type": "issue-tracker",
            "comment": "as detected from PackageJson property \"bugs.url\""
          }
        ]
      }
    ],
    "component": {
      "type": "application",
      "name": "ProductWithNpmAndSomeDependencies",
      "bom-ref": "-/ProductWithNpmAndSomeDependencies@-",
      "purl": "pkg:npm/ProductWithNpmAndSomeDependencies",
      "properties": [
        {
          "name": "cdx:npm:package:path",
          "value": ""
        }
      ]
    }
  },
  "components": [
    {
      "type": "library",
      "name": "type-fest",
      "version": "1.4.0",
      "bom-ref": "[email protected]",
      "author": "Sindre Sorhus",
      "description": "A collection of essential TypeScript types",
      "hashes": [
        {
          "alg": "SHA-512",
          "content": "c864b36bbe31934506f24fa92e1e687a8622aef2225a8e6dd3fa37cc71c03b6b510e265cc563fb7f2c0d95786a6fafebeac5afc22f91b3240c5a154b7b8055b8"
        }
      ],
      "licenses": [
        {
          "expression": "(MIT OR CC0-1.0)"
        }
      ],
      "purl": "pkg:npm/[email protected]",
      "externalReferences": [
        {
          "url": "sindresorhus/type-fest",
          "type": "vcs",
          "comment": "as detected from PackageJson property \"repository\""
        },
        {
          "url": "https://registry.npmjs.org/type-fest/-/type-fest-1.4.0.tgz",
          "type": "distribution",
          "comment": "as detected from npm-ls property \"resolved\""
        }
      ],
      "properties": [
        {
          "name": "cdx:npm:package:path",
          "value": "node_modules/type-fest"
        },
        {
          "name": "cdx:npm:package:development",
          "value": "true"
        }
      ]
    }
  ]
}

{
      "policies": [
            {
                  "id": "CC0-1.0",
                  "name": "Creative Commons Zero v1.0 Universal",
                  "family": "CC0",
                  "usagePolicy": "allow",
                  "annotationRefs": [
                        "APPROVED"
                  ],
                  "notes": [
                        ""
                  ]
            },
            {
                  "id": "MIT",
                  "name": "MIT License",
                  "family": "MIT",
                  "usagePolicy": "allow",
                  "annotationRefs": [
                        "APPROVED"
                  ],
                  "notes": [
                        ""
                  ],
                  "urls": [
                        "https://spdx.org/licenses/MIT.html"
                  ]
            }
      ]
}

.\sbom-utility.exe license list -i "mre-sbom.json" --format json --summary --quiet --config-license .\mre-license.json

When I invoke sbom-utility validate --input-file I always see the following warning:

[WARN] Invalid flag for command: `output-file` (`o`). Ignoring...

Need a testcase to validate the change of the "licenseChoice" schema defn.

Note: license "expression" is no longer a "string" type, but an "object" now with a "bom-ref".

    "licenseChoice": {
      "title": "License Choice",
      "description": "EITHER (list of SPDX licenses and/or named licenses) OR (tuple of one SPDX License Expression)",
      "type": "array",
      "oneOf": [
        {
          "title": "Multiple licenses",
          "description": "A list of SPDX licenses and/or named licenses.",
          "type": "array",
          "items": {
            "type": "object",
            "required": ["license"],
            "additionalProperties": false,
            "properties": {
              "license": {"$ref": "#/definitions/license"}
            }
          }
        },
        {
          "title": "SPDX License Expression",
          "description": "A tuple of exactly one SPDX License Expression.",
          "type": "array",
          "additionalItems": false,
          "minItems": 1,
          "maxItems": 1,
          "items": [{
            "type": "object",
            "additionalProperties": false,
            "required": ["expression"],
            "properties": {
              "expression": {
                "type": "string",
                "title": "SPDX License Expression",
                "examples": [
                  "Apache-2.0 AND (MIT OR GPL-2.0-only)",
                  "GPL-3.0-only WITH Classpath-exception-2.0"
                ]
              },
              "bom-ref": {
                "$ref": "#/definitions/refType",
                "title": "BOM Reference",
                "description": "An optional identifier which can be used to reference the license elsewhere in the BOM. Every bom-ref MUST be unique within the BOM."
              }
            }
          }]
        }
      ]
    },

See: https://blog.devgenius.io/graphs-in-golang-45f7ce31fd3f

and as one format DOT files: https://graphviz.org/doc/info/lang.html

This could include "assemblies" and also the formulation "worklflow->task" dependencies as well

Hi,
I'm trying to compare two SBoMs generated with two different versions of trivy.

sbom-utils thinks hard for a bit and then panics with panic: runtime error: slice bounds out of range [2004:1743]

Here are the files:

nats-box-49.sbom.json
nats-box-50.sbom.json

And this is the command line I've used:

sbom-utility diff --input-file nats-box-49.sbom.json --input-revision nats-box-50.sbom.json

Support the latest SPDX license templates: https://github.com/spdx/license-list-data

Which were made official in this tagged release: https://github.com/spdx/license-list-data/releases/tag/v3.20

Also, will need to make sure we can handle the "oneOf" constraint where the "legacy" object is an "array" type whereas the new type contains arrays of Components and Services:

 "tools": {
          "oneOf": [
            {
              "type": "object",
              "title": "Creation Tools",
              "description": "The tool(s) used in the creation of the BOM.",
              "additionalProperties": false,
              "properties": {
                "components": {
                  "type": "array",
                  "items": {"$ref": "#/definitions/component"},
                  "uniqueItems": true,
                  "title": "Components",
                  "description": "A list of software and hardware components used as tools"
                },
                "services": {
                  "type": "array",
                  "items": {"$ref": "#/definitions/service"},
                  "uniqueItems": true,
                  "title": "Services",
                  "description": "A list of services used as tools. This may include microservices, function-as-a-service, and other types of network or intra-process services."
                }
              }
            },
            {
              "type": "array",
              "title": "Creation Tools (legacy)",
              "description": "[Deprecated] The tool(s) used in the creation of the BOM.",
              "items": {"$ref": "#/definitions/tool"}
            }
          ]

Currently, JSF signatures are only validated for correctness (in thei; we need testcases to verify this. As part of this effort we may also want to add custom "structs" for JSF objects/fields and create custom marshallers/unmarshallers for them. Having these would allow us to create custom validation against the actual data in the future.

Attempt to use this tool to generate signed BOMs (which was used for the actual spec. examples):

• JSF reference implementation for Node
  • https://github.com/cyberphone/node-webpki.org

Please note that CDX does NOT support all fields of the IETFC RFC draft... but SHOULD support those generated by the tool.

A general mechanism to do this needs to provided and its complexity may go beyond what can be easily conveyed via a command line flag. This may lead to additional (custom) "configuration" items (i.e., custom.json) and allow per-command, per-column formatting as an abstraction.

with the change to pointers in CycloneDX structs, we need to assure functional path where a service has no "bom-ref" (i.e., marshalled as "nil") will not break future changes. Currently, all JSON test BOMs always have a valid "bom-ref".

It's unusual I feel to require config files on disk that are mainly going to be the same for all users. Has there been consideration of compiling that data into the executable, and then allowing it to be overridden by those users that want to do so?

A Slack request to capture just the JSON formatted validation errors (result set) was sent to me...

See the Golang schema file that dictates how CycloneDX JSON gets "marshalled" into Golang structs.

• schema/cyclonedx.go

Extracted this feature request from issue #35

I ran this on an SBOM with 9928 components. There were duplicate components.

	1. Type: [unique], Field: [components], Description: [array items[3,243] must be unique] 
	Failing object: [[
	  {
	    "name": "acl",
	    "publisher": "Guillem Jover <guillem@ ... (truncated)

The message is correct, components[3] and components[243] were duplicates. However, the "failing object" is truncated to show components[0]. But it wasn't related to the message. It would be much easier to read the message if the message showed components[3] instead of components[0].

Look to create special "handlers" for common error types starting with "duplicates". For example, the handler could actually identify and extract the "duplicate" object (the first one) and then better format the error message to show the entire json object.

TODO: figure out how to support both current (object)/legacy(array) tools in Metadata.Tools field. Currently, we use an interface{} placeholder in our struct bindings which is NOT ideal for many things we are trying to do with entity hashing, normalization, etc.

type CDXToolLegacy struct {
    Name               string                  `json:"name,omitempty"`
    Version            string                  `json:"version,omitempty"`
    Vendor             string                  `json:"vendor,omitempty"`
    Hashes             *[]CDXHash              `json:"hashes,omitempty"`
    ExternalReferences *[]CDXExternalReference `json:"externalReferences,omitempty"`
}

type CDXTools struct {
  Components *[]CDXComponent `json:"components,omitempty"`
  Services   *[]CDXService   `json:"services,omitempty"`
}

which are both referenced from:

type CDXMetadata struct {
	Timestamp    string                      `json:"timestamp,omitempty" scvs:"bom:core:timestamp"` // urn:owasp:scvs:bom:core:timestamp
	Tools        interface{}                 `json:"tools,omitempty"`                               // v1.2: added; v1.5: "tools" is now an interface{}
	Authors      *[]CDXOrganizationalContact `json:"authors,omitempty"`
	Component    *CDXComponent               `json:"component,omitempty"`
	Manufacturer *CDXOrganizationalEntity    `json:"manufacture,omitempty"` // NOTE: Typo is in spec.
	Supplier     *CDXOrganizationalEntity    `json:"supplier,omitempty"`
	Licenses     *[]CDXLicenseChoice         `json:"licenses,omitempty"`    // v1.3 added
	Properties   *[]CDXProperty              `json:"properties,omitempty"`  // v1.3 added
	Lifecycles   *[]CDXLifecycle             `json:"lifecycles,omitempty"`  // v1.5 added
	Manufacture  *CDXOrganizationalEntity    `json:"manufacture,omitempty"` // v1.5: deprecated
}

See: https://stackoverflow.com/questions/47057240/parsing-multiple-json-types-into-the-same-struct
on possible ways to do this efficiently...

The value (boolean) for "fsf" property should never be true if the cooresponding propert "osi" is `false. Issue a WARNING at the very least (during load/hash).

That is, it is possible to have these combinations:

"osi": true,
"fsf": false,

"osi": true,
"fsf": true,

but, this combination should never appear:

"osi": false,
"fsf": true,

When outputting data in JSON format, we need to remove all empty nested structs. We have done this for the license command. This reduces JSON output for downstream processing.

See this issue for more background: #18

See what was done for the license command here as part of the fix for issue 18: https://github.com/CycloneDX/sbom-utility/blob/custom-cmds/schema/cyclonedx_marshal.go

The workflow for release v0.13.0 has failed and no release assets have been published.

Apache Skywalking-eyes (action): https://github.com/apache/skywalking-eyes

Reference example: https://github.com/CycloneDX/cyclonedx-gomod/blob/main/.github/workflows/ci.yml

• Corresponding config file (path include/exclude): https://github.com/CycloneDX/cyclonedx-gomod/blob/main/.licenserc.yml

Now that I have been sent large SBOMs with 100s or 1000s of schema errors, we see that processing all of them is time intensive and user should be able to specify after x errors to stop as any more really does not do much other than to "pile on" how had the SBOM itself is.

Since the JSON patch functionality is based on a neutral record format and that it need only need be applied to valid JSON documents, in theory it should be possible to support SPDX. The primary need is creating a valid set of functional tests (equivalent to the current CDX ones) to validate this enablement.

If we want to have special processing for JSON encoding/decoding (e.g., marshal/unmarshal) it would be helpful to enable this via custom struct tags.

For example:

// v1.3 added "compositions"
Compositions []CDXCompositions `json:"compositions,omitempty" cdx:"v1.3"`
// v1.4 added "vulnerabilities", "signature"
Vulnerabilities []CDXVulnerability `json:"vulnerabilities,omitempty" cdx:"v1.4"`

See issue #13

As CycloneDX schema has many nested structs which are not, by default, removed if the are "empty" on output (--format json), we would need to implement (on a pre-struct basis) a custom marshaller (i.e., json.Marshaller) which allows us to decide which members of the struct should be omitted.

This is important for the LicenseChoice (specifically we need to remove the CDXAttachment child struct) and Vulnerability (specifically CDXAffect) structs to reduce output sizes.

Note: The CDXAttachment struct is reused elsewhere so the benefits should be seen in many places.

The CycloneDX schema file:

/Users/Matt_1/Projects/CycloneDX/sbom-utility/schema/cyclonedx_formulation.go

defines all the structures used for "formulation" which was added in v1.5; however, these structs should be updated to use pointers to structs in all cases to improve performance during JSON Marshal (decoding).

The same should be done for the added ModelCard data (as part of the Component type):

/Users/Matt_1/Projects/CycloneDX/sbom-utility/schema/cyclonedx_modelcard.go

Currently the output is directly outputted to the terminal directly.
But if you can format the output in a json format ,it would be easy to identify the error and act according to every error cases.
If the sbom-utility cannot support any specific version ,we just want to warn the user that the SBOM validity scan is not supported.But if its an error in SBOM itself, we need to consider it as an issue.

A Sample:

validation : (true/false/not_supported)
Description : [no error/json format error/ schema error]
errorlist:
[ {
type : <>
description : <>
feild : <>
},
{
type : <>
description : <>
feild : <>
}]

It is just an example.Feel free to change the variables,format or even the whole structure as per the requirement.But it would be easy if you provide an efficient way to capture the result.

With the advent of v1.19, we now have a mature replacement for all needed functions from the deprecated ioutil package (as of v1.16).

For example, the utility currently uses ioutil.ReadAll which would be replaced by io.ReadAll.

For a more complete set of package/function mappings see this answer: https://stackoverflow.com/questions/75206234/for-go-ioutil-readall-ioutil-readfile-ioutil-readdir-deprecated

I would like to parse the output of the license list summary and process it
Right now summary only supports csv,txt and md
I want to be able to get a JSON file that contains the following structure

> sbom-utility license list -i sbom.json --summary --format json
[{
    "usage-policy": "allow",
    "license-type":  "id",
    "license":  "MIT",    
    "resource-name": "@ace/otc-components-react-3.0.14.tgz",
    "bom-ref":  "",
    "bom-location": "components"
},
{
    "usage-policy": "UNDEFINED",
    "license-type":  "id",
    "license":  "Requires Review",        
    "resource-name": "@ace/otc-components-react-3.0.11.tgz",
    "bom-ref":  "",
    "bom-location": "components"
}]

Default is "colorize", but for large # of errors (

NOTE: we likely want to also find a faster way to "AddTabs" (i.e., reduce use of JSON marshal to speed up vulnerability formatting in the output)

i.e., add property Signature CDXSignaturejson:"signature,omitempty"to top-levelCDXBom` structure. Then support it with signing verification (validation) with testcases.

This will be a bit of a challenge as we may also need to implement JSF schema:

• https://github.com/CycloneDX/specification/blob/master/schema/jsf-0.82.schema.json

As referenced by the CycloneDX schema (external).

As we want the utility to work in a network-disconnected environment (e.g., a secure build pipeline), this would have to bring in a static encoding (marshal/unmarshal, etc.)

To help Windows consumers, see if we can autogen. an MSI file as part of the build/release process:

https://github.com/mh-cbon/go-msi

For those of us that love pipes, it would be useful to support stdin for reading the files.

some-command-that-returns-an-sbom | sbom-utility <command>

That way you could dynamically generate reports, or query an sboms content in one command, without writing anything to disk.

This would be explicit, eg. -i - is fairly common as a pay of specifying that it should read from stdin rather than a local file.

There's an interesting ecosystem of tools emerging around SBOMs that really play to this sort of pipelining. Happy to help with the implementation if this is of interest.

I tried sbom-utility against https://github.com/chainguard-dev/bom-shelter/blob/main/in-the-wild/cyclonedx/obsidian-sailboat-sbom-cdx.json . 407 errors were found, here is the first one:

	1. Type: [number_one_of], Field: [components.0.licenses.0.license], Description: [Must validate one and only one schema (oneOf)] 
	Failing object: [{
	  "url": "https://github.com/AngleSharp/AngleSharp/blob/master/LICENSE"
	}]

First question. sbom-utility logged Schema "schema/cyclonedx/1.2/bom-1.2.schema.json" loaded. but the error seems to be coming from validation in resources/schema/cyclonedx/1.2/bom-1.2-strict.schema.json, not resources/schema/cyclonedx/1.2/bom-1.2.schema.json.

The failing validation is

            "oneOf":[
              {
                "required": ["license"]
              },
              {
                "required": ["expression"]
              }

The failing component has

      "licenses": [
        {
          "license": {
            "url": "https://github.com/AngleSharp/AngleSharp/blob/master/LICENSE"
          }
        }
      ],

It isn't clear to me what is wrong with this license. In this case I could tell from the URL which fragment was the problem but I would like to see the JSON Path, e.g. to component. I would also like to see the more details of the failed validation. At first glance I see a license.

Type: [number_one_of], Field: [components.0.licenses.0.license], Description: [Must validate one and only one schema (oneOf)] No "license", no "expression"
Failing object components[0]/license[0] : [{
"url": "https://github.com/AngleSharp/AngleSharp/blob/master/LICENSE"
}]

The goal is to run all BOM examples found here: https://github.com/CycloneDX/bom-examples

This means creating a named unit test in validate_test.go for each JSON BOM (or VEX, etc.).

See standardized profiles: https://scvs.owasp.org/bom-maturity-model/profiles/examples/ntia-minimum-elements/

Also, see how they are being used in BOM generation (which could be used to create test/input data):

• https://github.com/CycloneDX/cdxgen and its "--profile" flag/option.

Note: profile usage/use cases will require some really good documentation to convey understanding with great references...