Methods of publishing scancode-toolkit-cache.json

1. Publish as a github action atifact:
   • our script must use Github API to download latest artifact
2. Publish to the same server as documentation.
   • if using existsting method, it can be published at:
     https://developer.nordicsemi.com/nRF_Connect_SDK_dev/doc/west-sbom/scancode-toolkit-cache.json
   • if file cannot be in the "doc" directory, modification of monitor scripts are required
3. Publish as a commit in master branch
   • it will create a commit once a day
   • no need to download from internet by the user - it will be already in the repository
4. As a commit in seperate repository:
   • what/where repository?

SITE NOTE:

Database can contain timestamp when it was created and when update has been checked last time.
The script will check for updates when, e.g.: creation_time > 25h AND last_check_time > 24h
(assuming that database is created once a day)

from fw-nrfconnect-nrf.

Planned CI use cases:

• Create and publish scancode-toolkit-cache.json.
• Check if all files contained in the PR have valid SPDX tag.
• Run check for entire NCS if there is no regression regarding licenses.
• Any more?

Other CI related:

• Add github action (runs e.g. once a week) that checks if SPDX license list is up to date and creates a PR with updated version if needed.
• Run nightly script that refreshes the cache database file and publishes it.

from fw-nrfconnect-nrf.

To do:

• ncs/nrf/samples/bluetooth/peripheral_lbs$ west sbom -d build/ --license-detectors spdx-tag @doki-nordic
  Exception. file: ```input_build.py line 100
  Exception: "ar -t "build/modules/nrf/drivers/hw_cc310/lib..__nrf__drivers__hw_cc310.a"" command exited with error code 9
• Case insensitive spdx license id. @doki-nordic
• Add -n parameter @maje-emb
• Add relative west top dir as path in database (helper function in common.py) @maje-emb
• Improve database detector @maje-emb
• Rename scancode-cache to cache @maje-emb
• Display warning when list of input files is empty @doki-nordic
• Use west logger @doki-nordic @maje-emb
• Add own exception @doki-nordic
• Add exception handling for scancode-toolkit (common.py) @doki-nordic
• Fix relative path returned by input-build @doki-nordic
• For the scancode-toolkit detector the logs are messed. It is due to the scancode prints the detection information on stderr. @maje-emb
• Check PEP8 standard in source code @maje-emb
• Source code documentation @doki-nordic @maje-emb
• Mark build-input as EXPERIMENTAL
• Fix: Close file handles in command_execute in some cases.
• Show help when no parameters are passed
• Remove old (dead) code
• Allow scancode from diffrent location
• Test scancode installation with "--version" argument

from fw-nrfconnect-nrf.

To do after PR:

• Add URLs to license exceptions if expression contains "WITH" operator
• Verify why scancode-toolkit does not always returns valid SDPX identifier?

from fw-nrfconnect-nrf.

To do:

Current PR

• Doc: We should add sample of the report (e.g. screenshot) with some explanation @doki-nordic
  Replaced by overview in the documentation and maybe later: https://doki-nordic.github.io/test-demo-docs/sbom_report.html
• Doc: Add information in the general license information and remove west spdx tool information @maje-emb
• Doc: Tell how we are generating the list of files from the application build. (from user perspective) @doki-nordic
• Doc: Add note that build must be successful and up to date, i.e. you have to rebuild it on each change of kconfig option. @doki-nordic
• Finalize current file extraction from a build directory. @doki-nordic
• Change the name to ncs-sbom
• Add SPDX output template and verify its correctness with SPDX verification tools @maje-emb
• Implement external file detector
• Change UUID in the custom licenses from the database to west-ncs-sbom- prefix
• Add special handling for libgcc.a,libc_nano.a,libm_nano.a - they are in the map file and not in the build system. Should be in the report
• Add note to the report that some licenses were detected by scancode-toolkit with link to it
• Implement map file parsing and verification @doki-nordic
• Use spdx_license_key from scancode output if possible
• Use license information (name and URL) from scancode output if possible
• Use Nordic colors in the report
• Use default build directory (if exists) if there is no input provided
• Show clickable (in linux and vcs terminal) link to the output
• Add dark and visual studio code CSS themes.
• Fix scancode toolkit for Nordic license @maje-emb

Later - may be after NCS 2.0

• Implement extracting origin of each file:
  • repository
  • commit and a version tag if available
  • information if it was modified
  • In SPDX each repository can be mapped into packages
  • In HTML each license details section can be divided on more sections for each repository
• Run the script on entire repository and detect other problems and report problems to people responsible or fix them
• Run the script on entire repository to compare results between scancode-toolkit and other detectors
• Run the final script on every sample from nrf and zephyr repository
• Create workflow for PR checks. Help from testers needed.
  • Add workflow also to nrfxlib, decision point: will the nrfxlib have its own license-allow-list.yaml?
• Test for the correctness of the script
• Implement cache in some temporary directory (e.g. Zephyr's cache: USER_CACHE_DIR)
• Test for entire repository
• Add URLs to license exceptions if expression contains "WITH" operator
• Add information (maybe some symbol) that specific file was detected with external tool: scancode-toolkit.
• Maybe consider adding more information for each file as a CSS popup:
  • List of detectors
  • Hash
  • Source of the file (repository and commit/tag)

from fw-nrfconnect-nrf.

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.

from fw-nrfconnect-nrf.