secvisogram / csaf-validator-lib Goto Github PK

Dokumentieren, dass in die validate Funktion nur vertrauenswürdige Funktionen übergeben werden dürfen

Unfortunately the upgrade to Node.js's ES Module package format broke istanbul which is used for test-coverage reporting. An option might be c8 which collects coverage using native v8 instruments.

Found in https://github.com/secvisogram/csaf-validator-lib/blob/main/lib/mandatoryTests/mandatoryTest_6_1_10.js and is needed as well in optional test 6.2.19

Currently, it is not possible to submit PRs successfully from remote repos as GH action will fail. This is potentially an issue with the use of GITHUB_TOKEN in

The README lists the example:

import validate from '../csaf-validator-lib/validate.js'

const document = '{}'
const tests = [
  {
    type: 'preset',
    name: 'mandatory'
  },
  {
    type: 'test',
    name: 'optionalTest_6_2_1'
  }
]

const result = await validate(tests, document)

However, that does not work for me. It throws an error:

file:///anywhere/csaf-validator-lib/lib/validate.js:11
    const result = await test(doc)
                         ^

TypeError: test is not a function
    at default (file:///anywhere/csaf-validator-lib/lib/validate.js:11:26)
    at file:///anywhere/csaf-validator-lib/testing.js:15:22
    at ModuleJob.run (internal/modules/esm/module_job.js:183:25)
    at async Loader.import (internal/modules/esm/loader.js:178:24)
    at async Object.loadESM (internal/process/esm_loader.js:68:5)
    at async handleMainPromise (internal/modules/run_main.js:59:12)

Based on a quick search, I guess the tuple of typeand name isn't implemented here. Correct?

Currently, I can't find the prerequisites for 6.3.8 in the README. We need to document them.

Test 6.1.8 is currently missing. I guess that should be implemented...

The CVSS SIG recently corrected a bug in the pattern part ((M)PR had an additional /non intentional extra U). We need to check whether our code has the same mistake.

The Interface documentation currently does not list, that the test name is provided as well. We need to check the documentation and update it accordingly.

Running the validation for csaf_2_0 and csaf_2_0_strict, I get the following error:

"csaf_2_0: must match pattern \"^(cpe:2\\.3:[aho\\*\\-](:(((\\?*|\\*?)([a-zA-Z0-9\\-\\._]|(\\\\[\\\\\\*\\?!\"#\\$%&'\\(\\)\\+,/:;<=>@\\[\\]\\^`\\{\\|\\}~]))+(\\?*|\\*?))|[\\*\\-])){5}(:(([a-zA-Z]{2,3}(-([a-zA-Z]{2}|[0-9]{3}))?)|[\\*\\-]))(:(((\\?*|\\*?)([a-zA-Z0-9\\-\\._]|(\\\\[\\\\\\*\\?!\"#\\$%&'\\(\\)\\+,/:;<=>@\\[\\]\\^`\\{\\|\\}~]))+(\\?*|\\*?))|[\\*\\-])){4})|([c][pP][eE]:/[AHOaho]?(:[A-Za-z0-9\\._\\-~%]*){0,6})$\"", "csaf_2_0_strict: must match pattern \"^(cpe:2\\.3:[aho\\*\\-](:(((\\?*|\\*?)([a-zA-Z0-9\\-\\._]|(\\\\[\\\\\\*\\?!\"#\\$%&'\\(\\)\\+,/:;<=>@\\[\\]\\^`\\{\\|\\}~]))+(\\?*|\\*?))|[\\*\\-])){5}(:(([a-zA-Z]{2,3}(-([a-zA-Z]{2}|[0-9]{3}))?)|[\\*\\-]))(:(((\\?*|\\*?)([a-zA-Z0-9\\-\\._]|(\\\\[\\\\\\*\\?!\"#\\$%&'\\(\\)\\+,/:;<=>@\\[\\]\\^`\\{\\|\\}~]))+(\\?*|\\*?))|[\\*\\-])){4})|([c][pP][eE]:/[AHOaho]?(:[A-Za-z0-9\\._\\-~%]*){0,6})$\""

Testing again the following CPE: cpe:/o:redhat:enterprise_linux:8::fastdatapath (which seems valid to me).

Testing with another regular expression too, I get an error that the regular expression is invalid for ECMAScript. Caused by two missing escape backslashes:

Diff:

- ^(cpe:2\.3:[aho\*\-](:(((\?*|\*?)([a-zA-Z0-9\-\._]|(\\[\\\*\?!"#\$%&'\(\)\+,/:;<=>@\[\]\^`\{\|\}~]))+(\?*|\*?))|[\*\-])){5}(:(([a-zA-Z]{2,3}(-([a-zA-Z]{2}|[0-9]{3}))?)|[\*\-]))(:(((\?*|\*?)([a-zA-Z0-9\-\._]|(\\[\\\*\?!"#\$%&'\(\)\+,/:;<=>@\[\]\^`\{\|\}~]))+(\?*|\*?))|[\*\-])){4})|([c][pP][eE]:/[AHOaho]?(:[A-Za-z0-9\._\-~%]*){0,6})$"", "csaf_2_0_strict: must match pattern "^(cpe:2\.3:[aho\*\-](:(((\?*|\*?)([a-zA-Z0-9\-\._]|(\\[\\\*\?!"#\$%&'\(\)\+,/:;<=>@\[\]\^`\{\|\}~]))+(\?*|\*?))|[\*\-])){5}(:(([a-zA-Z]{2,3}(-([a-zA-Z]{2}|[0-9]{3}))?)|[\*\-]))(:(((\?*|\*?)([a-zA-Z0-9\-\._]|(\\[\\\*\?!"#\$%&'\(\)\+,/:;<=>@\[\]\^`\{\|\}~]))+(\?*|\*?))|[\*\-])){4})|([c][pP][eE]:/[AHOaho]?(:[A-Za-z0-9\._\-~%]*){0,6})$
+ ^(cpe:2\.3:[aho\*\-](:(((\?*|\*?)([a-zA-Z0-9\-\._]|(\\[\\\*\?!"#\$%&'\(\)\+,/:;<=>@\[\]\^`\{\|\}~]))+(\?*|\*?))|[\*\-])){5}(:(([a-zA-Z]{2,3}(-([a-zA-Z]{2}|[0-9]{3}))?)|[\*\-]))(:(((\?*|\*?)([a-zA-Z0-9\-\._]|(\\[\\\*\?!"#\$%&'\(\)\+,/:;<=>@\[\]\^`\{\|\}~]))+(\?*|\*?))|[\*\-])){4})|([c][pP][eE]:\/[AHOaho]?(:[A-Za-z0-9\._\-~%]*){0,6})$"", "csaf_2_0_strict: must match pattern "^(cpe:2\.3:[aho\*\-](:(((\?*|\*?)([a-zA-Z0-9\-\._]|(\\[\\\*\?!"#\$%&'\(\)\+,/:;<=>@\[\]\^`\{\|\}~]))+(\?*|\*?))|[\*\-])){5}(:(([a-zA-Z]{2,3}(-([a-zA-Z]{2}|[0-9]{3}))?)|[\*\-]))(:(((\?*|\*?)([a-zA-Z0-9\-\._]|(\\[\\\*\?!"#\$%&'\(\)\+,/:;<=>@\[\]\^`\{\|\}~]))+(\?*|\*?))|[\*\-])){4})|([c][pP][eE]:\/[AHOaho]?(:[A-Za-z0-9\._\-~%]*){0,6})$

I postponed the implementation for a bit since we plan to use hunspell here and the implementation is therefore not "straight forward".

We should update to CWE 4.11

Die Typdefinitionen von Ajv beinhalten, dass es Fehler geben kann, die keine message Property besitzen. In diesem Fall soll die Standardmeldung unexpected empty error message generiert werden.

Test 6.3.9 has some bugs: (See https://github.com/oasis-tcs/csaf/pull/583/files for test files)

• valid files are considered invalid: e.g. oasis_csaf_tc-csaf_2_0-2021-6-3-09-12.json

  {
    "tests": [
  	{
  	  "isValid": true,
  	  "errors": [],
  	  "warnings": [],
  	  "infos": [
  		{
  		  "instancePath": "/product_tree/branches/0/branches/0/branches/0/branches/1",
  		  "message": "missing ancestor with category vendor"
  		}
  	  ],
  	  "name": "informativeTest_6_3_9"
  	}
    ],
    "isValid": true
  }
  

• invalid files do no show all errors: e.g. oasis_csaf_tc-csaf_2_0-2021-6-3-09-03.json reports only on instead of 4 errors (for 2 elements both defined error messages)

  	{
      "tests": [
    	{
    	  "isValid": true,
    	  "errors": [],
    	  "warnings": [],
    	  "infos": [
    		{
    		  "instancePath": "/product_tree/branches/0/branches/0/branches/0",
    		  "message": "missing ancestor with category vendor"
    		}
    	  ],
    	  "name": "informativeTest_6_3_9"
    	}
      ],
      "isValid": true
    }
  

Recently, a bug (#98) was fixed in the implementation of test 6.1.16 (#112). A similar one might exist in the implementation of 6.1.14. This needs to be investigated.

We should write CVE instead of cve.

As the CWE list gets updated now and then, we should add a script to generate the cwec.js file.

The latest version can be found here

Flagging @thomas-proell for attention.

Validation issue for identical date-times in revisions

Revisions with identical date-times cause the tracking/version to throw an version does not match latest revision error when the tracking->version is the higher/later number form the revision_history:

{
  "document": {
    "csaf_version": "2.0",
    "category": "ACME Security Advisory",
    "title": "ACME Security Advisory",
    "publisher": {
      "name": "ACME",
      "namespace": "a:b",
      "category": "vendor"
    },
    "tracking": {
      "id": "acme-adv-01",
      "current_release_date": "2023-02-11T23:00:00.000+00:00",
      "initial_release_date": "2023-02-11T23:00:00.000+00:00",
      "status": "final",
      "revision_history": [
        {
          "date": "2023-02-11T23:00:00.000+00:00",
          "number": "1",
          "summary": "Initial public release",
          "legacy_version": "1.0"
        },
        {
          "date": "2023-02-11T23:00:00.000+00:00",
          "number": "2",
          "summary": "Updates to products",
          "legacy_version": "2.6"
        }
      ],
      "version": "2",
      "generator": {
        "date": "2023-04-14T11:20:25.734+00:00",
        "engine": {
          "name": "ACME-Converter",
          "version": "1.0.0"
        }
      }
    }
  }
}

Also, changing the version to 1 should throw the version does not match latest revision error but does not.

A "normal" git clone followed by npm ci and npm test won't provide the intended result as the submodule has to be imported. Please adopt the documentation in README. (You can use the original one from Secvisogram)

Todos:

• Copy and adopt the original Readme

According to 6.1.27.9 the flags must be checked for an impact statement as well. However, that is currently not done.

This should be MIT

Is it relevant which node version (e.g. 14 or 16) is used to run that tests? Is there a minimal version? If so, please indicate in the Readme.

The same bug as secvisogram/secvisogram#216 (see ajv-validator/ajv-formats#55) applies here. We should use ajv-validator/ajv-formats#49 version 3.0.0 even if it is not stable yet.

In CDS02, there was a change to test 6.1.21. The change can be reviewed at https://github.com/oasis-tcs/csaf/pull/443/files. It basically binds the revision_history to a start...

Noted here.

Here's a list of issues I found reviewing the already checked in optional and informative tests. For tests not mentioned, I could not find any issues.

Optional Tests

• Test 6.2.1

  • Is the ?? false part necessary for all the checks? If so, I'd like to understand better why. I'm referring to these lines.

• Test 6.2.10

  • The test checks if the tlp label is present but it does not check for validity of the used label, i.e. if they are one of the allowed enum values AMBER, GREEN, RED, or WHITE.

• Test 6.2.11

  • The test does not check whether the url starts with https://.
  • It does falsely replace - characters for checking the string.
  • The specification has changed and now replaces sequences of not allowed characters with a single underscore. The new specification also supplies a regex that can be used to the the replacement: [^+\-a-z0-9]+

• Test 6.2.12

  • Running and modifying the existing test, it seems like skipping the lang property completely will pass the test. If I understand the specification correctly, it should also fail if the property isn't present at all.

Informative Tests

• Test 6.3.2
  • The schemas for CVSS v3.0 and v3.1 require the properties version, vectorString, baseScore, and baseSeverity while here they are listed as optionalProperties. It's probably out of the scope of this test, but I'm wondering if we could make it conform to the schema more. The same may apply for test 6.3.1.
  • An additional check for the pattern of vectorString could be performed to ensure v3.1 is used. Checking for the version prefix may be fine, though and this would be out of the scope of this test resp. extend it way beyond what is required by the specification.

I'm not sure yet how to implement this properly since the strict schema check is already in and checks other aspects as well. So I'm not sure if the naive idea of "labeling the strict schema as 'optional test 6.2.20'" is appropriate here. But I guess that we can discuss this in one of our status meetings.

Currently, npm test fails for me with the following error message:

> test
> tsc -b . && mocha tests

scripts/runTest.js:12:26 - error TS2307: Cannot find module 'fs/promises' or its corresponding type declarations.

12 import { readFile } from 'fs/promises'
                            ~~~~~~~~~~~~~

scripts/runTest.js:19:42 - error TS2339: Property 'argv' does not exist on type 'typeof process'.

19 const [, , filePath, testName] = process.argv
                                            ~~~~


Found 2 errors.

What did I miss?

The CSAF standard requires that all "keys in a CSAF document are sorted alphabetically". It looks like the current implementation also checking elements like /vulnerabilities/product_status[].

This must be corrected to reflect the standard.

The current implementation of 6.1.24 evaluates the involvement status over all vulnerabilities. However, the spec intends to do that per vulnerability.

We should add keywords to the package.json to improve the search results in the npm registry... At least the following should be added:

csaf
csaf-validator-lib
csaf full validator
secvisogram

We should have a GitHub action to run tests on each push and pull request to avoid merging invalid code.

Why do we use an AJV with strict=false for the schema validation? Shouldn't the strict_schema be checked with the strict option?

@domachine: Something to discuss in the meeting.

The files minimalSecurityAdvisoryDoc.js and minimalVexDoc.js are invalid. Both are missing the mandatory /vulnerabilites element. We added test 6.1.27.11 to detect that.

If a document contains a pre-release-version (like 1.0.1-1.0) in document.version and a matching revision_history entry with the same content in revision_history[].number, there are two warnings:
revision_history[].number -> contains prerelease part
document.version -> version does not match latest revision

{ "document": { "category": "csaf_base", "csaf_version": "2.0", "publisher": { "category": "discoverer", "name": "test", "namespace": "http://example.test" }, "title": "test_1", "tracking": { "current_release_date": "2022-01-01T00:00:00.000Z", "id": "111", "initial_release_date": "2022-01-01T00:00:00.000Z", "revision_history": [ { "date": "2022-01-01T00:00:00.000Z", "number": "1.0.0", "summary": "\"Initial Publication\"" }, { "date": "2022-01-01T00:00:00.001Z", "number": "1.0.1-1.0", "summary": "New Version" } ], "status": "draft", "version": "1.0.1-1.0" } } }

Maybe the check according the matching according the latest version with a pre-release-tag could be done against the latest revision history entry?

In order to more easily consume this functionality, it would be great if that could be published to npmjs.org

Test 6.3.6 is currently failing.
Trying out older releases doesn't change this, even though the test worked at the time of releasing.

6.3.6 valid
informative/oasis_csaf_tc-csaf_2_0-2021-6-3-06-11.json

I think it is a problem with the "https://csaf.io" domain referenced in the test files. Changing this url makes all tests pass.

But I can't figure out why csaf.io domain is failing the test now, please have a look for yourself @tschmidtb51, maybe you can figure out what is going on here.

We need to expose the group of tests as specified in the standard:

• basic: schemaTest (strict) + mandatoryTests (may skip 6.1.8 when covered by strict schema)
• extended: basic+ optionalTests
• full: extended + informativeTests

Currently, there are many different ways the json output is ordered in different tools (csaf-validator-lib, csaf-validator-service, csaf_validator).

We should add a script to test JSON files directly with the lib. It could look similar to the one @domachine provided during #3:

#!/usr/bin/env node

import { readFile } from 'fs/promises'
import * as schemaTests from '../schemaTests.js'
import * as mandatoryTests from '../mandatoryTests.js'
import * as optionalTests from '../optionalTests.js'
import * as informativeTests from '../informativeTests.js'
import validate from '../validate.js'

const [, , filePath, testName] = process.argv

const json = JSON.parse(await readFile(filePath))
console.log(JSON.stringify(json, null, 2))
const tests =
  /** @type {Array<import('../lib/shared/types.js').DocumentTest>} */ (
    Object.values(mandatoryTests)
  )
    .concat(Object.values(optionalTests))
    .concat(Object.values(informativeTests))
    .concat(Object.values(schemaTests))

const matchingTests = tests.filter((t) => t.name === testName)

if (!matchingTests.length)
  throw new Error(`No tests matching "${testName}" found`)
console.log(JSON.stringify(await validate(matchingTests, json), null, 2))

Maybe, we could adopt it, that the output of the JSON is optional...

The latest version can be found here

Currently, it is not well documented how the lib can be used in browsers... See csaf-poc/csaf_webview#1 (comment) for an example question.