mrl5 / metarepo-cpe-tag Goto Github PK

in CPE feed recent versions of nodejs programming language are for:

• vendor: nodejs
• product: node.js

this can result in false negatives because cpe-tag logic searches for product nodejs

use:

https://github.com/mrl5/metarepo-to-json/blob/master/bin/get_kits_json.py
https://github.com/mrl5/metarepo-to-json/blob/master/bin/get_catpkgs_json.py
https://github.com/mrl5/metarepo-to-json/blob/master/bin/get_categories_json.py

we dont care about converting metarepo or package to json in this project

it just accepts proper input(s), serialize it and works on it

Luke just use the pop-seed :D -> https://github.com/mrl5/private-wiki/blob/master/pop-framework-fairy-tale.md#luke-use-the-pop-seed

{"name": "firefox-bin", "versions": [{"version": "83.0"}]} matches cpe:2.3:a:mozilla:firefox:83.0:*:*:*:*:android:*:*

there should be no match for android (probably only * and linux)

steps to reproduce:

input='{"name": "firefox-bin", "versions": [{"version": "83.0"}]}'
./bin/tag_package_with_cpes.py "$input"

result:

{"name": "firefox-bin", "versions": [{"version": "83.0", "cpes": ["cpe:2.3:a:mozilla:firefox:83.0:*:*:*:*:android:*:*"]}]}

more info:

• https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir7696.pdf
• https://cpe.mitre.org/specification/CPE_Overview_and_Recent_Developments_20110603.pdf

related to #2

use:

1. https://github.com/mrl5/metarepo-to-json/blob/master/bin/get_kits_json.py
2. https://github.com/mrl5/metarepo-to-json/blob/master/bin/get_catpkgs_json.py
3. https://github.com/mrl5/metarepo-to-json/blob/master/bin/get_categories_json.py

pop contracts are applied only if contracted function is called directly
if it's called by another function that is in the hub contracts are ignored

overview:
this contract will be applied here:

$ pytest >/dev/null; echo $?
0

but will not be applied here:

$ input="/`uuidgen`/`uuidgen`/`uuidgen`" '{"name": "busybox", "versions": [{"version": "1.29.0"}]}'
$ ./bin/tag_package_with_cpes.py --cpe-match-feed $input
[ERROR   ] [:busybox:1.29.0:] gzip: /f739b6e6-dfb5-4f96-ab4f-b043d2cdd900/6f1f86e3-9002-42c0-9d3b-c01962b85d14/3db9fd82-c9a0-408e-8580-1b5e5f86e66a.gz: No such file or directory

here:

for some reason this line does not start many co-routines.

instead kits are processed one by one, however they should be processed in parallel

splits from #5

cpes can be retrieved from Official CPE Dictionary ia. by matching homepage.

in cpe xml it is under <reference href=""></reference> attribute

see also:
CPE 2.2 XML Schema
NVD API for CPE Retrieval

related to #1

incorporate existing tooling instead of metarepo-to-json

ego query versions $package
eix -nv -x -e $package

serialize_package_json() expects json

there should be some validator that:

• checks json schema
• doesnt accept input above some char limit

pop contracts: https://pop-book.readthedocs.io/en/latest/main/contracts.html

instead of

mkdir -p ~/feeds/json && cd $_
  wget https://nvd.nist.gov/feeds/json/cpematch/1.0/nvdcpematch-1.0.json.gz &&
  cd -

provide some script that downloads it

run pytest for tests/

1. read: https://docs.github.com/en/free-pro-team@latest/actions/guides/about-continuous-integration
2. implement: https://github.com/mrl5/metarepo-cpe-tag/actions/new
3. good to also have: https://docs.gitlab.com/ee/ci/yaml/

1. use cpe match feed (see json schema) or NVD API for CPE Retrieval
2. some requirements:

• cpe_product - strip -bin from package name
• cpe_version exclude 9999
• cpe_update should be _p2 part
• strip -r1 parts

see also:
Official CPE Dictionary
cpe python lib
nvdtools