HI,

Without any changes we are receiving the message KeyError: 'versionName' when running scans using
comand like docker
run --rm registry.imagesXXXX.com/il-blackduck-api:latest validate_project_by_scan_name.py -n project-name

result:
scanVersion = obj['versionName']
KeyError: 'versionName'

validate_project_by_scan_name.txt

version used :
VERSION = (0, 0, 56)
version = '.'.join(map(str, VERSION))

Any changes around the APIs regards?

Sometimes versions might have similar names, e.g.:

'1.10.2', '1.10.2-master-295', '1.10.2-master-295-master-296', '1.10.2-master-295-master-298'

The following search request returns "totalCount": 4 in this example:

/api/projects/{ProjectId}/versions?limit=100&q=versionName%3A1.10.2

Because of this line the version with the name 1.10.2 can never be deleted. The script says:

Did not find version with name 1.10.2 in project {Project}

Projects.py#L380

Saw this comment. Any idea or suggestion? As a workaround, I tried to do the POST manually, but I get a 412.

Thanks

The REST API of Blackduck supports assigning a license to a new project, upon creation, however there is no support for this in this Python library.
The issue is in
Versions.py:
def add_version_as_component(self, main_project_release, sub_project_release):
line 20 in the latest version
This is necessary to properly create new projects, which have an initial license.

This above will not find the group if it not in the first 10 (default) groups. This is a problem when linked to an Azure system with 6k+ security groups.

My hack to work around this:
assignable_user_groups_response = self.execute_get(f"{assignable_user_groups_link}?q=name:{user_group_name}")

However this does not deal properly with the potential multiple group matches and could still be affected by the limit issue above.

Hi, I am trying to play around with the examples and getting an error when trying to use print_vulnerability.py. My guess is the error is coming from

"classifications": [
NameError: name 'false' is not defined

Let me know if this can be fixed or what should I change from my side.

Hi all,

As part of a nightly dashboard update script, I grab the components that make up the projects->versions->version-risk-profile (which just shows counts) from projects->versions->vulnerable-components .

Is there a better way to get the highest scoring vulnerability per component than this.

    for project in bd.get_resource('projects'):
        latest = {'createdAt':'1970-01-01T00:00:00Z'}
        for version in bd.get_resource('versions', project, params={'sort':'createdAt ASC'}):
            if latest.get('createdAt') < version.get('createdAt'):
                latest = version
        vrc = {}
        lookup = ""      
        for vc in bd.get_resource('vulnerable-components', latest, params={'sort': ['componentname DESC', 'severity DESC']}):
            score = vc.get('vulnerabilityWithRemediation',{}).get('overallScore')
            // create key representing unique component-version
            lookup = f"{vc['componentName']}-{vc['componentVersionName']}"
            if vrc.get(lookup,{}).get('cvss', 0) < vc.get('vulnerabilityWithRemediation',{}).get('overallScore', 0):
                vrc[lookup] = {
                    'projectName': project['name'],
                    'projectVersion': latest['versionName'],
                    'componentName': vc['componentName'],
                    'componentVersion': vc['componentVersionName'],
                    'cvss': vc.get('vulnerabilityWithRemediation',{}).get('overallScore'),
                    'status': vc.get(vulnerabilityWithRemediation, {}).get('remediationStatus')
                }
        
        send_to_dashboard(vrc.items())

Any idea if the script can accept dynamic date parameter (sysdate – 3 days) or I must specify a specific date?

@skiyooka list_resource('components', version) is missing all resources, subsequently bd.get_resource('vulnerabilities', component) and other similar queries fail.

This works prior to #154, with the parent & name positioning reverted.

Sample:

for project in bd.get_resource('projects'):
    for version in bd.get_resource('versions', project):
        for component in bd.get_resource('components', version):
            pprint(bd.list_resources(component))
            exit()

Expected result:

{
    "comments": <url>,
    "component-issues": <url>,
    "policy-rules": <url>,
    "vulnerabilities": <url>,
    "matched-files": <url>,
    "origins": <url>,
    "policy-status": <url>,
    "custom-fields": <url>,
    "upgrade-guidance": <url>,
    "href": <url>
}

Actual result:

{
    "quick-filter": <url>,
    "dynamic-filter": <url>,
    "static-filter": <url>,
    "href": <url>
}

Hi,

I'm looking for a list of supported Api's.
Are the ones covered in this project all of them?

I'm interested in the vulnerabilities found in components or generating reports on the API level but an API url like this:

http://server/api/projects/id/versions/id/vulnerable-bom-components

Only redirects to the UI instead.

Any idea?

Trying to get a report of a project I found out the method get_version_components has a limit of 1000 items but the totalCount field returns the correct number of components.

I guess this is a limitation in the API call, or maybe they do pagination.

Is this a known issue?

As I understand, the get_version_components() method returns all the components for a specified project version, and get_vulnerable_bom_components() returns the subset of those components [for a specified project version] that have a security vulnerability.

I am observing that get_vulnerable_bom_components() sometimes returns a superset of components, potentially a much larger one. For example, I have a project with a single version that shows five components, three of them vulnerable, in the Web UI. get_version_components() returns five items, as expected. get_vulnerable_bom_components(), however, returns over 1500---all but three of which appear to be completely unrelated to the project version.

(I may be misunderstanding the intent of get_vulnerable_bom_components(), and if so, I'd appreciate being corrected.)

Trying to retrieve vulnerability information for CVE-2020-27216 (and potentially others I suppose) produces a stacktrace (or 3).
The following code snippet highlights the issue (noting that line 3 runs fine, but fails on line 4):

from blackduck.HubRestApi import HubInstance
hub = HubInstance()
hub.get_vulnerabilities('CVE-2021-26117')
hub.get_vulnerabilities('CVE-2020-27216')

and the stacktrace produced is below:

Traceback (most recent call last):
  File "C:\Users\me\.virtualenvs\blackduck\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  ...
  File "C:\Users\me\.virtualenvs\blackduck\python\lib\http\client.py", line 217, in parse_headers
    raise HTTPException("got more than %d headers" % _MAXHEADERS)
http.client.HTTPException: got more than 100 headers

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\me\.virtualenvs\blackduck\lib\site-packages\requests\adapters.py", line 439, in send
    resp = conn.urlopen(
  ...
  File "C:\Users\me\.virtualenvs\...\python\lib\http\client.py", line 217, in parse_headers
    raise HTTPException("got more than %d headers" % _MAXHEADERS)
urllib3.exceptions.ProtocolError: ('Connection aborted.', HTTPException('got more than 100 headers'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/me/work/git/utilcode/blackduck-scripts/headers_bug_poc.py", line 4, in <module>
    hub.get_vulnerabilities('CVE-2020-27216')
  File "c:\users\me\.virtualenvs\blackduck\src\blackduck\blackduck\Vulnerabilities.py", line 15, in get_vulnerabilities
    response = self.execute_get(url, custom_headers=headers)
  File "c:\users\me\.virtualenvs\blackduck\src\blackduck\blackduck\Core.py", line 138, in execute_get
    response = requests.get(url, headers=headers, verify = not self.config['insecure'])
  File "C:\Users\me\.virtualenvs\blackduck\lib\site-packages\requests\api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\me\.virtualenvs\blackduck\lib\site-packages\requests\api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\me\.virtualenvs\blackduck\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\me\.virtualenvs\blackduck\lib\site-packages\requests\sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\me\.virtualenvs\blackduck\lib\site-packages\requests\adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', HTTPException('got more than 100 headers'))

Hey @AR-Calder I don't think Exceptions are being properly imported into HubInstance. For example the following fails if you try and run it twice.

from blackduck.HubRestApi import HubInstance

hub = HubInstance(
    "https://your.blackduck.url",
    api_token="token",
    insecure=True,
)

r = hub.create_user_group_by_name("test-group")
print(r)
print(r.text)

Exception:

Traceback (most recent call last):
  File "/Users/skiyooka/Library/Application Support/JetBrains/IdeaIC2021.1/plugins/python-ce/helpers/pydev/pydevd.py", line 1483, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Users/skiyooka/Library/Application Support/JetBrains/IdeaIC2021.1/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/skiyooka/work_code/github/hub-rest-api-python/sumio_hub_instance.py", line 12, in <module>
    r = hub.create_user_group_by_name("test-group")
  File "/Users/skiyooka/work_code/github/hub-rest-api-python/blackduck/UserGroup.py", line 38, in create_user_group_by_name
    return self.create_user_group(user_group_info)
  File "/Users/skiyooka/work_code/github/hub-rest-api-python/blackduck/UserGroup.py", line 29, in create_user_group
    location = self._create(url, user_group_json)
  File "/Users/skiyooka/work_code/github/hub-rest-api-python/blackduck/Core.py", line 166, in _create
    raise CreateFailedAlreadyExists("Failed to create the object because it already exists - url {}, body {}, response {}".format(url, json_body, response))
NameError: name 'CreateFailedAlreadyExists' is not defined

I tried a quick fix to add the following to HubRestApi.py but I am not able to get it working. Do you have any cycles this weekend and/or do you see an obvious fix?

from .Exceptions import (
    CreateFailedAlreadyExists, CreateFailedUnknown, InvalidVersionPhase, UnknownVersion, UnsupportedBDVersion,
    EndpointNotFound, UnacceptableContentType
)

Downloading of CSV reports fails without -r option (default) "generate_csv_reports_for_project_version.py"
Might be due to invalid category 'cryptography': 'CRYPTO_ALGORITHMS', which does not exist in Version Detail Report

AssertionError: One or more selected report categories in ['VERSION', 'CODE_LOCATIONS', 'COMPONENTS', 'SECURITY', 'FILES', 'CRYPTO_ALGORITHMS', 'LICENSE_TERM_FULFILLMENT', 'BOM_COMPONENT_CUSTOM_FIELDS', 'PROJECT_VERSION_CUSTOM_FIELDS', 'VULNERABILITY_MATCH'] are not valid (['VERSION', 'CODE_LOCATIONS', 'COMPONENTS', 'SECURITY', 'FILES'])

Blackduck Server: v2021.4.0

When using BLACKDUCKv2020.2.0, the values list is already within the parameters sub key and not at the expression level. This means the fix_policy_expression is no longer required.

get_version_by_id() always returns a HTTP 404 error response, as it doesn't format the URL correctly, example:

/api/projects2cd1baa7-20ee-4bea-b0ea-a5d0561e02a9/versions/9603d400-9826-4f12-9015-e636d0300a7e

There is a missing forward slash after "projects", URL should be:

/api/projects/2cd1baa7-20ee-4bea-b0ea-a5d0561e02a9/versions/9603d400-9826-4f12-9015-e636d0300a7e

Fix is to change:

url = self._get_projects_url() + project_id + "/versions/" + version_id

to...

url = self._get_projects_url() + "/" + project_id + "/versions/" + version_id

in get_version_by_id()

In HubRestApi.py, the formats for the Notices File report are restricted to TEXT and HTML:

    valid_notices_formats = ["TEXT", "HTML"]
    def create_version_notices_report(self, version, format="TEXT"):
        assert format in HubInstance.valid_notices_formats, "Format must be one of {}".format(HubInstance.valid_notices_formats)

When my application code specifies "HTML", however, I get a HTTP 412 response status.

If I modify the code above to accept JSON instead of HTML, it works and I'm able to download a zip file containing the Notices File in JSON format. (TEXT format also works.)

I am learning how to use this environment and I want to know if it is possible to change the Usages in a component using the bd.session.put.
Using the examples that are in the documentation is possible to change the name and description of the project but how can we modify more attibutes?

`projects = [p for p in bd.get_resource('projects') if p['name'] == project_name]
assert len(projects) == 1, f"There should be one, and only one project named {project_name}. We found {len(projects)}"
project = projects[0]
project_url = bd.list_resources(project)['href']
print(f"project url: {project_url}")

versions = [v for v in bd.get_resource('versions', project) if v['versionName'] == version_name]
assert len(versions) == 1, f"There should be one, and only one version named {version_name}. We found {len(versions)}"
version = versions[0]

logging.debug(f"Found {project['name']}:{version['versionName']}")

all_bom_component_vulns = []

for bom_component_vuln in bd.get_resource('components', version):
bom_component_vuln['usages'] = [new_usage]
upgrade_guidance = bd.get_json(f"{bom_component_vuln['componentVersion']}/upgrade-guidance")
try:
r = bd.session.put(project_url, json=upgrade_guidance)
r.raise_for_status()
print("updated project")
except requests.HTTPError as err:
bd.http_error_handler(err)

project_data = {
'name': project_name,
'description': "AnaQ"
}

try:
r = bd.session.put(project_url, json=project_data)
r.raise_for_status()
print("updated project")
except requests.HTTPError as err:
bd.http_error_handler(err)`

Actually if there is another place to ask this kind of questions please let me know.

If you have a large Black Duck (Hub) instance, and you do not see an expected record in your output, you probably have exceeded a default limit hiding in
hub-rest-api-python/blackduck/HubRestApi.py

For example:
def get_projects(self, limit=100, parameters={}):

To change the default, add a parameter "limit=1000", like so:

hub = HubInstance(urlbase, username, password, insecure=True)
#projects = hub.get_projects()
projects = hub.get_projects(limit=1000)
print(json.dumps(projects.get('items', [])))

Hi ,

I am getting below error when running in azure datbricks envrironment , i provided all correct url,user and password

hub = HubInstance(urlbase, username, password, insecure=True)
SLError: HTTPSConnectionPool(host='jda.blackducksoftware.com', port=443): Max retries exceeded with url: /j_spring_security_check (Caused by SSLError(SSLError("bad handshake: SysCallError(110, 'ETIMEDOUT')")))

I am getting this issue in HubRestApi.py. I installed all the requirements but I am still getting this error.

Traceback:

File "my/path/upload_scan.py", line 11, in
hub.upload_scan(filename=str(f))
File "/root/.local/lib/python3.9/site-packages/blackduck/Scans.py", line 15, in upload_scan
response = requests.post(url, headers=headers, data=f, verify=not self.config['insecure'])
File "/root/.local/lib/python3.9/site-packages/requests/api.py", line 119, in post
return request('post', url, data=data, json=json, **kwargs)
File "/root/.local/lib/python3.9/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/root/.local/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/root/.local/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/root/.local/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/root/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/root/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/root/.local/lib/python3.9/site-packages/urllib3/connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/usr/local/lib/python3.9/http/client.py", line 1253, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.9/http/client.py", line 1299, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.9/http/client.py", line 1248, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.9/http/client.py", line 1037, in _send_output
for chunk in chunks:
File "/usr/local/lib/python3.9/http/client.py", line 996, in _read_readable
datablock = datablock.encode("iso-8859-1")
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 7619-7620: ordinal not in range(256)

Filename was
output/runs/2021-07-08-13-10-48-825/scan/BlackDuckScanOutput/2021-07-08_13-10-49-788_1/data/runner-7ydpvkq4-project-17242-concurrent-0cm5zn-my-project-2021-07-08T131051.618Z.json

By modifying L14 of https://github.com/blackducksoftware/hub-rest-api-python/blob/master/blackduck/Scans.py, it worked
FROM with open(filename,"r") as f:
TO with open(filename,"rb") as f:

@jackeekaplan delete_project_by_name does not delete the scans associated with the project

Here's what I got in the log output:

(blackduck) gsnyder@gsnyder-mac:~/Projects/hub-rest-api-python$ python examples/delete_project.py struts2-showcase
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ec2-18-217-189-8.us-east-2.compute.amazonaws.com:443
DEBUG:urllib3.connectionpool:https://ec2-18-217-189-8.us-east-2.compute.amazonaws.com:443 "POST /j_spring_security_check HTTP/1.1" 204 0
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ec2-18-217-189-8.us-east-2.compute.amazonaws.com:443
DEBUG:urllib3.connectionpool:https://ec2-18-217-189-8.us-east-2.compute.amazonaws.com:443 "GET /api/current-version HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ec2-18-217-189-8.us-east-2.compute.amazonaws.com:443
DEBUG:urllib3.connectionpool:https://ec2-18-217-189-8.us-east-2.compute.amazonaws.com:443 "GET /api/projects?q=name:struts2-showcase&limit=100 HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ec2-18-217-189-8.us-east-2.compute.amazonaws.com:443
DEBUG:urllib3.connectionpool:https://ec2-18-217-189-8.us-east-2.compute.amazonaws.com:443 "GET /api/projects/3947770d-2f62-47c7-89b6-2d8a46e8b4b3/versions?limit=100 HTTP/1.1" 200 None
DEBUG:root:delete_scans was True
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ec2-18-217-189-8.us-east-2.compute.amazonaws.com:443
DEBUG:urllib3.connectionpool:https://ec2-18-217-189-8.us-east-2.compute.amazonaws.com:443 "GET /api/projects/3947770d-2f62-47c7-89b6-2d8a46e8b4b3/versions/10901dfe-fde7-4976-b792-46b0569b38d8/codelocations?limit=100&offset=0 HTTP/1.1" 406 None
DEBUG:root:We did not find any codelocations (scans) in version 1.0
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ec2-18-217-189-8.us-east-2.compute.amazonaws.com:443
DEBUG:urllib3.connectionpool:https://ec2-18-217-189-8.us-east-2.compute.amazonaws.com:443 "GET /api/projects/3947770d-2f62-47c7-89b6-2d8a46e8b4b3/versions/e1fee724-dee4-4d4c-a80c-bf469c442fe1/codelocations?limit=100&offset=0 HTTP/1.1" 406 None
DEBUG:root:We did not find any codelocations (scans) in version 2.6-SNAPSHOT
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ec2-18-217-189-8.us-east-2.compute.amazonaws.com:443
DEBUG:urllib3.connectionpool:https://ec2-18-217-189-8.us-east-2.compute.amazonaws.com:443 "DELETE /api/projects/3947770d-2f62-47c7-89b6-2d8a46e8b4b3 HTTP/1.1" 204 0

https://github.com/blackducksoftware/hub-rest-api-python/blob/master/examples/delete_empty_projects_and_versions.py just seems to directly delete all empty projects and all empty versions in any projects.
Request to add parameter for its execution to provide number of days since that particular project or version was created.
Eg: Delete all projects and all versions in all projects that are empty, and created 30 days before or earlier.

https://github.com/blackducksoftware/hub-rest-api-python/blob/master/examples/delete_empty_projects.py does allow a time parameter. If similar logic to provide time were provided for versions also, it would be helpful.
It is better if the time parameter is in the form of number of days since creation, rather than in date format (YYYY-MM-DD).

Please see if possible to implement this. Would be helpful for multiple customers.

c:\Python37>python
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

from blackduck.HubRestApi import HubInstance
import json
username = "xxxxx"
password = "xxxxxx"
urlbase = "https://xxxxxxx"
hub = HubInstance(urlbase, username, password, insecure=True)

projects = hub.get_projects()
Traceback (most recent call last):
File "", line 1, in
File "c:\Python37\lib\site-packages\blackduck\HubRestApi.py", line 748, in get_projects
jsondata = response.json()
File "c:\Python37\lib\site-packages\requests\models.py", line 897, in json
return complexjson.loads(self.text, **kwargs)
File "c:\Python37\lib\json_init_.py", line 348, in loads
return _default_decoder.decode(s)
File "c:\Python37\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "c:\Python37\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Hi,

I'm trying to change the remediation data of vulnerabilities and when I try to use "hub.set_vulnerability_remediation" like in this example I get an error that the "'HubInstance' object has no attribute 'set_vulnerablity_remediation'". I tried re-installing the library a couple time but nothing has changed. Checking the library also shows that the function isn't installed in my local directory. I think this might be related to "set_vulnerability_remediation" being added recently.

Any help would be apprecicated!

Thanks!

To create a version report, there are different categories for a given report and as per the BlackDuck documentation, the possible values are [VERSION, CODE_LOCATIONS, COMPONENTS, SECURITY, FILES, CRYPTO_ALGORITHMS, PROJECT_VERSION_CUSTOM_FIELDS, BOM_COMPONENT_CUSTOM_FIELDS, LICENSE_TERM_FULFILLMENT, UPGRADE_GUIDANCE].

But if you see Reporting.py module, then valid_categories are defined as -
https://github.com/blackducksoftware/hub-rest-api-python/blob/master/blackduck/Reporting.py#L11
It doesn't contain UPGRADE_GUIDANCE as one of the category.

Hi,

It seems that when running the code for vulnerabilities batch update with the flag --no-process-origin-exclusion-list to update security remediation only, the code in the method process_vulnerabilities will always fail in the check of exclusion_action since it will not be initialized when updating security remediation only. (I guess that the oposit for updating origin only will fail as well on the check for remediation_action from the same reason)

The main issue is that the following 2 tests in process_vulnerabilities will not set value to remediation_action or exclusion_action in the described scenario and later on there is a check for these variables value.

if (remediation_data):
remediation_action = remediation_is_valid(vuln, remediation_data)
if (exclusion_data):
exclusion_action = origin_is_excluded(vuln, exclusion_data)

data_set construction in line 104, file deactivateUser.py should be reordered to get the script to work.

Traceback (most recent call last):
File "get_project_versions.py", line 14, in
from blackduck.HubRestApi import HubInstance
ImportError: No module named blackduck.HubRestApi

I need to get all project names in my blackduck
Tried
projects = hub.get_projects()
further massaging json data to get names.
but i get only 100 names instead of 300 + projects which I have

Traceback (most recent call last): File "./download-reports.py", line 11, in <module> from blackduck.HubRestApi import HubInstance File "/home/couchbase/jenkins/workspace/blackduck-reports/build-tools/blackduck/jenkins/generate-reports/.venv/lib/python3.6/site-packages/blackduck/HubRestApi.py", line 1685 def get_matched_components(self, version_obj, limit=9999): ^ TabError: inconsistent use of tabs and spaces in indentation

Hello,
method _get_license_info(self, license_obj), line 631 should be change from
self._get_license_info(license)
to
yield from self._get_license_info(license)

to also yield the recursive (nested) results.
Best wishes,
David

For each codelocation returned, the projectName, versionName and status sections are entirely absent. Also the list of links now contains only the "scan-summaries".

cls = hub.get_version_codelocations(version, 100, 0) print(json.dumps(cls, indent=4))

Below is the output of both versions aganst the same server, project and version.

0.0.43

{ "totalCount": 3, "items": [ { "name": "Foundation HE Infra Test Project [infra-test-project]/master/infra-test-project-main/com.cisco.sky.he/infra-test-project-main/3.7.5-11-SNAPSHOT maven/bom", "url": "urn:uuid:e4300a85-a7cb-338f-b843-c322bda5531b", "scanSize": 0, "createdAt": "2020-02-17T12:41:53.028Z", "updatedAt": "2020-03-05T13:01:45.609Z", "mappedProjectVersion": "https://synamedia.app.blackduck.com/api/projects/960f74e3-8342-4abe-a88e-d47569138e1a/versions/d85c2e4a-a4f0-466c-9e4b-d3cf3d23de09", "_meta": { "allow": [ "DELETE", "GET", "PUT" ], "href": "https://synamedia.app.blackduck.com/api/codelocations/d594de7f-2161-3df2-9b0b-15e1c3e3cc13", "links": [ { "rel": "scans", "href": "https://synamedia.app.blackduck.com/api/codelocations/d594de7f-2161-3df2-9b0b-15e1c3e3cc13/scan-summaries" } ] } }, { "name": "Foundation HE Infra Test Project [infra-test-project]/master/infra-test-project-packaging/com.cisco.sky.he/infra-test-project-packaging/3.7.5-11-SNAPSHOT maven/bom", "url": "urn:uuid:8c250d81-4f5a-3162-8522-e2e0bd9f5056", "scanSize": 0, "createdAt": "2020-02-17T12:41:52.079Z", "updatedAt": "2020-03-05T13:01:45.176Z", "mappedProjectVersion": "https://synamedia.app.blackduck.com/api/projects/960f74e3-8342-4abe-a88e-d47569138e1a/versions/d85c2e4a-a4f0-466c-9e4b-d3cf3d23de09", "_meta": { "allow": [ "DELETE", "GET", "PUT" ], "href": "https://synamedia.app.blackduck.com/api/codelocations/bd12ba59-a225-33b0-8c7f-796f6e3cb9f4", "links": [ { "rel": "scans", "href": "https://synamedia.app.blackduck.com/api/codelocations/bd12ba59-a225-33b0-8c7f-796f6e3cb9f4/scan-summaries" } ] } }, { "name": "Foundation HE Infra Test Project [infra-test-project]/master/infra-test-project/com.cisco.sky.he/infra-test-project/3.7.5-11-SNAPSHOT maven/bom", "url": "urn:uuid:7b28e76d-6035-3902-98a8-94af4d8458ce", "scanSize": 0, "createdAt": "2020-02-17T12:41:51.306Z", "updatedAt": "2020-03-05T13:01:44.738Z", "mappedProjectVersion": "https://synamedia.app.blackduck.com/api/projects/960f74e3-8342-4abe-a88e-d47569138e1a/versions/d85c2e4a-a4f0-466c-9e4b-d3cf3d23de09", "_meta": { "allow": [ "DELETE", "GET", "PUT" ], "href": "https://synamedia.app.blackduck.com/api/codelocations/cf4e2bb6-fb8d-35fd-af64-4e639a23fa6f", "links": [ { "rel": "scans", "href": "https://synamedia.app.blackduck.com/api/codelocations/cf4e2bb6-fb8d-35fd-af64-4e639a23fa6f/scan-summaries" } ] } } ], "appliedFilters": [], "_meta": { "allow": [ "GET" ], "href": "https://synamedia.app.blackduck.com/api/projects/960f74e3-8342-4abe-a88e-d47569138e1a/versions/d85c2e4a-a4f0-466c-9e4b-d3cf3d23de09/codelocations", "links": [] } }

0.0.42

{ "totalCount": 3, "items": [ { "name": "Foundation HE Infra Test Project [infra-test-project]/master/infra-test-project-main/com.cisco.sky.he/infra-test-project-main/3.7.5-11-SNAPSHOT maven/bom", "url": "urn:uuid:e4300a85-a7cb-338f-b843-c322bda5531b", "scanSize": 0, "createdAt": "2020-02-17T12:41:53.028Z", "updatedAt": "2020-03-05T13:01:45.609Z", "mappedProjectVersion": "https://synamedia.app.blackduck.com/api/projects/960f74e3-8342-4abe-a88e-d47569138e1a/versions/d85c2e4a-a4f0-466c-9e4b-d3cf3d23de09", "_meta": { "allow": [ "DELETE", "GET", "PUT" ], "href": "https://synamedia.app.blackduck.com/api/codelocations/d594de7f-2161-3df2-9b0b-15e1c3e3cc13", "links": [ { "rel": "projectVersion", "href": "https://synamedia.app.blackduck.com/api/projects/960f74e3-8342-4abe-a88e-d47569138e1a/versions/d85c2e4a-a4f0-466c-9e4b-d3cf3d23de09" }, { "rel": "mapping", "href": "https://synamedia.app.blackduck.com/api/v1/codelocations/d594de7f-2161-3df2-9b0b-15e1c3e3cc13/releasemapping/d85c2e4a-a4f0-466c-9e4b-d3cf3d23de09" }, { "rel": "scanlocations", "href": "https://synamedia.app.blackduck.com/api/v1/scanlocations/d594de7f-2161-3df2-9b0b-15e1c3e3cc13" }, { "rel": "enclosure", "href": "https://synamedia.app.blackduck.com/api/scan/data/d594de7f-2161-3df2-9b0b-15e1c3e3cc13.bdio" }, { "rel": "scans", "href": "https://synamedia.app.blackduck.com/api/codelocations/d594de7f-2161-3df2-9b0b-15e1c3e3cc13/scan-summaries" } ] }, "projectName": "Foundation HE Infra Test Project [infra-test-project]", "versionName": "master", "status": [ { "operationNameCode": "SnippetScanning", "operationName": "Snippet scanning", "status": "UNSTARTED", "createdAt": "2020-02-17T12:41:53.071Z", "updatedAt": "2020-02-17T12:41:53.071Z" }, { "operationNameCode": "ServerScanning", "operationName": "Scanning", "status": "COMPLETED", "statusMessage": "Version Bom Computation Job triggered for the scanId has been completed", "progress": { "message": "Completed", "value": 4, "percentage": 1.0 }, "scanSize": 0, "createdAt": "2020-02-17T12:41:53.059Z", "updatedAt": "2020-03-05T13:01:45.613Z" } ] }, { "name": "Foundation HE Infra Test Project [infra-test-project]/master/infra-test-project-packaging/com.cisco.sky.he/infra-test-project-packaging/3.7.5-11-SNAPSHOT maven/bom", "url": "urn:uuid:8c250d81-4f5a-3162-8522-e2e0bd9f5056", "scanSize": 0, "createdAt": "2020-02-17T12:41:52.079Z", "updatedAt": "2020-03-05T13:01:45.176Z", "mappedProjectVersion": "https://synamedia.app.blackduck.com/api/projects/960f74e3-8342-4abe-a88e-d47569138e1a/versions/d85c2e4a-a4f0-466c-9e4b-d3cf3d23de09", "_meta": { "allow": [ "DELETE", "GET", "PUT" ], "href": "https://synamedia.app.blackduck.com/api/codelocations/bd12ba59-a225-33b0-8c7f-796f6e3cb9f4", "links": [ { "rel": "projectVersion", "href": "https://synamedia.app.blackduck.com/api/projects/960f74e3-8342-4abe-a88e-d47569138e1a/versions/d85c2e4a-a4f0-466c-9e4b-d3cf3d23de09" }, { "rel": "mapping", "href": "https://synamedia.app.blackduck.com/api/v1/codelocations/bd12ba59-a225-33b0-8c7f-796f6e3cb9f4/releasemapping/d85c2e4a-a4f0-466c-9e4b-d3cf3d23de09" }, { "rel": "scanlocations", "href": "https://synamedia.app.blackduck.com/api/v1/scanlocations/bd12ba59-a225-33b0-8c7f-796f6e3cb9f4" }, { "rel": "enclosure", "href": "https://synamedia.app.blackduck.com/api/scan/data/bd12ba59-a225-33b0-8c7f-796f6e3cb9f4.bdio" }, { "rel": "scans", "href": "https://synamedia.app.blackduck.com/api/codelocations/bd12ba59-a225-33b0-8c7f-796f6e3cb9f4/scan-summaries" } ] }, "projectName": "Foundation HE Infra Test Project [infra-test-project]", "versionName": "master", "status": [ { "operationNameCode": "SnippetScanning", "operationName": "Snippet scanning", "status": "UNSTARTED", "createdAt": "2020-02-17T12:41:52.133Z", "updatedAt": "2020-02-17T12:41:52.133Z" }, { "operationNameCode": "ServerScanning", "operationName": "Scanning", "status": "COMPLETED", "statusMessage": "Version Bom Computation Job triggered for the scanId has been completed", "progress": { "message": "Completed", "value": 4, "percentage": 1.0 }, "scanSize": 0, "createdAt": "2020-02-17T12:41:52.121Z", "updatedAt": "2020-03-05T13:01:45.180Z" } ] }, { "name": "Foundation HE Infra Test Project [infra-test-project]/master/infra-test-project/com.cisco.sky.he/infra-test-project/3.7.5-11-SNAPSHOT maven/bom", "url": "urn:uuid:7b28e76d-6035-3902-98a8-94af4d8458ce", "scanSize": 0, "createdAt": "2020-02-17T12:41:51.306Z", "updatedAt": "2020-03-05T13:01:44.738Z", "mappedProjectVersion": "https://synamedia.app.blackduck.com/api/projects/960f74e3-8342-4abe-a88e-d47569138e1a/versions/d85c2e4a-a4f0-466c-9e4b-d3cf3d23de09", "_meta": { "allow": [ "DELETE", "GET", "PUT" ], "href": "https://synamedia.app.blackduck.com/api/codelocations/cf4e2bb6-fb8d-35fd-af64-4e639a23fa6f", "links": [ { "rel": "projectVersion", "href": "https://synamedia.app.blackduck.com/api/projects/960f74e3-8342-4abe-a88e-d47569138e1a/versions/d85c2e4a-a4f0-466c-9e4b-d3cf3d23de09" }, { "rel": "mapping", "href": "https://synamedia.app.blackduck.com/api/v1/codelocations/cf4e2bb6-fb8d-35fd-af64-4e639a23fa6f/releasemapping/d85c2e4a-a4f0-466c-9e4b-d3cf3d23de09" }, { "rel": "scanlocations", "href": "https://synamedia.app.blackduck.com/api/v1/scanlocations/cf4e2bb6-fb8d-35fd-af64-4e639a23fa6f" }, { "rel": "enclosure", "href": "https://synamedia.app.blackduck.com/api/scan/data/cf4e2bb6-fb8d-35fd-af64-4e639a23fa6f.bdio" }, { "rel": "scans", "href": "https://synamedia.app.blackduck.com/api/codelocations/cf4e2bb6-fb8d-35fd-af64-4e639a23fa6f/scan-summaries" } ] }, "projectName": "Foundation HE Infra Test Project [infra-test-project]", "versionName": "master", "status": [ { "operationNameCode": "SnippetScanning", "operationName": "Snippet scanning", "status": "UNSTARTED", "createdAt": "2020-02-17T12:41:51.348Z", "updatedAt": "2020-02-17T12:41:51.348Z" }, { "operationNameCode": "ServerScanning", "operationName": "Scanning", "status": "COMPLETED", "statusMessage": "Version Bom Computation Job triggered for the scanId has been completed", "progress": { "message": "Completed", "value": 4, "percentage": 1.0 }, "scanSize": 0, "createdAt": "2020-02-17T12:41:51.338Z", "updatedAt": "2020-03-05T13:01:44.741Z" } ] } ], "appliedFilters": [], "_meta": { "allow": [ "GET" ], "href": "https://synamedia.app.blackduck.com/api/projects/960f74e3-8342-4abe-a88e-d47569138e1a/versions/d85c2e4a-a4f0-466c-9e4b-d3cf3d23de09/codelocations", "links": [ { "rel": "static-filter", "href": "https://synamedia.app.blackduck.com/api/codelocations-filters?filterKey=codeLocationStatus", "name": "codeLocationStatus", "label": "Scan status" } ] } }

Search and replace "headers['Accept']:" with "headers['Accept'] ="

Few get_project_versions* methods are broken due to typos.

I get the following result on any call to assign_project_application_id:

File "assign_application_id_to_project.py", line 23, in <module> response = hub.assign_project_application_id(args.project_name, args.application_id, overwrite=args.overwrite) File "/usr/local/lib/python3.7/site-packages/blackduck/HubRestApi.py", line 1073, in assign_project_application_id existing_application_id, application_id_url = self.get_project_application_id(project_name) File "/usr/local/lib/python3.7/site-packages/blackduck/HubRestApi.py", line 1126, in get_project_application_id project_mapping_info = self.get_project_info(project_name, 'project-mappings') File "/usr/local/lib/python3.7/site-packages/blackduck/HubRestApi.py", line 1142, in get_project_info link = self.get_link(project, link_name) File "/usr/local/lib/python3.7/site-packages/blackduck/HubRestApi.py", line 211, in get_link if '_meta' in bd_rest_obj and 'links' in bd_rest_obj['_meta']: TypeError: argument of type 'NoneType' is not iterable
It appears that the 'links' field is empty, but I'm not clear on where that should be populated from, or the proper return value if that is in fact empty.

I'm running version 0.37, installed from pip.

Would be good to have a script to delete failed scans so that dB size can be reduced. Please incorporate if possible

The delete_project_version_by_name() function has a limitation. If the name of the version to be deleted is contained within other version names in the project, the delete will fail.

For example, if you have two versions of the same project, i.e. MyProject 1.0 and My Project 1.0.1, you can't delete MyProject 1.0 with delete_project_version_by_name() because the string "1.0" is contained in both version names.

If you set the logging level to DEBUG you get a misleading error message: "DEBUG:root:Did not find version with name 1.0 in project MyProject"

It seems that this could be fixed by using get_project_version_by_name() instead of get_project_versions() with the search query. This is at line 923 of HubRestApi.py.

After upgrading to version 1.0.1 from 0.0.56, my existing scripts utilizing the blackduck module break with the following exception:

Traceback (most recent call last):
  File "myScript.py", line 6, in <module>
    from blackduck.HubRestApi import HubInstance
  File "/opt/python-3.9.2/lib/python3.9/site-packages/blackduck/__init__.py", line 2, in <module>
    from .HubRestApi import HubInstance
  File "/opt/python-3.9.2/lib/python3.9/site-packages/blackduck/HubRestApi.py", line 55, in <module>
    from .Utils import object_id
  File "/opt/python-3.9.2/lib/python3.9/site-packages/blackduck/Utils.py", line 9, in <module>
    import dateutil.parser
ModuleNotFoundError: No module named 'dateutil'

Looking through the code, I do see python-dateutils defined in the requirements.txt file, but it doesn't seem like the full set of required packages are defined in the setup.py file, so pip doesn't know to install it.

The obvious workaround is for me to pip install python-dateutils manually when I need to use the blackduck module, but ideally running pip install blackduck would automatically install everything it needs.

This code is unnecessary and fails to handle pagination so will not find the group if it is not in the first 10 (default limit).

It would be much better to use the search field of the API e.g.

def get_user_group_by_name(self, group_name):
    return self.get_user_groups({"q": f"name:{group_name}"})

This works but may still return multiple rows beyond the limit. I think filter would be better but I couldn't get it to work.

Hi, I'm able to get a token using this code, but any further step returns
{u'error_description': u'Cannot convert access token to JSON', u'error': u'invalid_token'}
How to fix this?

When building this package from source, pytest fails as there is no code_locations.json in the directory test/data. This can be clearly seen in by browsing to the test/data folder on Github. Hence, on line 477 in the file test_hub_rest_api_python.py

data = json.load((shared_datadir / 'code_locations.json').open())

will always fail. I had to create my own using some json provided by a user in the issues page, which successfully passes all the pytest tests.

There seems to be an issue if the API Client is used in an environment where the working directory is not writable for the process using the .restconfig.json.

I would prefer that HubInstance can be used in a way without ever trying to access that file altogether if it is instantiated with proper url and credentials. Configuration should be doable external, HubInstance should not try to leak/persist the secrets anywhere else than in memory.

I stumbled upon this when i tried to use the client in an Azure Function. Sure one could simply deploy a working .restconfig.json along with the function, but that would be insecure.

We get the error when scanning the docker image https://hub.docker.com/r/confluentinc/cp-kafka-connect

at

Error log: after running scan_docker_image.py

Traceback (most recent call last):
  File "examples/scan_docker_image.py", line 458, in <module>
    sys.exit(main())
  File "examples/scan_docker_image.py", line 451, in main
    scan_container_image(args.imagespec)
  File "examples/scan_docker_image.py", line 379, in scan_container_image
    scanner.generate_project_structures()
  File "examples/scan_docker_image.py", line 216, in generate_project_structures
    parameters['description'] = layer['command']['created_by']
KeyError: 'created_by'

The end of manifest for that docker image is

    {
        "name": "confluentinc/cp-kafka-connect_layer_13",
        "path": "20b561df82a25b49c9773e8c8aa15786e66e3f475e28e406c2a0eded57f4f105/layer.tar",
        "command": {
            "created": "2021-07-01T21:40:40.151286297Z",
            "created_by": "/bin/sh -c #(nop) COPY --chown=appuser:appuserdir:aa284bb2f49cdbb5d62fce98ce7990f61f89bc78f41ee00166f8f3ead6a79b3e in /etc/confluent/docker "
        },
        "shaid": "sha256:305ba58c0de1d8e2ac30ee05c0e0c97d34658219f55b4c8dadf3eec6263bc99c"
    },
    {
        "name": "confluentinc/cp-kafka-connect_layer_14",
        "path": "6f5ccf9d04a5f95d285248f96c894f055480b5a964348ee8432f57168ed4334c/layer.tar",
        "command": {
            "created": "2021-07-01T21:40:49.639411935Z",
            "created_by": "|6 ARTIFACT_ID=cp-kafka-connect BUILD_NUMBER=4 CONFLUENT_PACKAGES_REPO=https://packages.confluent.io/rpm/6.1 CONFLUENT_VERSION=6.1.2 GIT_COMMIT=15316f2d PROJECT_VERSION=6.1.2 /bin/sh -c echo \"===> Installing ${COMPONENT}...\"     && echo \"===> Cleaning up ...\"     && yum clean all     && rm -rf /tmp/*"
        },
        "shaid": "sha256:4a997b96114329f621daa7c3604f726e938883e8cf519be8e83e5f05b168cf1e"
    }
]

In file HubRestApi.py too many codes:
response = requests.get(url...)
jsondata = response.json()
return jsondata
but if reponse is failed then raise JSONDecodeError("Expecting value", s, err.value) from None
in "json\decoder.py", line 355, in raw_decode

so I think we will write code like:
response = requests.get(url, headers=headers, verify = not self.config['insecure'])
if response.status_code == 200:
jsondata = response.json()
return jsondata
elif response.status_code == 403:
logging.warning("Failed to retrieve code locations (aka scans) probably due to lack of permissions, status code {}".format(
response.status_code))
else:
logging.error("Failed to retrieve code locations (aka scans), status code {}".format(
response.status_code))

Hi,
Let me come straight to the point. Can we get a proper fix to the issue described in this Synopsys article?
https://community.synopsys.com/s/article/Intermittent-JSONDecodeError-in-Hub-REST-API-Python

The article provides the workaround to "instantiate a new HubInstance object" whenever one gets a JSONDecodeError but by doing it we would be obscuring real issues causing JSONDecodeError and not because the token has expired.

So IMO a proper fix should be done in the HubRestAPI client.

The HubInstance object does not allow us to specify proxy configurations for the request library.

Can we add this?

Hi Black Duck Team

When using the assign_user_group_to_project() function, a user [with Project Manager rights - i.e default role as project creator] cannot add groups with roles within its own project. The reason why is because, you are not using the same endpoint when project roles have to be assigned or not:

1. If you do not provide roles to the function assign_user_group_to_project(), you use the projects endpoint (https://SERVER/api/projects/PROJECT_HASH/roles)

It will work in any case because project creator try to access is own project. OK.

2. However, if you provide roles (a role list or whatever), you use the usergroups endpoint (https://SERVER/api/usergroups/USERGROUP_HASH/roles)

It will not work if the user don't have the global role to access the usergroups endpoint. NOK

If the user don't have enough privileges, he will not be able to access this group endpoint, and the function return a 403 error, due to insufficient privileges.

I'm attaching to this issue a PR.
My proposal is to use in both cases the projects endpoint as it seems more natural in that situation.
Also, I've modified a json parameter that seemed to be outdated.
Regards,
Nicolas