stefan6419846 / license_tools Goto Github PK
View Code? Open in Web Editor NEWCollection of tools for working with Open Source licenses
Home Page: https://license-tools.readthedocs.io/
License: Apache License 2.0
Collection of tools for working with Open Source licenses
Home Page: https://license-tools.readthedocs.io/
License: Apache License 2.0
At the moment, run_on_directory
uses TemporaryDirectoryWithFixedName
to handle nested archives by unpacking them and deleting them after the analysis. This does not allow to use this method to permanently store the extracted files for further analysis.
There are two possible approaches to fix this:
run_on_directory
to receive a dedicated parameter which avoids deleting the unpacked archive directory after the analysis.Nested archives are not handled at the moment, which is required for .src.rpm
files for example which usually ship an additional archive with the actual source code.
At the moment, the FileResults
class uses the NOT_REQUESTED
object to indicate not requested information. This seems to cause issues when combined with _get_dummy_file_results
, as the NOT_REQUESTED
object seems to have different values.
To make this more consistent with the remaining optional fields as well, rewrite this to use | None = None
instead of | object = NOT_REQUESTED
.
Example reproducing this issue:
from license_tools import retrieval
file_results = retrieval.run(
directory="directory_containing_archive_file",
retrieve_copyrights=True,
retrieve_emails=True,
retrieve_file_info=False,
retrieve_urls=True,
retrieve_ldd_data=True,
retrieve_font_data=True,
)
for file_result in file_results:
print(file_result.short_path)
if file_result.copyrights != retrieval.NOT_REQUESTED and (file_result.copyrights.copyrights or file_result.copyrights.holders or file_result.copyrights.authors):
print(file_result.copyrights)
if file_result.emails != retrieval.NOT_REQUESTED and file_result.emails.emails:
print(file_result.emails.emails)
if file_result.urls != retrieval.NOT_REQUESTED and file_result.urls.urls:
print(file_result.urls.urls)
The conditions will fail for the archive file:
AttributeError: 'object' object has no attribute 'copyrights'
OTF files are currently being skipped, although already supported.
At the moment, run_on_file
will print the Cargo.toml
metadata in all cases. This complicates library usage, thus there should be a corresponding flag for it instead.
When the linking analysis reports the shared objects, there should be an option to resolve these to the OS package it they are not package-local. The output should be the package and the installed version, maybe even a source download link if it can be determined.
Possible implementation: https://unix.stackexchange.com/questions/158041/
At the moment, cleaning up unpacked archives being part of a directory does not work correctly when not being downloaded on the fly. The reason is that the corresponding logic mostly has temporary downloads in mind (unpacking the archive into a subdirectory of the directory the archive is part of), which tends to be my primary use case:
license_tools/license_tools/retrieval.py
Line 259 in 95b0a10
Example:
from license_tools import retrieval
file_results = retrieval.run(
directory="/home/user/directory_with_archive",
retrieve_copyrights=True,
retrieve_emails=True,
retrieve_file_info=False,
retrieve_urls=True,
retrieve_ldd_data=True,
retrieve_font_data=True,
)
This will unpack file.jar
and generate a corresponding /home/user/directory/file_jar
directory, which should be removed on exit to not modify the file system state from before the run.
This has some side effects as well when running analysis on the directory multiple times:
At the moment, LDD analysis is restricted to shared objects. It should be extended to all types of Linux binaries.
At the moment, the RPM file modes are wrong due to an upstream bug: https://github.com/srossross/rpmfile/issues/48
. Afterwards, we should be able to provide a meaningful mapping for the file modes based upon the stat
module as well.
Add a switch for the --package
option to download either a binary wheel or the source distribution. This is especially useful for native code to have a look at the source code copyrights as well.
get_package_versions
currently prints all cases where a package is skipped due to the repository URL. We should use logging instead to allow further customization with log levels and redirection.
Expose a simple API to download Rust/Cargo crates for a Cargo.lock
file for further analysis. Maybe add a way for getting the repository URL and the version license as well (see package
section of Cargo.toml
file or corresponding API).
Basic draft (not yet respecting the API limits/requirements from https://crates.io/data-access):
import hashlib
import sys
from pathlib import Path
import requests
import tomli
def get_package_versions(lock_file):
with open(lock_file, mode='rb') as fd:
data = tomli.load(fd)
for package in data['package']:
if package.get('source') != 'registry+https://github.com/rust-lang/crates.io-index':
print('Skipping', package)
continue
yield package['name'], package['version'], package['checksum']
def download_from_lock_file(lock_file, target_directory):
session = requests.Session()
target_directory = Path(target_directory)
for name, version, checksum in get_package_versions(lock_file):
url = f'https://crates.io/api/v1/crates/{name}/{version}/download'
response = session.get(url)
if response.status_code != 200:
print(response)
continue
target_directory.joinpath(f'{name}_{version}.crate').write_bytes(response.content)
digest = hashlib.sha256(response.content).hexdigest()
assert checksum == digest, url
def main():
lock_file = sys.argv[1]
target_directory = sys.argv[2]
download_from_lock_file(lock_file, target_directory)
if __name__ == '__main__':
main()
Images might have additional metadata which should be retrieved if a specific flag is active.
Possible metadata formats to evaluate/consider:
The latest release of scancode_toolkit==32.1.0
introduced some incompatible changes which require some changes in our code as well.
The direct failures are all related to the same source code line:
======================================================================
ERROR: test_rpm (test_retrieval.RunOnPackageArchiveFileTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/runner/work/license_tools/license_tools/tests/test_retrieval.py", line 441, in test_rpm
self._check_call(
File "/home/runner/work/license_tools/license_tools/tests/test_retrieval.py", line 414, in _check_call
result = list(
File "/home/runner/work/license_tools/license_tools/license_tools/retrieval.py", line 292, in run_on_package_archive_file
archive_results = _run_on_archive_file(path=archive_path, short_path=archive_path.name, default_to_none=True)
File "/home/runner/work/license_tools/license_tools/license_tools/retrieval.py", line 131, in _run_on_archive_file
rpm_results = PackageResults.from_rpm(path)
File "/home/runner/work/license_tools/license_tools/license_tools/tools/scancode_tools.py", line 335, in from_rpm
return cls(**data)
File "<string>", line 40, in __init__
File "/home/runner/work/license_tools/license_tools/license_tools/tools/scancode_tools.py", line 318, in __post_init__
self.license_detections = [
File "/home/runner/work/license_tools/license_tools/license_tools/tools/scancode_tools.py", line 319, in <listcomp>
LicenseDetection(**x) if not isinstance(x, LicenseDetection) else x for x in self.license_detections # type: ignore[arg-type]
TypeError: __init__() got an unexpected keyword argument 'license_expression_spdx'
The get_files_from_directory
functionality is generic enough to be moved to the path utils instead to allow further usage if required.
At the moment, extracting (source) RPM files requires manual attention. This should probably be automated.
For the best experience, this should probably be implemented after #6.
Prepare for pip-licenses-lib>=0.3.0
(not yet released) which uses dataclasses
as containers instead of dictionaries.
This might require some custom support for license_files
as dataclasses.as_dict
only covers fields, but not custom properties.
After the initial preparations, pip-licenses-lib==0.3.0
can be released safely.
Python 3.13 currently has a bad test speed (1:15 min for Python < 3.13, 4:30 for Python 3.13).
It seems to be related to tests.test_main.MainTestCase.test_retrieval
. Since Python 3.13, it does not allow running in parallel any more due to some internal timeouts:
joblib.externals.loky.process_executor._RemoteTraceback:
"""
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.13.0-beta.3/x64/lib/python3.13/site-packages/joblib/externals/loky/process_executor.py", line 463, in _process_worker
r = call_item()
File "/opt/hostedtoolcache/Python/3.13.0-beta.3/x64/lib/python3.13/site-packages/joblib/externals/loky/process_executor.py", line 291, in __call__
return self.fn(*self.args, **self.kwargs)
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.0-beta.3/x64/lib/python3.13/site-packages/joblib/parallel.py", line 598, in __call__
return [func(*args, **kwargs)
~~~~^^^^^^^^^^^^^^^^^
File "/home/runner/work/license_tools/license_tools/license_tools/retrieval.py", line 214, in run_on_file
return FileResults(
path=path, short_path=short_path, retrieve_licenses=True,
...<3 lines>...
retrieve_file_info=retrieval_kwargs["retrieve_file_info"],
)
File "<string>", line 15, in __init__
File "/home/runner/work/license_tools/license_tools/license_tools/tools/scancode_tools.py", line 522, in __post_init__
self.licenses = Licenses(**api.get_licenses(path_str))
~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.0-beta.3/x64/lib/python3.13/site-packages/scancode/api.py", line 200, in get_licenses
for detection in detections:
^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.0-beta.3/x64/lib/python3.13/site-packages/licensedcode/detection.py", line 1924, in detect_licenses
index = cache.get_index()
File "/opt/hostedtoolcache/Python/3.13.0-beta.3/x64/lib/python3.13/site-packages/licensedcode/cache.py", line 459, in get_index
return get_cache(
~~~~~~~~~^
only_builtin=only_builtin,
^^^^^^^^^^^^^^^^^^^^^^^^^^
...<2 lines>...
additional_directory=additional_directory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
).index
^
File "/opt/hostedtoolcache/Python/3.13.0-beta.3/x64/lib/python3.13/site-packages/licensedcode/cache.py", line 399, in get_cache
return populate_cache(
only_builtin=only_builtin,
...<2 lines>...
additional_directory=additional_directory,
)
File "/opt/hostedtoolcache/Python/3.13.0-beta.3/x64/lib/python3.13/site-packages/licensedcode/cache.py", line 419, in populate_cache
_LICENSE_CACHE = LicenseCache.load_or_build(
only_builtin=only_builtin,
...<6 lines>...
additional_directory=additional_directory,
)
File "/opt/hostedtoolcache/Python/3.13.0-beta.3/x64/lib/python3.13/site-packages/licensedcode/cache.py", line 136, in load_or_build
with lockfile.FileLock(lock_file).locked(timeout=timeout):
...<55 lines>...
return license_cache
File "/opt/hostedtoolcache/Python/3.13.0-beta.3/x64/lib/python3.13/contextlib.py", line 141, in __enter__
return next(self.gen)
File "/opt/hostedtoolcache/Python/3.13.0-beta.3/x64/lib/python3.13/site-packages/scancode/lockfile.py", line 29, in locked
raise LockTimeout(timeout)
scancode.lockfile.LockTimeout: 360
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/home/runner/work/license_tools/license_tools/license_tools/__main__.py", line 203, in <module>
main()
~~~~^^
File "/home/runner/work/license_tools/license_tools/license_tools/__main__.py", line 182, in main
retrieval.run(
~~~~~~~~~~~~~^
directory=arguments.directory,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<14 lines>...
retrieve_cargo_metadata=arguments.retrieve_cargo_metadata,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/runner/work/license_tools/license_tools/license_tools/retrieval.py", line 438, in run
results = list(
run_on_downloaded_package_file(
...<5 lines>...
)
)
File "/home/runner/work/license_tools/license_tools/license_tools/retrieval.py", line 360, in run_on_downloaded_package_file
yield from run_on_package_archive_file(
...<3 lines>...
)
File "/home/runner/work/license_tools/license_tools/license_tools/retrieval.py", line 296, in run_on_package_archive_file
yield from run_on_directory(
...<3 lines>...
)
File "/home/runner/work/license_tools/license_tools/license_tools/retrieval.py", line 243, in run_on_directory
results = Parallel(n_jobs=job_count)(
delayed(run_on_file)(
...<4 lines>...
for path, short_path in files
)
File "/opt/hostedtoolcache/Python/3.13.0-beta.3/x64/lib/python3.13/site-packages/joblib/parallel.py", line 2007, in __call__
return output if self.return_generator else list(output)
~~~~^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.0-beta.3/x64/lib/python3.13/site-packages/joblib/parallel.py", line 1650, in _get_outputs
yield from self._retrieve()
File "/opt/hostedtoolcache/Python/3.13.0-beta.3/x64/lib/python3.13/site-packages/joblib/parallel.py", line 1754, in _retrieve
self._raise_error_fast()
~~~~~~~~~~~~~~~~~~~~~~^^
File "/opt/hostedtoolcache/Python/3.13.0-beta.3/x64/lib/python3.13/site-packages/joblib/parallel.py", line 1789, in _raise_error_fast
error_job.get_result(self.timeout)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/opt/hostedtoolcache/Python/3.13.0-beta.3/x64/lib/python3.13/site-packages/joblib/parallel.py", line 745, in get_result
return self._return_or_raise()
~~~~~~~~~~~~~~~~~~~~~^^
File "/opt/hostedtoolcache/Python/3.13.0-beta.3/x64/lib/python3.13/site-packages/joblib/parallel.py", line 763, in _return_or_raise
raise self._result
scancode.lockfile.LockTimeout: 360
Sequential processing works, but its performance is bad as well without any apparent reason. Isolated tests with bash-based parallel processing (instead of joblib) did not show this behavior: https://github.com/stefan6419846/scancode-tests/blob/main/.github/workflows/run.yml
We already analyze the metadata of font and RPM files and could extend this to Python packages as well. Some basic implementations are available from https://github.com/stefan6419846/pip-licenses-lib
Some things which could be considered:
At the moment, our tests are only running on Ubuntu 22.04. This does not allow for the full suite of RPM-based tooling/testing, as required for #25. Therefore, a corresponding RPM-based distribution, probably OpenSUSE Leap due to already using it for development, should be added to GitHub Actions.
Docs: https://docs.github.com/en/actions/using-jobs/running-jobs-in-a-container, containers are at https://hub.docker.com/r/opensuse/leap/tags
It seems like we need at least OpenSUSE Leap 15.2 for https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/), but as the currently supported minimum version is 15.5 anyway, this should be no big deal.
check_shared_objects
currently uses sys.stderr
to report shared objects using symlinks. We should use logging instead to allow further customization with log levels and redirection.
Currently, we just forward the output of ldd
for ELF binaries from a subprocess and print it to stdout directly. This probably can be improved.
Due to already using the ScanCode toolkit, I evaluated the elf-inspector
tool, but this currently lacks important features as it does not provide the full dependency paths (to decide whether the originate from the OS or whether they are shipped with the package itself) and somehow omits the libc dependency in some cases: aboutcode-org/elf-inspector#4.
At the moment, run_on_downloaded_archive_file
and run_on_downloaded_package_file
download the files directly. As this functionality could be useful for other cases as well, the corresponding download code should be moved to the download utils (for generic archive downloads) and the pip tools (for pip-specific package downloads without URL).
The application might emit info messages. For this reason, allow the user to configure the desired log level with a CLI parameter.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.