armijnhemel / binaryanalysis-ng Goto Github PK
View Code? Open in Web Editor NEWBinary Analysis Next Generation (BANG)
License: GNU Affero General Public License v3.0
Binary Analysis Next Generation (BANG)
License: GNU Affero General Public License v3.0
I'm trying a program to explore the possibilities of scanning
[kea@localhost src]$ python3 ./bang-scanner -c ./bang.config -f ./openwrt-18.06.1-brcm2708-bcm2710-rpi-3-squashfs-factory.img.gz
and get errors
Process Process-4:
Traceback (most recent call last):
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 673, in processfile scanjob.check_for_signatures(unpacker)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 297, in check_for_signatures signature, offset)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/Unpacker.py", line 198, in try_unpack_file_for_signatures return bangsignatures.signaturetofunction[signature](fileresult, scanenvironment, offset, self.dataunpackdirectory)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/bangunpack.py", line 12346, in unpack_compress p = subprocess.Popen(['uncompress'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "/usr/lib64/python3.7/subprocess.py", line 775, in init restore_signals, start_new_session)
File "/usr/lib64/python3.7/subprocess.py", line 1522, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'uncompress': 'uncompress'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib64/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run()
File "/usr/lib64/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 745, in processfile raise ScanJobError(scanjob, e)
ScanJob.ScanJobError: Exception for scanjob:
file:
openwrt-18.06.1-brcm2708-bcm2710-rpi-3-squashfs-factory.img.gz-gzip-1/openwrt-18.06.1-brcm2708-bcm2710-rpi-3-squashfs-factory.img
labels:
Traceback (most recent call last):
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 673, in processfile scanjob.check_for_signatures(unpacker)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 297, in check_for_signatures signature, offset)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/Unpacker.py", line 198, in try_unpack_file_for_signatures return bangsignatures.signaturetofunction[signature](fileresult, scanenvironment, offset, self.dataunpackdirectory)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/bangunpack.py", line 12346, in unpack_compress p = subprocess.Popen(['uncompress'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "/usr/lib64/python3.7/subprocess.py", line 775, in init restore_signals, start_new_session)
File "/usr/lib64/python3.7/subprocess.py", line 1522, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'uncompress': 'uncompress'
if interested, then this file https://cloud.mail.ru/public/4Znn/sNB3zAYja
openwrt-18.06.1-brcm2708-bcm2710-rpi-3-squashfs-factory.img.gz
Error with Python's testtar.tar archive:
$ python3 bang-scanner -f /tmp/testtar.tar
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib64/python3.9/logging/__init__.py", line 1082, in emit
stream.write(msg + self.terminator)
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 50-56: surrogates not allowed
Call stack:
File "/home/armijn/tmp/binaryanalysis-ng/src/bang-scanner", line 414, in <module>
main(sys.argv)
File "/home/armijn/tmp/binaryanalysis-ng/src/bang-scanner", line 294, in main
process.start()
File "/usr/lib64/python3.9/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/usr/lib64/python3.9/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/usr/lib64/python3.9/multiprocessing/context.py", line 277, in _Popen
return Popen(process_obj)
File "/usr/lib64/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/usr/lib64/python3.9/multiprocessing/popen_fork.py", line 71, in _launch
code = process_obj._bootstrap(parent_sentinel=child_r)
File "/usr/lib64/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib64/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/armijn/tmp/binaryanalysis-ng/src/ScanJob.py", line 694, in processfile
scanjob.check_entire_file(unpacker)
File "/home/armijn/tmp/binaryanalysis-ng/src/ScanJob.py", line 556, in check_entire_file
log(logging.DEBUG, "TRYING %s %s at offset: 0" %
File "/home/armijn/tmp/binaryanalysis-ng/src/banglogging.py", line 7, in log
logging.log(level, message)
Message: 'TRYING testtar.tar-0x00000000-tar-1/ustar/umlauts-\udcc4\udcd6\udcdc\udce4\udcf6\udcfc\udcdf script at offset: 0'
When I run the bang-scanner I get:
Traceback (most recent call last):
File "/usr/src/bang/src/ScanJob.py", line 689, in processfile
scanjob.carve_file_data(unpacker)
File "/usr/src/bang/src/ScanJob.py", line 481, in carve_file_data
outfile = open(outfile_full, 'wb')
OSError: [Errno 36] File name too long: '/usr/src/bang/src/unpacked/bang-scan-zzd2vg9u/unpack/firmware.bin-0x00000076-lz4-1/unpacked-from-lz4-0x00def93d-synthesized-3/unpacked-0xdef93d-0x109123c-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900'
Is there a way to prevent this?
The new scanjob setup has a problem with broken links:
$ time python3 bang-scanner -c bang.config -f /tmp/TEW-636APB-1002.bin
Process Process-2:
Traceback (most recent call last):
File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "bang-scanner", line 1120, in processfile
scanjob.check_for_signatures(unpacker, fileresult, scanfilequeue, scanenvironment, unpackdirectory, temporarydirectory)
File "bang-scanner", line 378, in check_for_signatures
j = ScanJob(pathlib.Path(unpackedfile), unpackedlabel, self.filename, [], {})
File "bang-scanner", line 107, in init
self._stat_file()
File "bang-scanner", line 119, in _stat_file
self.stat = os.stat(self.filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/armijn/tmp/bang-scan-2dchgsnf/unpack/TEW-636APB-1002.bin-squashfs-1/usr/sbin/rc'
This file is a broken symlink.
The init() method of the ScanJob class calls self.stat_file() which throws an exception.
If specifying the unpack directory or temporary directory as a relative path name, the scan takes forever even on a small file.
Are you interested perhaps in re-packing capabilities for reengineering?
Think e.g. round-tripping apk-s through apktool, repacking archives/installers after editing the contents, etc.
The GIF .ksy file is not correct for application identifiers.
After cloning today from master
and building the Docker container:
$ docker build -t bang
(...)
python3-pytz-2018.5-1.fc29.noarch
python3-webencodings-0.5.1-6.fc29.noarch
yajl-2.1.0-11.fc29.x86_64
Complete!
Removing intermediate container 727e3b8f8134
---> 798710dfd0fb
Step 5/5 : CMD ["python3","bangshell"]
---> Running in de71b6f2177b
Removing intermediate container de71b6f2177b
---> eceae9b2e1dc
Successfully built eceae9b2e1dc
Successfully tagged bang:latest
Then inside the container itself interactively, bang-scanner
does not seem to behave as specified in the README.md's invocation:
$ docker run -it bang /bin/bash
[root@b6cc367e62c7 bang]# ./src/bang-scanner
bash: ./src/bang-scanner: Permission denied
[root@b6cc367e62c7 src]# chmod +x bang-scanner
[root@b6cc367e62c7 src]# ./bang-scanner
Traceback (most recent call last):
File "./bang-scanner", line 52, in <module>
import elasticsearch
ModuleNotFoundError: No module named 'elasticsearch'
what should I do after nix-shell is ready?
When I run "python3 -m bang.cli scan -u xxx xxx", It shows error: Error while finding module specification for 'bang.cli' (ModuleNotFoundError: No module named 'bang')
I've just been through the nix setup (ubuntu 23.04) and now at the stage of running for the first time, but i get the below error:
Are you able to expand on the usage info as there isn't much on the readme?
maybe some more examples or a yt video
thanks
[nix-shell:~/tools/binaryanalysis-ng]$ python3 -m bang.cli
/nix/store/4agknr9yslk6rd1n5s45pgxlmpfb4vvq-python3-3.10.11-env/bin/python3.10: Error while finding module specification for 'bang.cli' (ModuleNotFoundError: No module named 'bang')
We installed BANG successfully and able to unpack binary files (.bin-flashwares).
But we are not able to locate result files for Open Source License scanning.
We see "LicenseIdentifierScanner.py" file in \src but not sure if it is called during unpacking or has to be called separately?
Hi, I am trying to get BANG to work, believing that it's a tool that could be very useful, but I have not had a lot of success with it yet.
Since I used the command listed in the README, I would suggest to put a sentence below that (in the "Invocation" section) what kind of output one should expect.
I tried to get binaryanalysis-ng working in docker. Here are the problems i encountered:
Dockerfile.kaitai
the filename kaitai-struct-compiler-0.10-SNAPSHOT.zip
is hardcoded but have since changed (same as issue #121 )Step 7/12 : RUN unzip -d / jvm/target/universal/kaitai-struct-compiler-0.9-SNAPSHOT.zip
---> Running in b74c7540079c
unzip: cannot find or open jvm/target/universal/kaitai-struct-compiler-0.9-SNAPSHOT.zip, jvm/target/universal/kaitai-struct-compiler-0.9-SNAPSHOT.zip.zip or jvm/target/universal/kaitai-struct-compiler-0.9-SNAPSHOT.zip.ZIP.
The command '/bin/sh -c unzip -d / jvm/target/universal/kaitai-struct-compiler-0.9-SNAPSHOT.zip' returned a non-zero code: 9
make: *** [Makefile:12: docker-kaitai] Error 9
src/Makefile
assumes that kaitai-struct-compiler
is in PATH but it is notStep 5/16 : RUN make
---> Running in b21cb46187e5
kaitai-struct-compiler -t python --outdir `dirname "parsers/font/pcf/pcf_font.ksy"` parsers/font/pcf/pcf_font.ksy
/bin/sh: 1: kaitai-struct-compiler: not found
Makefile:25: recipe for target 'parsers/font/pcf/pcf_font.py' failed
make: *** [parsers/font/pcf/pcf_font.py] Error 127
The command '/bin/sh -c make' returned a non-zero code: 2
src/Dockerfile
assumes that bangshell
is in the PATH which it is notpython3: can't open file '/kaitai_struct/runtime/python/bangshell': [Errno 2] No such file or directory
I'm going to create a pull request that fixes these issues.
Besides that I think the following things could be changed:
Can you tag a release and package it in nixpkgs?
Related to #26
What is the expected behaviour for signature matching in the following situation?
signature = aba
string in which to search = abababc
There are three possibilities: position 0 only, position 2 only, or both positions. The current code picks the first alternative, but it might miss some signatures. Some signatures might overlap, e.g. ico and truetype combined with padding.
Upon Running docker image build -t bang .
I get the following error:
> docker image build -t bang .
[+] Building 2.0s (5/5) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 1.24kB 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 45B 0.0s
=> CANCELED [internal] load metadata for docker.io/library/fedora:33 1.9s
=> ERROR [internal] load metadata for docker.io/library/kaitai:latest 1.9s
=> [auth] library/kaitai:pull token for registry-1.docker.io 0.0s
> [internal] load metadata for docker.io/library/kaitai:latest:
failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
I have authenticated to Docker Hub, so I'm not sure where the issue is stemming from here. I am running this on a Debian system under WSL2.
I'm trying a program to explore the possibilities of scanning
[kea@localhost src]$ python3 ./bang-scanner -c ./bang.config -f ./fw-UR-825AC-12.11.15.bin
the program creates a file of infinite size ( I interrupted on a file size "sda" of 100GB )
bang-scan-ejl4sdbz/unpack/fw-UR-825AC-12.11.15.bin-0x0014c822-squashfs-1/dev/sda
if interested, then this file https://cloud.mail.ru/public/481V/5kbYd9dLz
HI
I run bang-scanner for an error
root@ubuntu:/home/test/binaryanalysis-ng/src# python3 bang-scanner -c bang.config -f /home/test/Desktop/S29AL016D90-200000H.bin
Traceback (most recent call last):
File "bang-scanner", line 403, in
main(sys.argv)
File "bang-scanner", line 168, in main
startedfile = open(scandirectory / "STARTED", 'wb')
TypeError: invalid file: PosixPath('/root/tmp/bang-scan-r42t3f24/STARTED')
root@ubuntu:/home/test/binaryanalysis-ng/src#
file test_password.7z found at https://github.com/fkie-cad/FACT_core/tree/master/src/plugins/unpacking/sevenz/test/data cannot be unpacked without hitting "enter".
The commit in 299da18 seems to break logging.
So you can easily set up a python virtual env? If you would consider merging I can create a fork.
Regards,
Jaap
yqbboy@ubuntu:~/binaryanalysis-ng-master/src$ python3 bang-scanner -c bang.config -f /home/yq/samples/
/usr/lib/python3/dist-packages/requests/init.py:80: RequestsDependencyWarning: urllib3 (1.25.10) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
Traceback (most recent call last):
File "bang-scanner", line 55, in
from bangsignatures import maxsignaturesoffset
File "/home/secneo/binaryanalysis-ng-master/src/bangsignatures.py", line 487, in
extension_to_unpackparser = get_unpackers_for_extensions()
File "/home/secneo/binaryanalysis-ng-master/src/bangsignatures.py", line 481, in get_unpackers_for_extensions
for u in get_unpackers():
File "/home/secneo/binaryanalysis-ng-master/src/bangsignatures.py", line 476, in get_unpackers
pathlib.Path(os.path.dirname(parsers.file)), pathlib.Path('.'))
File "/home/secneo/binaryanalysis-ng-master/src/bangsignatures.py", line 470, in _get_unpackers_recursive
unpackers_root, full_module_path ))
File "/home/secneo/binaryanalysis-ng-master/src/bangsignatures.py", line 461, in _get_unpackers_recursive
module = importlib.import_module(module_name)
File "/usr/lib/python3.6/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/secneo/binaryanalysis-ng-master/src/parsers/archivers/cpio/UnpackParser.py", line 5, in
from . import cpio_new_ascii
ImportError: cannot import name 'cpio_new_ascii'
How does this relate to diffoscope?
I'm trying a program to explore the possibilities of scanning
[kea@localhost src]$ python3 ./bang-scanner -c ./bang.config -f ./openwrt-realtek-A2004NS-AP-fw_SDK_v2.4_20150724_release_by_Igor_T.bin
and program hung ( not response over 1 hour )
if interested, then this file https://cloud.mail.ru/public/3snL/3gnzzsBGx
Since e2tools 0.1.0 the file mode is by default pretty printed with a 10 character string instead of numeric. There doesn't seem to be a way to get it in numeric.
printing a ScanJobError fails:
AttributeError: 'ScanJob' object has no attribute 'filename'
@timhemel what did you exactly have in mind with ScanJobError?
[davidak@ethmoid:~/code/binaryanalysis-ng]$ nix-shell
error: undefined variable 'dockerfile-parse' at /home/davidak/code/binaryanalysis-ng/shell.nix:6:5
(use '--show-trace' to show detailed location information)
On which channel are you on?
"x86_64-linux"
Linux 4.19.49, NixOS, 19.03.172866.4649b6ef4b5 (Koi)
yes
yes
nix-env (Nix) 2.2.2
"nixos-19.03.172979.8634c3b6199, nixos-hardware, nixos-unstable-19.09pre183392.83ba5afcc96"
/nix/var/nix/profiles/per-user/root/channels/nixos
tagging duplicate files doesn't seem to work anymore
I used the Dockerfile to set up BANG and first, I couldn't figure out how to actually scan a file using the default bangshell. So instead, I modified the Dockerfile to just open up a bash instance and I'm trying to execute the following:
[root@0b7e5774a21c src]# python3 bang-scanner -c bang.config -f firmware.bin
I get:
Base unpack directory /root/tmp does not exist, exiting
At first, I thought this was due to where I had the . bin file but then I moved it to the current directory and I still get the error. In fact, I get it even if I just call python3 bang-scanner with no args at all.
What am I doing wrong? And Is there a way to do this from within the bangshell? Thanks!
Also note: root/tmp directory DOES exist...
(venv) freedom@freedom-virtual-machine:~/project/newbinaryscan/binaryanalysis-ng/src$ sudo /home/freedom/project/newbinaryscan/venv/bin/python bang-scanner -c bang.config -f test/testdata/unpackers/
Traceback (most recent call last):
File "bang-scanner", line 55, in <module>
from bangsignatures import maxsignaturesoffset
File "/home/freedom/project/newbinaryscan/binaryanalysis-ng/src/bangsignatures.py", line 27, in <module>
import bangandroid
File "/home/freedom/project/newbinaryscan/binaryanalysis-ng/src/bangandroid.py", line 37, in <module>
import bangunpack
File "/home/freedom/project/newbinaryscan/binaryanalysis-ng/src/bangunpack.py", line 58, in <module>
import snappy
File "/home/freedom/project/newbinaryscan/venv/lib/python3.6/site-packages/snappy/__init__.py", line 7, in <module>
from .SnapPy import (AbelianGroup, HolonomyGroup, FundamentalGroup,
File "cython/core/basic.pyx", line 45, in init SnapPy
File "/home/freedom/project/newbinaryscan/venv/lib/python3.6/site-packages/snappy/horoviewer.py", line 3, in <module>
from .CyOpenGL import (HoroballScene, OpenGLOrthoWidget,
File "opengl/CyOpenGL.pyx", line 38, in init CyOpenGL
AttributeError: type object 'CyOpenGL.vector3' has no attribute '__reduce_cython__'
(venv) freedom@freedom-virtual-machine:~/project/newbinaryscan/binaryanalysis-ng/src$
python3.6.9
ubuntu18.04
Currently the assumption is that if the System Use field is used the data stored is a Rockridge extension. This is not necessarily true: in the '90s Apple used the System Use field to store HFS specific data (see http://www.reverse-engineering.info/CD/iso9660.pdf page 33)
there are various files that all have the same MZ
signature. Right now BANG does not consistently unpack these: sometimes the PE unpacker is run first, sometimes the DOS MZ unpacker is run first. Merge these (as far as possible) so scanning is more consistent.
I'm trying out BANG for the first time. Executing nix-shell analysis.nix
results in the following error message:
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)
I'm using an Ubuntu 22.04.2 LTS.
The archive at https://archive.synology.com/download/DSM/release/6.2.1/23824/DSM_DS112+_23824.pat contains a certificate file that is unpacked to
unpack/DSM_DS112+_23824.pat-tar-1/hda1.tgz-xz-1/unpacked-from-xz-tar-1/usr/lib/python2.7/ensurepip/_bundled/pip-8.1.1-py2.py3-none-any.whl-zip-1/pip/_vendor/requests/cacert.pem
unpackCertificate rejects this file and says that it is not a certificate.
I am getting this after setting json = yes
in my config:
Traceback (most recent call last):
File "bang-scanner", line 414, in <module>
main(sys.argv)
File "bang-scanner", line 374, in main
JsonReporter(jsonfile).report(scanresult)
File "/home/hmeine/tmp/tern-test/binaryanalysis-ng/src/JsonReporter.py", line 43, in report
for a, h in fileresult.get_hashresult().items():
AttributeError: 'dict' object has no attribute 'get_hashresult'
Hi I am currently looking into the different unpackers of BANG. I notice that although BANG supports more than 100 different filetypes, only a few exists in the test directory. Do you have a test corpus of all the filetypes that BANG supports? If so, would you mind sharing the test corpus so I can also test BANG from my end? Thanks!
squashfs 4.4 treats non-fatal errors the same as fatal errors and always exits with 1. In case a squashfs file system has files that cannot be unpacked, but which are irrelevant, such as device files, pipes, etc. then unsquashfs will exit with 1 which BANG will treat as an error although.
Newer versions of squashfs have a workaround, see plougher/squashfs-tools#94
My test file openwrt-18.06.1-brcm2708-bcm2710-rpi-3-ext4-sysupgrade.img.gz
contains an image, which file
identifies as MBR image:
$ file openwrt-18.06.1-brcm2708-bcm2710-rpi-3-ext4-sysupgrade.img
openwrt-18.06.1-brcm2708-bcm2710-rpi-3-ext4-sysupgrade.img: DOS/MBR boot sector; partition 1 : ID=0xc, active, start-CHS (0x20,2,3), end-CHS (0xc3,0,12), startsector 8192, 40960 sectors; partition 2 : ID=0x83, start-CHS (0xe3,2,15), end-CHS (0x14,0,16), startsector 57344, 524288 sectors
Bang does not find it. It mistakenly identifies a FAT filesystem because it recognizes the magic number for MBR \x55\xaa
. As a result, bang is trying many other filetypes and clutters the unpack directory.
There should be an unpacker for MBR images.
load a css file with some non printable characters, my example had
['0xe2', '0x9e', '0x99'] in them.
file should have label the text, not binary
file has both text and binary as labels
The extension unpacker adds the labels text and css, the IsTextComputer adds binary.
What is the correct labeling in this case?
There is a race condition that affects the android_sparse_data unpacker: successful unpacking of this format depends on the presence of other files (such as .transferlist
). Since files are yielded by the other unpackers it could happen that the main file has already been yielded and in the scan queue, but that the transferlist file has not been yielded yet or is empty. This does not happen when running with a single thread.
I'm trying a program to explore the possibilities of scanning
python3 ./bang-scanner -c ./bang.config -f ./gpt99_vfat16_sdb88.bin
and get errors
Process Process-2:
Traceback (most recent call last):
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 673, in processfile scanjob.check_for_signatures(unpacker)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 297, in check_for_signatures signature, offset)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/Unpacker.py", line 198, in try_unpack_file_for_signatures return bangsignatures.signaturetofunction[signature](fileresult, scanenvironment, offset, self.dataunpackdirectory)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/bangfilesystems.py", line 3032, in unpack_fat chainindex = clustervals[chainindex]
IndexError: list index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib64/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run()
File "/usr/lib64/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 745, in processfile raise ScanJobError(scanjob, e)
ScanJob.ScanJobError: Exception for scanjob:
file:
gpt99_vfat16_sdb88.bin
labels: root
Traceback (most recent call last):
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 673, in processfile scanjob.check_for_signatures(unpacker)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 297, in check_for_signatures signature, offset)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/Unpacker.py", line 198, in try_unpack_file_for_signatures return bangsignatures.signaturetofunction[signature](fileresult, scanenvironment, offset, self.dataunpackdirectory)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/bangfilesystems.py", line 3032, in unpack_fat chainindex = clustervals[chainindex]
IndexError: list index out of range
if interested, then this file https://cloud.mail.ru/public/5MZc/4GfiFTVwq
From https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT section 4.3.9.1:
For ZIP64(tm) format archives, the compressed and uncompressed sizes are 8 bytes each.
Currently this is not supported correctly.
Example: two GIF files concatenated together:
test.gif-0x00000000-gif-1/unpacked.gif
test.gif-0x000351b5-gif-1/unpacked.gif
The counter should have been increased.
When scanning a JPEG file successfully the following is printed to the logs:
Error closing: 'NoneType' object has no attribute 'close'
This error message is coming from pillow and probably this is because of load() that is called unnecessarily and can likely be removed without affecting functionality.
$ python3 bang-scanner -f /tmp/test.wad
Traceback (most recent call last):
File "bang-scanner", line 414, in
main(sys.argv)
File "bang-scanner", line 380, in main
HumanReadableReporter(reportfile).report(scanresult)
File "/home/armijn/tmp/binaryanalysis-ng/src/reporter/humanreadablereport.py", line 101, in report
s += self._fileunpackedfiles(fn)
File "/home/armijn/tmp/binaryanalysis-ng/src/reporter/humanreadablereport.py", line 65, in _fileunpackedfiles
l['offset'], l['type'], " ".join(sorted(l['files']))
TypeError: sequence item 0: expected str instance, PosixPath found
docker image build -t bang .
Sending build context to Docker daemon 2.027MB
Step 1/16 : FROM kaitai as builder
pull access denied for kaitai, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
Makefile:52: recipe for target 'ctrbuild' failed
make: *** [ctrbuild] Error 1
I tried a few Docker images for kaitai:
blacktop/kaitai
only provides 0.8librespace/kaitai
does not contain /kaitai_struct
kaitai/ksv
also does not contain /kaitai_struct
So where does the kaitai
image come from?
When building the environment with nix-shell and then launching bang-scanner an error is thrown:
Traceback (most recent call last):
File "bang-scanner", line 414, in <module>
main(sys.argv)
File "bang-scanner", line 87, in main
options = BangScannerOptions().get()
File "/home/armijn/tmp/binaryanalysis-ng/src/bangscanneroptions.py", line 43, in __init__
self._read_configuration_file()
File "/home/armijn/tmp/binaryanalysis-ng/src/bangscanneroptions.py", line 127, in _read_configuration_file
self.config = configparser.ConfigParser(os.environ)
File "/nix/store/yl69v76azrz4daiqksrhb8nnmdiqdjg9-python3-3.8.8/lib/python3.8/configparser.py", line 639, in __init__
self._read_defaults(defaults)
File "/nix/store/yl69v76azrz4daiqksrhb8nnmdiqdjg9-python3-3.8.8/lib/python3.8/configparser.py", line 1219, in _read_defaults
self.read_dict({self.default_section: defaults})
File "/nix/store/yl69v76azrz4daiqksrhb8nnmdiqdjg9-python3-3.8.8/lib/python3.8/configparser.py", line 752, in read_dict
raise DuplicateOptionError(section, key, source)
configparser.DuplicateOptionError: While reading from '<dict>': option 'shell' in section 'DEFAULT' already exists
The bangscanneroptions.py code does the following:
self.config = configparser.ConfigParser(os.environ)
os.environ has an environment variable SHELL. The nix-shell command introduces a variable 'shell' to the environment. These conflict as configparser first lowercases everything before adding it to a ConfigParser instance.
Proposal: rewrite to something using YAML or so.
It is a bit problematic to see all the specs in this repo when they reside in different branches. So I propose to organize an own separate repo mirroring kaitai_struct_formats (ksf) structure (so its contents can be copied over it (except the dotted dirs, of course) for the specs not yet merged into ksf.
Probably it may make sense to create an org first and move this repo into it.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.