I have some error when install from pip on Ubuntu 16.04 (Backbox Linux).
The python version is "Python 3.5.2 (default, Nov 12 2018, 13:43:14)".

root[~] >> pip install filebytes
Collecting filebytes
  Downloading https://files.pythonhosted.org/packages/0b/3a/9fc0c62bd74583137a8bbc3c8020d6a8234b9cf8bc1e99fe929688b19093/filebytes-0.9.20.tar.gz
  WARNING: Generating metadata for package filebytes produced metadata for project name unknown. Fix your #egg=filebytes fragments.
Requirement already satisfied (use --upgrade to upgrade): unknown from https://files.pythonhosted.org/packages/0b/3a/9fc0c62bd74583137a8bbc3c8020d6a8234b9cf8bc1e99fe929688b19093/filebytes-0.9.20.tar.gz#sha256=6c33986ca048e49cf1a5e2f167af9f02c7f866576b3b91a8a9124d32e57f935d in /usr/local/lib/python3.5/dist-packages/UNKNOWN-0.9.20-py3.5.egg
Building wheels for collected packages: unknown, unknown
  Building wheel for unknown (setup.py) ... done
  Stored in directory: /root/.cache/pip/wheels/5e/69/2b/08660fcb6cd841d60ff8980be781f40ee15ff014c5aaaace29
  Building wheel for unknown (setup.py) ... error
  ERROR: Complete output from command /usr/bin/python3.5 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-13mrh900/unknown/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-x8zwjl3s --python-tag cp35:
  ERROR: Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/usr/lib/python3.5/tokenize.py", line 454, in open
      buffer = _builtin_open(filename, 'rb')
  FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-install-13mrh900/unknown/setup.py'
  ----------------------------------------
  ERROR: Failed building wheel for unknown
  Running setup.py clean for unknown
  ERROR: Complete output from command /usr/bin/python3.5 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-13mrh900/unknown/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' clean --all:
  ERROR: Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/usr/lib/python3.5/tokenize.py", line 454, in open
      buffer = _builtin_open(filename, 'rb')
  FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-install-13mrh900/unknown/setup.py'
  ----------------------------------------
  ERROR: Failed cleaning build dir for unknown
Successfully built unknown
Failed to build unknown

Now I get bytes data(from elf file) from network, I want to analysis it use filebytes, so how convert bytes format to filebytes.elf.ELF format?

Found a case where a PE section parser was thrown off by a mismatch between section's raw size and its virtual size. It would read the section bytes sequentially, encounter the padding zeroes in the end, and crash.

PE stores the size of the section in a loaded PE file in the second field of the section header, VirtualSize. Maybe the library should initialize section.bytes with a VirtualSize-sized blob instead. Note the case where VirtualSize is greater than SizeOfRawData ; in those cases the bytes should be zero padded.

The filebytes library surfaces that field as PhysicalAddress_or_VirtualSize. The doc at https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#section-table-section-headers doesn't mention the circumstances when this field can mean PhysicalAddress, but maybe the maintainers know something I don't.

Hi,

I've been manipulating "large" PE files (i.e. more than 5MB) and filebytes fails when trying to parse the structure.

For example with ntoskrnl.exe:

# ls -lah /vagrant/ntoskrnl.exe
-rw-r--r-- 1 root root 5.5M Jan 11 09:50 /vagrant/ntoskrnl.exe
#  python3
Python 3.6.4 (default, Dec 19 2017, 14:09:48) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from filebytes.pe import *
>>> p = PE('/vagrant/ntoskrnl.exe')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/filebytes/pe.py", line 432, in __init__
    self.__dataDirectory = self._parseDataDirectory(self._bytes, self.sections, self.imageNtHeaders)
  File "/usr/local/lib/python3.6/dist-packages/filebytes/pe.py", line 535, in _parseDataDirectory
    import_data_directory_data = self._parseDataDirectoryImport(import_data_directory, import_section)
  File "/usr/local/lib/python3.6/dist-packages/filebytes/pe.py", line 592, in _parseDataDirectoryImport
    import_address_table =  self.__parseThunks(import_descriptor.FirstThunk, importSection)
  File "/usr/local/lib/python3.6/dist-packages/filebytes/pe.py", line 651, in __parseThunks
    self.__parseThunkData(thunkData, importSection)
  File "/usr/local/lib/python3.6/dist-packages/filebytes/pe.py", line 662, in __parseThunkData
    ibn = IMAGE_IMPORT_BY_NAME.from_buffer(importSection.raw, offset)
ValueError: Buffer size too small (8192 instead of at least 3093560 bytes)

Any idea?

Thanks!

I've been trying to install this for two days now and keep hitting errors.

When I try to install this via pip "pip install filebytes", I get the following error message:

ERROR: Complete output from command python setup.py egg_info:
    ERROR: Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-rjhyqG/filebytes/setup.py", line 15, in <module>
        version = extractMetaInfo((currentDir / "filebytes" / "__init__.py").read_text())["VERSION"]
    AttributeError: 'PosixPath' object has no attribute 'read_text'
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-rjhyqG/filebytes/

I've tried updating setuptools etc, but it still doesnt work. I am not sure if this is a problem with pip or filebytes though.

The MachO constructor reads the whole file into memory, whether it's fat or not. In my usage scenarios, only one architecture slice is needed at a time. Reading the whole thing wastes I/O and memory.

Both pip3 install filebytes and python3.5 setup.py install --user install the package as UNKNOWN-0.9.21 instead of filebytes-0.9.21 which means packages that depend on this one don't find it.

This can be reproduced in an ubuntu:16.04 docker image.

First of all, thank you for this great project!

Would it be possible to add support for Universal binaries that are frequently used on iOS (armv7 and arm64 binary), for example? The header is defined in fat.h, it is the fat_header struct.

I noticed this bug while trying to package ropper 1.8.0 via the ropper test suite sashs/Ropper#18.
Please also maybe consider adding some unit tests for this project directly to make it able to test for regressions before any release.

PS: the tests were running on i686 (32bit) because of the packaging process

ERROR: test_gadgets (testcases.test_x86_64.MACHO_x86_84)

Traceback (most recent call last):
File "/build/ropper/src/Ropper-1.8.0/testcases/test_x86_64.py", line 132, in setUp
self.file = Loader.open('test-binaries/ls-macho-x86_64')
File "/build/ropper/src/Ropper-1.8.0/ropper/loaders/loader.py", line 154, in open
return subclass(fileName)
File "/build/ropper/src/Ropper-1.8.0/ropper/loaders/mach_o.py", line 33, in init
super(MachO, self).init(filename)
File "/build/ropper/src/Ropper-1.8.0/ropper/loaders/loader.py", line 72, in init
self.binary = self._loadFile(filename)
File "/build/ropper/src/Ropper-1.8.0/ropper/loaders/mach_o.py", line 92, in _loadFile
return macho.MachO(fileName)
File "/usr/lib/python3.5/site-packages/filebytes/mach_o.py", line 328, in __init
self.__loadCommands = self._parseLoadCommands(self._bytes, self.machHeader)
File "/usr/lib/python3.5/site-packages/filebytes/mach_o.py", line 384, in _parseLoadCommands
command = self.__parseSegmentCommand(data, offset, raw)
File "/usr/lib/python3.5/site-packages/filebytes/mach_o.py", line 404, in __parseSegmentCommand
sections = self.__parseSections(data, sc, offset+sizeof(self._classes.SegmentCommand))
File "/usr/lib/python3.5/site-packages/filebytes/mach_o.py", line 442, in __parseSections
raw = (c_ubyte * sec.size).from_buffer(data, sec.offset)
OverflowError: cannot fit 'int' into an index-sized integer

ERROR: test_general (testcases.test_x86_64.MACHO_x86_84)

Traceback (most recent call last):
File "/build/ropper/src/Ropper-1.8.0/testcases/test_x86_64.py", line 132, in setUp
self.file = Loader.open('test-binaries/ls-macho-x86_64')
File "/build/ropper/src/Ropper-1.8.0/ropper/loaders/loader.py", line 154, in open
return subclass(fileName)
File "/build/ropper/src/Ropper-1.8.0/ropper/loaders/mach_o.py", line 33, in init
super(MachO, self).init(filename)
File "/build/ropper/src/Ropper-1.8.0/ropper/loaders/loader.py", line 72, in init
self.binary = self._loadFile(filename)
File "/build/ropper/src/Ropper-1.8.0/ropper/loaders/mach_o.py", line 92, in _loadFile
return macho.MachO(fileName)
File "/usr/lib/python3.5/site-packages/filebytes/mach_o.py", line 328, in __init
self.__loadCommands = self._parseLoadCommands(self._bytes, self.machHeader)
File "/usr/lib/python3.5/site-packages/filebytes/mach_o.py", line 384, in _parseLoadCommands
command = self.__parseSegmentCommand(data, offset, raw)
File "/usr/lib/python3.5/site-packages/filebytes/mach_o.py", line 404, in __parseSegmentCommand
sections = self.__parseSections(data, sc, offset+sizeof(self._classes.SegmentCommand))
File "/usr/lib/python3.5/site-packages/filebytes/mach_o.py", line 442, in __parseSections
raw = (c_ubyte * sec.size).from_buffer(data, sec.offset)
OverflowError: cannot fit 'int' into an index-sized integer

ERROR: test_jmpreg (testcases.test_x86_64.MACHO_x86_84)

Traceback (most recent call last):
File "/build/ropper/src/Ropper-1.8.0/testcases/test_x86_64.py", line 132, in setUp
self.file = Loader.open('test-binaries/ls-macho-x86_64')
File "/build/ropper/src/Ropper-1.8.0/ropper/loaders/loader.py", line 154, in open
return subclass(fileName)
File "/build/ropper/src/Ropper-1.8.0/ropper/loaders/mach_o.py", line 33, in init
super(MachO, self).init(filename)
File "/build/ropper/src/Ropper-1.8.0/ropper/loaders/loader.py", line 72, in init
self.binary = self._loadFile(filename)
File "/build/ropper/src/Ropper-1.8.0/ropper/loaders/mach_o.py", line 92, in _loadFile
return macho.MachO(fileName)
File "/usr/lib/python3.5/site-packages/filebytes/mach_o.py", line 328, in __init
self.__loadCommands = self._parseLoadCommands(self._bytes, self.machHeader)
File "/usr/lib/python3.5/site-packages/filebytes/mach_o.py", line 384, in _parseLoadCommands
command = self.__parseSegmentCommand(data, offset, raw)
File "/usr/lib/python3.5/site-packages/filebytes/mach_o.py", line 404, in __parseSegmentCommand
sections = self.__parseSections(data, sc, offset+sizeof(self._classes.SegmentCommand))
File "/usr/lib/python3.5/site-packages/filebytes/mach_o.py", line 442, in __parseSections
raw = (c_ubyte * sec.size).from_buffer(data, sec.offset)
OverflowError: cannot fit 'int' into an index-sized integer

ERROR: test_ppr (testcases.test_x86_64.MACHO_x86_84)

Traceback (most recent call last):
File "/build/ropper/src/Ropper-1.8.0/testcases/test_x86_64.py", line 132, in setUp
self.file = Loader.open('test-binaries/ls-macho-x86_64')
File "/build/ropper/src/Ropper-1.8.0/ropper/loaders/loader.py", line 154, in open
return subclass(fileName)
File "/build/ropper/src/Ropper-1.8.0/ropper/loaders/mach_o.py", line 33, in init
super(MachO, self).init(filename)
File "/build/ropper/src/Ropper-1.8.0/ropper/loaders/loader.py", line 72, in init
self.binary = self._loadFile(filename)
File "/build/ropper/src/Ropper-1.8.0/ropper/loaders/mach_o.py", line 92, in _loadFile
return macho.MachO(fileName)
File "/usr/lib/python3.5/site-packages/filebytes/mach_o.py", line 328, in __init
self.__loadCommands = self._parseLoadCommands(self._bytes, self.machHeader)
File "/usr/lib/python3.5/site-packages/filebytes/mach_o.py", line 384, in _parseLoadCommands
command = self.__parseSegmentCommand(data, offset, raw)
File "/usr/lib/python3.5/site-packages/filebytes/mach_o.py", line 404, in __parseSegmentCommand
sections = self.__parseSections(data, sc, offset+sizeof(self._classes.SegmentCommand))
File "/usr/lib/python3.5/site-packages/filebytes/mach_o.py", line 442, in __parseSections
raw = (c_ubyte * sec.size).from_buffer(data, sec.offset)
OverflowError: cannot fit 'int' into an index-sized integer

Ran 35 tests in 64.721s

FAILED (errors=4)

looks like you forgot to push the git tag for 0.9.16.

cheers

There seems to be a v0.9.15 on pypi but no git tag pushed.
cheers,
Levente

I've been using Ropper for some time now, and I really enjoy the program. Over the last week, I've been receiving issues with what terminal says to be Filebyte. It basically says something along the lines of not being able to download and use the most current version of Filebyte. I've tried removing Ropper, and all files associated with it, this didn't resolve the issue. I've tried manually downloading Filebyte and putting it where it should be, this doesn't resolve the issue. I've tested this on two separate operating systems, and the issue still persists. If you like, we could do a screen sharing session so that you can see. I sent you an E-Mail in regards to this as well.

Trying to open a DWARF debug symbols file, as generated by Xcode on Mac, with filebytes.MachO. Getting the following error message from the MachO constructor:

>>> macho = MachO(filename)
Traceback (most recent call last):
File "", line 1, in
File "/home/seva/.local/lib/python2.7/site-packages/filebytes/mach_o.py", line 365, in init
self.__fatArches = self._tryParseFat(self._bytes)
File "/home/seva/.local/lib/python2.7/site-packages/filebytes/mach_o.py", line 438, in _tryParseFat
thin = MachO('{}.{}'.format(self.fileName, cputype), thin_data)
File "/home/seva/.local/lib/python2.7/site-packages/filebytes/mach_o.py", line 374, in init
self.__loadCommands = self._parseLoadCommands(self._bytes, self.machHeader)
File "/home/seva/.local/lib/python2.7/site-packages/filebytes/mach_o.py", line 461, in _parseLoadCommands
command = self.__parseSegmentCommand(data, offset, raw)
File "/home/seva/.local/lib/python2.7/site-packages/filebytes/mach_o.py", line 481, in __parseSegmentCommand
sections = self.__parseSections(data, sc, offset+sizeof(self._classes.SegmentCommand))
File "/home/seva/.local/lib/python2.7/site-packages/filebytes/mach_o.py", line 522, in __parseSections
raw = (c_ubyte * sec.size).from_buffer(data, sec.offset)
ValueError: Buffer size too small (2089089 instead of at least 2406496 bytes)

Looks like some kind of assumption about section size doesn't hold. What should I check?

DWARFDUMP on Mac has no problem with that file.

I recently tried to use MachO to read a large (1.3GB) Mach-O dSYM file, and found that it ended up using about 4GB of memory and didn't free that memory even when I tried to get it to be garbage collected. So, it seems like there may be two issues here, one of unnecessarily large memory usage, and the other of a memory leak. Is MachO not just collecting metadata? It seems like that would use almost no memory at all.

Cleaning them from history with BFG repo cleaner and rewriting it is recommended

Hi again.

When developing a tool FrozenTable.py editing a data structure in cpython binaries, I had to create some modules it may make some sense to integrate them into your tool

• BinPatchTools - an abstraction layer providing an unified interface to some libs for parsing and editing binaries. Surprisingly even LIEF doesn't, even though it was one of the goals of that project. It may make sense to throw away own parsers used in filebytes and use third-party ones, concentrating on providing a unified interface.
• ELFRelocs.py - unfortunately none of the libs I have encountered was able to calculate relocs. So I had created a lib allowing integration of relocs computation in third-party libs.
• ELFMachine.py - just a enum for e_machine header field. Different libs use own enum classes, this is bad for interoperability, so I have created a one class to rule them all. setup.py visits some hardcoded addresses of projects under permissive licenses and parses this enum from their sources and creates a python file.

In class LC, the following is missing:

ENCRYPTION_INFO_64 = 0x0000002C