mkorman90 / regipy Goto Github PK

In the registry.py file on line 114 I receved the following exception when processing a userclass.dat registry file.

Traceback (most recent call last):
  File "C:\Program Files\Python37\lib\site-packages\regipy\registry.py" line 199, in dump_hive_to_json
    for entry in tqdm(self.recurse_subkeys(name_key_entry, as_json=True), disable=not verbose)
  File "C:\Program Files\Python37\lib\site-packages\tqdm\_tqdm.py" line 1022, in __iter__
    for obj in iterable:
  File "C:\Program Files\Python37\lib\site-packages\regipy\registry.py" line 119, in recurse_subkeys
    yield from self.recurse_subkeys(nk_record=subkey, path=r'\{}'.format(subkey.name), as_json=as_json)
  File "C:\Program Files\Python37\lib\site-packages\regipy\registry.py" line 117, in recurse_subkeys
    as_json=as_json)
  File "C:\Program Files\Python37\lib\site-packages\regipy\registry.py" line 114, in recurse_subkeys
    if subkey.subkey_count:
AttributeError: 'LIRecord' object has no attribute 'subkey_count'

It appears that try_decode_binary applies the max_len to the data regardless off whether as_json is set or not. This causes data to be truncated when using something like get_values with as_json=False

This issue provides visibility into Renovate updates and their statuses. Learn more

Edited/Blocked

These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.

• Update dependency pytest to v7.1.2

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update actions/setup-python action to v4
• Update python orb to v2

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• Update dependency click to v8.1.3

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Using the tool, I see that the returned data size of a key is always no more than 128 characters, even with keys that longer than this.

I assured that the length is more than 128 both with the windows registry viewer and in hivexsh.

Is there an explanation for this?

both with using: registry-dump.exe SAM.hiv -o test.json and Python [for entry in reg.recurse_subkeys(as_json=True):]

always see like this

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

At this stage, libfwsi-python is not available as prebuilt wheels for Linux or macOS and this makes it difficult to install in a few cases. I offered to help there but there is no reply yet at libyal/libfwsi#19

In addition your requirement is pinned to an exact version which is problematic when using regipy as a library if there is an upgrade in the future. See this excellent article on the topic https://iscinumpy.dev/post/bound-version-constraints/

It would be great if you could relax the setup.py bounded requirements this way in

libfwsi-python >= 20220123
inflection >= 0.5.1

And wrt. libfwsi-python, it would be even better to make it an extra requires such that the plugin(s) that need it would only import it privately if used?

I can submit a patch alright if you think that can work for you.

FWIW, we are using regipy in https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/win_reg.py and this helps us extract installed package information from registry hives of Windows VM and Docker containers while running this on Linux... so thank you ++ as this is the only Python registry library that can actually parse correctly these data! (This is also used in https://scancodeio.readthedocs.io indirectly)

I have a registry file that I concocted (more later) that has a single key with five values in it. Those values should be as follows:

$ hivexget test.registry '\key' 
"valueempty"=""
"value0"="0"
"value1"="1"
"valuea"="A"
"valueaa"="AA"

All value types are set to REG_SZ.

However, regipy does not parse the single-character strings as, well, strings. Given this script:

from regipy import registry

hive = registry.RegistryHive("test.registry")
key = hive.get_key("key")

for value in key.iter_values():
    print(value)

the output is:

Value(name='valueempty', value=0, value_type='REG_SZ', is_corrupted=False)
Value(name='value0', value=48, value_type='REG_SZ', is_corrupted=False)
Value(name='value1', value=49, value_type='REG_SZ', is_corrupted=False)
Value(name='valuea', value=65, value_type='REG_SZ', is_corrupted=False)
Value(name='valueaa', value='AA', value_type='REG_SZ', is_corrupted=False)

For valueempty/value0/value1/valuea, the resulting .value is an integer rather than a string. Note that chr(48) == "0", chr(49) == "1", chr(65) == "A"; and then of course the "AA" value is correct in the output.

I've tracked this down to here, in registry.py:

    def iter_values(self, as_json=False, max_len=MAX_LEN, trim_values=True):
        ...
        for _ in range(self.values_count):
            ...
            with boomerang_stream(self._stream) as substream:
                ...
                if data_type in ['REG_SZ', 'REG_EXPAND', 'REG_EXPAND_SZ']:
                    if vk.data_size >= 0x80000000:
                        # data is contained in the data_offset field
                        value.size -= 0x80000000
                        actual_value = vk.data_offset                                      # <-----
                    elif vk.data_size > 0x3fd8 and value.value[:2] == b'db':
                        ...
                    ...
                ...

where there is no string conversion in this case, vk.data_offset is just an integer.

I assume that because REG_SZ is (typically) UTF-16, a one-character string plus null termination fits into a four-byte size field in the binary format, but a two-character string plus null termination does not, and that's why two characters and up seems to work.

At least for my test case, I can fix this with a bit cast of vk.data_offset into bytes, which I then try to decode following the example of the other parts of the if data_type in [....] true branch:

                    if vk.data_size >= 0x80000000:
                        # data is contained in the data_offset field
                        value.size -= 0x80000000
                        packed = struct.pack("=l", vk.data_offset)
                        actual_value = try_decode_binary(packed, as_json=False, trim_values=False)

(you'll need to import struct of course).

This seems to work with higher Unicode code points as well; if I add a value "valueunicode"="ሴ" then I get Value(name='valueunicode', value=4660, value_type='REG_SZ', is_corrupted=False) on the release version and Value(name='valueunicode', value='ሴ', value_type='REG_SZ', is_corrupted=False) with my fix. I'm not positive that the pack call shouldn't use <l instead of =l, or L instead of l, or i instead of l, etc. For reporting purposes I tried to be conservative and go big, and extra null bytes after the string wouldn't hurt.

I suspect there's the same problem with REG_BINARY just below:

                elif data_type in ['REG_BINARY', 'REG_NONE']:
                    if vk.data_size >= 0x80000000:
                        # data is contained in the data_offset field
                        actual_value = vk.data_offset             # actual_value = struct.pack(...) ?

but I don't have a test case for this and am not set up to easily create one.

Speaking of the test case, I constructed mine "manually" using libhivex. I am attaching it here. However, I have seen the empty string version of this (coming out with a value of 0 instead of "") in the wild, with my actual registry.

Finally:

$ python
Python 3.11.7 (main, Dec 14 2023, 01:49:40) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import regipy
>>> regipy.__version__
'4.2.1'

test.registry.zip

Hello everybody,

I am trying to use regipy in a script within a Docker container. As soon as my script tries to import the RegistryHive, the following error gets thrown:

Traceback (most recent call last):
  File "/my_project/./Registry/registry.py", line 7, in <module>
    from regipy.registry import RegistryHive
  File "/usr/local/lib/python3.9/site-packages/regipy/__init__.py", line 1, in <module>
    from .registry import *
  File "/usr/local/lib/python3.9/site-packages/regipy/registry.py", line 16, in <module>
    from regipy.structs import REGF_HEADER, HBIN_HEADER, CM_KEY_NODE, LF_LH_SK_ELEMENT, VALUE_KEY, INDEX_ROOT, \
  File "/usr/local/lib/python3.9/site-packages/regipy/structs.py", line 3, in <module>
    REGF_HEADER = Struct(
  File "/usr/local/lib/python3.9/site-packages/construct/core.py", line 444, in compile
    module = importlib.util.module_from_spec(module_spec)
AttributeError: module 'importlib' has no attribute 'util'

It seems like core.py is importing importlib.util the wrong way (based on other articles with similar problems).

I use the following line within my script to import the RegistryHive

from regipy.registry import RegistryHive

The important lines of my Dockerfile are the following:

FROM python:3.9-slim-buster
....
COPY my_project/ .
...
RUN pip3 install -r requirements.txt
...
ENTRYPOINT ["python3", "my_project.py"]

The requirements.txt includes a line "regipy", which installs version 2.0.1 of regipy. I already tried to use older version of python (3.8, 3.7, 3.6), which resulted in the same error.

I am aware that this is not directly an issue of Regipy, but I would appreciate any help/input I can get here.

Thanks in advance and Best Regards
Moe

It is proposed to solve this by replacing registry- prefix witn regipy- one

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

The latest tag is 1.9.0 from March.
All the releases since are missing a tag: 2.0.0, 1.9.4, 1.9.3, 1.9.2, 1.9.1

Hi,

Pls see following for an example of a hive whose transactions were not successfully recovered:

registry-transaction-logs Amcache.hve -p Amcache.hve.LOG1 -s Amcache.hve.LOG2 -o Amcache.hve.regipy
[2020-09-10 02:41:17.219000] INFO: regipy.cli: Processing hive Amcache.hve with transaction log Amcache.hve.LOG1
[2020-09-10 02:41:17.219997] INFO: regipy.cli: Processing hive Amcache.hve with secondary transaction log Amcache.hve.LOG1
[2020-09-10 02:41:17.220994] INFO: regipy.recovery: Log Size: 196608
[2020-09-10 02:41:17.220994] INFO: regipy.recovery: Log Size: 196608
[2020-09-10 02:41:17.221991] INFO: regipy.recovery: Parsing hvle block at 512
[2020-09-10 02:41:17.221991] INFO: regipy.recovery: Currently at start of dirty pages: 616
[2020-09-10 02:41:17.222989] INFO: regipy.recovery: seq number: 313
[2020-09-10 02:41:17.222989] INFO: regipy.recovery: dirty pages: 8
[2020-09-10 02:41:17.222989] INFO: regipy.recovery: Restored 4096 bytes to offset 4096 from offset 616
[2020-09-10 02:41:17.222989] INFO: regipy.recovery: Restored 8192 bytes to offset 339968 from offset 4712
[2020-09-10 02:41:17.222989] INFO: regipy.recovery: Restored 8192 bytes to offset 364544 from offset 12904
[2020-09-10 02:41:17.222989] INFO: regipy.recovery: Restored 8192 bytes to offset 380928 from offset 21096
[2020-09-10 02:41:17.223987] INFO: regipy.recovery: Restored 4096 bytes to offset 946176 from offset 29288
[2020-09-10 02:41:17.223987] INFO: regipy.recovery: Restored 4096 bytes to offset 970752 from offset 33384
[2020-09-10 02:41:17.223987] INFO: regipy.recovery: Restored 4096 bytes to offset 1056768 from offset 37480
[2020-09-10 02:41:17.223987] INFO: regipy.recovery: Restored 4096 bytes to offset 1097728 from offset 41576
[2020-09-10 02:41:17.223987] INFO: regipy.recovery: Parsing hvle block at 49152
[2020-09-10 02:41:17.224983] INFO: regipy.recovery: Currently at start of dirty pages: 49336
[2020-09-10 02:41:17.224983] INFO: regipy.recovery: seq number: 314
[2020-09-10 02:41:17.224983] INFO: regipy.recovery: dirty pages: 18
[2020-09-10 02:41:17.224983] INFO: regipy.recovery: Restored 4096 bytes to offset 4096 from offset 49336
[2020-09-10 02:41:17.224983] INFO: regipy.recovery: Restored 4096 bytes to offset 28672 from offset 53432
[2020-09-10 02:41:17.225981] INFO: regipy.recovery: Restored 4096 bytes to offset 45056 from offset 57528
[2020-09-10 02:41:17.225981] INFO: regipy.recovery: Restored 4096 bytes to offset 221184 from offset 61624
[2020-09-10 02:41:17.225981] INFO: regipy.recovery: Restored 4096 bytes to offset 229376 from offset 65720
[2020-09-10 02:41:17.225981] INFO: regipy.recovery: Restored 4096 bytes to offset 258048 from offset 69816
[2020-09-10 02:41:17.225981] INFO: regipy.recovery: Restored 4096 bytes to offset 364544 from offset 73912
[2020-09-10 02:41:17.225981] INFO: regipy.recovery: Restored 4096 bytes to offset 380928 from offset 78008
[2020-09-10 02:41:17.225981] INFO: regipy.recovery: Restored 4096 bytes to offset 389120 from offset 82104
[2020-09-10 02:41:17.226978] INFO: regipy.recovery: Restored 4096 bytes to offset 450560 from offset 86200
[2020-09-10 02:41:17.226978] INFO: regipy.recovery: Restored 4096 bytes to offset 888832 from offset 90296
[2020-09-10 02:41:17.226978] INFO: regipy.recovery: Restored 4096 bytes to offset 913408 from offset 94392
[2020-09-10 02:41:17.226978] INFO: regipy.recovery: Restored 8192 bytes to offset 925696 from offset 98488
[2020-09-10 02:41:17.226978] INFO: regipy.recovery: Restored 4096 bytes to offset 946176 from offset 106680
[2020-09-10 02:41:17.227976] INFO: regipy.recovery: Restored 4096 bytes to offset 970752 from offset 110776
[2020-09-10 02:41:17.227976] INFO: regipy.recovery: Restored 4096 bytes to offset 983040 from offset 114872
[2020-09-10 02:41:17.227976] INFO: regipy.recovery: Restored 8192 bytes to offset 1077248 from offset 118968
[2020-09-10 02:41:17.227976] INFO: regipy.recovery: Restored 8192 bytes to offset 1097728 from offset 127160
[2020-09-10 02:41:17.227976] INFO: regipy.recovery: Parsing hvle block at 139264
[2020-09-10 02:41:17.227976] INFO: regipy.recovery: Reached a non HvLE object. stopping
[2020-09-10 02:41:17.229970] INFO: regipy.recovery: Log Size: 1101824
[2020-09-10 02:41:17.230967] INFO: regipy.recovery: Parsing hvle block at 512
[2020-09-10 02:41:17.230967] INFO: regipy.recovery: Currently at start of dirty pages: 608
[2020-09-10 02:41:17.230967] INFO: regipy.recovery: seq number: 311
[2020-09-10 02:41:17.230967] INFO: regipy.recovery: dirty pages: 7
[2020-09-10 02:41:17.231965] INFO: regipy.recovery: Restored 323584 bytes to offset 4096 from offset 608
[2020-09-10 02:41:17.231965] INFO: regipy.recovery: Restored 610304 bytes to offset 331776 from offset 324192
[2020-09-10 02:41:17.231965] INFO: regipy.recovery: Restored 4096 bytes to offset 946176 from offset 934496
[2020-09-10 02:41:17.231965] INFO: regipy.recovery: Restored 53248 bytes to offset 954368 from offset 938592
[2020-09-10 02:41:17.231965] INFO: regipy.recovery: Restored 49152 bytes to offset 1011712 from offset 991840
[2020-09-10 02:41:17.232962] INFO: regipy.recovery: Restored 20480 bytes to offset 1073152 from offset 1040992
[2020-09-10 02:41:17.232962] INFO: regipy.recovery: Restored 16384 bytes to offset 1097728 from offset 1061472
[2020-09-10 02:41:17.232962] INFO: regipy.recovery: Parsing hvle block at 1081344
[2020-09-10 02:41:17.232962] INFO: regipy.recovery: Currently at start of dirty pages: 1081416
[2020-09-10 02:41:17.232962] INFO: regipy.recovery: seq number: 312
[2020-09-10 02:41:17.232962] INFO: regipy.recovery: dirty pages: 4
[2020-09-10 02:41:17.232962] INFO: regipy.recovery: Restored 4096 bytes to offset 4096 from offset 1081416
[2020-09-10 02:41:17.232962] INFO: regipy.recovery: Restored 4096 bytes to offset 24576 from offset 1085512
[2020-09-10 02:41:17.233960] INFO: regipy.recovery: Restored 4096 bytes to offset 380928 from offset 1089608
[2020-09-10 02:41:17.233960] INFO: regipy.recovery: Restored 4096 bytes to offset 393216 from offset 1093704
Recovered 37 dirty pages. Restored hive is at Amcache.hve.regipy

Based on the log above, it seems like it had ignored the second log file - Amcache.hve.LOG2.
Files to reproduce the issue:
Amcache.hve
Amcache.hve.LOG1
Amcache.hve.LOG2
regipy output
yarp output

Regards.

system os: windows 10
python version: Python 3.6.8
code:

def main():
strfile = r'c:\reg\system'
reg = RegistryHive(strfile)

for entry in reg.recurse_subkeys(as_json=True):
    print(entry)

if name == 'main':
main()

error:
File "C:\Users\ax\PycharmProjects\regtest\venv\lib\site-packages\regipy\registry.py", line 160, in
values = [attr.asdict(x) for x in subkey.iter_values(as_json=as_json)]
File "C:\Users\ax\PycharmProjects\regtest\venv\lib\site-packages\regipy\registry.py", line 443, in iter_values
value = self.read_value(vk, substream)
File "C:\Users\ax\PycharmProjects\regtest\venv\lib\site-packages\regipy\registry.py", line 387, in read_value
data = stream.read(vk.data_size)
OverflowError: Python int too large to convert to C ssize_t

Hello!

This project looks amazing! I read all of the documentation I could find on the repo, but I'm struggling with what kind of files I need to input. I'm currently trying to open the extension-less SOFTWARE file found in C:\Windows\System32\config like so:

from regipy.registry import RegistryHive
reg = RegistryHive('C:\Windows\System32\config\SOFTWARE')

I get no error on initializing reg, but when I try to use any of the functions in the readme (get_key, get_subkeys, etc.) I get different errors related to empty lists or non-existent functions. I'm guessing that I'm giving the function invalid input, so I'm wondering is there support in this project for those types of files and if so which function should I use?

Thanks in advance!

test_requires is now deprecated per https://setuptools.readthedocs.io/en/latest/userguide/dependency_management.html#optional-dependencies

Hi,

thanks for your library, it looks very easy to use and integrate into larger python applications.
I'm facing an issue with duplicated keys yielded by the recurse_subkeys method.

example, a JSON output obtained with registry-dump -v SYSTEM -o system.json:

{
  "subkey_name": "CertStore",
  "path": "\\ControlSet001\\Control\\AppID\\CertStore",
  "timestamp": "2009-07-14T04:54:39.318948+00:00",
  "values_count": 0,
  "values": [],
  "actual_path": null
}
{
  "subkey_name": "AppID",
  "path": "\\ControlSet001\\Control\\AppID",
  "timestamp": "2009-07-14T04:54:39.318948+00:00",
  "values_count": 0,
  "values": [],
  "actual_path": null
}
{
  "subkey_name": "AppID",
  "path": "\\ControlSet001\\Control\\AppID",
  "timestamp": "2009-07-14T04:54:39.318948+00:00",
  "values_count": 0,
  "values": [],
  "actual_path": null
}

The AppID keys are exactly the same

I don't know how many keys are duplicated, but there seems to be plenty:

...
    {
      "name": "BrokenVideo",
      "value": "2805000000000000000000000000000000000000000000000000000001000000010001002800000000010000000000000000000000000000ec02000000000000",
      "value_type": "REG_RESOURCE_REQUIREMENTS_LIST",
      "is_corrupted": false
    },
    {
      "name": "PCStandard",
      "value": "0801000000000000000000000000000000000000000000000000000001000000010001000700000000010000000000000000000000000000f802000000000000",
      "value_type": "REG_RESOURCE_REQUIREMENTS_LIST",
      "is_corrupted": false
    }
  ],
  "actual_path": null
}
{
  "subkey_name": "Arbiters",
  "path": "\\ControlSet001\\Control\\Arbiters",
  "timestamp": "2009-07-14T04:49:01.365188+00:00",
  "values_count": 0,
  "values": [],
  "actual_path": null
}
{
  "subkey_name": "Arbiters",
  "path": "\\ControlSet001\\Control\\Arbiters",
  "timestamp": "2009-07-14T04:49:01.365188+00:00",
  "values_count": 0,
  "values": [],
  "actual_path": null
}
{
  "subkey_name": "FilesNotToBackup",
  "path": "\\ControlSet001\\Control\\BackupRestore\\FilesNotToBackup",
  "timestamp": "2019-10-16T02:43:11.468800+00:00",
  "values_count": 22,
  "values": [
    {
      "name": "WER",
      "value": [
...

I used the latest release available in PyPI: 1.7.0

Here is a hive you can play with to repro:
https://www.dropbox.com/s/pexash635lguuuk/SYSTEM?dl=0

Could you take a look ?

Thanks !

In

tested with the SYSTEM hive and transaction logs from from https://www.ashemery.com/dfir.html#Challenge3
Centos 8, regipy installed via pip today.

$ registry-transaction-logs -p SYSTEM.LOG1 -s SYSTEM.LOG2 -o system.merged -v SYSTEM
...
Recovered 488 dirty pages.

registry-parse-header says the Header checksum does not match.

$ registry-diff -o merged.diff SYSTEM system.merged
INFO:regipy.regdiff:Enumerating subkeys in SYSTEM
INFO:regipy.regdiff:Enumerating subkeys in system.merged
ERROR:regipy.registry:Could not parse VK at 7208934, registry hive is probably corrupted.
ERROR:regipy.registry:Could not parse VK at 7170390, registry hive is probably corrupted.
ERROR:regipy.registry:Could not parse VK at 4102, registry hive is probably corrupted.
ERROR:regipy.registry:Could not parse VK at 4103, registry hive is probably corrupted.
ERROR:regipy.registry:Could not parse VK at 2869900088, registry hive is probably corrupted.
ERROR:regipy.registry:Could not parse VK at 1176627265, registry hive is probably corrupted.
ERROR:regipy.registry:Could not parse VK at 1037319478, registry hive is probably corrupted.
ERROR:regipy.registry:Could not parse VK at 4329524, registry hive is probably corrupted.
INFO:regipy.registry:Skipping bad registry VK at 4036404501
ERROR:regipy.registry:Failed to parse hive value at path: \ControlSet001\Enum\PCI\VEN_80EE&DEV_CAFE&SUBSYS_00000000&REV_00\3&267a616a&2&20\Properties
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/regipy/registry.py", line 429, in iter_values
vk_offset = Int32ul.parse_stream(self._stream)
File "/usr/local/lib/python3.6/site-packages/construct/core.py", line 300, in parse_stream
return self._parsereport(stream, context, "(parsing)")
File "/usr/local/lib/python3.6/site-packages/construct/core.py", line 312, in _parsereport
obj = self._parse(stream, context, path)
File "/usr/local/lib/python3.6/site-packages/construct/core.py", line 1041, in _parse
data = stream_read(stream, self.length, path)
File "/usr/local/lib/python3.6/site-packages/construct/core.py", line 91, in stream_read
raise StreamError("stream read less than specified amount, expected %d, found %d" % (length, len(data)), path=path)
construct.core.StreamError: Error in path (parsing)
stream read less than specified amount, expected 4, found 0

RegistryExplorer (EricZimmerman) produces a clean hive
$ registry-diff SYSTEM SYSTEM_clean -o ez.diff
INFO:regipy.regdiff:Enumerating subkeys in SYSTEM
INFO:regipy.regdiff:Enumerating subkeys in SYSTEM_clean
Comparing SYSTEM vs SYSTEM_clean
Detected 10 differences

Hi there, thanks for this tool, I find it very useful.

The tool outputs logging to STDOUT by default. When processing stdout, info messages from logging breaks the JSON stream. Would it be possible to switch to STDINFO logging to preserve the integrity of the STDOUT data stream?

Many thanks,

Tom

The current shellbags parser does not return childless entries from the registry keys. This is somewhat of a crucial item to miss, as the last_write time of the childless keys is what's known as the "First Interacted" timestamp for the shellbag entry. Currently, the only interaction timestamps that are possible to collect with both the usrclass and ntuser shellbags parsers in regipy are "Last Interacted" (the Last Write time of the parent key for any key with an MRU position of 0).

The root cause of this seems to be checking if the value name is a digit, here: https://github.com/mkorman90/regipy/blob/master/regipy/plugins/ntuser/shellbags_ntuser.py#L222

However, removing this line will obviously throw a lot of other errors, as other values will be missing when parsing the childless entries.

For more documentaiton around the "First Interacted" timestamp, this is a great blog https://www.4n6k.com/2013/12/shellbags-forensics-addressing.html:

*Technically, we could determine the first interacted times of "Pix" and "Songs" if they were the youngest child subkeys in the branch, but they are not. Per Fig. 2, we would be able to tell the times that the "Plans" and "PDFs" folders were first interacted with because no folders inside of them were interacted with. The subkeys representing both of these folders are therefore never updated (since they contain no numbered values and their MRUListEx is blank (i.e. FF FF FF FF); the LastWrite times stay the same as when the subkeys were first created -- that is, until the MRUListEx value is put into use and there is at least one numbered value present within the subkey.

same SYSTEM registry hive as in the registry-transaction-log issue

$ registry-dump -o system.dump SYSTEM
$ grep 83da6326 system.dump | grep 2d7ae1ff
{"subkey_name":"0003","path":"\ControlSet001\Enum\USB\VID_80EE&PID_0021\5&2d7ae1ff&0&1\Properties\{83da6326-97a6-4088-9453-a1923f573b29}\0003","timestamp":"2015-12-12T02:18:32.065852+00:00","values_count":0,"values":[],"actual_path":null}
{"subkey_name":"000A","path":"\ControlSet001\Enum\USB\VID_80EE&PID_0021\5&2d7ae1ff&0&1\Properties\{83da6326-97a6-4088-9453-a1923f573b29}\000A","timestamp":"2015-12-12T02:18:31.847936+00:00","values_count":0,"values":[],"actual_path":null}
{"subkey_name":"0064","path":"\ControlSet001\Enum\USB\VID_80EE&PID_0021\5&2d7ae1ff&0&1\Properties\{83da6326-97a6-4088-9453-a1923f573b29}\0064","timestamp":"2015-12-12T02:18:32.065852+00:00","values_count":0,"values":[],"actual_path":null}
{"subkey_name":"0065","path":"\ControlSet001\Enum\USB\VID_80EE&PID_0021\5&2d7ae1ff&0&1\Properties\{83da6326-97a6-4088-9453-a1923f573b29}\0065","timestamp":"2015-12-12T02:18:32.065852+00:00","values_count":0,"values":[],"actual_path":null}
{"subkey_name":"0066","path":"\ControlSet001\Enum\USB\VID_80EE&PID_0021\5&2d7ae1ff&0&1\Properties\{83da6326-97a6-4088-9453-a1923f573b29}\0066","timestamp":"2015-12-12T03:26:35.106732+00:00","values_count":0,"values":[],"actual_path":null}
{"subkey_name":"{83da6326-97a6-4088-9453-a1923f573b29}","path":"\ControlSet001\Enum\USB\VID_80EE&PID_0021\5&2d7ae1ff&0&1\Properties\{83da6326-97a6-4088-9453-a1923f573b29}","timestamp":"2015-12-12T02:18:32.065852+00:00","values_count":0,"values":[],"actual_path":null}

When trying to access reg.get_key(r"System\ControlSet001\Control\hivelist") it raises a RegistryKeyNotFound Error. When opening the specified Hive file with Regedit.exe i can see this "hivelist" key though (HKLM\System\ControlSet001\Control\hivelist) and view it's values.

Printing all subkeys of System\ControlSet001\Control with reg.get_key(r"System\ControlSet001\Control").iter_subkeys() it look like all keys are there except "hivelist". It's just missing.

Not sure if this is a bug or actually expected behavior.

On the release page
https://github.com/mkorman90/regipy/releases/tag/2.0.1

The package download link
https://github.com/mkorman90/regipy/releases/download/2.0.1/regipy-2.0.1.tar.gz
and the source archive link
https://github.com/mkorman90/regipy/archive/refs/tags/2.0.1.tar.gz
contain different versions of the source tree.

Compare _setup_logging() definitions from utils.py:
the source archive has PR #180 applied, while the download artifact does not.

The version uploaded to PyPI also does not have the patch applied even though it's tagged as 2.0.1.

It would be really to publish a wheel on PyPI too.
I would like to retire this https://pypi.org/project/regipy2/ temporary package.

Hi,
it would pe possible use regipy as library to parse recursively all windows hive and get json of each hive?

When using the plugin "services" on the SYSTEM Hive, Python 3.7 is throwing a TypeError.

$ registry-plugins-run SYSTEM -p services -o out.json                                                                                          
Loaded 17 plugins
[2020-04-08 21:16:03.152736] INFO: regipy.plugins.system.services: Started Services enumeration Plugin...
[2020-04-08 21:16:03.154465] INFO: regipy.registry: Found control sets: ['\\ControlSet001\\Services']
Traceback (most recent call last):
  File "/usr/local/bin/registry-plugins-run", line 10, in <module>
    sys.exit(run_plugins())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/regipy/cli.py", line 107, in run_plugins
    plugin_results = run_relevant_plugins(registry_hive, as_json=True, plugins=plugins)
  File "/usr/local/lib/python3.7/site-packages/regipy/plugins/utils.py", line 21, in run_relevant_plugins
    plugin.run()
  File "/usr/local/lib/python3.7/site-packages/regipy/plugins/system/services.py", line 29, in run
    'timestamp': subkey.header.last_modified
TypeError: list indices must be integers or slices, not str

I exported some registry keys from my Windows VM using

reg export HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration C:\ICA1.reg

Then I try to parse the file using regipy.

The error I get is:

> registry-dump ICA1.reg
Traceback (most recent call last):
  File "/home/bastian/.local/bin/registry-dump", line 8, in <module>
    sys.exit(hive_to_json())
  File "/usr/lib/python3/dist-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/bastian/.local/lib/python3.10/site-packages/regipy/cli.py", line 52, in hive_to_json
    registry_hive = RegistryHive(hive_path, hive_type=hive_type, partial_hive_path=partial_hive_path)
  File "/home/bastian/.local/lib/python3.10/site-packages/regipy/registry.py", line 103, in __init__
    self.header = REGF_HEADER.parse_stream(s)
  File "/home/bastian/.local/lib/python3.10/site-packages/construct/core.py", line 300, in parse_stream
    return self._parsereport(stream, context, "(parsing)")
  File "/home/bastian/.local/lib/python3.10/site-packages/construct/core.py", line 312, in _parsereport
    obj = self._parse(stream, context, path)
  File "/home/bastian/.local/lib/python3.10/site-packages/construct/core.py", line 787, in _parse
    return self.parsefunc(stream, context)
  File "", line 103, in parseall
  File "", line 31, in parse_struct_1
  File "", line 21, in parse_const
construct.core.ConstError

Why is it failing? I take as this library cannot be used to read (text) reg-files created with regedit.

This fails for the user NTUSER.DAT hive in Win 11 image available at https://www.khyrenz.com/resources:

#!/bin/python

import sys
from regipy.registry import RegistryHive

#Getting argument = NTUSER.DAT file - successful
reg = RegistryHive(sys.argv[1])

#This correctly gets subkeys under NTUSER\Software
for sk in reg.get_key('Software').iter_subkeys():
print(sk.name)
#This fails to find any subkeys under Software (e.g. Microsoft) despite above list showing keys exist
for subk in reg.get_key('Software\'+sk.name).iter_subkeys():
print(subk.name)

Error:
Traceback (most recent call last):
File "test.py", line 9, in
for subk in reg.get_key('Software\'+sk.name).iter_subkeys():
File "/home/ubuntu/.local/lib/python3.8/site-packages/regipy/registry.py", line 228, in get_key
raise RegistryKeyNotFoundException('Did not find subkey at {}'.format(key_path))
regipy.exceptions.RegistryKeyNotFoundException: Did not find subkey at Software\7-Zip

Hi:
Logbook as a project seems like dead. Would you care for a PR to remove it and replace it with plain simpler logging?

From HIVE file NTUSER.DAT, I want to extract the content of Software\Microsoft\Windows\CurrentVersion\Explorer\UserAssist\{GUID}\Count.

All seems good, except the binary data, just 64 bytes are returned; however the complete binary string is 72 bytes.

Example:

This is what should return:

This is what returns:
9200 0000 0000 0000 0000 0000 0000 0000
9976 043d 5c5c a73d 96d2 8c3d 9550 333d
1a1a 1b3c b31a 4a3d 455d c63b b524 c93b
d598 393d caff fc3d 0800 0000 5026 b568

The code I am using that writes the retrieved data into a CSV file:

import regipy
import Rot13

hive = regipy.registry.RegistryHive(r"<path_to_hive>\NTUSER.DAT")
key = "NTUSER.DAT\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Explorer\\UserAssist"

uaf = open("userassist.csv","w",encoding="UTF-8")
uaf.write("timestamp,User,Index,Object,Encoded,binary,Type\n")

for entry in hive.recurse_subkeys(hive.get_key(key),as_json=True):
    if entry.values_count <= 1:
        continue
    GUID_Type = ""
    index_i = 0

    if entry.path == "\{CEBFF5CD-ACE2-4F4F-9178-9926F41749EA}\Count":
        GUID_Type = "Objects that have been accessed."
    elif entry.path == "\{F4E57C4B-2036-45F0-A9AB-443BCFE33D9F}\Count":
        GUID_Type = "Shortcut links used to start programs."

    for value in entry.values:
        application_encoded = value["name"]
        application_fullpath = Rot13.func(value["name"])
        binary_value = value["value"]
        uaf.write(f"timestamp,User,{index_i},{application_fullpath},{application_encoded},{binary_value},{GUID_Type}\n")
        index_i += 1

uaf.close()

Hey,

we seem to have a problem with the new get_network_info function (network data from SYSTEM hive) introduced in 3.1.6 #250

We are triggering the plugins using...

from regipy.plugins.utils import run_relevant_plugins
...
system_reg_json = run_relevant_plugins(system_reg, as_json=True)

Traceback

Traceback (most recent call last):
  File "/registry-analyzer/registry-analyzer.py", line 168, in <module>
    run_regipy_plugins()
  File "/registry-analyzer/registry-analyzer.py", line 132, in run_regipy_plugins
    system_reg_json = run_relevant_plugins(system_reg, as_json=True)
  File "/usr/local/lib/python3.9/site-packages/regipy/plugins/utils.py", line 47, in run_relevant_plugins
    plugin.run()
  File "/usr/local/lib/python3.9/site-packages/regipy/plugins/system/network_data.py", line 82, in run
    interfaces = self.get_network_info(subkey, interfaces)
  File "/usr/local/lib/python3.9/site-packages/regipy/plugins/system/network_data.py", line 61, in get_network_info
    self.get_network_info(self, interface, sub_interfaces)
TypeError: get_network_info() takes from 2 to 3 positional arguments but 4 were given

Pip Freeze

root@3798ebdd8fae:/registry-analyzer# pip freeze
attrs==23.1.0
construct==2.10.68
inflection==0.5.1
pytz==2023.3
regipy==3.1.6

I am not quite sure what the best way of handing you guys information needed is, so if you need any further information, please just ask.

Thanks for any help in advance!

Hi: you may not have (yet) pushed a wheel to PyPI for 2.3.1?
If you care I could help with a script (for instance a simple GitHub action)to automate this whenever you tag your repo?

It turns out Wine registry format differs somehow from Windows one.

I see that from inflection import underscore is used in a few places like here

The current requirements are a source of problems:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
regipy 2.0.0 requires attrs~=21.2.0, but you have attrs 21.4.0 which is incompatible.

When trying to open the key System\<CurrentControlSet>\Services\Tcpip from a Win10 SYSTEM hive it works fine. However, when trying to open the same key from a Windows Server SYSTEM hive this will fail.

In this case Services was named services, so to open the key i had to use System\<CurrentControlSet>\services\Tcpip.

I would expect services to match Services and vice versa. My workaround so far is to check for both keys with try-catch but that seems dirty and i might miss alot of cases. Maybe there is an alternative?

for entry in reg.recurse_subkeys(as_json=True):
... print(entry)
...
Subkey(subkey_name='1024', path='\Apple Computer, Inc.\iPod\RegisteredApps\1024', timestamp='2020-07-13T12:11:12.411520+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='RegisteredApps', path='\Apple Computer, Inc.\iPod\RegisteredApps', timestamp='2020-07-13T12:19:31.697374+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='RegisteredApps', path='\Apple Computer, Inc.\iPod\RegisteredApps', timestamp='2020-07-13T12:19:31.697374+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='iPod', path='\Apple Computer, Inc.\iPod', timestamp='2020-07-13T12:19:31.735122+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='iPod', path='\Apple Computer, Inc.\iPod', timestamp='2020-07-13T12:19:31.735122+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='Apple Computer, Inc.', path='\Apple Computer, Inc.', timestamp='2020-07-13T12:19:31.697374+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='Apple Computer, Inc.', path='\Apple Computer, Inc.', timestamp='2020-07-13T12:19:31.697374+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='Apple Application Support', path='\Apple Inc.\Apple Application Support', timestamp='2020-07-13T12:23:08.363750+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='Apple Inc.', path='\Apple Inc.', timestamp='2020-07-13T12:27:09.542870+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='Apple Inc.', path='\Apple Inc.', timestamp='2020-07-13T12:27:09.542870+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='BlueStacksInstaller', path='\BlueStacksInstaller', timestamp='2020-03-20T00:10:16.109034+00:00', values_count=2, values=[{'name': 'MachineID', 'value': '2d599309-ecac-4f72-a2fc-a3e20b5cba31', 'value_type': 'REG_SZ', 'is_corrupted': False}, {'name': 'VersionMachineId_4.180.10.9302', 'value': '023aa6fc-0b35-4bbb-89c6-4eafa60f5d11', 'value_type': 'REG_SZ', 'is_corrupted': False}], actual_path=None)
Subkey(subkey_name='Excel.exe', path='\Classes\\OpenWithList\Excel.exe', timestamp='2019-03-19T04:55:56.532884+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='IExplore.exe', path='\Classes\\OpenWithList\IExplore.exe', timestamp='2019-03-19T04:55:56.532884+00:00', values_count=0, values=[], actual_path=None)
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Gebruiker\AppData\Local\Programs\Python\Python38-32\lib\site-packages\regipy\registry.py", line 147, in recurse_subkeys
yield from self.recurse_subkeys(nk_record=subkey,
File "C:\Users\Gebruiker\AppData\Local\Programs\Python\Python38-32\lib\site-packages\regipy\registry.py", line 147, in recurse_subkeys
yield from self.recurse_subkeys(nk_record=subkey,
File "C:\Users\Gebruiker\AppData\Local\Programs\Python\Python38-32\lib\site-packages\regipy\registry.py", line 147, in recurse_subkeys
yield from self.recurse_subkeys(nk_record=subkey,
File "C:\Users\Gebruiker\AppData\Local\Programs\Python\Python38-32\lib\site-packages\regipy\registry.py", line 154, in recurse_subkeys
values = [attr.asdict(x) for x in subkey.iter_values(as_json=as_json)]
File "C:\Users\Gebruiker\AppData\Local\Programs\Python\Python38-32\lib\site-packages\regipy\registry.py", line 154, in
values = [attr.asdict(x) for x in subkey.iter_values(as_json=as_json)]
File "C:\Users\Gebruiker\AppData\Local\Programs\Python\Python38-32\lib\site-packages\regipy\registry.py", line 416, in iter_values
value = self.read_value(vk, substream)
File "C:\Users\Gebruiker\AppData\Local\Programs\Python\Python38-32\lib\site-packages\regipy\registry.py", line 365, in read_value
data = stream.read(vk.data_size)
OverflowError: cannot fit 'int' into an index-sized integer

Hello,
when i try to use recurse_subkeys function on particularly path instead of whole hive. Instead of getting data from part of registry I get whole hive listed with path joined in returned data. Like shown bellow:

for entry in reg.recurse_subkeys(path=r"System\ControlSet001\Services", as_json=True):
print(entry.path)

returned data:
System\ControlSet001\Services\ControlSet001\Control\ACPI
System\ControlSet001\Services\ControlSet001\Control\AGP
System\ControlSet001\Services\ControlSet001\Control\AppID\CertChainStore
System\ControlSet001\Services\ControlSet001\Control\AppID\CertStore

I might be doing something horribly wrong but here it goes.

I'm trying to follow the readme with the following code:

from regipy.registry import RegistryHive
reg = RegistryHive('SYSTEM')
for sk in reg.get_key(r'SYSTEM\ControlSet001').get_subkeys():
print(sk.name, convert_wintime(sk.header.last_modified).isoformat())

And I get the following error
AttributeError: 'NKRecord' object has no attribute 'get_subkeys'

I've tried replacing ControlSet001 with SYSTEM\ControlSet001, same deal.

Following command works in Windows PowerShell and Windows Terminal:
registry-parse-header HIVE_PATH

The same command is not working in PowerShell ISE and Visual Studio Code:
registry-parse-header : Traceback (most recent call last):
In Zeile:1 Zeichen:1

• registry-parse-header "C:\Users\evild3ad\Desktop\Artifacts-Collector\ ...

•   + CategoryInfo          : NotSpecified: (Traceback (most recent call last)::String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
  
  File "C:\Program Files\Python37\Scripts\registry-parse-header-script.py", line 11, in <module>
    load_entry_point('regipy==1.6.3', 'console_scripts', 'registry-parse-header')()
  File "C:\Program Files\Python37\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Program Files\Python37\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Program Files\Python37\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Program Files\Python37\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Program Files\Python37\lib\site-packages\regipy\cli.py", line 30, in parse_header
    click.secho(tabulate(registry_hive.header.items(), tablefmt='fancy_grid'))
  File "C:\Program Files\Python37\lib\site-packages\click\termui.py", line 548, in secho
    return echo(message, file=file, nl=nl, err=err, color=color)
  File "C:\Program Files\Python37\lib\site-packages\click\utils.py", line 272, in echo
    file.write(message)
  File "C:\Program Files\Python37\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
  

UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-59: character maps to

I tried to fix the encoding issue in Visual Studio Code without success. Any tips for me?

My file encoding is UTF-8 with BOM.

Thank you!

Hi:
We are adding regipy to https://github.com/nexB/scancode-toolkit and use it as a library there.
Would you be open to have a lightweight wheel version with fewer deps for use as a library?
That would be super useful!

tqdm is awesome, but its usage is minimal here and may be could be replaced by the Click built-in progress bar to have overall fewer deps.