mkorman90 / regipy Goto Github PK

When trying to access reg.get_key(r"System\ControlSet001\Control\hivelist") it raises a RegistryKeyNotFound Error. When opening the specified Hive file with Regedit.exe i can see this "hivelist" key though (HKLM\System\ControlSet001\Control\hivelist) and view it's values.

Printing all subkeys of System\ControlSet001\Control with reg.get_key(r"System\ControlSet001\Control").iter_subkeys() it look like all keys are there except "hivelist". It's just missing.

Not sure if this is a bug or actually expected behavior.

for entry in reg.recurse_subkeys(as_json=True):
... print(entry)
...
Subkey(subkey_name='1024', path='\Apple Computer, Inc.\iPod\RegisteredApps\1024', timestamp='2020-07-13T12:11:12.411520+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='RegisteredApps', path='\Apple Computer, Inc.\iPod\RegisteredApps', timestamp='2020-07-13T12:19:31.697374+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='RegisteredApps', path='\Apple Computer, Inc.\iPod\RegisteredApps', timestamp='2020-07-13T12:19:31.697374+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='iPod', path='\Apple Computer, Inc.\iPod', timestamp='2020-07-13T12:19:31.735122+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='iPod', path='\Apple Computer, Inc.\iPod', timestamp='2020-07-13T12:19:31.735122+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='Apple Computer, Inc.', path='\Apple Computer, Inc.', timestamp='2020-07-13T12:19:31.697374+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='Apple Computer, Inc.', path='\Apple Computer, Inc.', timestamp='2020-07-13T12:19:31.697374+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='Apple Application Support', path='\Apple Inc.\Apple Application Support', timestamp='2020-07-13T12:23:08.363750+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='Apple Inc.', path='\Apple Inc.', timestamp='2020-07-13T12:27:09.542870+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='Apple Inc.', path='\Apple Inc.', timestamp='2020-07-13T12:27:09.542870+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='BlueStacksInstaller', path='\BlueStacksInstaller', timestamp='2020-03-20T00:10:16.109034+00:00', values_count=2, values=[{'name': 'MachineID', 'value': '2d599309-ecac-4f72-a2fc-a3e20b5cba31', 'value_type': 'REG_SZ', 'is_corrupted': False}, {'name': 'VersionMachineId_4.180.10.9302', 'value': '023aa6fc-0b35-4bbb-89c6-4eafa60f5d11', 'value_type': 'REG_SZ', 'is_corrupted': False}], actual_path=None)
Subkey(subkey_name='Excel.exe', path='\Classes\\OpenWithList\Excel.exe', timestamp='2019-03-19T04:55:56.532884+00:00', values_count=0, values=[], actual_path=None)
Subkey(subkey_name='IExplore.exe', path='\Classes\\OpenWithList\IExplore.exe', timestamp='2019-03-19T04:55:56.532884+00:00', values_count=0, values=[], actual_path=None)
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Gebruiker\AppData\Local\Programs\Python\Python38-32\lib\site-packages\regipy\registry.py", line 147, in recurse_subkeys
yield from self.recurse_subkeys(nk_record=subkey,
File "C:\Users\Gebruiker\AppData\Local\Programs\Python\Python38-32\lib\site-packages\regipy\registry.py", line 147, in recurse_subkeys
yield from self.recurse_subkeys(nk_record=subkey,
File "C:\Users\Gebruiker\AppData\Local\Programs\Python\Python38-32\lib\site-packages\regipy\registry.py", line 147, in recurse_subkeys
yield from self.recurse_subkeys(nk_record=subkey,
File "C:\Users\Gebruiker\AppData\Local\Programs\Python\Python38-32\lib\site-packages\regipy\registry.py", line 154, in recurse_subkeys
values = [attr.asdict(x) for x in subkey.iter_values(as_json=as_json)]
File "C:\Users\Gebruiker\AppData\Local\Programs\Python\Python38-32\lib\site-packages\regipy\registry.py", line 154, in
values = [attr.asdict(x) for x in subkey.iter_values(as_json=as_json)]
File "C:\Users\Gebruiker\AppData\Local\Programs\Python\Python38-32\lib\site-packages\regipy\registry.py", line 416, in iter_values
value = self.read_value(vk, substream)
File "C:\Users\Gebruiker\AppData\Local\Programs\Python\Python38-32\lib\site-packages\regipy\registry.py", line 365, in read_value
data = stream.read(vk.data_size)
OverflowError: cannot fit 'int' into an index-sized integer

same SYSTEM registry hive as in the registry-transaction-log issue

$ registry-dump -o system.dump SYSTEM
$ grep 83da6326 system.dump | grep 2d7ae1ff
{"subkey_name":"0003","path":"\ControlSet001\Enum\USB\VID_80EE&PID_0021\5&2d7ae1ff&0&1\Properties\{83da6326-97a6-4088-9453-a1923f573b29}\0003","timestamp":"2015-12-12T02:18:32.065852+00:00","values_count":0,"values":[],"actual_path":null}
{"subkey_name":"000A","path":"\ControlSet001\Enum\USB\VID_80EE&PID_0021\5&2d7ae1ff&0&1\Properties\{83da6326-97a6-4088-9453-a1923f573b29}\000A","timestamp":"2015-12-12T02:18:31.847936+00:00","values_count":0,"values":[],"actual_path":null}
{"subkey_name":"0064","path":"\ControlSet001\Enum\USB\VID_80EE&PID_0021\5&2d7ae1ff&0&1\Properties\{83da6326-97a6-4088-9453-a1923f573b29}\0064","timestamp":"2015-12-12T02:18:32.065852+00:00","values_count":0,"values":[],"actual_path":null}
{"subkey_name":"0065","path":"\ControlSet001\Enum\USB\VID_80EE&PID_0021\5&2d7ae1ff&0&1\Properties\{83da6326-97a6-4088-9453-a1923f573b29}\0065","timestamp":"2015-12-12T02:18:32.065852+00:00","values_count":0,"values":[],"actual_path":null}
{"subkey_name":"0066","path":"\ControlSet001\Enum\USB\VID_80EE&PID_0021\5&2d7ae1ff&0&1\Properties\{83da6326-97a6-4088-9453-a1923f573b29}\0066","timestamp":"2015-12-12T03:26:35.106732+00:00","values_count":0,"values":[],"actual_path":null}
{"subkey_name":"{83da6326-97a6-4088-9453-a1923f573b29}","path":"\ControlSet001\Enum\USB\VID_80EE&PID_0021\5&2d7ae1ff&0&1\Properties\{83da6326-97a6-4088-9453-a1923f573b29}","timestamp":"2015-12-12T02:18:32.065852+00:00","values_count":0,"values":[],"actual_path":null}

Hi: you may not have (yet) pushed a wheel to PyPI for 2.3.1?
If you care I could help with a script (for instance a simple GitHub action)to automate this whenever you tag your repo?

both with using: registry-dump.exe SAM.hiv -o test.json and Python [for entry in reg.recurse_subkeys(as_json=True):]

always see like this

Hi:
We are adding regipy to https://github.com/nexB/scancode-toolkit and use it as a library there.
Would you be open to have a lightweight wheel version with fewer deps for use as a library?
That would be super useful!

On the release page
https://github.com/mkorman90/regipy/releases/tag/2.0.1

The package download link
https://github.com/mkorman90/regipy/releases/download/2.0.1/regipy-2.0.1.tar.gz
and the source archive link
https://github.com/mkorman90/regipy/archive/refs/tags/2.0.1.tar.gz
contain different versions of the source tree.

Compare _setup_logging() definitions from utils.py:
the source archive has PR #180 applied, while the download artifact does not.

The version uploaded to PyPI also does not have the patch applied even though it's tagged as 2.0.1.

Hello everybody,

I am trying to use regipy in a script within a Docker container. As soon as my script tries to import the RegistryHive, the following error gets thrown:

Traceback (most recent call last):
  File "/my_project/./Registry/registry.py", line 7, in <module>
    from regipy.registry import RegistryHive
  File "/usr/local/lib/python3.9/site-packages/regipy/__init__.py", line 1, in <module>
    from .registry import *
  File "/usr/local/lib/python3.9/site-packages/regipy/registry.py", line 16, in <module>
    from regipy.structs import REGF_HEADER, HBIN_HEADER, CM_KEY_NODE, LF_LH_SK_ELEMENT, VALUE_KEY, INDEX_ROOT, \
  File "/usr/local/lib/python3.9/site-packages/regipy/structs.py", line 3, in <module>
    REGF_HEADER = Struct(
  File "/usr/local/lib/python3.9/site-packages/construct/core.py", line 444, in compile
    module = importlib.util.module_from_spec(module_spec)
AttributeError: module 'importlib' has no attribute 'util'

It seems like core.py is importing importlib.util the wrong way (based on other articles with similar problems).

I use the following line within my script to import the RegistryHive

from regipy.registry import RegistryHive

The important lines of my Dockerfile are the following:

FROM python:3.9-slim-buster
....
COPY my_project/ .
...
RUN pip3 install -r requirements.txt
...
ENTRYPOINT ["python3", "my_project.py"]

The requirements.txt includes a line "regipy", which installs version 2.0.1 of regipy. I already tried to use older version of python (3.8, 3.7, 3.6), which resulted in the same error.

I am aware that this is not directly an issue of Regipy, but I would appreciate any help/input I can get here.

Thanks in advance and Best Regards
Moe

Hey,

we seem to have a problem with the new get_network_info function (network data from SYSTEM hive) introduced in 3.1.6 #250

We are triggering the plugins using...

from regipy.plugins.utils import run_relevant_plugins
...
system_reg_json = run_relevant_plugins(system_reg, as_json=True)

Traceback

Traceback (most recent call last):
  File "/registry-analyzer/registry-analyzer.py", line 168, in <module>
    run_regipy_plugins()
  File "/registry-analyzer/registry-analyzer.py", line 132, in run_regipy_plugins
    system_reg_json = run_relevant_plugins(system_reg, as_json=True)
  File "/usr/local/lib/python3.9/site-packages/regipy/plugins/utils.py", line 47, in run_relevant_plugins
    plugin.run()
  File "/usr/local/lib/python3.9/site-packages/regipy/plugins/system/network_data.py", line 82, in run
    interfaces = self.get_network_info(subkey, interfaces)
  File "/usr/local/lib/python3.9/site-packages/regipy/plugins/system/network_data.py", line 61, in get_network_info
    self.get_network_info(self, interface, sub_interfaces)
TypeError: get_network_info() takes from 2 to 3 positional arguments but 4 were given

Pip Freeze

root@3798ebdd8fae:/registry-analyzer# pip freeze
attrs==23.1.0
construct==2.10.68
inflection==0.5.1
pytz==2023.3
regipy==3.1.6

I am not quite sure what the best way of handing you guys information needed is, so if you need any further information, please just ask.

Thanks for any help in advance!

Hi,

thanks for your library, it looks very easy to use and integrate into larger python applications.
I'm facing an issue with duplicated keys yielded by the recurse_subkeys method.

example, a JSON output obtained with registry-dump -v SYSTEM -o system.json:

{
  "subkey_name": "CertStore",
  "path": "\\ControlSet001\\Control\\AppID\\CertStore",
  "timestamp": "2009-07-14T04:54:39.318948+00:00",
  "values_count": 0,
  "values": [],
  "actual_path": null
}
{
  "subkey_name": "AppID",
  "path": "\\ControlSet001\\Control\\AppID",
  "timestamp": "2009-07-14T04:54:39.318948+00:00",
  "values_count": 0,
  "values": [],
  "actual_path": null
}
{
  "subkey_name": "AppID",
  "path": "\\ControlSet001\\Control\\AppID",
  "timestamp": "2009-07-14T04:54:39.318948+00:00",
  "values_count": 0,
  "values": [],
  "actual_path": null
}

The AppID keys are exactly the same

I don't know how many keys are duplicated, but there seems to be plenty:

...
    {
      "name": "BrokenVideo",
      "value": "2805000000000000000000000000000000000000000000000000000001000000010001002800000000010000000000000000000000000000ec02000000000000",
      "value_type": "REG_RESOURCE_REQUIREMENTS_LIST",
      "is_corrupted": false
    },
    {
      "name": "PCStandard",
      "value": "0801000000000000000000000000000000000000000000000000000001000000010001000700000000010000000000000000000000000000f802000000000000",
      "value_type": "REG_RESOURCE_REQUIREMENTS_LIST",
      "is_corrupted": false
    }
  ],
  "actual_path": null
}
{
  "subkey_name": "Arbiters",
  "path": "\\ControlSet001\\Control\\Arbiters",
  "timestamp": "2009-07-14T04:49:01.365188+00:00",
  "values_count": 0,
  "values": [],
  "actual_path": null
}
{
  "subkey_name": "Arbiters",
  "path": "\\ControlSet001\\Control\\Arbiters",
  "timestamp": "2009-07-14T04:49:01.365188+00:00",
  "values_count": 0,
  "values": [],
  "actual_path": null
}
{
  "subkey_name": "FilesNotToBackup",
  "path": "\\ControlSet001\\Control\\BackupRestore\\FilesNotToBackup",
  "timestamp": "2019-10-16T02:43:11.468800+00:00",
  "values_count": 22,
  "values": [
    {
      "name": "WER",
      "value": [
...

I used the latest release available in PyPI: 1.7.0

Here is a hive you can play with to repro:
https://www.dropbox.com/s/pexash635lguuuk/SYSTEM?dl=0

Could you take a look ?

Thanks !

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

From HIVE file NTUSER.DAT, I want to extract the content of Software\Microsoft\Windows\CurrentVersion\Explorer\UserAssist\{GUID}\Count.

All seems good, except the binary data, just 64 bytes are returned; however the complete binary string is 72 bytes.

Example:

This is what should return:

This is what returns:
9200 0000 0000 0000 0000 0000 0000 0000
9976 043d 5c5c a73d 96d2 8c3d 9550 333d
1a1a 1b3c b31a 4a3d 455d c63b b524 c93b
d598 393d caff fc3d 0800 0000 5026 b568

The code I am using that writes the retrieved data into a CSV file:

import regipy
import Rot13

hive = regipy.registry.RegistryHive(r"<path_to_hive>\NTUSER.DAT")
key = "NTUSER.DAT\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Explorer\\UserAssist"

uaf = open("userassist.csv","w",encoding="UTF-8")
uaf.write("timestamp,User,Index,Object,Encoded,binary,Type\n")

for entry in hive.recurse_subkeys(hive.get_key(key),as_json=True):
    if entry.values_count <= 1:
        continue
    GUID_Type = ""
    index_i = 0

    if entry.path == "\{CEBFF5CD-ACE2-4F4F-9178-9926F41749EA}\Count":
        GUID_Type = "Objects that have been accessed."
    elif entry.path == "\{F4E57C4B-2036-45F0-A9AB-443BCFE33D9F}\Count":
        GUID_Type = "Shortcut links used to start programs."

    for value in entry.values:
        application_encoded = value["name"]
        application_fullpath = Rot13.func(value["name"])
        binary_value = value["value"]
        uaf.write(f"timestamp,User,{index_i},{application_fullpath},{application_encoded},{binary_value},{GUID_Type}\n")
        index_i += 1

uaf.close()

tqdm is awesome, but its usage is minimal here and may be could be replaced by the Click built-in progress bar to have overall fewer deps.

I might be doing something horribly wrong but here it goes.

I'm trying to follow the readme with the following code:

from regipy.registry import RegistryHive
reg = RegistryHive('SYSTEM')
for sk in reg.get_key(r'SYSTEM\ControlSet001').get_subkeys():
print(sk.name, convert_wintime(sk.header.last_modified).isoformat())

And I get the following error
AttributeError: 'NKRecord' object has no attribute 'get_subkeys'

I've tried replacing ControlSet001 with SYSTEM\ControlSet001, same deal.

It appears that try_decode_binary applies the max_len to the data regardless off whether as_json is set or not. This causes data to be truncated when using something like get_values with as_json=False

test_requires is now deprecated per https://setuptools.readthedocs.io/en/latest/userguide/dependency_management.html#optional-dependencies

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

Hello!

This project looks amazing! I read all of the documentation I could find on the repo, but I'm struggling with what kind of files I need to input. I'm currently trying to open the extension-less SOFTWARE file found in C:\Windows\System32\config like so:

from regipy.registry import RegistryHive
reg = RegistryHive('C:\Windows\System32\config\SOFTWARE')

I get no error on initializing reg, but when I try to use any of the functions in the readme (get_key, get_subkeys, etc.) I get different errors related to empty lists or non-existent functions. I'm guessing that I'm giving the function invalid input, so I'm wondering is there support in this project for those types of files and if so which function should I use?

Thanks in advance!

The current requirements are a source of problems:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
regipy 2.0.0 requires attrs~=21.2.0, but you have attrs 21.4.0 which is incompatible.

When trying to open the key System\<CurrentControlSet>\Services\Tcpip from a Win10 SYSTEM hive it works fine. However, when trying to open the same key from a Windows Server SYSTEM hive this will fail.

In this case Services was named services, so to open the key i had to use System\<CurrentControlSet>\services\Tcpip.

I would expect services to match Services and vice versa. My workaround so far is to check for both keys with try-catch but that seems dirty and i might miss alot of cases. Maybe there is an alternative?

I exported some registry keys from my Windows VM using

reg export HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration C:\ICA1.reg

Then I try to parse the file using regipy.

The error I get is:

> registry-dump ICA1.reg
Traceback (most recent call last):
  File "/home/bastian/.local/bin/registry-dump", line 8, in <module>
    sys.exit(hive_to_json())
  File "/usr/lib/python3/dist-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/bastian/.local/lib/python3.10/site-packages/regipy/cli.py", line 52, in hive_to_json
    registry_hive = RegistryHive(hive_path, hive_type=hive_type, partial_hive_path=partial_hive_path)
  File "/home/bastian/.local/lib/python3.10/site-packages/regipy/registry.py", line 103, in __init__
    self.header = REGF_HEADER.parse_stream(s)
  File "/home/bastian/.local/lib/python3.10/site-packages/construct/core.py", line 300, in parse_stream
    return self._parsereport(stream, context, "(parsing)")
  File "/home/bastian/.local/lib/python3.10/site-packages/construct/core.py", line 312, in _parsereport
    obj = self._parse(stream, context, path)
  File "/home/bastian/.local/lib/python3.10/site-packages/construct/core.py", line 787, in _parse
    return self.parsefunc(stream, context)
  File "", line 103, in parseall
  File "", line 31, in parse_struct_1
  File "", line 21, in parse_const
construct.core.ConstError

Why is it failing? I take as this library cannot be used to read (text) reg-files created with regedit.

It is proposed to solve this by replacing registry- prefix witn regipy- one

Hi,
it would pe possible use regipy as library to parse recursively all windows hive and get json of each hive?

It would be really to publish a wheel on PyPI too.
I would like to retire this https://pypi.org/project/regipy2/ temporary package.

Hi:
Logbook as a project seems like dead. Would you care for a PR to remove it and replace it with plain simpler logging?

Using the tool, I see that the returned data size of a key is always no more than 128 characters, even with keys that longer than this.

I assured that the length is more than 128 both with the windows registry viewer and in hivexsh.

Is there an explanation for this?

When using the plugin "services" on the SYSTEM Hive, Python 3.7 is throwing a TypeError.

$ registry-plugins-run SYSTEM -p services -o out.json                                                                                          
Loaded 17 plugins
[2020-04-08 21:16:03.152736] INFO: regipy.plugins.system.services: Started Services enumeration Plugin...
[2020-04-08 21:16:03.154465] INFO: regipy.registry: Found control sets: ['\\ControlSet001\\Services']
Traceback (most recent call last):
  File "/usr/local/bin/registry-plugins-run", line 10, in <module>
    sys.exit(run_plugins())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/regipy/cli.py", line 107, in run_plugins
    plugin_results = run_relevant_plugins(registry_hive, as_json=True, plugins=plugins)
  File "/usr/local/lib/python3.7/site-packages/regipy/plugins/utils.py", line 21, in run_relevant_plugins
    plugin.run()
  File "/usr/local/lib/python3.7/site-packages/regipy/plugins/system/services.py", line 29, in run
    'timestamp': subkey.header.last_modified
TypeError: list indices must be integers or slices, not str

It turns out Wine registry format differs somehow from Windows one.

I see that from inflection import underscore is used in a few places like here

system os: windows 10
python version: Python 3.6.8
code:

def main():
strfile = r'c:\reg\system'
reg = RegistryHive(strfile)

for entry in reg.recurse_subkeys(as_json=True):
    print(entry)

if name == 'main':
main()

error:
File "C:\Users\ax\PycharmProjects\regtest\venv\lib\site-packages\regipy\registry.py", line 160, in
values = [attr.asdict(x) for x in subkey.iter_values(as_json=as_json)]
File "C:\Users\ax\PycharmProjects\regtest\venv\lib\site-packages\regipy\registry.py", line 443, in iter_values
value = self.read_value(vk, substream)
File "C:\Users\ax\PycharmProjects\regtest\venv\lib\site-packages\regipy\registry.py", line 387, in read_value
data = stream.read(vk.data_size)
OverflowError: Python int too large to convert to C ssize_t

The latest tag is 1.9.0 from March.
All the releases since are missing a tag: 2.0.0, 1.9.4, 1.9.3, 1.9.2, 1.9.1

Following command works in Windows PowerShell and Windows Terminal:
registry-parse-header HIVE_PATH

The same command is not working in PowerShell ISE and Visual Studio Code:
registry-parse-header : Traceback (most recent call last):
In Zeile:1 Zeichen:1

• registry-parse-header "C:\Users\evild3ad\Desktop\Artifacts-Collector\ ...

•   + CategoryInfo          : NotSpecified: (Traceback (most recent call last)::String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
  
  File "C:\Program Files\Python37\Scripts\registry-parse-header-script.py", line 11, in <module>
    load_entry_point('regipy==1.6.3', 'console_scripts', 'registry-parse-header')()
  File "C:\Program Files\Python37\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Program Files\Python37\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Program Files\Python37\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Program Files\Python37\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Program Files\Python37\lib\site-packages\regipy\cli.py", line 30, in parse_header
    click.secho(tabulate(registry_hive.header.items(), tablefmt='fancy_grid'))
  File "C:\Program Files\Python37\lib\site-packages\click\termui.py", line 548, in secho
    return echo(message, file=file, nl=nl, err=err, color=color)
  File "C:\Program Files\Python37\lib\site-packages\click\utils.py", line 272, in echo
    file.write(message)
  File "C:\Program Files\Python37\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
  

UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-59: character maps to

I tried to fix the encoding issue in Visual Studio Code without success. Any tips for me?

My file encoding is UTF-8 with BOM.

Thank you!

Hello,
when i try to use recurse_subkeys function on particularly path instead of whole hive. Instead of getting data from part of registry I get whole hive listed with path joined in returned data. Like shown bellow:

for entry in reg.recurse_subkeys(path=r"System\ControlSet001\Services", as_json=True):
print(entry.path)

returned data:
System\ControlSet001\Services\ControlSet001\Control\ACPI
System\ControlSet001\Services\ControlSet001\Control\AGP
System\ControlSet001\Services\ControlSet001\Control\AppID\CertChainStore
System\ControlSet001\Services\ControlSet001\Control\AppID\CertStore

In

This issue provides visibility into Renovate updates and their statuses. Learn more

Edited/Blocked

These updates have been manually edited so Renovate will no longer make changes. To discard all commits and start over, click on a checkbox.

• Update dependency pytest to v7.1.2

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update actions/setup-python action to v4
• Update python orb to v2

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• Update dependency click to v8.1.3

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

In the registry.py file on line 114 I receved the following exception when processing a userclass.dat registry file.

Traceback (most recent call last):
  File "C:\Program Files\Python37\lib\site-packages\regipy\registry.py" line 199, in dump_hive_to_json
    for entry in tqdm(self.recurse_subkeys(name_key_entry, as_json=True), disable=not verbose)
  File "C:\Program Files\Python37\lib\site-packages\tqdm\_tqdm.py" line 1022, in __iter__
    for obj in iterable:
  File "C:\Program Files\Python37\lib\site-packages\regipy\registry.py" line 119, in recurse_subkeys
    yield from self.recurse_subkeys(nk_record=subkey, path=r'\{}'.format(subkey.name), as_json=as_json)
  File "C:\Program Files\Python37\lib\site-packages\regipy\registry.py" line 117, in recurse_subkeys
    as_json=as_json)
  File "C:\Program Files\Python37\lib\site-packages\regipy\registry.py" line 114, in recurse_subkeys
    if subkey.subkey_count:
AttributeError: 'LIRecord' object has no attribute 'subkey_count'

Hi there, thanks for this tool, I find it very useful.

The tool outputs logging to STDOUT by default. When processing stdout, info messages from logging breaks the JSON stream. Would it be possible to switch to STDINFO logging to preserve the integrity of the STDOUT data stream?

Many thanks,

Tom

tested with the SYSTEM hive and transaction logs from from https://www.ashemery.com/dfir.html#Challenge3
Centos 8, regipy installed via pip today.

$ registry-transaction-logs -p SYSTEM.LOG1 -s SYSTEM.LOG2 -o system.merged -v SYSTEM
...
Recovered 488 dirty pages.

registry-parse-header says the Header checksum does not match.

$ registry-diff -o merged.diff SYSTEM system.merged
INFO:regipy.regdiff:Enumerating subkeys in SYSTEM
INFO:regipy.regdiff:Enumerating subkeys in system.merged
ERROR:regipy.registry:Could not parse VK at 7208934, registry hive is probably corrupted.
ERROR:regipy.registry:Could not parse VK at 7170390, registry hive is probably corrupted.
ERROR:regipy.registry:Could not parse VK at 4102, registry hive is probably corrupted.
ERROR:regipy.registry:Could not parse VK at 4103, registry hive is probably corrupted.
ERROR:regipy.registry:Could not parse VK at 2869900088, registry hive is probably corrupted.
ERROR:regipy.registry:Could not parse VK at 1176627265, registry hive is probably corrupted.
ERROR:regipy.registry:Could not parse VK at 1037319478, registry hive is probably corrupted.
ERROR:regipy.registry:Could not parse VK at 4329524, registry hive is probably corrupted.
INFO:regipy.registry:Skipping bad registry VK at 4036404501
ERROR:regipy.registry:Failed to parse hive value at path: \ControlSet001\Enum\PCI\VEN_80EE&DEV_CAFE&SUBSYS_00000000&REV_00\3&267a616a&2&20\Properties
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/regipy/registry.py", line 429, in iter_values
vk_offset = Int32ul.parse_stream(self._stream)
File "/usr/local/lib/python3.6/site-packages/construct/core.py", line 300, in parse_stream
return self._parsereport(stream, context, "(parsing)")
File "/usr/local/lib/python3.6/site-packages/construct/core.py", line 312, in _parsereport
obj = self._parse(stream, context, path)
File "/usr/local/lib/python3.6/site-packages/construct/core.py", line 1041, in _parse
data = stream_read(stream, self.length, path)
File "/usr/local/lib/python3.6/site-packages/construct/core.py", line 91, in stream_read
raise StreamError("stream read less than specified amount, expected %d, found %d" % (length, len(data)), path=path)
construct.core.StreamError: Error in path (parsing)
stream read less than specified amount, expected 4, found 0

RegistryExplorer (EricZimmerman) produces a clean hive
$ registry-diff SYSTEM SYSTEM_clean -o ez.diff
INFO:regipy.regdiff:Enumerating subkeys in SYSTEM
INFO:regipy.regdiff:Enumerating subkeys in SYSTEM_clean
Comparing SYSTEM vs SYSTEM_clean
Detected 10 differences

Hi,

Pls see following for an example of a hive whose transactions were not successfully recovered:

registry-transaction-logs Amcache.hve -p Amcache.hve.LOG1 -s Amcache.hve.LOG2 -o Amcache.hve.regipy
[2020-09-10 02:41:17.219000] INFO: regipy.cli: Processing hive Amcache.hve with transaction log Amcache.hve.LOG1
[2020-09-10 02:41:17.219997] INFO: regipy.cli: Processing hive Amcache.hve with secondary transaction log Amcache.hve.LOG1
[2020-09-10 02:41:17.220994] INFO: regipy.recovery: Log Size: 196608
[2020-09-10 02:41:17.220994] INFO: regipy.recovery: Log Size: 196608
[2020-09-10 02:41:17.221991] INFO: regipy.recovery: Parsing hvle block at 512
[2020-09-10 02:41:17.221991] INFO: regipy.recovery: Currently at start of dirty pages: 616
[2020-09-10 02:41:17.222989] INFO: regipy.recovery: seq number: 313
[2020-09-10 02:41:17.222989] INFO: regipy.recovery: dirty pages: 8
[2020-09-10 02:41:17.222989] INFO: regipy.recovery: Restored 4096 bytes to offset 4096 from offset 616
[2020-09-10 02:41:17.222989] INFO: regipy.recovery: Restored 8192 bytes to offset 339968 from offset 4712
[2020-09-10 02:41:17.222989] INFO: regipy.recovery: Restored 8192 bytes to offset 364544 from offset 12904
[2020-09-10 02:41:17.222989] INFO: regipy.recovery: Restored 8192 bytes to offset 380928 from offset 21096
[2020-09-10 02:41:17.223987] INFO: regipy.recovery: Restored 4096 bytes to offset 946176 from offset 29288
[2020-09-10 02:41:17.223987] INFO: regipy.recovery: Restored 4096 bytes to offset 970752 from offset 33384
[2020-09-10 02:41:17.223987] INFO: regipy.recovery: Restored 4096 bytes to offset 1056768 from offset 37480
[2020-09-10 02:41:17.223987] INFO: regipy.recovery: Restored 4096 bytes to offset 1097728 from offset 41576
[2020-09-10 02:41:17.223987] INFO: regipy.recovery: Parsing hvle block at 49152
[2020-09-10 02:41:17.224983] INFO: regipy.recovery: Currently at start of dirty pages: 49336
[2020-09-10 02:41:17.224983] INFO: regipy.recovery: seq number: 314
[2020-09-10 02:41:17.224983] INFO: regipy.recovery: dirty pages: 18
[2020-09-10 02:41:17.224983] INFO: regipy.recovery: Restored 4096 bytes to offset 4096 from offset 49336
[2020-09-10 02:41:17.224983] INFO: regipy.recovery: Restored 4096 bytes to offset 28672 from offset 53432
[2020-09-10 02:41:17.225981] INFO: regipy.recovery: Restored 4096 bytes to offset 45056 from offset 57528
[2020-09-10 02:41:17.225981] INFO: regipy.recovery: Restored 4096 bytes to offset 221184 from offset 61624
[2020-09-10 02:41:17.225981] INFO: regipy.recovery: Restored 4096 bytes to offset 229376 from offset 65720
[2020-09-10 02:41:17.225981] INFO: regipy.recovery: Restored 4096 bytes to offset 258048 from offset 69816
[2020-09-10 02:41:17.225981] INFO: regipy.recovery: Restored 4096 bytes to offset 364544 from offset 73912
[2020-09-10 02:41:17.225981] INFO: regipy.recovery: Restored 4096 bytes to offset 380928 from offset 78008
[2020-09-10 02:41:17.225981] INFO: regipy.recovery: Restored 4096 bytes to offset 389120 from offset 82104
[2020-09-10 02:41:17.226978] INFO: regipy.recovery: Restored 4096 bytes to offset 450560 from offset 86200
[2020-09-10 02:41:17.226978] INFO: regipy.recovery: Restored 4096 bytes to offset 888832 from offset 90296
[2020-09-10 02:41:17.226978] INFO: regipy.recovery: Restored 4096 bytes to offset 913408 from offset 94392
[2020-09-10 02:41:17.226978] INFO: regipy.recovery: Restored 8192 bytes to offset 925696 from offset 98488
[2020-09-10 02:41:17.226978] INFO: regipy.recovery: Restored 4096 bytes to offset 946176 from offset 106680
[2020-09-10 02:41:17.227976] INFO: regipy.recovery: Restored 4096 bytes to offset 970752 from offset 110776
[2020-09-10 02:41:17.227976] INFO: regipy.recovery: Restored 4096 bytes to offset 983040 from offset 114872
[2020-09-10 02:41:17.227976] INFO: regipy.recovery: Restored 8192 bytes to offset 1077248 from offset 118968
[2020-09-10 02:41:17.227976] INFO: regipy.recovery: Restored 8192 bytes to offset 1097728 from offset 127160
[2020-09-10 02:41:17.227976] INFO: regipy.recovery: Parsing hvle block at 139264
[2020-09-10 02:41:17.227976] INFO: regipy.recovery: Reached a non HvLE object. stopping
[2020-09-10 02:41:17.229970] INFO: regipy.recovery: Log Size: 1101824
[2020-09-10 02:41:17.230967] INFO: regipy.recovery: Parsing hvle block at 512
[2020-09-10 02:41:17.230967] INFO: regipy.recovery: Currently at start of dirty pages: 608
[2020-09-10 02:41:17.230967] INFO: regipy.recovery: seq number: 311
[2020-09-10 02:41:17.230967] INFO: regipy.recovery: dirty pages: 7
[2020-09-10 02:41:17.231965] INFO: regipy.recovery: Restored 323584 bytes to offset 4096 from offset 608
[2020-09-10 02:41:17.231965] INFO: regipy.recovery: Restored 610304 bytes to offset 331776 from offset 324192
[2020-09-10 02:41:17.231965] INFO: regipy.recovery: Restored 4096 bytes to offset 946176 from offset 934496
[2020-09-10 02:41:17.231965] INFO: regipy.recovery: Restored 53248 bytes to offset 954368 from offset 938592
[2020-09-10 02:41:17.231965] INFO: regipy.recovery: Restored 49152 bytes to offset 1011712 from offset 991840
[2020-09-10 02:41:17.232962] INFO: regipy.recovery: Restored 20480 bytes to offset 1073152 from offset 1040992
[2020-09-10 02:41:17.232962] INFO: regipy.recovery: Restored 16384 bytes to offset 1097728 from offset 1061472
[2020-09-10 02:41:17.232962] INFO: regipy.recovery: Parsing hvle block at 1081344
[2020-09-10 02:41:17.232962] INFO: regipy.recovery: Currently at start of dirty pages: 1081416
[2020-09-10 02:41:17.232962] INFO: regipy.recovery: seq number: 312
[2020-09-10 02:41:17.232962] INFO: regipy.recovery: dirty pages: 4
[2020-09-10 02:41:17.232962] INFO: regipy.recovery: Restored 4096 bytes to offset 4096 from offset 1081416
[2020-09-10 02:41:17.232962] INFO: regipy.recovery: Restored 4096 bytes to offset 24576 from offset 1085512
[2020-09-10 02:41:17.233960] INFO: regipy.recovery: Restored 4096 bytes to offset 380928 from offset 1089608
[2020-09-10 02:41:17.233960] INFO: regipy.recovery: Restored 4096 bytes to offset 393216 from offset 1093704
Recovered 37 dirty pages. Restored hive is at Amcache.hve.regipy

Based on the log above, it seems like it had ignored the second log file - Amcache.hve.LOG2.
Files to reproduce the issue:
Amcache.hve
Amcache.hve.LOG1
Amcache.hve.LOG2
regipy output
yarp output

Regards.

This fails for the user NTUSER.DAT hive in Win 11 image available at https://www.khyrenz.com/resources:

#!/bin/python

import sys
from regipy.registry import RegistryHive

#Getting argument = NTUSER.DAT file - successful
reg = RegistryHive(sys.argv[1])

#This correctly gets subkeys under NTUSER\Software
for sk in reg.get_key('Software').iter_subkeys():
print(sk.name)
#This fails to find any subkeys under Software (e.g. Microsoft) despite above list showing keys exist
for subk in reg.get_key('Software\'+sk.name).iter_subkeys():
print(subk.name)

Error:
Traceback (most recent call last):
File "test.py", line 9, in
for subk in reg.get_key('Software\'+sk.name).iter_subkeys():
File "/home/ubuntu/.local/lib/python3.8/site-packages/regipy/registry.py", line 228, in get_key
raise RegistryKeyNotFoundException('Did not find subkey at {}'.format(key_path))
regipy.exceptions.RegistryKeyNotFoundException: Did not find subkey at Software\7-Zip

At this stage, libfwsi-python is not available as prebuilt wheels for Linux or macOS and this makes it difficult to install in a few cases. I offered to help there but there is no reply yet at libyal/libfwsi#19

In addition your requirement is pinned to an exact version which is problematic when using regipy as a library if there is an upgrade in the future. See this excellent article on the topic https://iscinumpy.dev/post/bound-version-constraints/

It would be great if you could relax the setup.py bounded requirements this way in

libfwsi-python >= 20220123
inflection >= 0.5.1

And wrt. libfwsi-python, it would be even better to make it an extra requires such that the plugin(s) that need it would only import it privately if used?

I can submit a patch alright if you think that can work for you.

FWIW, we are using regipy in https://github.com/nexB/scancode-toolkit/blob/develop/src/packagedcode/win_reg.py and this helps us extract installed package information from registry hives of Windows VM and Docker containers while running this on Linux... so thank you ++ as this is the only Python registry library that can actually parse correctly these data! (This is also used in https://scancodeio.readthedocs.io indirectly)