log2timeline / dftimewolf Goto Github PK

• Docker
• pypi
• CD with Github actions

The current implementation is convoluted, and error prone.

Currently timesketch is not working in dftimewolf.

What needs to happen is:

• Remove timesketch_utils
• Add the API and importer clients as dependencies to dftimewolf
• Make use of the configuration object to get access to the API client and use the importer to do the uploads of data.

If the disk image being processed by plaso has multiple partitions, then local_plaso will hang while log2timeline attempts to ask the user which partition they would like to process. Since we are trying to minimize user interaction on stdin, this isn't displayed which causes dfTimewolf to appear hung.

To get around this, we can just process all partitions with the --partition=all flag.

• change method names to CamelCase
  • #192
  • #193
• update pylintrc to enforce style
  • #194
• update l2tdocs style guide
  • log2timeline/l2tdocs#44
• deprecate timewolf style guide (if present)
  • per tomchop no additional documentation

Split module loader from config so that modules can use the loader to load other modules

e.g. when a processor fails and produces no output, the exporters still run. Errors should be caught and reported correctly.

The launch_collector() method in GRRHostCollector launches different collectors depending on the arguments provided. Ideally, we would like to know which collector to launch in the recipe rather than determining this with the arguments.

TLS certificate verification is disabled in the Timesketch API client by this line:

It should probably be enabled by default with the ability to disable, like TLS certificate validation for the GRR server URL.

I am trying to run and upload grr artificate with dftimewolf grr_artifact_ts . Once the plaso file got created, it is failed to upload to timesketch with following error.

Running external command: "log2timeline.py -q --status_view none --partition all --logfile /tmp/tmpr9c8zdrf/plaso.log /tmp/tmpr9c8zdrf/2d1c69396b2f48c3a23f275ba0fddd9a.plaso /tmp/tmpuu70uhlm/msedgewin10"
Module LocalPlasoProcessor completed
Uploading /tmp/tmpr9c8zdrf/2d1c69396b2f48c3a23f275ba0fddd9a.plaso to timeline msedgewin10
Module TimesketchExporter completed
dfTimewolf encountered one or more errors:
CRITICAL: An unknown error occurred: 'objects'
Full traceback:
Traceback (most recent call last):
File "/opt/dftimewolf/dftimewolf/lib/state.py", line 167, in _RunModuleThread
module.Process()
File "/opt/dftimewolf/dftimewolf/lib/exporters/timesketch.py", line 95, in Process
self.timesketch_api.ExportArtifacts(named_timelines, self.sketch_id)
File "/opt/dftimewolf/dftimewolf/lib/timesketch_utils.py", line 141, in ExportArtifacts
new_timeline_id = self._UploadTimeline(timeline_name, artifact_path)
File "/opt/dftimewolf/dftimewolf/lib/timesketch_utils.py", line 124, in _UploadTimeline
return response_dict['objects'][0]['id']
KeyError: 'objects'

Critical error found. Aborting.

Some code clean up: https://github.com/log2timeline/dftimewolf/blob/master/dftimewolf/config.py#L79

Currently uses a hard exit without any explanation. I opt to replace it by a BadConfig exception (or equiv)

• Delete the temporary directory in cleanup()
• Doctring says """"Execute the grep command""" though that's not happening =)
• _final_output is never used? Should it be initialized to '' so you can start appending to it right away?
• Probably don't need to filter(None, found) as you already check [item for item in found if item]

root@plaso:~# cd dftimewolf-master/
root@plaso:~/dftimewolf-master# pip install .

[...]
    byte-compiling /usr/local/lib/python2.7/dist-packages/dftimewolf/cli/dftimewolf_recipes.py to dftimewolf_recipes.pyc
    running install_data
    creating /usr/local/dftimewolf
    error: can't copy 'dftimewolf/dftimewolf.json': doesn't exist or not a regular file

    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-nw3biE-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-7n3gw2-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-nw3biE-build/

root@plaso:~/dftimewolf-master# ls dftimewolf/

cli/         config.json  config.py    __init__.py  lib/

This (https://github.com/log2timeline/dftimewolf/blob/master/setup.py#L32) should probably be corrected to config.json

alternatively

config.json should be renamed to dftimewolf.json (as per
https://github.com/log2timeline/dftimewolf/blob/master/dftimewolf/cli/dftimewolf_recipes.py#L54)

Boolean flags are processed as strings, fix this.

Running module GRRHuntArtifactCollector
Traceback (most recent call last):
File "dftimewolf/cli/dftimewolf_recipes.py", line 125, in
main()
File "dftimewolf/cli/dftimewolf_recipes.py", line 113, in main
module.setup(**new_args)
TypeError: setup() got an unexpected keyword argument 'verify'

Traceback:

Running linter on changed files.
ERROR: No valid .dftimewolfrc file found. See README for details.
test_dftimewolf (unittest.loader.ModuleImportFailure) ... ERROR

======================================================================
ERROR: test_dftimewolf (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_dftimewolf
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/home/onager/code/onager_dftimewolf/tests/test_dftimewolf.py", line 5, in <module>
    from dftimewolf.cli import dftimewolf_recipes
  File "/home/onager/code/onager_dftimewolf/dftimewolf/cli/dftimewolf_recipes.py", line 44, in <module>
    MODULES = import_modules()
  File "/home/onager/code/onager_dftimewolf/dftimewolf/internals.py", line 49, in import_modules
    module_directories = get_config()['module_dirs']
  File "/home/onager/code/onager_dftimewolf/dftimewolf/internals.py", line 131, in get_config
    exit(-1)
  File "/home/onager/code/venvs/plaso/lib/python2.7/site.py", line 403, in __call__
    raise SystemExit(code)
SystemExit: -1

recipes appear to be JSON in Python files, consider moving recipes to JSON (configuration) files

Maybe as a first step move args into the contents definition

Currently, critical errors are just labeled so but execution flow is not interrupted. The caller has to manually return when they add a critical error - it would be better to raise an exception and catch it in the main loop.

• The authentication_information tuple in the timesketch modules is awkward. Change this and pass different parameters.
• The grr_auth tuple in the GRR collector modules has the same problem and is making parameter specification in config.js difficult.

• bs4 is not the correct package name for beautifulsoup

dftimewolf timesketch_upload now supports a single file (at least according to the help information) add support for multiple file upload

Make sure that setup() methods are called correctly with different set of recipe parameters.

incident_id in the gcp_turbinia_import recipe is set to now by default, though this is the only recipe that does this (other recipes that define incident_id don't)

Determine the desired behavior and make this uniform

Issue surfaced while working on #169

https://travis-ci.org/log2timeline/dftimewolf/jobs/565314908

======================================================================
ERROR: lib.processors.turbinia (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: lib.processors.turbinia
Traceback (most recent call last):
  File "/opt/python/3.6.7/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/opt/python/3.6.7/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/travis/build/log2timeline/dftimewolf/tests/lib/processors/turbinia.py", line 19, in <module>
    from dftimewolf.lib.processors import turbinia
  File "./dftimewolf/lib/processors/turbinia.py", line 11, in <module>
    from turbinia import client as turbinia_client
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/turbinia/client.py", line 31, in <module>
    from turbinia import task_manager
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/turbinia/task_manager.py", line 35, in <module>
    from google.cloud import pubsub
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/google/cloud/pubsub.py", line 20, in <module>
    from google.cloud.pubsub_v1 import PublisherClient
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/google/cloud/pubsub_v1/__init__.py", line 18, in <module>
    from google.cloud.pubsub_v1 import publisher
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/google/cloud/pubsub_v1/publisher/__init__.py", line 17, in <module>
    from google.cloud.pubsub_v1.publisher.client import Client
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/google/cloud/pubsub_v1/publisher/client.py", line 29, in <module>
    from google.cloud.pubsub_v1.gapic import publisher_client
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/google/cloud/pubsub_v1/gapic/publisher_client.py", line 27, in <module>
    import google.api_core.client_options
ModuleNotFoundError: No module named 'google.api_core.client_options'

Docstrings should be descriptive rather than imperative.

Recent changes, require Python >=3.6 for dfTimewolf

Add user feedback that they are using an unsupported Python version to:

• setup.py,
• run_tests.py
• dftimewolf_recipes.py

e.g.

version_tuple = (sys.version_info[0], sys.version_info[1])
if version_tuple[0] != 3 or version_tuple < (3, 6):
  print((
      'Unsupported Python version: {0:s}, version 3.6 or higher '
      'required.').format(sys.version))
  sys.exit(1)

The stdout / syslog combination was removed in favor of a simpler print statement but we still want a more powerful logging mechanism.

the tests create tests/lib/processors/test_data/turbinia.confc

• change the test to clean up after itself
• remove the .gitignore entry for the tests create tests/lib/processors/test_data/turbinia.confc

Doesn't make sense to have two functions do the same thing.

Pipenv is pretty good, consider using this for packaging.

It would be nice to have a recipe to export a disk to GCS (maybe gcp_export). We already have most of the code to do this, so I believe we just have to add the last export step.

Some wrappers around dftimewolf might want to replace an already registered module or recipe. Currently registering the same module / recipe twice with the new registry classes raises an exception. Should we change the codebase to unregister the module / recipe first, or should we pass a force or overwrite parameter to the method to ignore the exception?

• Description for what recipes do
• Description for what Collectors, Processors, Exporters do
• Use-cases and example command lines