log2timeline / dftimewolf Goto Github PK
View Code? Open in Web Editor NEWA framework for orchestrating forensic collection, processing and data export
License: Apache License 2.0
A framework for orchestrating forensic collection, processing and data export
License: Apache License 2.0
The current implementation is convoluted, and error prone.
Currently timesketch is not working in dftimewolf.
What needs to happen is:
timesketch_utils
If the disk image being processed by plaso has multiple partitions, then local_plaso will hang while log2timeline attempts to ask the user which partition they would like to process. Since we are trying to minimize user interaction on stdin, this isn't displayed which causes dfTimewolf to appear hung.
To get around this, we can just process all partitions with the --partition=all flag.
Split module loader from config so that modules can use the loader to load other modules
e.g. when a processor fails and produces no output, the exporters still run. Errors should be caught and reported correctly.
The launch_collector()
method in GRRHostCollector
launches different collectors depending on the arguments provided. Ideally, we would like to know which collector to launch in the recipe rather than determining this with the arguments.
File "dftimewolf/lib/state.py", line 195, in check_errors
print('{0:s} {1:s}'.format('CRITICAL: ' if critical else '', error))
TypeError: unsupported format string passed to TypeError.__format__
TLS certificate verification is disabled in the Timesketch API client by this line:
It should probably be enabled by default with the ability to disable, like TLS certificate validation for the GRR server URL.
I am trying to run and upload grr artificate with dftimewolf grr_artifact_ts . Once the plaso file got created, it is failed to upload to timesketch with following error.
Running external command: "log2timeline.py -q --status_view none --partition all --logfile /tmp/tmpr9c8zdrf/plaso.log /tmp/tmpr9c8zdrf/2d1c69396b2f48c3a23f275ba0fddd9a.plaso /tmp/tmpuu70uhlm/msedgewin10"
Module LocalPlasoProcessor completed
Uploading /tmp/tmpr9c8zdrf/2d1c69396b2f48c3a23f275ba0fddd9a.plaso to timeline msedgewin10
Module TimesketchExporter completed
dfTimewolf encountered one or more errors:
CRITICAL: An unknown error occurred: 'objects'
Full traceback:
Traceback (most recent call last):
File "/opt/dftimewolf/dftimewolf/lib/state.py", line 167, in _RunModuleThread
module.Process()
File "/opt/dftimewolf/dftimewolf/lib/exporters/timesketch.py", line 95, in Process
self.timesketch_api.ExportArtifacts(named_timelines, self.sketch_id)
File "/opt/dftimewolf/dftimewolf/lib/timesketch_utils.py", line 141, in ExportArtifacts
new_timeline_id = self._UploadTimeline(timeline_name, artifact_path)
File "/opt/dftimewolf/dftimewolf/lib/timesketch_utils.py", line 124, in _UploadTimeline
return response_dict['objects'][0]['id']
KeyError: 'objects'
Critical error found. Aborting.
Some code clean up: https://github.com/log2timeline/dftimewolf/blob/master/dftimewolf/config.py#L79
Currently uses a hard exit without any explanation. I opt to replace it by a BadConfig exception (or equiv)
[item for item in found if item]
root@plaso:~# cd dftimewolf-master/
root@plaso:~/dftimewolf-master# pip install .
[...]
byte-compiling /usr/local/lib/python2.7/dist-packages/dftimewolf/cli/dftimewolf_recipes.py to dftimewolf_recipes.pyc
running install_data
creating /usr/local/dftimewolf
error: can't copy 'dftimewolf/dftimewolf.json': doesn't exist or not a regular file
----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-nw3biE-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-7n3gw2-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-nw3biE-build/
root@plaso:~/dftimewolf-master# ls dftimewolf/
cli/ config.json config.py __init__.py lib/
This (https://github.com/log2timeline/dftimewolf/blob/master/setup.py#L32) should probably be corrected to config.json
alternatively
config.json should be renamed to dftimewolf.json (as per
https://github.com/log2timeline/dftimewolf/blob/master/dftimewolf/cli/dftimewolf_recipes.py#L54)
Boolean flags are processed as strings, fix this.
Running module GRRHuntArtifactCollector
Traceback (most recent call last):
File "dftimewolf/cli/dftimewolf_recipes.py", line 125, in
main()
File "dftimewolf/cli/dftimewolf_recipes.py", line 113, in main
module.setup(**new_args)
TypeError: setup() got an unexpected keyword argument 'verify'
Traceback:
Running linter on changed files.
ERROR: No valid .dftimewolfrc file found. See README for details.
test_dftimewolf (unittest.loader.ModuleImportFailure) ... ERROR
======================================================================
ERROR: test_dftimewolf (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_dftimewolf
Traceback (most recent call last):
File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
module = self._get_module_from_name(name)
File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
__import__(name)
File "/home/onager/code/onager_dftimewolf/tests/test_dftimewolf.py", line 5, in <module>
from dftimewolf.cli import dftimewolf_recipes
File "/home/onager/code/onager_dftimewolf/dftimewolf/cli/dftimewolf_recipes.py", line 44, in <module>
MODULES = import_modules()
File "/home/onager/code/onager_dftimewolf/dftimewolf/internals.py", line 49, in import_modules
module_directories = get_config()['module_dirs']
File "/home/onager/code/onager_dftimewolf/dftimewolf/internals.py", line 131, in get_config
exit(-1)
File "/home/onager/code/venvs/plaso/lib/python2.7/site.py", line 403, in __call__
raise SystemExit(code)
SystemExit: -1
recipes appear to be JSON in Python files, consider moving recipes to JSON (configuration) files
Maybe as a first step move args into the contents definition
Currently, critical errors are just labeled so but execution flow is not interrupted. The caller has to manually return
when they add a critical error - it would be better to raise an exception and catch it in the main loop.
authentication_information
tuple in the timesketch modules is awkward. Change this and pass different parameters.grr_auth
tuple in the GRR collector modules has the same problem and is making parameter specification in config.js
difficult.dftimewolf timesketch_upload now supports a single file (at least according to the help information) add support for multiple file upload
Make sure that setup() methods are called correctly with different set of recipe parameters.
incident_id in the gcp_turbinia_import recipe is set to now by default, though this is the only recipe that does this (other recipes that define incident_id don't)
Determine the desired behavior and make this uniform
Issue surfaced while working on #169
https://travis-ci.org/log2timeline/dftimewolf/jobs/565314908
======================================================================
ERROR: lib.processors.turbinia (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: lib.processors.turbinia
Traceback (most recent call last):
File "/opt/python/3.6.7/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/opt/python/3.6.7/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/home/travis/build/log2timeline/dftimewolf/tests/lib/processors/turbinia.py", line 19, in <module>
from dftimewolf.lib.processors import turbinia
File "./dftimewolf/lib/processors/turbinia.py", line 11, in <module>
from turbinia import client as turbinia_client
File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/turbinia/client.py", line 31, in <module>
from turbinia import task_manager
File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/turbinia/task_manager.py", line 35, in <module>
from google.cloud import pubsub
File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/google/cloud/pubsub.py", line 20, in <module>
from google.cloud.pubsub_v1 import PublisherClient
File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/google/cloud/pubsub_v1/__init__.py", line 18, in <module>
from google.cloud.pubsub_v1 import publisher
File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/google/cloud/pubsub_v1/publisher/__init__.py", line 17, in <module>
from google.cloud.pubsub_v1.publisher.client import Client
File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/google/cloud/pubsub_v1/publisher/client.py", line 29, in <module>
from google.cloud.pubsub_v1.gapic import publisher_client
File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/google/cloud/pubsub_v1/gapic/publisher_client.py", line 27, in <module>
import google.api_core.client_options
ModuleNotFoundError: No module named 'google.api_core.client_options'
Docstrings should be descriptive rather than imperative.
Recent changes, require Python >=3.6 for dfTimewolf
Add user feedback that they are using an unsupported Python version to:
e.g.
version_tuple = (sys.version_info[0], sys.version_info[1])
if version_tuple[0] != 3 or version_tuple < (3, 6):
print((
'Unsupported Python version: {0:s}, version 3.6 or higher '
'required.').format(sys.version))
sys.exit(1)
The stdout / syslog combination was removed in favor of a simpler print
statement but we still want a more powerful logging mechanism.
the tests create tests/lib/processors/test_data/turbinia.confc
.gitignore
entry for the tests create tests/lib/processors/test_data/turbinia.confcDoesn't make sense to have two functions do the same thing.
Pipenv is pretty good, consider using this for packaging.
======================================================================
ERROR: lib.processors.turbinia (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: lib.processors.turbinia
Traceback (most recent call last):
File "/usr/lib64/python3.7/unittest/loader.py", line 436, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib64/python3.7/unittest/loader.py", line 377, in _get_module_from_name
__import__(name)
File "/home/user/Projects/dftimewolf/dftimewolf/tests/lib/processors/turbinia.py", line 19, in <module>
from dftimewolf.lib.processors import turbinia
File "./dftimewolf/lib/processors/turbinia.py", line 11, in <module>
from turbinia import client as turbinia_client
File "/home/user/Projects/dftimewolf/lib/python3.7/site-packages/turbinia/client.py", line 29, in <module>
from turbinia import task_manager
File "/home/user/Projects/dftimewolf/lib/python3.7/site-packages/turbinia/task_manager.py", line 30, in <module>
import psq
File "/home/user/Projects/dftimewolf/lib/python3.7/site-packages/psq/__init__.py", line 24, in <module>
from .broadcast_queue import BroadcastQueue
File "/home/user/Projects/dftimewolf/lib/python3.7/site-packages/psq/broadcast_queue.py", line 20, in <module>
from . import queue
File "/home/user/Projects/dftimewolf/lib/python3.7/site-packages/psq/queue.py", line 37
name='default', storage=None, extra_context=None, async=True):
^
SyntaxError: invalid syntax
It would be nice to have a recipe to export a disk to GCS (maybe gcp_export). We already have most of the code to do this, so I believe we just have to add the last export step.
Some wrappers around dftimewolf might want to replace an already registered module or recipe. Currently registering the same module / recipe twice with the new registry classes raises an exception. Should we change the codebase to unregister the module / recipe first, or should we pass a force
or overwrite
parameter to the method to ignore the exception?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.