cert-polska / drakvuf-sandbox Goto Github PK

If the error log contains the following error about device model:

libxl: error: libxl_create.c:1676:domcreate_devmodel_started: Domain 4:device model did not start: -3

subprocess.CalledProcessError: Command 'xl create /etc/drakrun/configs/vm-0.cfg' returned non-zero exit status 3.

then one should inspect /var/log/xen/qemu*.log in order to determine what actually happened. This should be docummented in "Troubleshooting" section in README.

At the moment there are few separate issues that are preventing DRAKVUF Sandbox from working on Ubuntu 20.04:

• dh-virtualenv is not shipped in Ubuntu 20.04 because it doesn't build properly - Python 2 being deprecated (found workaround)
• drakvuf-bundle depends on libjson-c3 and libnettle6 which are present in Ubuntu 20.04 under bumped versions (can be easily adjusted)
• Xen is yielding compile errors under 20.04 (found workaround)
• Xen is crashing under 20.04 (found patch)

patches:

• adjustment PR to drakvuf
• adjustment PR to drakvuf-sandbox

DRAKVUF is already capable of producing network-related logs (socketmon plugin) and generic WinAPI call logs (apimon plugin). We do also have ordinary pcaps. This can be correlated together into the form of network activity report.

We are mostly interested in basic info about:

• DNS queries
• TCP streams and UDP packets
• HTTP(S) traffic

All of the above should be annotated with PIDs, whenever possible.

It was reported few times already that on an incompatible kernel, the GRUB Xen entry is not being added at all, which is pretty unintuitive. We should detect such situation and warn accordingly.

Describe the bug

During Setup at the point when i run the command sudo draksetup postinstall i run into a timeout.

root@debian:/home/martin# draksetup postinstall
Traceback (most recent call last):
  File "/usr/lib/python3.7/subprocess.py", line 474, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/usr/lib/python3.7/subprocess.py", line 939, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/lib/python3.7/subprocess.py", line 1682, in _communicate
    self._check_timeout(endtime, orig_timeout)
  File "/usr/lib/python3.7/subprocess.py", line 982, in _check_timeout
    raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['vmi-win-guid', 'name', 'vm-0']' timed out after 30 seconds

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/draksetup", line 7, in <module>
    ds.main()
  File "/opt/venvs/drakrun/lib/python3.7/site-packages/drakrun/draksetup.py", line 389, in main
    generate_profiles()
  File "/opt/venvs/drakrun/lib/python3.7/site-packages/drakrun/draksetup.py", line 233, in generate_profiles
    output = subprocess.check_output(['vmi-win-guid', 'name', 'vm-0'], timeout=30).decode('utf-8')
  File "/usr/lib/python3.7/subprocess.py", line 395, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.7/subprocess.py", line 479, in run
    stderr=stderr)
subprocess.TimeoutExpired: Command '['vmi-win-guid', 'name', 'vm-0']' timed out after 30 seconds

I guessing this is, because the command vmi-win-guid name vm-0 takes around 1 minute to complete.

I am running all of this in a KVM on proxmox.

How to reproduce

Steps to reproduce the behavior:

1. Install drakcore and drakrun
2. Start and install a Windows 10 x64 image
3. Execute sudo draksetup postinstall

Output of the status checking commands

root@debian:/home/martin# drak-healthcheck
Checking daemon status...
drak-web.service              OK
drak-system.service           OK
drak-minio.service            OK
drak-postprocess.service      OK

Checking worker status...
[email protected]             ERROR

Right now we don't assist with setting up the networking inside the guest VM. This can be achieved using plain Xen bridges or Open vSwitch.

Related:
http://docs.openvswitch.org/en/latest/howto/kvm/
https://wiki.xenproject.org/wiki/Xen_Networking#Setting_up_Open_vSwitch_networking

When building the deb package for drakcore, build target from rules is run twice.
It triggers rebuild of frontend and download of minio which increases build times.

Looks like the current Redis connection timeout is set to 120 seconds. This should be lowered to some sane value.

Apr 26 16:19:13 zen systemd[1]: Started drakrun service.
Apr 26 16:21:12 zen drakrun[417]: [2020-04-26 16:21:12,503][INFO] Service karton.drakrun-prod started
Apr 26 16:21:12 zen drakrun[417]: [2020-04-26 16:21:12,505][INFO] Service binds created.
Apr 26 16:21:12 zen drakrun[417]: [2020-04-26 16:21:12,505][INFO] Binding on: {'type': 'sample', 'stage': 'recognized', 'platform': 'win32'}
Apr 26 16:21:12 zen drakrun[417]: [2020-04-26 16:21:12,506][INFO] Binding on: {'type': 'sample', 'stage': 'recognized', 'platform': 'win64'}

Create read-only instance for demonstration purposes

This project should be easy to edit locally after installation from DEB package if pip install -e is applied. This should be documented in README.

We should setup a custom simple package repository to make distribution and installation easier.

Currently we do have hardcoded timeout of 10 minutes. This should be configurable globally (in config) and locally (if somebody would like to override this value for a single analysis).

Since we do have a functional networking, we may:

• spin up a temporary VM
• spin up a temporary web server
• inject a PowerShell command that would grab the interesting DLLs (e.g. syswow64\ntdll.dll) and upload them to the web server
• retrieve them on the host side and generate necessary profiles

Right now, only the analysis ID is shown on the list of analyses. We would want to also show at least:

• SHA256 of the sample,
• the output from file sample.bin command,
• the analysis date and time.

During build process minio is downloaded a few times from official, but slow CDN.

Such an error may indicate that there is not enough memory in total:

subprocess.CalledProcessError: Command 'xl create /etc/drakrun/configs/vm-0.cfg' returned non-zero exit status 3.

xc: error: panic: xc_dom_boot.c:122: xc_dom_boot_mem_init: can't allocate low memory for domain: Out of memory

this may be resolved by editing either /etc/default/grub.d/xen.cfg to adjust Dom0 memory (update-grub and reboot required) or by editing /etc/drakrun/scripts/cfg.template to adjust DomU memory. This should be documented.

Currently the DRAKVUF Sandbox requires either bare metal or nesting in a VMware product. We should document the good and bad paths (e.g. KVM nesting doesnt work at the moment).

Default /etc/drakrun/config.ini:

[minio]
access_key=
secret_key=
address=localhost:9000
bucket=karton2
secure=0

we should detect whether access_key or secret_key is empty and warn the user appropriately.

Fix this leftover in code:

Project description should be enhanced with screenshots, videos and other materials that would better describe how to install/use this project and what is it's main purpose.

Depends on #1, #2.

• A screenshot in README.md
• A demo movie which shows the installation process as well as the basic operation of version v0.2.0
• Demo instance of the sandbox

We should implement e2e tests within this project's CI in order to avoid breaking core functionality when making changes.

Describe the bug

when im try this command: sudo draksetup install --iso /opt/path_to_windows.iso

logs:

root@ubuntu:/home/fmt# draksetup install --iso win7.iso
[2020-05-09 05:50:33,413][INFO] Ensuring that drakrun@* services are stopped...
[2020-05-09 05:50:35,586][INFO] Performing installation...
[2020-05-09 05:50:35,766][INFO] Checking xen-detect...
Running in PV context on Xen V4.13.
[2020-05-09 05:50:35,921][INFO] Testing if xl tool is sane...
[2020-05-09 05:50:38,503][INFO] Generated VM configuration for vm-0
Parsing config from /etc/drakrun/configs/vm-0.cfg
xc: error: panic: xc_dom_boot.c:122: xc_dom_boot_mem_init: can't allocate low memory for domain: Out of memory
libxl: error: libxl_dom.c:762:libxl__build_dom: xc_dom_boot_mem_init failed: Cannot allocate memory
libxl: error: libxl_create.c:1420:domcreate_rebuild_done: Domain 1:cannot (re-)build domain: -3
libxl: error: libxl_domain.c:1177:libxl__destroy_domid: Domain 1:Non-existant domain
libxl: error: libxl_domain.c:1131:domain_destroy_callback: Domain 1:Unable to destroy guest
libxl: error: libxl_domain.c:1058:domain_destroy_cb: Domain 1:Destruction of domain failed
[2020-05-09 05:50:48,029][ERROR] Failed to launch VM vm-0
Traceback (most recent call last):
File "/opt/venvs/drakrun/lib/python3.6/site-packages/drakrun/draksetup.py", line 201, in install
subprocess.run('xl create {}'.format(shlex.quote(cfg_path)), shell=True, check=True)
File "/usr/lib/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'xl create /etc/drakrun/configs/vm-0.cfg' returned non-zero exit status 3.

my vm-0.cfg info:

root@ubuntu:/home/fmt# cat /etc/drakrun/configs/vm-0.cfg
arch = 'x86_64'
name = "vm-0"
maxmem = 2048
memory = 2048
vcpus = 2
maxcpus = 2
builder = "hvm"
boot = "cd"
hap = 1
acpi = 1
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "destroy"
vnc=1
vnclisten="0.0.0.0"
vncdisplay=0
vncpasswd="lAHA0VnXQv0ElbHfugK0"
usb = 1
usbdevice = "tablet"
altp2m = 2
shadow_memory = 16
audio=1
soundhw='hda'
vif = [ 'type=ioemu,model=e1000,bridge=drak0' ]
disk = [ "tap:qcow2:/var/lib/drakrun//volumes/vm-0.img,xvda,w", "file:/home/fmt/win7.iso,hdc:cdrom,r" ]

https://github.com/tklengyel/drakvuf/blob/master/package/extra/etc/default/grub.d/xen.cfg

Probably half CPUs and half RAM to Dom0 by default?

Plus warning if the current configuration doesn't allow to run any virtual machine (total ram < dom0_ram + single_vm_ram).

Applies to

• #10
• #11
• #13
• #43

Describe the bug

When installing the drakvuf bundle on Ubuntu Server 18.04 LTS, xen is not showing up in the grub bootloader, and it does not boot into xen by default.

How to reproduce

Steps to reproduce the behavior:

1. Install drakvuf bundle
2. Execute sudo reboot
3. xen-detect always returns not running on xen

(To be designed)

• Sample uploading through API
• The full context of possible usage (uploading samples + collecting results)

(reported on Twitter: https://twitter.com/matte_lodi/status/1255080965556391936 -> reply when ticket is resolved)

In Xubuntu 18.04, there is a problem with drakcore UI caused by lack of dataclasses module (builtin since Python 3.7, but Xubuntu has Python 3.6).

This is not on our list of officially supported systems but if this is an only issue and it is related just to the compatibility with Python 3.6 then we could probably study the topic.

Apr 28 16:52:46 ubuntu uwsgi[3909]: Traceback (most recent call last): Apr 28 16:52:46 ubuntu uwsgi[3909]:   File "/opt/venvs/drakcore/lib/python3.6/site-packages/drakcore/app.py", line 16, in <module> Apr 28 16:52:46 ubuntu uwsgi[3909]:     from drakcore.pstree import generate_process_tree Apr 28 16:52:46 ubuntu uwsgi[3909]:   File "/opt/venvs/drakcore/lib/python3.6/site-packages/drakcore/pstree.py", line 2, in <module> Apr 28 16:52:46 ubuntu uwsgi[3909]:     from dataclasses import dataclass, field Apr 28 16:52:46 ubuntu uwsgi[3909]: ModuleNotFoundError: No module named 'dataclasses'

If the ProcDOT graph is generated, it is graphically replacing process tree.

Expected: both graph and process tree appearing.

We could pretty easily integrate noVNC, e.g. in the "please wait" subpage while the analysis is pending.

https://novnc.com/info.html

Currently, the apimon plugin doesn't work at all because the distributed hook list (/etc/drakrun/hooks.txt) is empty.

Currently, only qcow2 is supported. We should also support guest installation on ZFS volumes.

Packages should depend (or include) sudo in dependencies, as scripts require it.

We do need some material to compare how far we are currently with the ProcDOT integration. This could be basically achieved by dropping a few samples into a VM monitored by procmon, generating graphs and finally comparing these graphs against ones that are generated by our integration.

Parse procmon output into a form of a process tree that can be easily read by human (and visualized).

E.g. output structure:

[
  {
    "pid":2408,
    "process":"C:\\Users\\Administrator\\AppData\\Local\\Temp\\cmd.exe",
    "children":[
      {
        "pid":2968,
        "process":"C:\\Users\\Administrator\\AppData\\Local\\Temp\\emo2.exe",
        "children":[
          {
            "pid":3048,
            "process":"C:\\Users\\Administrator\\AppData\\Local\\Temp\\emo3.exe",
            "children":[

            ]
          }
        ]
      },
      {
        "pid":3028,
        "process":"C:\\Users\\Administrator\\AppData\\Local\\Temp\\emo4.exe",
        "children":[

        ]
      }
    ]
  }
]

It's not possible to zoom the behavioral graph generated by ProcDOT integration, which makes it hard/impossible to read the graph contents.

Depends on #13. Frontend changes are required to display network report (once it's implemented) in the analysis UI.

Check whether drakvuf-sandbox is feasible on GCP on at least one of the supported systems. Info needed: whether some custom hacks/adjustments are needed and if we could document them/implement some improvements dedicated for GCP.

This is a feature request based on community feedback.

Signatures should allow to extract some interesting high-level facts out of behavioral logs, e.g. process injected itself to other process and exitted, process created files in C:\sth\sth\Autostart\ etc.

Depends on #16. The comparison will allow to identify how accurate is the integration right now.

depends #10

Add a new tab to analysis report in the web UI that would show apimon logs in a human-friendly form, with the possibility to filter them per process.

Depends on #10. Visualize DRAKVUF logs in the web UI in order to make them more discoverable.

For each process separately, we would like to show (at least):

• related process activity (procmon logs)
• registry activity (apimon, regmon logs)
• file operations (apimon, filetracer, filedelete logs)

related #43

ZFS installation on Debian Buster:
https://github.com/openzfs/zfs/wiki/Debian

Necessary commands:

# zpool create tank <partition_name>
# zfs create tank/vms
# draksetup install --iso win7.iso --unattended-xml unattended.xml --storage-backend zfs --zfs-tank-name tank/vms

Describe the bug

After completing the necessary steps to complete the installation of the sandbox, I tried to go to http://localhost:6300. Instead of the the web UI, I was greeted with an internal server error.

How to reproduce

Steps to reproduce the behavior:

1. Install sandbox as instructed
2. Execute sudo draksetup postinstall
3. Visit localhost:6300

Output of the status checking commands

systemctl status drak-web.service

drak-web.service - drak-web service
   Loaded: loaded (/etc/systemd/system/drak-web.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2020-05-09 23:49:53 PDT; 10min ago
 Main PID: 783 (uwsgi)
    Tasks: 6 (limit: 2207)
   CGroup: /system.slice/drak-web.service
           ├─ 783 /opt/venvs/drakcore/bin/uwsgi --ini /etc/drakcore/uwsgi.ini
           ├─1014 /opt/venvs/drakcore/bin/uwsgi --ini /etc/drakcore/uwsgi.ini
           ├─1016 /opt/venvs/drakcore/bin/uwsgi --ini /etc/drakcore/uwsgi.ini
           ├─1018 /opt/venvs/drakcore/bin/uwsgi --ini /etc/drakcore/uwsgi.ini
           ├─1029 /opt/venvs/drakcore/bin/uwsgi --ini /etc/drakcore/uwsgi.ini
           └─1030 /opt/venvs/drakcore/bin/uwsgi --ini /etc/drakcore/uwsgi.ini

May 09 23:49:59 ubuntu uwsgi[783]: *** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
May 09 23:49:59 ubuntu uwsgi[783]: *** uWSGI is running in multiple interpreter mode ***
May 09 23:49:59 ubuntu uwsgi[783]: spawned uWSGI master process (pid: 783)
May 09 23:49:59 ubuntu uwsgi[783]: spawned uWSGI worker 1 (pid: 1014, cores: 1)
May 09 23:49:59 ubuntu uwsgi[783]: spawned uWSGI worker 2 (pid: 1016, cores: 1)
May 09 23:50:00 ubuntu uwsgi[783]: spawned uWSGI worker 3 (pid: 1018, cores: 1)
May 09 23:50:00 ubuntu uwsgi[783]: spawned uWSGI worker 4 (pid: 1029, cores: 1)
May 09 23:50:00 ubuntu uwsgi[783]: spawned uWSGI http 1 (pid: 1030)
May 09 23:52:48 ubuntu uwsgi[783]: --- no python application found, check your startup logs for errors ---
May 09 23:52:48 ubuntu uwsgi[783]: [pid: 1014|app: -1|req: -1/1] 127.0.0.1 () {36 vars in 563 bytes} [Sat May  9 23:52:48 2020] GET / => generated 21 bytes in 2 msecs (HTTP/1.1 500) 2 headers in 83 bytes (0 switches on core 0)

sudo systemctl status [email protected]

[email protected] - drakrun service
   Loaded: loaded (/etc/systemd/system/[email protected]; indirect; vendor preset: enabled)
   Active: active (running) since Sat 2020-05-09 23:50:07 PDT; 16min ago
 Main PID: 1334 (drakrun)
    Tasks: 1 (limit: 2207)
   CGroup: /system.slice/system-drakrun.slice/[email protected]
           └─1334 /opt/venvs/drakrun/bin/python /opt/venvs/drakrun/bin/drakrun 1

May 09 23:50:07 ubuntu systemd[1]: Started drakrun service.
May 09 23:52:07 ubuntu drakrun[1334]: iptables: Bad rule (does a matching rule exist in that chain?).
May 09 23:52:07 ubuntu drakrun[1334]: iptables: Bad rule (does a matching rule exist in that chain?).
May 09 23:52:07 ubuntu drakrun[1334]: [2020-05-09 23:52:07,174][INFO] Service karton.drakrun-prod started
May 09 23:52:07 ubuntu drakrun[1334]: [2020-05-09 23:52:07,187][INFO] Service binds created.
May 09 23:52:07 ubuntu drakrun[1334]: [2020-05-09 23:52:07,195][INFO] Binding on: {'type': 'sample', 'stage': 'recognized', 'platform': 'win32'}
May 09 23:52:07 ubuntu drakrun[1334]: [2020-05-09 23:52:07,198][INFO] Binding on: {'type': 'sample', 'stage': 'recognized', 'platform': 'win64'}

tree /var/lib/drakcore

.
└── minio
    ├── drakrun
    ├── karton2
    └── .minio.sys
        ├── backend-encrypted
        ├── buckets
        │   ├── .bloomcycle.bin
        │   ├── .minio.sys
        │   │   └── buckets
        │   │       ├── .bloomcycle.bin
        │   │       │   └── fs.json
        │   │       ├── .usage-cache.bin
        │   │       │   └── fs.json
        │   │       └── .usage.json
        │   │           └── fs.json
        │   ├── .tracker.bin
        │   ├── .usage-cache.bin
        │   └── .usage.json
        ├── config
        │   ├── config.json
        │   └── iam
        │       └── format.json
        ├── format.json
        ├── multipart
        └── tmp
            ├── 64f8397f-8834-4223-83c1-336df8a504ef
            ├── 72d1d724-222f-44a6-810c-a24e2c742133
            ├── b7219802-97ef-4e4a-b647-55979848954f
            └── f8c8d0cb-6d09-4a91-bc9b-53fbebf64459

18 directories, 11 files

Add a CLI command to drakcore that would generate basic troubleshooting report.

Related bug: #90

The default azure kernel shipped with Azure Cloud Ubuntu 18.04 doesn't declare Xen support, thus the grub hook doesn't see it as a suitable Dom0 system.

Some guide on how to prepare a custom Ubuntu VM with a generic kernel:
https://docs.microsoft.com/en-us/azure/virtual-machines/linux/create-upload-ubuntu

Another options is to simply try to replace azure kernel with a stock one, but the Xen Dom0 doesn't work out of the box in such setup:
https://ubuntu.com/blog/microsoft-and-canonical-increase-velocity-with-azure-tailored-kernel

Depends on #10. Frontend changes are required to incorporate the process tree into the analysis UI.