ahlashkari / dohlyzer Goto Github PK

reading from file ./MaliciousDoH-dns2tcp-Pcap-001_600/dns2tcp_tunnel_1111_doh1_2020-03-31T21_54_32.055088.pcap, link-type LINUX_SLL (Linux cooked v1)
Packet count: 969
Garbage Collection Began. Flows = 2
/usr/lib/python3/dist-packages/scipy/stats/stats.py:263: RuntimeWarning: The input array could not be properly checked for nan values. nan values will b
e ignored.
warnings.warn("The input array could not be properly checked for nan "
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.8/dist-packages/scapy/sendrecv.py", line 989, in _run
session.on_packet_received(p)
File "/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter/flow_session.py", line 105, in on_packet_received
self.garbage_collect(packet.time)
File "/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter/flow_session.py", line 119, in garbage_collect
data = flow.get_data()
File "/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter/flow.py", line 80, in get_data
'PacketTimeMode': packet_time.get_mode(),
File "/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter/features/packet_time.py", line 114, in get_mode
mode = float(stat.mode(self._get_packet_times())[0])
File "/usr/lib/python3/dist-packages/scipy/stats/stats.py", line 458, in mode
if a.dtype == object and np.nan in set(a.ravel()):
TypeError: unhashable type: 'EDecimal'
mohittembe@DESKTOP-8FG1VFR:/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter$ python3 dohlyzer.py -f ./MaliciousDoH-dns2tcp
-Pcap-001_600/dns2tcp_tunnel_1111_doh1_2020-03-31T22_17_01.945354.pcap -c ./output.csv
reading from file ./MaliciousDoH-dns2tcp-Pcap-001_600/dns2tcp_tunnel_1111_doh1_2020-03-31T22_17_01.945354.pcap, link-type LINUX_SLL (Linux cooked v1)
Packet count: 945
Garbage Collection Began. Flows = 2
/usr/lib/python3/dist-packages/scipy/stats/stats.py:263: RuntimeWarning: The input array could not be properly checked for nan values. nan values will b
e ignored.
warnings.warn("The input array could not be properly checked for nan "
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.8/dist-packages/scapy/sendrecv.py", line 989, in _run
session.on_packet_received(p)
File "/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter/flow_session.py", line 105, in on_packet_received
self.garbage_collect(packet.time)
File "/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter/flow_session.py", line 119, in garbage_collect
data = flow.get_data()
File "/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter/flow.py", line 80, in get_data
'PacketTimeMode': packet_time.get_mode(),
File "/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter/features/packet_time.py", line 114, in get_mode
mode = float(stat.mode(self._get_packet_times())[0])
File "/usr/lib/python3/dist-packages/scipy/stats/stats.py", line 458, in mode
if a.dtype == object and np.nan in set(a.ravel()):
TypeError: unhashable type: 'EDecimal'
mohittembe@DESKTOP-8FG1VFR:/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter$ python3 dohlyzer.py -f ./MaliciousDoH-dns2tcp
-Pcap-001_600 -c ./output.csv
tcpdump: error reading dump file: Is a directory
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.8/dist-packages/scapy/sendrecv.py", line 882, in _run
sniff_sockets[PcapReader(
File "/usr/local/lib/python3.8/dist-packages/scapy/utils.py", line 975, in call
raise Scapy_Exception(
scapy.error.Scapy_Exception: No data could be read!
mohittembe@DESKTOP-8FG1VFR:/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter$ python3 dohlyzer.py -f ./MaliciousDoH-dns2tcp
-Pcap-001_600/dns2tcp_tunnel_1111_doh1_2020-03-31T22_17_01.945354.pcap -c ./output.csv
reading from file ./MaliciousDoH-dns2tcp-Pcap-001_600/dns2tcp_tunnel_1111_doh1_2020-03-31T22_17_01.945354.pcap, link-type LINUX_SLL (Linux cooked v1)
Packet count: 945
Garbage Collection Began. Flows = 2
/usr/lib/python3/dist-packages/scipy/stats/stats.py:263: RuntimeWarning: The input array could not be properly checked for nan values. nan values will b
e ignored.
warnings.warn("The input array could not be properly checked for nan "
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.8/dist-packages/scapy/sendrecv.py", line 989, in _run
session.on_packet_received(p)
File "/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter/flow_session.py", line 105, in on_packet_received
self.garbage_collect(packet.time)
File "/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter/flow_session.py", line 119, in garbage_collect
data = flow.get_data()
File "/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter/flow.py", line 80, in get_data
'PacketTimeMode': packet_time.get_mode(),
File "/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter/features/packet_time.py", line 114, in get_mode
mode = float(stat.mode(self._get_packet_times())[0])
File "/usr/lib/python3/dist-packages/scipy/stats/stats.py", line 458, in mode
if a.dtype == object and np.nan in set(a.ravel()):
TypeError: unhashable type: 'EDecimal'
mohittembe@DESKTOP-8FG1VFR:/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter$ python3 dohlyzer.py -f ./MaliciousDoH-dns2tcp
-Pcap-001_600/dns2tcp_tunnel_1111_doh1_2020-03-31T22_17_01.945354.pcap -c ./output.csv
reading from file ./MaliciousDoH-dns2tcp-Pcap-001_600/dns2tcp_tunnel_1111_doh1_2020-03-31T22_17_01.945354.pcap, link-type LINUX_SLL (Linux cooked v1)
Packet count: 945
Garbage Collection Began. Flows = 2
/usr/lib/python3/dist-packages/scipy/stats/stats.py:263: RuntimeWarning: The input array could not be properly checked for nan values. nan values will b
e ignored.
warnings.warn("The input array could not be properly checked for nan "
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.8/dist-packages/scapy/sendrecv.py", line 989, in _run
session.on_packet_received(p)
File "/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter/flow_session.py", line 105, in on_packet_received
self.garbage_collect(packet.time)
File "/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter/flow_session.py", line 119, in garbage_collect
data = flow.get_data()
File "/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter/flow.py", line 80, in get_data
'PacketTimeMode': packet_time.get_mode(),
File "/mnt/c/Users/mohit/Onedrive/Desktop/studentjob/project/DOHmeter/DoHlyzer/meter/features/packet_time.py", line 114, in get_mode
mode = float(stat.mode(self._get_packet_times())[0])
File "/usr/lib/python3/dist-packages/scipy/stats/stats.py", line 458, in mode
if a.dtype == object and np.nan in set(a.ravel()):
TypeError: unhashable type: 'EDecimal'

I am getting this error while uploading the pcap file. Can you please help me with this?
This is the link for pcap file I am trying to upload
http://205.174.165.80/CICDataset/DoHBrw-2020/Dataset/MaliciousDoH-dns2tcp-Pcaps/

I've tried Python 2.7, 3.5, 3.6, 3.7, and 3.8 each with its own issues.
Python 2.7, Python 3.5 & Python 3.8 are not relevant - due to syntax and unsatisfied requirements.

All python versions producing the following exception:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/Desktop/DoHlyzer-master/venv36/lib/python3.6/site-packages/scapy/sendrecv.py", line 836, in _run
    *session_args, **session_kwargs)
  File "/home/ubuntu/Desktop/DoHlyzer-master/meter/flow_session.py", line 31, in __init__
    super(FlowSession, self).__init__(None, True, *args, **kwargs)
TypeError: __init__() got multiple values for argument 'prn'

As no use is done with args and kwargs at FlowSession, we can remove the asterisks from the super constructor:
super(FlowSession, self).__init__(None, True, *args, **kwargs)
Not a good solution, but it's working.

After "fixing" the above issue, with Python 3.6 we get the following exception:

Packet count: 10000
Garbage Collection Began. Flows = 191
Garbage Collection Finished. Flows = 191
Packet count: 20000
Garbage Collection Began. Flows = 485
/home/ubuntu/Desktop/DoHlyzer-master/venv36/lib/python3.6/site-packages/scipy/stats/stats.py:269: RuntimeWarning: The input array could not be properly checked for nan values. nan values will be ignored.
  "values. nan values will be ignored.", RuntimeWarning)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/Desktop/DoHlyzer-master/venv36/lib/python3.6/site-packages/scapy/sendrecv.py", line 989, in _run
    session.on_packet_received(p)
  File "/home/ubuntu/Desktop/DoHlyzer-master/meter/flow_session.py", line 105, in on_packet_received
    self.garbage_collect(packet.time)
  File "/home/ubuntu/Desktop/DoHlyzer-master/meter/flow_session.py", line 119, in garbage_collect
    data = flow.get_data()
  File "/home/ubuntu/Desktop/DoHlyzer-master/meter/flow.py", line 80, in get_data
    'PacketTimeMode': packet_time.get_mode(),
  File "/home/ubuntu/Desktop/DoHlyzer-master/meter/features/packet_time.py", line 116, in get_mode
    mode = float(stat.mode(self._get_packet_times())[0])
  File "/home/ubuntu/Desktop/DoHlyzer-master/venv36/lib/python3.6/site-packages/scipy/stats/stats.py", line 469, in mode
    if a.dtype == object and np.nan in set(a.ravel()):
TypeError: unhashable type: 'EDecimal'

With Python 3.7 we have the following exception:

Packet count: 10000
Garbage Collection Began. Flows = 191
Garbage Collection Finished. Flows = 191
Packet count: 20000
Garbage Collection Began. Flows = 485
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/Desktop/DoHlyzer-master/venv37/lib/python3.7/site-packages/scapy/sendrecv.py", line 989, in _run
    session.on_packet_received(p)
  File "/home/ubuntu/Desktop/DoHlyzer-master/meter/flow_session.py", line 105, in on_packet_received
    self.garbage_collect(packet.time)
  File "/home/ubuntu/Desktop/DoHlyzer-master/meter/flow_session.py", line 119, in garbage_collect
    data = flow.get_data()
  File "/home/ubuntu/Desktop/DoHlyzer-master/meter/flow.py", line 78, in get_data
    'PacketTimeMean': packet_time.get_mean(),
  File "/home/ubuntu/Desktop/DoHlyzer-master/meter/features/packet_time.py", line 94, in get_mean
    mean = numpy.mean(self._get_packet_times())
  File "<__array_function__ internals>", line 6, in mean
  File "/home/ubuntu/Desktop/DoHlyzer-master/venv37/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 3420, in mean
    out=out, **kwargs)
  File "/home/ubuntu/Desktop/DoHlyzer-master/venv37/lib/python3.7/site-packages/numpy/core/_methods.py", line 190, in _mean
    ret = ret / rcount
  File "/home/ubuntu/Desktop/DoHlyzer-master/venv37/lib/python3.7/site-packages/scapy/utils.py", line 84, in __truediv__
    return EDecimal(Decimal.__truediv__(self, Decimal(other), **kwargs))
TypeError: conversion from numpy.int64 to Decimal is not supported

Python 3.8 couldn't have worked as well:

ERROR: Could not find a version that satisfies the requirement tensorflow==2.1.0
ERROR: No matching distribution found for tensorflow==2.1.0

Tested on Windows 10 & Ubuntu 20.04
Command-line: (venv3X) ubuntu@ubuntu-vm:~/Desktop/DoHlyzer-master$ python meter/dohlyzer.py -f '/home/ubuntu/Desktop/dump_00005_20200113142226.pcap' -c ./output.csv

It would help if you could write what Python interpreter version you have used.

Tnx :)

When I ryn the meter/dohlyzer.py file I get the error
"" File "./dohlyzer.py", line 8, in
from meter.flow_session import generate_session_class
ModuleNotFoundError: No module named 'meter'
""
I tried to add the meter folder to the packages but I still get errors
Any idea please

C:\Users\hp\Desktop\DoHlyzer-master\meter>python3 dohlyzer.py -n eth0 -c ./output.csv
Traceback (most recent call last):
File "dohlyzer.py", line 8, in
from meter.flow_session import *
ModuleNotFoundError: No module named 'meter'

The import in this file should be
from meter.features.context.packet_direction import PacketDirection not from meter.features.context import packet_direction
and the line 33 should be if direction == PacketDirection.FORWARD:

I am trying to run the Analyzer module from the example command given in the README, but I am getting back the following error -

PS -- I have all the directories cloned and set up just like the github repository.
Kindly help out with this issue. Thanks in advance!