rshk / python-pcapng Goto Github PK

Because of the sanity check at structs.py:258, we get an OSError: [Errno 29] Illegal seek when we try to use FileScanner with a pipe (e.g. Popen(tshark -w -)).

When iterating over pcapng blocks using PcapngReader, the reader will fail when an InterfaceStatistics or NameResolution block is encountered:

  File "/foo/pcapng/reader.py", line 216, in _parse_block
    return NameResolution.unpack(blk.contents, self._endianness)
AttributeError: type object 'NameResolution' has no attribute 'unpack'

I was having issues with the timestamp field losing accuracy when being calculated in a single line. I haven't looked much more in to it, but I needed to patch with the code below to get the accuracy back after calculation. I'm working with nanosecond resolution timestamp captures.

    @property
    def timestamp(self):
        # First, get the accuracy from the ts_resol option
        high_scaled = (self.timestamp_high << 32) * self.timestamp_resolution
        low_scaled  = (self.timestamp_low) * self.timestamp_resolution
        scaled = high_scaled + low_scaled
        return scaled

with open('file.pcap') as fp:
scanner = FileScanner(fp)
for block in scanner:
pass

running:
raise ValueError("File not starting with a proper section header") -- yield self._read_next_block()

hi

we have several appliances that produces output similar to "tcpdump -x" hexdumps.
they can contain several interfaces at once, when using text2pcap infos are lost,
like: direction / captured interface.

timeformat can be from start of capture or full time
sample:

0.367441 wan1 in 54.239.26.180.443 -> 192.168.13.31.58723: ack 1152503568
0x0000 0000 0000 0001 f4ec 38fc 24c2 0800 4500 ........8.$...E.
0x0010 0028 cc67 4000 d906 b5fd 36ef 1ab4 c0a8 .([email protected].....
0x0020 0d1f 01bb e563 9336 03f3 44b1 cf10 5010 .....c.6..D...P.
0x0030 012c fd33 0000 .,.3..

0.367837 internal7 out 54.239.26.180.443 -> 10.255.86.140.58723: ack 1152503568
0x0000 0000 0000 0000 085b 0eaa 66e1 0800 4500 .......[..f...E.
0x0010 0028 cc67 4000 d806 233a 36ef 1ab4 0aff .(.g@...#:6.....
0x0020 568c 01bb e563 9336 03f3 44b1 cf10 5010 V....c.6..D...P.
0x0030 012c 6970 0000

Steps:
Attempt to load attached pcap file pcap_with_error.zip
I am using something like bellow
scanner = pcapng.scanner.FileScanner(fp)
for block in scanner:
pass

The attached pcap is not 100% according with the pcapng spec. The library attempts to log following warning: "repeated option 2 'if_name' not permitted by pcapng spec"

Instead, we will see a crash:

for block in scanner:

File "C:\Users\cacatana.KEYSIGHT\AppData\Local\Programs\Python\Python37\lib\site-packages\pcapng\scanner.py", line 46, in iter
yield self._read_next_block()
File "C:\Users\cacatana.KEYSIGHT\AppData\Local\Programs\Python\Python37\lib\site-packages\pcapng\scanner.py", line 62, in _read_next_block
block = self._read_block(block_type)
File "C:\Users\cacatana.KEYSIGHT\AppData\Local\Programs\Python\Python37\lib\site-packages\pcapng\scanner.py", line 90, in _read_block
blocks.KNOWN_BLOCKS[block_type], raw=data
File "C:\Users\cacatana.KEYSIGHT\AppData\Local\Programs\Python\Python37\lib\site-packages\pcapng\blocks.py", line 207, in new_member
blk = cls(section=self, endianness=self.endianness, **kwargs)
File "C:\Users\cacatana.KEYSIGHT\AppData\Local\Programs\Python\Python37\lib\site-packages\pcapng\blocks.py", line 148, in init
super(SectionMemberBlock, self).init(**kwargs)
File "C:\Users\cacatana.KEYSIGHT\AppData\Local\Programs\Python\Python37\lib\site-packages\pcapng\blocks.py", line 50, in init
self.schema, io.BytesIO(kwargs["raw"]), kwargs["endianness"]
File "C:\Users\cacatana.KEYSIGHT\AppData\Local\Programs\Python\Python37\lib\site-packages\pcapng\structs.py", line 1033, in struct_decode
decoded[name] = field.load(stream, endianness=endianness, seen=decoded)
File "C:\Users\cacatana.KEYSIGHT\AppData\Local\Programs\Python\Python37\lib\site-packages\pcapng\structs.py", line 377, in load
return Options(schema=self.options_schema, data=options, endianness=endianness)
File "C:\Users\cacatana.KEYSIGHT\AppData\Local\Programs\Python\Python37\lib\site-packages\pcapng\structs.py", line 731, in init
self._update_data(data)
File "C:\Users\cacatana.KEYSIGHT\AppData\Local\Programs\Python\Python37\lib\site-packages\pcapng\structs.py", line 823, in _update_data
"repeated option {} not permitted by pcapng spec".format(name)
File "C:\Users\cacatana.KEYSIGHT\AppData\Local\Programs\Python\Python37\lib\site-packages\pcapng\strictness.py", line 38, in warn
if strict_level > Strictness.NONE:
TypeError: '>' not supported between instances of 'Strictness' and 'Strictness'

This is happening because the Strictness is derived from Enum instead of IntEnum. Operators like < > work only with IntEnum.

Hi! I was wondering if there was any interest in adding type annotations to this library.

I saw on README that python2 is still supported, so this eliminates the option of having inline type annotations. As an alternative, we could have use type comments:

# add.py

def add(a, b):
    # type (int, int) -> int
    return a + b

Or make seperate .pyi type stub files:

# add.pyi

def add(a: int, b: int) -> int: ...

These files could be stored in this repository, or stored in typeshed. which allows type checkers access to type stubs for lots of standard and 3rd party libraries.

Let me know what you guys think! I dont mind putting in the effort, just thought I might ask before I get anything started.

For a use-case at work, we need support for essentially skipping an entire section in a FileScanner based on some information in the section header block. The fastest way to do this is to seek the underlying stream if it supports it. (We know ours will). FileScanner caches the header for the current section and we cannot modify this cache externally. All we're left with is manually shoveling blocks until the next section header (unless we want to come up with some scheme that opens a file pointer, allocates a scanner and somehow closes the scanner whenever we want to skip ahead).

We've looked into how to do this and are willing to open a PR to address this by exposing a skip_section method on FileScanner. skip_section would check if the underlying stream is seekable and if so, seek to the correct offset. If not, we shovel blocks until we come to a SectionHeaderBlock. We need to cache this section header block so that we can return it in _read_next_block (or directly in __iter__) if it exists (because we'll have already read the header and the stream is not seekable).

Does this seem reasonable? Is this a feature that you'd be willing to support?

I'm using dumpcap to write a file. I have been unsuccessful in getting python-pcapng to "follow" the growing file (it races to the "current" end-of-file and stops) Please provide the capability (or an example if I'm just missing how to) pause for more data rather than assume EOF.

Ideally, dumpcap would write to a named pipe (fifo), and python-pcapng would read the fifo but dumpcap refuses to do that.

edit: I've since learned that more modern versions of dumpcap can write to both a pipe or a named pipe.

Looks like travis CI discontinued their free plan, as a result CI hasn't been running.
We should probably migrate to an alternative (CircleCI / GitHub actions?)

Traceback (most recent call last):
File "E:/Master_Paper/sanchr/python-pcapng-master/examples/dump_pcapng_info.py", line 16, in
dump_information(scanner)
File "E:/Master_Paper/sanchr/python-pcapng-master/examples/dump_pcapng_info.py", line 10, in dump_information
for block in scanner:
File "C:\Python27\lib\site-packages\pcapng\scanner.py", line 40, in iter
yield self._read_next_block()
File "C:\Python27\lib\site-packages\pcapng\scanner.py", line 45, in _read_next_block
block_type = self._read_int(32, False)
File "C:\Python27\lib\site-packages\pcapng\scanner.py", line 108, in _read_int
endianness=self.endianness)
File "C:\Python27\lib\site-packages\pcapng\structs.py", line 48, in read_int
data = read_bytes(stream, size_bytes)
File "C:\Python27\lib\site-packages\pcapng\structs.py", line 143, in read_bytes
data = stream.read(size)
AttributeError: 'str' object has no attribute 'read'

I'm using pcapng to create test inputs for another system, and I tried to write a big-endian pcapng to make sure that big-endian files are handled correctly. I discovered that the block type and block length are written with native endianness even when the non-native endianness is requested by setting the endianness on the section header block.

I will try to find some time to put together a PR fixing this.

Line 108 in blocks.py is not compatible with py3: interface_id = self._interfaces_id.next()

It should be interface_id = next(self._interfaces_id) if it is to work with python3.

edit: I see in this Github repo this has already been fixed. It doesn't seem to have been pushed to PyPI though?

Sever code

#-*- coding: utf-8 -*- 
#1번줄 지우지 말것. 한글 입력을 위한 필수 라인임

import socket               # Import socket module 라이브러리 삭제하지마시오
import os		    # filesize 알아내기 위한 라이브러리 삭제하지마시오
import struct
from thread import *
import threading

Path = 'C:\\File\\T_REX_FBX.fbx'
print 'Sever Start'

def getFileSize():
	"""
	파일 사이즈 알아내는 함수
	"""
	#size = os.path.getsize("./Assets/TocusTest00.fbx")
	size = int(os.path.getsize(Path))
	#return c_int(size).value #c 체계의 int 형태로 변환해주는 코드 필요하면 밑의 return 지우고 이 부분을 사용하셈
	return size


def Filethreaded(c):
        size = getFileSize()
        print 'size', size
        c.send(struct.pack("i", size))

        f = open(Path, 'rb')
        print 'File_Sending...'
        l = f.read(5000)
        while (l):
            print 'Sending...'
            c.sendall(l)
            l = f.read(5000)
        f.close()
def Main():
    host = '192.168.0.18'
    port = 10002
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.bind((host, port))
    print 'socket binded to post', port
    s.listen(5)
    print 'socket is listening'

    c, addr = s.accept()
    print 'Got connection from', addr

    start_new_thread(Filethreaded, (c,))
    s.close()

if __name__ == '__main__':
    Main()


Client code
#-*- coding: utf-8 -*-

from socket import *
import os, sys

HOST = '192.168.0.18'
PORT = 10002
ADDR = (HOST, PORT)
BUFF_SIZE = 5000

clientSocket = socket(AF_INET, SOCK_STREAM)
clientSocket.connect(ADDR)

### serverFile receive ###
serverData = clientSocket.recv(BUFF_SIZE)

serverFile = 'C:\\File\\SocketTest\\SocketTest.fbx'
with open(serverFile, 'wb') as f:
    print("serverFile opened...")
    count = int(int(serverData.decode("utf-8","ignore")) / BUFF_SIZE + 1)
    while count:
        data = clientSocket.recv(BUFF_SIZE)
        f.write(data)
        count -= 1
    f.close()
print("file received complete")

clientSocket.close()
print('connection closed')
sys.exit()

It should open pcapng files as binary files not text files add 'rb'

with open('/tmp/mycapture.pcap', 'rb') as fp:

Seems like there is some incompatibility, since Python3.9 deprecated fractions.gcd in favor of math.gcd function, see the stacktrace below

WARNING: No route found for IPv6 destination :: (no default route?). This affects only IPv6
WARNING: can't import layer ipsec: cannot import name 'gcd' from 'fractions' (/usr/lib/python3.9/fractions.py)
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.9/site-packages/pcapng/structs.py", line 562, in _get_raw
    return self.raw_data[_name][0]
KeyError: 2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/user/.local/lib/python3.9/site-packages/hytera_homebrew_bridge-2021.1-py3.9.egg/hytera_homebrew_bridge/tests/pcap_test.py", line 270, in <module>
    pprint_enhanced_packet(block)
  File "/home/user/.local/lib/python3.9/site-packages/hytera_homebrew_bridge-2021.1-py3.9.egg/hytera_homebrew_bridge/tests/pcap_test.py", line 137, in pprint_enhanced_packet
    col256(str(block.interface.options["if_name"]), fg="140"),
  File "/home/user/.local/lib/python3.9/site-packages/pcapng/structs.py", line 490, in __getitem__
    return self._get_converted(name)
  File "/home/user/.local/lib/python3.9/site-packages/pcapng/structs.py", line 574, in _get_converted
    value = self._get_raw(name)
  File "/home/user/.local/lib/python3.9/site-packages/pcapng/structs.py", line 564, in _get_raw
    raise KeyError(name)
KeyError: 'if_name'

Is it possible to release a new python-pcapng official package?
Write pcapng functionality is not available in current pkg, but seems to work fine what is on master.

The use case is rewriting the timestamp for each packet - the source being either a pcap or a pcap-ng file.

Hi guys,
i need to parse beacon packets and i want to know how to do. For example, i want to understand how to get something like this or something like this. How can i do with python-pcapng?

I can't see an example starting from a pcap file!

Hi

files generated from text2pcap not suppored

raises:
File "/usr/local/lib/python2.7/dist-packages/pcapng/structs.py", line 538, in _get_raw
raise KeyError(name)
KeyError: 'if_name'

Section version: 1.0 - Little endian - unspecified size -- opt_comment: Generated from input file out.txt. shb_userappl: text2pcap (Wireshark) 2.2.7 (Git Rev Unknown from unknown)
Interface #0 Link type: 1 D/I/X and 802.3 Ethernet Snap length: 65535 -- if_tsresol:

It would be great to have a test suite of raw pcapng files to read by the library and compare expected results:

• load files by URL and/or locally
• run each file
• report results

Is there any standard test suite for this ?

I found some files there:

http://wiki.wireshark.org/Development/PcapNg