hgn / captcp Goto Github PK
View Code? Open in Web Editor NEWA open source program for TCP analysis of PCAP files
Home Page: http://research.protocollabs.com/captcp/
License: GNU General Public License v3.0
A open source program for TCP analysis of PCAP files
Home Page: http://research.protocollabs.com/captcp/
License: GNU General Public License v3.0
Hi,
I discovered that the way captcp hashes/identifies flows in order to associate them with connections is flawed.
Connections and subflows are identified by a kind of unique identifier/hash, uid
computed in TcpConn.__init__
beginning at line 2273:
l = [ord(a) ^ ord(b) for a,b in zip(self.sipnum, self.dipnum)]
self.uid = "%s:%s:%s" % (
str(self.ipversion),
str(l),
str(long(self.sport) + long(self.dport)))
This identifier is used in various places as an index/key into hashtables (well, dicts :) ) containing connections, and is notably used for associating subflows (uni-directional) with the connection (bi-directional) they are part of. For this purpose, the hash needs to be the same for any given port-pair between the same two addresses, regardless of source/destination ordering. I suppose this is why they are added together, since this is a commutative operation.
The problem is that uid
is used as a direct key into tables, without checking for hash collisions. The above function is obviously not a perfect hash function, and will produce collisions if you are unlucky enough to have two connections with port pairs that add up to the same thing, which corrupts the data for those connections.
I am proposing a patch to rectify this by concatenating the (hex string representations of the) ordered port pair together instead of summation. This preserves the commutativity, but ensures we will not get collisions for the port pair part of the identifier. Note that the hash function overall still isn't perfect, you could get collisions between the IP address pairs, but that is perhaps slightly less likely.
Best wishes,
Kristian
PS. If you wonder how I came across this, I managed to trigger this bug by starting a bunch of connections to sequentially increasing port numbers almost simultaneously under FreeBSD. When FreeBSD has to generate a lot of port numbers in a very short time, it temporarily switches ephemeral port selection strategy to simple sequential incrementation, see: https://www.cymru.com/jtk/misc/ephemeralports.html
Sometimes a pair of connect()
calls would get inversed and I ended up with port pairs such as
Feature request for an option to be able to ignore packets that are corrupt/mangled instead of erroring out like so:
root@host ~/t/g/captcp# ./captcp.py throughput --output-dir /root/traffic-stats/ /media/storage/3-20.pcap
captcp 2010-2013 Hagen Paul Pfeifer and others (c)
http://research.protocollabs.com/captcp/
Traceback (most recent call last):
File "./captcp.py", line 5298, in
sys.exit(captcp.run())
File "./captcp.py", line 5266, in run
pcap_parser = PcapParser(self.pcap_file_path, self.pcap_filter)
File "./captcp.py", line 534, in init
self.pc = dpkt.pcap.Reader(self.pcap_file)
File "/usr/lib/pymodules/python2.7/dpkt/pcap.py", line 103, in init
raise ValueError, 'invalid tcpdump header'
ValueError: invalid tcpdump header
some pcap files captcp failes to parse data.
For example, attached small pcap file with ~300 packets of flow control in rate of
30 packets per second. Running captcp on that file cause following backtrace:
$ captcp throughput --verbose=debug -o vm2 ./vm2.pcap # captcp 2010-2013 Hagen Paul Pfeifer (c) # http://research.protocollabs.com/captcp/ # python: 2.7.5 [releaselevel: final, serial: 0] # pcap file: ./vm2.pcap # call pre_process_packet [1/4] # call pre_process_final [2/4] # call process_packet [3/4] # skip processing step # call pre_process_final [4/4] Traceback (most recent call last): File "/usr/bin/captcp", line 5210, in sys.exit(captcp.run()) File "/usr/bin/captcp", line 5196, in run ret = classinstance.process_final() File "/usr/bin/captcp", line 2820, in process_final timediff = Utils.ts_tofloat(self.end_time - self.start_time) File "/usr/bin/captcp", line 457, in ts_tofloat return float(ts.seconds) + ts.microseconds / 1E6 + ts.days * 86400 AttributeError: 'int' object has no attribute 'seconds' $>
Here is the trace from pcap:
01:58:21.282251 MPCP, Opcode Pause, length 46
01:58:21.312596 MPCP, Opcode Pause, length 46
01:58:21.342912 MPCP, Opcode Pause, length 46
01:58:21.373227 MPCP, Opcode Pause, length 46
.....
01:58:31.225154 MPCP, Opcode Pause, length 46
01:58:31.255468 MPCP, Opcode Pause, length 46
python captcp.py statistic file.txt
Traceback (most recent call last):
File "captcp.py", line 5301, in
sys.exit(captcp.run())
File "captcp.py", line 5269, in run
pcap_parser = PcapParser(self.pcap_file_path, self.pcap_filter)
File "captcp.py", line 534, in init
self.pc = dpkt.pcap.Reader(self.pcap_file)
File "/usr/lib/pymodules/python2.7/dpkt/pcap.py", line 97, in init
self.fh = FileHdr(buf)
File "/usr/lib/pymodules/python2.7/dpkt/dpkt.py", line 75, in __init
raise NeedData
dpkt.dpkt.NeedData
It can be checked with "FileHdr" class in the dpkt module.
captcp is Ethernet centric. Some values like link layer overhead is always calculated with a Ethernet assumption. Since bb00a87 and PPP support this is not true.
There is no easy workaround because the API as based on a Ethernet assumption. Solution: shift packet data argument to packet and not packet.data and provide real link layer access. This needs modification to all modules!
The -t option for timesequence doesn't appear to be working.
captcp timesequence -o timesequence/ -z -f 1.1 -i -t 4:6 *.pcap
The resulting pdf seems to ignore the -t option and includes the entire time sequence (10 seconds in my case).
I'm struggeling with the graphical part. I'm able to parse and analyze capture files, but generating PDF/PNG doesn't work. All packages are installed except texlive-latex-extra, which is not availabe under BSD.
OS:
[root@psrISS001 ~]# uname -a
FreeBSD psrISS001 8.3-RELEASE-p11 FreeBSD 8.3-RELEASE-p11 #0: Tue Sep 10 00:06:55 UTC 2013 [email protected]:/usr/obj/usr/src/sys/GENERIC amd64
[root@psrISS001 /data/HSI-10G/captcp]# captcp timesequence -i -o timesequence -f 1.1 data.pcap
Traceback (most recent call last):
File "/usr/bin/captcp", line 4932, in
sys.exit(captcp.run())
File "/usr/bin/captcp", line 4893, in run
classinstance.initialize()
File "/usr/bin/captcp", line 1275, in initialize
self.parse_local_options()
File "/usr/bin/captcp", line 1515, in parse_local_options
parser.add_option( "-g", "--gnuplot-options", dest="gnuplotoptions", default=none,
NameError: global name 'none' is not defined
any help appreciated !
Regards
Daniel
Hi
I´m getting an error while trying to graph the throughput with captcp on actual ubuntu 12.04
the pcap file generate using ns3 simulator and i would like to graph the output with captcp
root@salma:~# captcp statistic Amrr2.pcap
captcp 2010-2013 Hagen Paul Pfeifer (c)
Traceback (most recent call last):
File "/usr/bin/captcp", line 4364, in
sys.exit(captcp.run())
File "/usr/bin/captcp", line 4332, in run
pcap_parser = PcapParser(self.pcap_file_path, self.pcap_filter)
File "/usr/bin/captcp", line 518, in init
self.pc = dpkt.pcap.Reader(self.pcap_file)
File "/usr/lib/pymodules/python2.7/dpkt/pcap.py", line 105, in init
self.dloff = dltoff[self.__fh.linktype]
KeyError: 105
please could you tell me why ?
View merge request comment for detailed info.
Issues:
Dumps:
all-dumpfiles.zip
Cmds:
./captcp.py spurious-retransmissions -f 1.1 -m summary ~/megadump.pcap
./captcp.py spurious-retransmissions -f 1.1 -m retransmissions ~/megadump.pcap
./captcp.py spurious-retransmissions -f 1.1 -m spurious ~/megadump.pcap
./captcp.py spurious-retransmissions -f 1.1 -m list ~/megadump.pcap
Hi,
when read the source of captcp.py, sack start is left edge, and stop is right edge, also in memory. but the function linear_sackblocks_array() exchange start and stop.
I need to calculate the throughput over a trace contains only UDP packets.
I try the program first but it terminated therefore I check the code and I find that:
In throughput module class, the program check if the packet is part of TCP flow in line (2805). TCP or UDP has no effect in the remain of the code.
can you help me?
I have ONE TCP flow of 64kb download a pcap file. When I run the following command, I get throughput.data but it is empty! Please correct me if I have missed anything.
Let me know if you need more info.
Hello!
Thank you for implementing the captcp.
I have a small question.
I am not sure whether bug or not.
When I was running on captcp about 802.11 pcap files, captcp had a error following as:
captcp 2010-2013 Hagen Paul Pfeifer and others (c)
http://research.protocollabs.com/captcp/
Packet link type not know (105)! Interpret at Ethernet now - but be carefull!
Traceback (most recent call last):
File "captcp.py", line 5225, in
sys.exit(captcp.run())
File "captcp.py", line 5196, in run
pcap_parser.run()
File "captcp.py", line 574, in run
packet = self.decode(pkt)
File "C:\Python27\lib\site-packages\dpkt\dpkt.py", line 75, in init
raise NeedData
dpkt.dpkt.NeedData
As I know the dpkt in python, dpkt have supported 802.11 (i.e. dpkt.ieee80211.IEEE80211). But, PcapParser class in source does not support 802.11.
why does not captcp support 802.11?
please consider the 802.11 pcap.
Thank you in advance.
There is quite a big amount of white space above and below of plotted graphs. Especially when embedding in other documents this is inconvenient. Following some additions to the default makefile to automatically crop graph files:
GNUPLOT_FILES = $(wildcard *.gpi)
PNG_OBJ = $(patsubst %.gpi,%.png, $(GNUPLOT_FILES))
PDF_OBJ = $(patsubst %.gpi,%.pdf, $(GNUPLOT_FILES))
->TEMPFILE := $(shell mktemp)
%.pdf: %.eps
@echo "conversion in pdf format"
->@ps2epsi $< TEMPFILE
->@epstopdf --outfile=$*.pdf TEMPFILE
->@rm TEMPFILE
@echo "end"
Daniel
Guessing this is not a captcp problem, but wondering if you have some tips? Trying to run this on OSX.
$> python captcp.py sequencegraph -v debug -i 1 -l 192.168.0.199 ~/Desktop/archive.pcap
# captcp 2010,2011 Hagen Paul Pfeifer (c)
# python: 2.7.1 [releaselevel: final, serial: 0]
# pcap file: /Users/igrigorik/Desktop/archive.pcap
# visualization limited to the following connections: ['1']
# call pre_process_packet [1/4]
# call pre_process_final [2/4]
# now draw 974 packets
# call process_packet [3/4]
# call pre_process_final [4/4]
Traceback (most recent call last):
File "captcp.py", line 3753, in <module>
sys.exit(captcp.run())
File "captcp.py", line 3740, in run
ret = classinstance.process_final()
File "captcp.py", line 1848, in process_final
self.cr.show_page()
MemoryError
Cairo issue / bug? pcap file: http://dl.dropbox.com/u/2132185/archive.pcap
Actually, if I have 3 flows and want to plot their throughput, I need to run 3 times captcp, using each time a different -f parameter .
What about adding a range capability for the -f parameter ? The output .dat file will be something like
Time Throughput flow 1 Throughput flow 2
0.0 0.0 0.0
1.0 100.0 85.0
2.0 120.0 95.0
.....
Thank you
When using the t-range of the timesequence module, the arrows in plots are not bounded by the gnuplot y-range and thus be drawn in the whole plot outside of the axes.
$ captcp statistic 500M.pcap
# captcp 2010-2013 Hagen Paul Pfeifer and others (c)
# http://research.protocollabs.com/captcp/
Traceback (most recent call last):
File "/usr/bin/captcp", line 6054, in <module>
sys.exit(captcp.run())
File "/usr/bin/captcp", line 6040, in run
ret = classinstance.process_final()
File "/usr/bin/captcp", line 4993, in process_final
self.process_final_data()
File "/usr/bin/captcp", line 4987, in process_final_data
self.rexmt_final(connection.sc1)
File "/usr/bin/captcp", line 4671, in rexmt_final
sc.user_data["tl-ps-avg"] = "%.2f" % numpy.mean(sc.user_data["_tl_pkt_sizes"])
AttributeError: 'NoneType' object has no attribute 'mean'
I've read the doc again and again but found nothing. Then I looked into the source code and seems that this feature is not provided?
Please consider migrating the application to python 3: python 2 is EOL from January 1st 2020 on.
When running captcp with --init for creating gnuplots, I've encountered an error that is raised due to the TemplateContainer instance not having an attribute name. I've traced this issue to the init_db() method in TemplateMod. There is possibly an erroneous if-statement creating attributes name and type on TemplateContainers. I tried adding name and type always (empty strings if neither of the conditions are true) and this seemed to fix the problem.
I'm using python version 2.7.1, and the commit I'm currently on is 0eb9fdf (which I think is HEAD).
== OUTPUT ==
$ captcp inflight -v debug -f 1.1 send.pcap -o test -i
Traceback (most recent call last):
File "/usr/bin/captcp", line 3749, in
sys.exit(captcp.run())
File "/usr/bin/captcp", line 3711, in run
classinstance.pre_initialize()
File "/usr/bin/captcp", line 2417, in pre_initialize
self.create_gnuplot_environment()
File "/usr/bin/captcp", line 2471, in create_gnuplot_environment
fd.write("%s" % (TemplateMod().get_content_by_name("inflight")))
File "/usr/bin/captcp", line 844, in get_content_by_name
if i.name == name:
AttributeError: TemplateContainer instance has no attribute 'name'
== Possible fix? ==
Adding the following on line 880
else
tc.name = ""
tc.type = ""
The pypcap module seems to be abandoned, and it is nearly impossible to install it on Fedora. What about moving to something more widespread and maintained?
I am using captcp to extract tcp throughput for point to point protocol links (PPP) from pcap files. Now using same captcp, I am not able to extract WiFi 802.11ac pcap files which are of the type IEEE_11_RADIO (802.11 plus radiotap header).
I have tried to add a line into captcp.py as:
dpkt.pcap.DLT_IEEE802_11_RADIO: dpkt.ethernet.Ethernet
But still it is generating error and I do not understand what I need to do.
Please find attached out.pcap files.
Please help me in this issue.
Thanks!!
Hi,
I´m getting an error while trying to graph the throughput with captcp on actual Kubuntu 13.04 64bit. I would fix this mysef if i could write python but I can´t so here is the output for you:
floh1111@floh1111-VirtualBox:~/Rechnernetze/ns-allinone-3.16/ns-3.16$ captcp throughput -s 0.1 -i -f 45.2 -o throughput-graph/ meine_aufgabe-5-2.pcap
# captcp 2010-2013 Hagen Paul Pfeifer (c)
# Packet link type not know (9)! Interpret at Ethernet now - but be carefull!
# Packet link type not know (9)! Interpret at Ethernet now - but be carefull!
Traceback (most recent call last):
File "/usr/bin/captcp", line 4345, in <module>
sys.exit(captcp.run())
File "/usr/bin/captcp", line 4331, in run
ret = classinstance.process_final()
File "/usr/bin/captcp", line 2711, in process_final
timediff = Utils.ts_tofloat(self.end_time - self.start_time)
File "/usr/bin/captcp", line 455, in ts_tofloat
return float(ts.seconds) + ts.microseconds / 1E6 + ts.days * 86400
AttributeError: 'int' object has no attribute 'seconds'
I´m running a plain installation in an virtual machine but I think I installed all needed packages: sudo apt-get install python-dpkt python-numpy make gnuplot texlive-latex-extra mupdf
Regards
Floh1111
Hello people. When I use the module " TIMESECUENCE " I keep everything in a folder, that folder and income to do "make all" or "make png" displays on screen " compillation of time- sequence.gpi " and stays down waiting indefinitely . I check the time- code sequence.gpi and me is because the lines :
1. load " data- arrow.data "
2. load " data- arrow- retrans.data "
3. load " data- arrow- sack.data "
4. load " data- arrow- push.data "
5. load " data- arrow- ece.data "
6. load " data- arrow- cwr.data "
If I comment that the scrip works, but do not think all that well as prints 3 equal lines .
Someone happens the same? What could we be doing?
Thanks , I hope your help please
Comands:
mkdir time
captcp timesequence -i --zero -e -o time -f 1.1 eth1G2.pcap
cd time
make all
./captcp.py animation test.pcap -f 13.1 --output-dir foo
# captcp 2010-2013 Hagen Paul Pfeifer (c)
# http://research.protocollabs.com/captcp/
# connection: 13 (local flow: 1, remote flow: 2)
Traceback (most recent call last):
File "./captcp.py", line 5210, in <module>
sys.exit(captcp.run())
File "./captcp.py", line 5191, in run
pcap_parser.run()
File "./captcp.py", line 562, in run
self.callback(dt, packet.data)
File "./captcp.py", line 4360, in process_packet
self.process_local_side(ts, packet, tpi)
File "./captcp.py", line 4302, in process_local_side
(td, len(packet) / 2, self.rtt_data["twh-delay"] / self.acceleration))
KeyError: 'twh-delay'
Recent version of captcp fails to analyze pcap file.
Version 1.7 works fine so possibly this issue was introduced by recent commit c18793c
"Consolidate logging.getLogger() calls"
$ captcp throughput -i -o test ./test.pcap # captcp 2010-2013 Hagen Paul Pfeifer and others (c) # http://research.protocollabs.com/captcp/ Traceback (most recent call last): File "/usr/bin/captcp", line 5220, in sys.exit(captcp.run()) File "/usr/bin/captcp", line 5181, in run classinstance.initialize() File "/usr/bin/captcp", line 2702, in initialize self.create_gnuplot_environment() File "/usr/bin/captcp", line 2652, in create_gnuplot_environment tmpl = string.Template(TemplateMod().get_content_by_name("throughput")) File "/usr/bin/captcp", line 1014, in __init__ self.init_db() File "/usr/bin/captcp", line 1041, in init_db self.logger.debug("initialize local template database") AttributeError: TemplateMod instance has no attribute 'logger'
Pcap file is a capture of few SSH packets from the network.
Captcp v1.7 analyzes same capture successfully.
Fix can be:
diff --git a/captcp.py b/captcp.py index d3bf33f..930d69e 100755 --- a/captcp.py +++ b/captcp.py @@ -1011,6 +1011,7 @@ class TemplateMod(Mod): def __init__(self): + Mod.__init__(self) self.init_db()
Hi,
When I run the following cmd: (captcp 1.7)
captcp throughput -o out -i tcp-large-transfer-0-0.pcap
I got this error:
# captcp 2010-2013 Hagen Paul Pfeifer and others (c)
# http://research.protocollabs.com/captcp/
Traceback (most recent call last):
File "/usr/bin/captcp", line 5269, in <module>
sys.exit(captcp.run())
File "/usr/bin/captcp", line 5228, in run
classinstance.initialize()
File "/usr/bin/captcp", line 2710, in initialize
self.create_gnuplot_environment()
File "/usr/bin/captcp", line 2663, in create_gnuplot_environment
gpi_cmd = tmpl.substitute(TITLE=title,
UnboundLocalError: local variable 'title' referenced before assignment
File can be found at https://goo.gl/AWATmN
$captcp statistic file.pcap
Traceback (most recent call last):
File "/usr/bin/captcp", line 5868, in
sys.exit(captcp.run())
File "/usr/bin/captcp", line 5854, in run
ret = classinstance.process_final()
File "/usr/bin/captcp", line 4807, in process_final
self.process_final_data()
File "/usr/bin/captcp", line 4801, in process_final_data
self.rexmt_final(connection.sc1)
File "/usr/bin/captcp", line 4485, in rexmt_final
sc.user_data["tl-ps-avg"] = "%.2f" % numpy.mean(sc.user_data["_tl_pkt_sizes"])
AttributeError: 'NoneType' object has no attribute 'mean'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.