netsniff-ng / netsniff-ng Goto Github PK
View Code? Open in Web Editor NEWA Swiss army knife for your daily Linux network plumbing.
Home Page: http://netsniff-ng.org
License: Other
A Swiss army knife for your daily Linux network plumbing.
Home Page: http://netsniff-ng.org
License: Other
////////////////////////////////////////////////////////////////////////////// netsniff-ng - the packet sniffing beast \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ . . netsniff-ng is a free, performant /( )\ Linux network analyzer and .' {______} '. networking toolkit. If you will, \ ^, ,^ / the Swiss army knife for network |'O\ /O'| _.<0101011>-- packets. > `' '` < / ) ,.==., ( | Web: http://netsniff-ng.org .-(|/--~~--\|)-' ( ___ The gain of performance is \__.=|___E reached by built-in zero-copy mechanisms, so that on packet reception and transmission the kernel does not need to copy packets from kernel space to user space, and vice versa. The netsniff-ng toolkit's primary usage goal is to facilitate a network developer's / hacker's daily Linux plumbing. It can be used for network development, debugging, analysis, auditing or network reconnaissance. It consists of the following fixed set of utilities: * netsniff-ng: a zero-copy packet analyzer, pcap capturing/replaying tool * trafgen: a multithreaded low-level zero-copy network packet generator * mausezahn [*]: high-level packet generator for appliances with Cisco-CLI * ifpps: a top-like kernel networking and system statistics tool * curvetun [*]: a lightweight curve25519-based multiuser IP tunnel * astraceroute: an autonomous system trace route and DPI testing utility * flowtop: a top-like netfilter connection tracking tool * bpfc: a [seccomp-]BPF (Berkeley packet filter) compiler, JIT disassembler Note that tools marked with [*] should be considered as experimental for now, and not used in production environments as they still need more work to be fully stable and in line with others. You have been warned! Each release can be verified with Git and GPG, here are the steps to do so: 1) Import the maintainers public keys: git show maint-tklauser-pgp-pub | gpg --import git show maint-dborkman-pgp-pub | gpg --import 2) Verify the Git tag: git tag -v <tag-name> Carefully read the INSTALL document for the next steps in building netsniff-ng. Note that the toolkit is still quite young and under heavy development, not yet feature complete and in a quality level where we're satisfied with (i.e. for mausezahn). However, we're on a good way towards tackling all these goals. The netsniff-ng toolkit is an open source project covered by the GNU General Public License, version 2.0. For any questions or feedback about netsniff-ng you are welcome to leave us a message at <[email protected]>. netsniff-ng is non-profit and provided in the hope, that it is found useful. The current project status can be considered as "working". In general, all tools have been tested by us to a great extend including their command-line options. In fact, many of our tools are used in a lot of production systems. However, we give no guarantee that our tools are free of bugs! If you spot some issues, contact us as described in REPORTING-BUGS. Also, have a look at our online FAQ for answering your questions. This project has received support from companies and institutions listed in the according section in the AUTHORS file. Thanks for contributing, we're thrilled to provide you with netsniff-ng! Happy packet hacking!
Using pthread-based multi-threading might make things easier to spread the load over the CPU thus achieving better performance.
It would also make the first step towards more flexibility such as the possibility to pause/resume capture or replay in real-time.
First of all, a proof-of-concept has to be prepared, tested and benchmarked.
In debugging curvetun, I'm kicking off the client and receiving no error status whatsoever... nothing in /var/log/messages... Aside from "curvetun client booting!" But clearly my curvetun client is dying for some reason.
Turns out the panic() calls in xio.c are not getting to me... I'm wondering if stderr has been redirected? Changing the panic() calls to syslog_panic() within xio.c is helping me along - now I see my problem has to do with the ioctl call. Should these calls be implemented with syslog_panic()? I tried flushing the stderr after the panic() call in question, no dice (not getting output to terminal).
Evaluate the support of ERF packet format to capture/read/replay it
Manpages + Readme's needs to be written!
When compiling without the lib the compiler can't find the include files. Error:
[ 87%] Building C object flowtop/CMakeFiles/flowtop.dir/__/flowtop.c.o
.../temp/netsniff-ng/src/flowtop.c:37:18: fatal error: urcu.h: No such file or directory
Integrate cmake with contrib/devel/doc/kernel-doc in order to generate a source documentation.
libGeoIP must access a file which contains the database of the location of each IP address.
There is a free version which can be downloaded from the project website and it is also present in the package management of the all big Linux distribution.
It is also tricky to come up with a "one-size-fits-all" solution because such databases can be installed anywhere on the target but also different versions are delivered between different distributions.
For instance, Debian seems to deliver a GeoIP city database whereas Ubuntu delivers the country edition.
trafgen 0.5.8-rc2+ (Ziggomatic), Git id: v0.5.8-rc2-26-gc2887b3
I'm having two separate issues.
I'm having an issue with trafgen where the only way I can get it to generate the exact number of packets specified by --num is if I do this AND by issuing it to the shell in contrast from using the same command multiple times in script:
$ trafgen --in configs/64.cfg --out eth2 --verbose --kernel-pull 5 --cpus 1 --num 15000000
--kernel-pull 6 works as well.
I've tried various kernel pulls, CPU, and ring size combinations and found that the
above is the only one where my system will generate all 15000000 packets.
I'm reporting no tx loss on the sending host. I've been watching /proc/net/dev and ethtool and do not see any tx discards or drops.
If I launch trafgen with the same arguments and options from a shell script or atd job
the number of packets actually sent vary greatly. For my tests I'm running trafgen
every 3 minutes. The script is executed via crond on the sending host every hour.
sleep 10
for i in {0..29..3}
do
echo "trafgen --in configs/64.cfg --out eth2 --verbose --kernel-pull 5 --cpus 1 --num 15000000" | at now +${i}min
done
The receiving and sending machine's /proc/net/dev packet column
show the following values. 1 for each of 10 runs per tool is listed.
I'm not sure why the first trafgen instance for each tool sends the full
15000000 packets and the rest slack off.
root@recv:~/64# grep ^eth2: *.stat | awk '{ print $1, $3 }'
daemonlogger.stat:eth2: 15000000
daemonlogger.stat:eth2: 13691387
daemonlogger.stat:eth2: 13441450
daemonlogger.stat:eth2: 13884259
daemonlogger.stat:eth2: 13757665
daemonlogger.stat:eth2: 13714474
daemonlogger.stat:eth2: 13581056
daemonlogger.stat:eth2: 13530621
daemonlogger.stat:eth2: 13562485
daemonlogger.stat:eth2: 13460224
dumpcap.stat:eth2: 15000000
dumpcap.stat:eth2: 13984965
dumpcap.stat:eth2: 13974693
dumpcap.stat:eth2: 13772195
dumpcap.stat:eth2: 13914422
dumpcap.stat:eth2: 14044146
dumpcap.stat:eth2: 13782769
dumpcap.stat:eth2: 13813973
dumpcap.stat:eth2: 13869773
dumpcap.stat:eth2: 13756627
...
Here's two tests to demonstrate the difference between the # packets sent on my machine and the the trafgen options used. The /proc/net/dev output below is on the sending machine from which trafgen is running.
$ trafgen --in configs/64.cfg --out eth2 --verbose --kernel-pull 5 --cpus 1 --num 15000000
eth2: 0 0 0 0 0 0 0 0 1020000000 15000000 0 0 0 0 0 0
$ trafgen --in configs/64.cfg --out eth2 --verbose --num 15000000
eth2: 0 0 0 0 0 0 0 0 1016840720 14953540 0 0 0 0 0 0
Here's a diff between the ethtool output of each:
$ diff good.ethtool.txt bad.ethtool.txt
28c28
tx_octets: 1016840720
53c53
< tx_ucast_packets: 15000000
tx_ucast_packets: 14953540
65,66c65,66
< dma_readq_full: 5006163< dma_read_prioq_full: 0
dma_readq_full: 104878 dma_read_prioq_full: 235
68,72c68,72
< ring_set_send_prod_index: 15000000
< ring_status_update: 1143980
< nic_irqs: 488244
< nic_avoided_irqs: 655736< nic_tx_threshold_hit: 1118132
ring_set_send_prod_index: 14952642 ring_status_update: 2150811 nic_irqs: 367275 nic_avoided_irqs: 1783536 nic_tx_threshold_hit: 2124554
The following output can be seen on x86 build:
# ./build_nacl.sh ~/nacl
Building NaCl (this takes a while) ...
readlink: missing operand
Try `readlink --help' for more information.
dirname: missing operand
Try `dirname --help' for more information.
readlink: missing operand
Try `readlink --help' for more information.
dirname: missing operand
Try `dirname --help' for more information.
Path for linking:
Path for including:
Please input the path where NaCl is like the following:
././nacl_path.sh.sh <include_path> <lib_path>
Done!
$ uname -a
Linux thinkbox 3.2.11 #1 SMP Fri Mar 16 13:00:39 CET 2012 i686 GNU/Linux
The script seems to work fine on x86_64. Current stable Debian was used.
Flowtop will be a top-like tool to track netfilter connections and extract statistics out of it.
@borkmann, Could you please evaluate the feature needed to implement flowtop in the toolkit?
Well... I've tried to configure and make it work, but it's not working and the error is as follows:
Building netsniff-ng toolkit (0.5.8-rc1) for x86_64-linux-gnu:
Building netsniff-ng:
-e CC dissector.c
In file included from dissector.h:18:0,
from dissector.c:16:
pcap_io.h: In function ‘pcap_devtype_to_linktype’:
pcap_io.h:210:7: error: ‘ARPHRD_IEEE802154_MONITOR’ undeclared (first use in this function)
pcap_io.h:210:7: note: each undeclared identifier is reported only once for each function it appears in
make: *** [netsniff-ng/dissector.o] Error 1
Thanks for the tool!
Request for an cmdline option to disable all logging for the curvetun VPN, so that it cannot be determined which user signed on on a server. This could improve anonymity.
See if it is possible to use gcov to spot where the code is most used and where it is almost used none for testing purpose
SCTP, DCCP at least and improvements for the current ones, i.e. --less output.
Ifpps throws an error after a few lines when I use a 10 second interval.
ifpps --dev eth2 --csv --loop --interval 10000
ifpps: ifpps.c:539: stats_diff: Assertion `!((new->rx_bytes - old->rx_bytes) > (new->rx_bytes))' failed.
The code makes the strong assumption that we are working on Ethernet hardware with IP on it.
This code needs to be more generic at this place and detect which kind of hardware is which interface
See, if it possible to implement an anonymous mode where packets would be anonymized while capturing
Hi,
When I make traffic capture via the ethernet interface, the pcap file is created and packets are received but without 802.1Q header. If I use tshark, I get all headers but netsniff-ng remove 802.1Q headers.
I compiled netsniff-ng 0.5.7 from source.
Is that normal behavior ?
Thanks,
Guillaume
Test framework would allow us to quickly write some tests to perform sanity checks.
Such tests would be very useful to spot regressions and could be also used to validate a release.
For now, I see two serious possibilty:
Ctest integrates nicely with CMake and it is a very powerful tool.
By writing small test programs, specific functions can be isolated and fully tested.
Shareness has simplicity on its side and would be an extra tool to integrate. It has a clear output syntax (TAP) and it very efficient to test actions on larger scale.
Maybe the two can be combined to cover specific testing and more generic testing
Take a look at the locality concepts presented in hwloc and see if they
can be leveraged to have more control over CPU cores like in
the context of hyper threading.
I think I've hit an issue with netsniff-ng release 0.5.6 generating corrupt pcaps. I have two Amazon EC2 virtual machines running, one with Linux and one with Windows. I was running an HTTP proxy and netsniff-ng on the Linux VM with the following command line:
$ sudo netsniff-ng --in eth0 --out dump.pcap --silent --dump-path . --filter bpf.txt
After browsing for a couple minutes from the Windows VM, I had two 3+ MB pcaps, both of which I copied to dropbox:
http://dl.dropbox.com/u/70279462/dump_00000.pcap
http://dl.dropbox.com/u/70279462/dump_00001.pcap
I initially thought it was an xplico problem since xplico did not recognize most of the sessions within the pcaps. However, when I tried to open with Wireshark, I got an error. The capinfos utility has a clear explanation of the issue:
$ capinfos dump_00000.pcap
capinfos: An error occurred after reading 135 packets from "dump_00000.pcap": File contains a record that's not valid.
(pcap: File has 5242946-byte packet, bigger than maximum of 65535)
$ capinfos dump_00001.pcap
capinfos: An error occurred after reading 48 packets from "dump_00001.pcap": File contains a record that's not valid.
(pcap: File has 5242946-byte packet, bigger than maximum of 65535)
Let me know if you are struggling to repro/test fix and I can give you access to my VM's. Thanks.
Jonathan
Check how we could make the rx ring functionality so generic that it could also process packets from other hardware layer such as bluetooth, IRDA, token ring etc.
Our tools are mostly based on PF_PACKET sockets which need elevated privileges in order to work properly.
As a workaround, you can run the tool using sudo but it gives privileges to the whole process.
We have to find a way to give the process the needed permissions just to open PF_PACKET sockets.
Put sharness into netsniff-ng/src/t/ with some basic tests.
A new switch has been added with 65d0aa1 to allow the user to capture Super Jumbo Frames properly by making the frame size equal 64kB.
However, when the user does not use this option and capture a packet larger than 2kB, his capture PCAP file will be corrupted.
To avoid this problem, two possibilities:
ifpps, show the 't' value (== interval) in ncurses.
I send the same # of packets each time (15000000) and tcpdump reports the correct amount while netsniff-ng reports an extra 2 or 4 packets.
rx: 972.748 MiB 15000000 pkts 0 drops 0 errors
tx: 0.000 MiB 0 pkts 0 drops 0 errors
$ tcpdump -nni eth2 -w /dev/null
^C15000000 packets captured
15000000 packets received by filter
0 packets dropped by kernel
$ netsniff-ng --in eth2 --ring-size 1GiB --out /dev/null --silent --verbose
RX: 1024.00 MiB, 16384 Frames, each 65536 Byte allocated
Running! Hang up with ^C!
15000002 packets incoming (0 unread on exit)
15000002 packets passed filter
0 packets failed filter (out of space)
0.0000% packet droprate
113 sec, 98840 usec in total
$ netsniff-ng --in eth2 --ring-size 512MiB --out /dev/null --silent --verbose
RX: 512.00 MiB, 8192 Frames, each 65536 Byte allocated
Running! Hang up with ^C!
15000002 packets incoming (2 unread on exit)
15000004 packets passed filter
0 packets failed filter (out of space)
0.0000% packet droprate
131 sec, 615943 usec in total
It looks like ifpps prints some values which can be NaN to the user.
Could you investigate on it? thanks!
Show percentage of line-rate would be a useful feature.
Over mid till long term, we need to migrate mausezahn code from staging into the normal tree.
Curvetun Server-side appears to be up and running fine... syslog says it's ready for connections. That's on a remote Fedora 17 build.
My local client, however, is causing me trouble (Fedora 18 on a macbook pro).
Turns out the ct_client.c call to curve25519_encode is never returning (line 254 of ct_client.c).
254 clen = curve25519_encode(c, p, (unsigned char ) us, us_len,
255 (unsigned char *) &cbuff);
256 if (unlikely(clen <= 0))
257 syslog_panic("Init encrypt error!\n");
I was able to trace it to a spinning loop in randombytes()
xio.c:
162 static void randombytes(unsigned char *x, unsigned long long xlen)
163 {
164 int ret;
165
166 if (fd_rnd < 0) {
167 for (;;) {
168 fd_rnd = open("/dev/urandom", O_RDONLY);
169 if (fd_rnd < 0)
170 break;
171 sleep(1);
172 }
173 }
I have to say, I'm confused as to how this portion of randombytes works! I added some debugging statements to the loops...
First time through fd_rnd = -1, prior to open().
Open returns file descriptor, fd_rnd = 6
This isn't negative, so we don't break. We sleep one second and repeat! Why?
We re-open /dev/urandom... get a NEW file descriptor, now:
fd_rnd = 7!
Etc... Based upon my log my client continues to open /dev/urandom every second... I'm currently on file id 304!
I believe this code came from nacl. I'm just struggling to see how it works =D
Clean up code issue & TODO file things ... here should be discussed what needs to be done regarding cleaning code!
Evaluate the LibE1000 user-space library to receive and transmit packets from/to Intel E1000 adapters. LibE1000 implements "OS bypass" mechanism, where fast path processing and RX/TX data transfers are done without kernel assistance (i.e. no system calls).
http://sourceforge.net/projects/libe1000/
If it's faster for e1000(e) adapters than RX/TX rings, we should include this into netsniff-ng + trafgen via compile option, so that either the libe1000 or RX/TX ring can be used.
I got some help to prepare a new design for the netsniff-ng project website.
The main goal is to improve content readability by having all information in a single screen and to improve code layout for easy maintenance.
Accessibility is also improved for people having disabilities.
A preview can be seen at http://test.netsniff-ng.org.
It does not need PHP and JQuery is used to organise content. Using it also opens, new perspective in terms of flexibility.
I created a new branch starting by eroullit/netsniff-ng@5861a7d03ed9872547d31bd9a01592e87376c192
Some things are still missing for now:
on ubuntu almost two main issue 👍 Wunsused.... flag is not recognized:
NACL_LIB_DIR/NACL_INC_DIR is undefined, building libnacl with curvetun!
Building netsniff-ng toolkit (0.5.8-rc0) for i686-linux-gnu:
Building netsniff-ng:
-e CC hash.c
cc1: error: unrecognized command line option "-Wunused-but-set-variable"
make: *** [netsniff-ng/hash.o] Errore 1
and:
mac80211.c:20:38: fatal error: libnl3/netlink/genl/genl.h: File o directory non esistente
compilation terminated.
make: *** [netsniff-ng/mac80211.o] Errore 1
need to compile only on redHat or Debian ? i don't think so ... wath wrong in my enviroment ?
Add a -U_FORTIFY_SOURCE in HARDENING=1 [1] to prevent [2].
x86_64-pc-linux-gnu-gcc -march=native -O2 -pipe -std=gnu99 -pipe -fPIE -pie -Wl,-z,relro,-z,now -fstack-protector-all -Wstack-protector --param=ssp-buffer-size=4 -ftrapv -D_FORTIFY_SOURCE=2 -fexceptions -fomit-frame-pointer -fno-strict-aliasing -fasynchronous-unwind-tables -fno-delete-null-pointer-checks -D_REENTRANT -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -DVERSION_STRING=""0.5.8-rc2"""" -DVERSION_LONG=""0.5.8-rc2"" (Ziggomatic)"" -DPREFIX_STRING="" -Wall -I. -I/usr/include/libnl3 -I/usr/include/libnl3 -DHAVE_DISSECTOR_PROTOS -DNEED_TCPDUMP_LIKE_FILTER -I/usr/include/libnl3 -I/usr/include/libnl3 -DHAVE_DISSECTOR_PROTOS -DNEED_TCPDUMP_LIKE_FILTER -o netsniff-ng/proto_igmp.o -c proto_igmp.c
:0:0: warning: "_FORTIFY_SOURCE" redefined [enabled by default]
proto_igmp.c:1:0: note: this is the location of the previous definition
/*
[2] http://b-4.xmw.de/var/log/portage/build/net-analyzer/netsniff-ng-0.5.8_rc2:20130805-070734.log
We need to extend astraceroute to allow for dns tracerouting (there was a nice SIGCOMM paper in 2012), plus we need to extend the backend to enable paris-traceroute like analysis.
libgeoip is installed in version 1.4.8.
The error message:
root@tinkerbell:~/temp/netsniff-ng/build# flowtop
OUI UDP TCP ETH
Error Opening file /usr/share/GeoIP/GeoIPCity.dat
Cannot open GeoIP database!
The following files are present:
root@tinkerbell:~/temp/netsniff-ng/build# ls -la /usr/share/GeoIP/
total 6520
drwxr-xr-x 2 root root 4096 2011-10-15 12:44 .
drwxr-xr-x 267 root root 12288 2012-02-15 00:14 ..
-rw-r--r-- 1 root root 1585923 2011-08-18 01:43 GeoIP.dat
-rw-r--r-- 1 root root 5068179 2011-08-18 01:43 GeoIPv6.dat
I'm using Ubuntu 11.10. following the instructions in INSTALL:
$ cd netsniff-ng/src/
$ mkdir build
$ cd build
$ cmake ..
$ make
and after that I got ifpps: command not found
It would be very practical to be able to dump ifpps results in file.
This way such data could be imported in a classical spreadsheet software for further data crunching or give it to gnuplot to generate a graph out of it.
To achieve this, data could be dump as a CSV file. Such data can be then easily imported.
A shell script called csv2gnuplot.sh can convert to a gnuplot input file.
As more and more device on the network are addressed by IPv6, it would be great if ashunt could report them properly.
Flowtop does not correctly install from the current source or the ubuntu packages.
I can not find the binary anywhere.
I use EL6 for my passive monitoring platform and would very much like to use the latest netsniff-ng features. It seems that '91e3a4c5' might be the last commit that support capture from a v2 interface. I understand the rationale for efficiency purposes, but eliminating TPACKET_V2 prevents anyone on RHEL from using netsniff-ng, at least until maybe RHEL7 comes out (if it ships with kernel > 3.2).
I don't know what happened to documentation like we had in 0.5.7: netsniff-ng-0.5.7/Documentation ?
Please bring it back or let me know how I can generate it!
Thanks!
Cocinelle is a tool to perform automatic code review.
It would allow us to check the syntax of newly introduced code and catch bugs early on.
It is already used successfully on the Linux kernel.
More info on http://coccinelle.lip6.fr/
The older versions lack include <sys/time.h> in libcli.h, [1] fixed the configure script to not fail in in the test [2].
[1] http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/net-analyzer/netsniff-ng/files/netsniff-ng-0.5.8-libcli-test.patch?revision=1.1&view=markup
[2] http://b-4.xmw.de/var/log/portage/build/net-analyzer/netsniff-ng-0.5.8_rc2:20130805-055918.log
Some library names have small and captial letters (LibGeoIP_LIBRARY), some only captial letters (NETFILTER_CONNTRACK_LIBRARY) and so on. Please make all of those names consistent in regard of naming.
If neither udp or tcp is specified in a server config file, tcp defaults (say, if one were to misspell udp). I ended up digging into the source code before realizing my error.
Could parsing in ct_servmgmt.c throw warnings or errors to syslog? Or is that too polluting? I would have benefitted from such.
The goal of this task is to prepare the project to support a CMake build infrastructure.
It will be very useful to optimize compile times, perform cross-compiling, support a test framework and probe needed dependencies.
Cmake >= 2.6 must be supported for a good compatibility.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.