ntop / ndpi Goto Github PK
View Code? Open in Web Editor NEWOpen Source Deep Packet Inspection Software Toolkit
Home Page: http://www.ntop.org
License: GNU Lesser General Public License v3.0
Open Source Deep Packet Inspection Software Toolkit
Home Page: http://www.ntop.org
License: GNU Lesser General Public License v3.0
Add nDPI support for Instagram (social network focused on photo publication).
Trace will be sent separately
Hello, folks!
Thanks for your brilliant work!
I have run nDPI on Network with 500 kpps load (per packet analitics mode) and expect significant performance issues, I can't achieve more than 250 kpps. Thus, I have tried to debug it but have some troubles with debugging symbols.
In perf top I saw:
Samples: 775K of event 'cycles', Event count (approx.): 49347133671
19.00% libc-2.19.so [.] __memcmp_sse4_1
4.31% libc-2.19.so [.] memset
3.20% libndpi.so.1.0.0 [.] 0x00000000000089ca
3.20% libndpi.so.1.0.0 [.] 0x00000000000089b0
2.98% libndpi.so.1.0.0 [.] 0x00000000000089c5
2.58% libndpi.so.1.0.0 [.] memcmp@plt
1.81% libndpi.so.1.0.0 [.] 0x00000000000089bf
1.79% libndpi.so.1.0.0 [.] 0x0000000000009d88
1.75% libc-2.19.so [.] _int_free
1.44% libndpi.so.1.0.0 [.] ndpi_detection_process_packet
1.39% libc-2.19.so [.] malloc
1.34% libndpi.so.1.0.0 [.] 0x0000000000009d98
1.08% libc-2.19.so [.] _int_malloc
0.96% libndpi.so.1.0.0 [.] 0x000000000000a404
0.90% ndpicallback.so [.] fastnetmon_parse_pkt
But my binary is not stripped:
file /opt/ndpi/lib/libndpi.so.1.0.0
/opt/ndpi/lib/libndpi.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=3fbf0ef7daee47da17d55984d3184baf414340a2, not stripped
And have all important symbols:
nm /opt/ndpi/lib/libndpi.so.1.0.0 |grep process
000000000000fc10 T ndpi_detection_process_packet
0000000000007400 t ndpi_patricia_process
I have installed my nDPI binary with following reference:
cd /usr/src
git clone https://github.com/ntop/nDPI.git
cd nDPI/
./autogen.sh
./configure --prefix=/opt/ndpi
make install
echo "/opt/ndpi/lib" > /etc/ld.so.conf.d/ndpi.conf
ldconfig
Maybe you could help me with this "unidentified" functions?
Add nDPI support for Deezer Application, a Music Streaming service. Trace will be sent separately.
Add nDPI support for TIM Beta Application, a chargeable application that allows the subscriber to pay accordingly their use of Voice, Internet and SMS service. The connection can be done via Facebook login or using a password that need to be generated via BETA site. Trace will be sent separately
To work with >10Gps traffic I want to share the traffic detection in different threads. Shall I use one detection module for all threads or own detection module for each thread?
Add nDPI support for Meu application. PCAP file with example sent separately.
I don't know why this is the case, but as shown at
Line 189 in b2816d7
As a follow up to issue #77 , currently there is inconsistent usage of the byte order of the in_addr *pin
parameter of ndpi_network_ptree_match
Based on the existing implementation of ndpi_network_ptree_match
it appears ndpi_network_ptree_match
expects in_addr *pin
to be in host byte order, as you convert to network byte order looking at your comment here:
pin->s_addr = ntohl(pin->s_addr); /* Make sure all in network byte order otherwise compares wont work */
I'm assuming ndpi_patricia_search_best
expects the prefix to be in network byte order then. However the problem is currently it's a bit of a mess how ndpi_network_ptree_match
is called:
ndpi_host_ptree_match
- calls ndpi_network_ptree_match
with pin
in host byte order
tor_ptree_match
- calls ndpi_network_ptree_match
with pin
in network byte order
ndpi_detection_process_packet
- calls ndpi_network_ptree_match
with pin
in network byte order
ndpi_guess_undetected_protocol
- calls ndpi_network_ptree_match
with pin
in host byte order
Also related:
ndpi_init_ptree_ipv4
fills the prefix in network byte order
ndpi_add_host_ip_subprotocol
fills the prefix in host byte order
Shouldn't this all be normalized so that whenever a struct in_addr
is used that it should already be in network byte order, as the documentation and convention of struct in_addr
is documented as network byte order: http://man7.org/linux/man-pages/man7/ip.7.html ?
In ndpi_detection_process_packet
the packet parameter is defined as const unsigned char *packet
:
ndpi_protocol ndpi_detection_process_packet(struct ndpi_detection_module_struct *ndpi_struct,
struct ndpi_flow_struct *flow,
const unsigned char *packet,
const unsigned short packetlen,
const u_int64_t current_tick_l,
struct ndpi_id_struct *src,
struct ndpi_id_struct *dst)
However on line https://github.com/ntop/nDPI/blob/dev/src/lib/ndpi_main.c#L3316 you cast this const away:
flow->packet.iph = (struct ndpi_iphdr *)packet;
Which eventually gets use as a parameter to ndpi_network_ptree_match
in line https://github.com/ntop/nDPI/blob/dev/src/lib/ndpi_main.c#L3423 :
struct ndpi_packet_struct *packet = &flow->packet;
if((ret.master_protocol = ndpi_network_ptree_match(ndpi_struct, (struct in_addr *)&packet->iph->saddr)) == NDPI_PROTOCOL_UNKNOWN)
ret.master_protocol = ndpi_network_ptree_match(ndpi_struct, (struct in_addr *)&packet->iph->daddr);
Inside ndpi_network_ptree_match
you write to this memory https://github.com/ntop/nDPI/blob/dev/src/lib/ndpi_main.c#L1673:
pin->s_addr = ntohl(pin->s_addr); /* Make sure all in network byte order otherwise compares wont work */
This causes a segfault if packet
is in read-only memory. Surely you should not be writing anything to the library consumer packet buffer? I.e. flow->packet.iph
should be read-only at all times?
nDPI is required to differentiate between different types of WhatsApp media i.e. WhatsApp Voice vs. WhatsApp Messaging.
PCAP traces of WhatsApp Voice and WhatsApp Messaging sent by mail to [email protected]
Add nDPI support for Waze, community based traffic and navigation Application. Trace will be sent separately
Please add nDPI support for TED (video conference platform application).
Trace is here:
http://www.netdeep.com.br/tmp/TED.pcap
Thanks!
Add nDPI support for Portas Abertas Application, a Mobile Operator Portal. Trace will be sent separately
Function ndpi_detection_process_packet hangs when process the packet mpegts.
Captured packet:
https://dl.dropboxusercontent.com/u/5135944/Github/mpegts.cap
This problem occur in version 1.6.1. Version 1.5.1 works normally.
Secure Socket Tunneling Protocol was introduced by Microsoft in Windows Vista SP1, and it is now available for Linux, RouterOS and SEIL. SSTP uses SSL v3, and therefore offers similar advantages to OpenVPN (such as the ability to use to TCP port 443 to avoid NAT firewall issues), and because it is integrated into Windows may be easier to use and more stable.
The following header structure is common to all types of SSTP packets:[8]
https://en.wikipedia.org/wiki/Secure_Socket_Tunneling_Protocol
here is pcap file https://drive.google.com/file/d/0B0SCwy1irn3qZE1kd2xPTWhOOGM/view?usp=sharing
Hello,
Please add nDPI support for Evernote (notes application generally in mobile devices)
Follows trace (tested login, synchronization, and changing a note):
http://www.netdeep.com.br/tmp/evernote.pcap
Thanks!
Add nDPI support for HOOQ application. HOOQ is a Video Streaming service. https://www.hooq.tv/
Traces will be sent separately.
The release notes for nDPI 1.6 mention "New API call for converting nDPI protocols IDs to names ndpi_protocol2name()", but the function name is not included in libndpi.sym and thus not exported in the resulting shared library.
natano@ketzer:~$ nm /usr/lib/libndpi.so.1.0.0|grep protocol2name 000000000000efd0 t ndpi_protocol2name
Notice the lower-case 't', which indicates that the symbol is local.
Is this on purpose?
I'm glad to see ntop / nDPI coming to git!
I see you're starting with no history though, so I wanted to say feel free to take my svn mirror from https://github.com/nyov/ndpi and continue from there.
(I haven't watched it for a while now, but I hope it has all the ndpi branches and that no commits were missed by the mirror script.)
You can of course drop the glue and opendpi code after, if you wish.
It would be a boon to drop my mirror script if git becomes the canonical source.
Add nDPI support for TIM_WiFi application. Trace sent in a separate email.
Ability to return multi-protocol detection (e.g. IP, UDP, DNS, Twitter) instead of just Twitter for a DNS query for www.twitter.com
This App is very popular in the Far East. Can you support it?
How can I send you traces for Kakao talk? I get a message 'unfortunately, we don't support that file type'
Add nDPI support for EAQ application. PCAP file with example sent separately.
Add nDPI support for Torcedor application. PCAP file with example sent separately.
Add nDPI support for TIM Menu Application, which gives subscribers access all Mobile Operator's apps and services. Trace will be sent separately
I remember there was windows update protocol in the old days. What happened to it? Lots of people will migrate from windows 7, 8, 8.1 to windows 10... it will be 3GB update that will run in background. It will be handy to have WSUS filter to put it to low priority...
Add nDPI support for Beta application. PCAP file with example sent separately.
For eg. It talks about using the demo pcapReader application, which is not present in the latest code.
Instead ndpiReader should be used.
Add nDPI support for Globo.tv application. An application providing access to Globo, a TV channel in Brazil. Trace will be sent separately
I am ddwrt user and some time ago ddwrt developers drop using l7 filters and started using nDPI with iptables (for marking-matching traffic for QoS and Access Restriction purposes). I am not developer nor the network guru. Just simple SOHO user with very basic understanding of routers.
ddwrt has big community and now we have problem because some nDPI protocols (filters) do not work. ddwrt devs says it is not ddwrt problem. Youtube, Steam do not work... my findings about this problem are here in bug report on ddwrt TRAC http://svn.dd-wrt.com/ticket/4117#comment:7
If I can provide more informations I would be happy, just to solve this issue. Thx for your work on nDPI it is already great!
ICQ traffic (Oscar protocol) detects correct only when clients connects to icq server. Later, after client idle, when a flow was deleted as an old flow, when an icq-messages goes over the established connection, nDPI can't detect this flow as a oscar protocol.
Add nDPI support for EasyTaxi, an application to call taxis. Trace file will be sent separately.
Add nDPI support for 99 Taxis, an Application to call taxis. Trace will be sent separately.
Viber is no longer detected, identified as unknown. Will upload pcap file soon.
Hi,
I'm downloading some data using ftp and it hasn't detected how many bytes i've downloaded. Otherwise i can see it has been detected as Unknown protocol. If you need anything else please let me know.
Add nDPI support for Recarga Application, an Application to recharge mobile credits
I'd like to request for Snapchat traffic to be classified by nDPI.
Server names should be the following:
feelsonice-hrd.appspot.com
feelsonice.appspot.com
Two pcap files of sample traffic are available at
https://drive.google.com/file/d/0B-7GNfxPaSADekxKUUdKdUlzWlE/view?usp=sharing
Hi,
Is it possible to work with ndpi to capture multiple requests and responses in a single http session (http pipeline support)?
The example code demonstrates only single request-response detection.
thanks,
Shirley.
un cliente mi ha segnalato che la sua connessione con la casa madre viene rilevata come bittorrent ma è una connessione Teredo.
i pcap li trovi in dropbox:
Dropbox/Wurth/baddpi-bittorrent-teredo.pcap.gz
Dropbox/Wurth/baddpi-bittorrent-teredo-small.pcap
Add nDPI support for Som de Chamada Application, allowing Mobile Operator subscribers to personalize the ringtone with different tones and music. Trace will be sent separately.
Add https://www.hsselite.com protocol dissector
Add nDPI support for SIMET, a speed test application. Trace will be sent separately
Currently nDPI is a monolithic library. Instead it would be desirable to split nDPI into engine + plugins where protocol dissectors are loaded at runtime (e.g. via shared libraries)
Hello guys. Congratulations for the work!
In my tests, the version 1.6 is not properly detecting/blocking Whatsapp messages.
Can confirm me if I have done wrong?
I don't see it in /src/lib/protocols. Is it detected in some other way? Just in case pcap files for icmp and icmpv6 added... Thx
https://drive.google.com/file/d/0B0SCwy1irn3qTDlMVGhSUXc1dU0/view?usp=sharing
https://drive.google.com/file/d/0B0SCwy1irn3qZVJVRjd3X0x0Tm8/view?usp=sharing
Add nDPI support for OpenSignal, a Speed test application. Trace will be sent separately
There is application called Popcorn Time. It is very popular in our county (Serbia) and I believe in others too. Because of the local laws you can watch movies online but it is prohibited to download them, it is against the law. This app do exactely that. Popcorn Time streams movies, popular series and TV shows from torrents. So, this application is widely used. It can be used on Windows, Andoids etc. Popcorn Time uses, among others protocol, Bittorrent client for temporary downloading content (other P2P users downloads from you at the same time). And that is a problem. nDPI recognizes Popcorn Time as Bittorrent so it can not be properly prioritized. Most users wants it with higher priority while downloading other (P2P) stuff in backgroung. Can you make filter (protocol) for Popcorn Time? Aplication can be downloaded here https://popcorntime.io/ It is only 40-50MB...
here is pcap file https://drive.google.com/file/d/0B0SCwy1irn3qOGY4WFdxalQyVFk/view?usp=sharing and here is fiew screenshots of it:
http://www.dodaj.rs/f/3Y/2d/3SYEjWhH/popcorntimescshot.png
http://www.dodaj.rs/f/S/Gj/2MNcZwaG/popcorntimemv.png
I've used this DPI for many months; I always installed it in the past with the svn commands, as well as the git repository wasn't available yet.
So, for my new Ubuntu VMs as well as for my Ubuntu PC, I started installing it with the procedure here illustrated; since I have always used the json output, I installed the libraries before the autogen and the configuration, and the compilation was successful. But, when started the dpi (i.e. sudo ./ndpiReader -i eth0 -v 2 -j capture.json), the file isn't created at all. I tried it on many VMs and on my local computer, but nothing changed.
So I got back the last version from an older machine and repetead the previous (autogen, configure, make) and the same procedure gave me the output json file, so I thought it is a version problem!
Kakao Chat
Add nDPI support for MEU Application, allowing Mobile Operator's subscribers have access to all information regarding their account, bills, credits, activate or deactivate promotions, etc. Trace will be sent separately
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.