Giter Club home page Giter Club logo

samplicator's Introduction

UDP Samplicator

This small program receives UDP datagrams on a given port, and resends those datagrams to a specified set of receivers. In addition, a sampling divisor N may be specified individually for each receiver, which will then only receive one in N of the received packets.

DOWNLOADING

This package is distributed under https://github.com/sleinen/samplicator/

INSTALLATION

See INSTALL.md.

AUTHORS

See the AUTHORS file.

USAGE

The usage convention for the program is

$ samplicate [<option>...] [<destination>...]

Where each <option> can be one of

-d <level>	to set the debugging level
-s <address>	to set interface address on which to listen
		for incoming packets (default any)
-p <port>	to set the UDP port on which to listen for
		incoming packets (default 2000)
-b <buflen>	size of receive buffer (default 65536)
-c <configfile>	specify a config file to read
-x <delay>	to specify a transmission delay after each packet,
	    in units of	microseconds
-S		maintain (spoof) source addresses
-n		don't compute UDP checksum (only relevant with -S)
-f		fork program into background
-m <pidfile>	write the process ID to a file
-4		IPv4 only
-6		IPv6 only
-h		to print a usage message and exit
-u <pdulen>	size of max pdu on listened socket (default 65536)

and each <destination> should be specified as <addr>[/<port>[/<interval>[,ttl]]], where

<addr>		IP address of the receiver
<port>		port UDP number of the receiver (default 2000)
<freq>		number of received datagrams between successive
		copied datagrams for this receiver.
<ttl>		The TTL (IPv4) or hop-limit (IPv6) for
		outgoing datagrams.

Config file format:

a.b.c.d[/e.f.g.h]: receiver ...

where:

a.b.c.d     is the sender's IP address
e.f.g.h     is a mask to apply to the sender (default 255.255.255.255)
receiver    see above.

Receivers specified on the command line will get all packets, those specified in the config-file will get only packets with a matching source.

samplicator's People

Contributors

aheybey avatar cbueche avatar cfsmp3 avatar chleggett avatar pangorgo avatar sleinen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

samplicator's Issues

using samplicator with ffmpeg

some hints for using samplicator to delay ffmpeg stream correctly ?
my build always destroys h264 data and creates LAG / fragments and other decoding errors.
cant use it in production, tried a lot of different bufsizes but i dont get it to work smoothly

only first 1500 bytes from large UDP datagrams is copied

I am working with traffic from application which sends UDP datagrams longer than 1500 Bytes so they are fragmented in IP layer.

in samplicate.c PDU_SIZE is fixed to 1500 here
#define PDU_SIZE 1500

result of which - maximum 1500 bytes from buffer is copied and no warning nor error is thrown in case of longer UDP datagram.

I see that further there is some test that should catch longer datagram:

if ((n = recvfrom (ctx->fsockfd, (char*)fpdu, sizeof (fpdu), 0, (struct sockaddr *) &remote_address, &addrlen)) == -1) {
  fprintf (stderr, "recvfrom(): %s\n", strerror(errno));
  exit (1);
}
if (n > PDU_SIZE) {
  fprintf (stderr, "Warning: %d excess bytes discarded\n", n-PDU_SIZE);
  n = PDU_SIZE;
}

But at least in my case (kernel 2.6.18) it fails to catch it.
I think the reason is lack of MSG_TRUNC flag in recvfrom() call for modern kernels (since 2.2).

My workaround was changing PDU_SIZE to larger value, and it solved problem for my long UDP case but not sure if it won't broke something else - for example TCP handling.

Src IP or string based filter?

Hi,

Scenario - devices are sending syslog and netflow to one samplicator, so one udp port (514) for syslog and one udp port (2500) for netflow. Syslog & netflow saved into same device and also forwarded to two other devices.

Now I have need to forward traffic from only some sender syslog and/or netflow to third device, so samplicator internal filtering would be best for such purpose. Using regex for strings will definitely be (slight)performance issue, but using sender src IP for the filter, should be fine.

Can You please add such functionality?

Code review request

From [email protected] on March 13, 2012 14:54:57

Purpose of code changes on this branch: To support replication of large UDP datagrams. When reviewing my code changes, please focus on: Possible issues when sending out copies of large packets, especially when raw mode (i.e. spoofing) is used. After the review, I'll merge this branch into: /trunk Well, actually I guess I'll release this as 1.3.7. But maybe I get some other small changes in as well.

Original issue: http://code.google.com/p/samplicator/issues/detail?id=19

Packets truncated to 1500 bytes

From [email protected] on March 11, 2012 08:33:49

What steps will reproduce the problem? Use samplicator to forward UDP packets with a size greater than 1500 bytes What is the expected output? What do you see instead? Expect packets bigger than 1500 bytes to be forwarded in full. Instead, packet is truncated to a maximum size of 1500 bytes What version of the product are you using? On what operating system? 1.3.6, Linux Please provide any additional information below. Could the #define PDU_SIZE in samplicate.c be increased to 65535 - the maximum possible UDP packet size.

Original issue: http://code.google.com/p/samplicator/issues/detail?id=18

Configuration file parser buggy/too strict

From [email protected] on March 09, 2011 16:32:24

What steps will reproduce the problem? n/a What is the expected output? What do you see instead? the device should be displayed in the Netflow Tracker What version of the product are you using? On what operating system? Older version of Samplicator. Red Hat rel.5 is the platform Please provide any additional information below. Our Samplicator is running fine, however one thing I noticed is that when I am adding new devices it doesn't seem to display the new device to the Netflow Tracker. But for older device entries, the samplicator continuous to export data to the Tracker no problem at all. What could be a possible cause that hampers the samplicator to export new list of devices after adding to samplicator.conf file?

Original issue: http://code.google.com/p/samplicator/issues/detail?id=4

Reading from multicast stream

The following

/usr/local/bin/samplicate -s 239.168.100.7 -p 5000 127.0.0.1/50405 127.0.0.1/50406

Doesn't work for me. 239.168.100.7:5000 is a multicast stream that I can read with VLC, etc. However samplicate doesn't seem to join the multicast group.

Samplicate crashes over and over

From [email protected] on September 02, 2011 21:02:51

What steps will reproduce the problem? 1. running Samplicate
2. on RHEL 4.1
3. in VMware What is the expected output? What do you see instead? the samplicate process crashes constantly. current work around is a cron job to check process and start if no running every 1 min. What version of the product are you using? On what operating system? 5 year old code Not sure of version. RHEL4 Please provide any additional information below. has anyone seen this behavioer before? and is this a known issue and does latest code fix this?

Original issue: http://code.google.com/p/samplicator/issues/detail?id=7

Allow receiver hostname definition

Please add the ability to define the receiver by hostname and not just IP address. This would allow to forward to a cluster hostname using RR DNS or a service discovery hostname. This would increase the availability of the replicated packet.

Using just a VIP would require something like a hardware load balancer or managing an IP address with something like corosync. Being able to use modern techniques like service discover allows for an active-active load balance (although a harware LB ie. F5 would do this, I don't want to pay for one of those), which adds a scale out feature.

r126 build errors

From [email protected] on October 19, 2011 21:24:14

What steps will reproduce the problem? aclocal
autoheader
automake -a
autconf
./configure
make What is the expected output? What do you see instead? autoheader produces warnings:

autoheader: WARNING: Using auxiliary files such as acconfig.h',config.h.bot'
autoheader: WARNING: and config.h.top', to define templates forconfig.h.in'
autoheader: WARNING: is deprecated and discouraged.
autoheader:
autoheader: WARNING: Using the third argument of AC_DEFINE' and autoheader: WARNING:AC_DEFINE_UNQUOTED' allows one to define a template without
autoheader: WARNING: acconfig.h': autoheader: autoheader: WARNING: AC_DEFINE([NEED_FUNC_MAIN], 1, autoheader: [Define if a functionmain' is needed.])
autoheader:
autoheader: WARNING: More sophisticated templates can also be produced, see the
autoheader: WARNING: documentation.

automake -a produces warnings:

automake: configure.in: installing ./install-sh' automake: configure.in: installing./mkinstalldirs'
automake: configure.in: installing ./missing' automake: configure.in: installing./config.guess'
automake: configure.in: installing ./config.sub' Makefile.am:7: @LIBOBJS@ seen but never set inconfigure.in'

make produces:

gcc -DHAVE_CONFIG_H -I. -I. -I. -g -O2 -c samplicate.c
samplicate.c:38:18: error: inet.h: No such file or directory
make: *** [samplicate.o] Error 1

I have arpa/inet.h, so I tried making the #include "inet.h" to be in a #else to the arpa/inet.h check. This was in samplicate.c and read_config.c. I got farther, but then make yielded:

make: *** No rule to make target inet.o', needed bysamplicate'. Stop. What version of the product are you using? On what operating system? r126 , from the repository, on Linux -- Debian 6.0.2. Please provide any additional information below. autoheader version 2.67.

automake version 1.4-p6.

Original issue: http://code.google.com/p/samplicator/issues/detail?id=12

Netflow v9 source_id

Hello,

I'm trying to debug Netflow V9 from CISCO without source_id value
The goal is to add IP source (router who generates flow) to the source_id https://netflow.caligare.com/netflow_v9.htm before duplication.

This can be very useful to find the source device
Any advice on how to do this would be greatly appreciated

For now I'm trying to see the buffer without any success
https://github.com/sleinen/samplicator/blob/master/samplicate.c#L346

I added this right after
if (n > 0) { fpdu[n] = '\0'; printf("Test received message: %s size: %ld\n", fpdu, sizeof (fpdu) ); }

But fpdu message is always empty (size not of course), I don't understand why
I guess there is something specific here, a basic client/server UDP test works well

I guess that show the socket data can also be very interesting for debugging purpose in many cases

Thanks

Lack of IPv6 support

From [email protected] on March 11, 2009 09:48:02

What steps will reproduce the problem? 1. Start the samplicator on some port
2. Send UDP packets to that port over IPv6 What is the expected output? What do you see instead? Expected: The packets should be replicated to the configured destinations.
Instead, they are dropped by the host's network stack, because the
samplicator process uses an IPv4-only socket.

Similarly, when IPv6 addresses are specified as destinations, the
samplicator complains about malformed addresses.

Ideally, the samplicator should even be able to receive datagrams over IPv4
and distribute them over IPv6 and vice versa.

Original issue: http://code.google.com/p/samplicator/issues/detail?id=1

Setable source port for samplicated packets

From [email protected] on August 06, 2010 11:47:40

In our company we need to have always same source port for samplicated packets, so I wrote a small patch, which allows that.
I think it might be useful also for other users, at least as a optional patch.

Patch adds option to set source port into receiver settings.

A.B.C.D[/port[/freq][,ttl][.src_port]]

It's not optimal, because Samplicator uses only one socket for all outgoing packets so source port is set only once, when first modified receiver is found.

Source port settings are ignored in spoof mode (-S).

Attachment: samplicator-1.3.6-src_port.patch

Original issue: http://code.google.com/p/samplicator/issues/detail?id=3

Multicast

Hi,-

in a large environment with multiple destinations, it would make sense simplicator to put the flow on multicast streams. i would reduce the processing power and bandwidth.
Any thoughts ?

thanks,
idsteiner

Ability to change source IP to arbitrary value

In flow testing, it's sometimes beneficial to choose an alternate source IP.

Open ended:
Allow setting the IP via the command line.

Tighter controlled: (if you're worried about UDP floods)
Allow setting the IP from any given IP on the box. (primary or secondary ips)

Samlicate not forwarding to receivers

From [email protected] on March 28, 2014 18:57:41

What steps will reproduce the problem? I don't know, I followed the steps for installing the Samplicator application as documented. What is the expected output? What do you see instead? What version of the product are you using? samplicator-1.3.5
On what operating system? CentOS 6.5 Please provide any additional information below.

Attachment: Samplicator Issue with Forwarding to Receivers on port 2055.txt

Original issue: http://code.google.com/p/samplicator/issues/detail?id=28

Fix for issue #20

From [email protected] on March 14, 2012 22:30:17

Purpose of code changes on this branch: Fix for issue #20 When reviewing my code changes, please focus on: Whether it could break any systems, or whether you can think of other systems than Solaris where this variant is necessary. After the review, I'll merge this branch into: /branches/1-3-maint

This is also in 1.3.7-beta3.

Original issue: http://code.google.com/p/samplicator/issues/detail?id=21

special config consideration using 0.0.0.0/0.0.0.0 in config file (high cpu usage)

From [email protected] on April 06, 2011 16:51:22

Hi,

We have many flow packets that arrive from a few hundred routers and the need is to duplicate some of the packets to different destinations without changing the dst ip on the routers.

so we used
samplicate -p 9996 -f -S 1.1.1.1/9996 2.2.2.2/9996

but then the need came up to duplicate certain routers to additional ones or remove some from the 2 already defined.

so we started using:

samplicate -p 9996 -f -S -c /etc/samplicate.conf

samplicate.conf:
5.5.5.5:1.1.1.1/9996 2.2.2.2/9996 3.3.3.3/9996
0.0.0.0/0.0.0.0:1.1.1.1/9996 2.2.2.2/9996

Now while doing a top I saw that the cpu spikes are getting quite high from samplicate and I wonder if this is really a problem or can this be ignored.

We want to add more destination and maybe remove certain destinations from matching the 0.0.0.0 rule.

I wonder if it would be easier to specify some kind of catch all without the program having to see if the 0.0.0.0 line matches?

Original issue: http://code.google.com/p/samplicator/issues/detail?id=5

setsockopt() sometimes gets passed arguments of the wrong type

From [email protected] on April 22, 2012 16:49:21

John Heasley points out that arguments to setsockopt() should be int, but the current code passes longs. This would make the code fail on 64-bit machines.

I must say that I have never noticed this, and I have been running samplicator on 64-bit GNU/Linux machines for quite some time. But it is plausible that this would cause problems on other systems.

Original issue: http://code.google.com/p/samplicator/issues/detail?id=24

Duplicating lots of snmp traps : some get lost on second destination?

From [email protected] on January 08, 2013 09:06:12

What steps will reproduce the problem? Command which is running on the server to forward all snmp traps to two other servers :
/usr/local/bin/samplicate -fS -p 162 192.168.0.197/162 192.168.3.181/162

The first server (197) seems to get all traps fine, but the second looses lots of traps. Those are 2 different monitoring machines working with the same configuration, code, and store the traps in the database. But the 2nd server (3.181) does not get all the traps.

This is now ongoing for about 4 weeks and because it is an internal network I can't think of loosing those packets on the cable internally.
But it is of course really lots of traps that get duplicated, around 30 - 40k per day.
I can also always see a time difference when the traps are entered in the database.
The first server always stores the traps around 20 minutes before the second server receives the trap and stores it. What is the expected output? What do you see instead? I am expecting that both machines would receive the traps around the same time and also receiving the same amount of traps. What version of the product are you using? On what operating system? OpenSuse 12.2 x64

If you need further details I will provide.

Original issue: http://code.google.com/p/samplicator/issues/detail?id=27

how does samplicator.netflow get processed?

i've seen two behaviors. When sending flow to mulitple collects i've experienced where samplicate will process the flow from one rule then stop. I've also seen it where it will process two rules.

Use case:

I have a rule for 192.168.10.0/255.255.255.0: 10.1.1.1/2055
I have a rule 0.0.0.0/0.0.0.0: 10.1.1.2/2055

some times when samplicate is running and a flow comes in for a device in 192.168.10.0/24 subnet it will just be sent to 10.1.1.1. then i'll restart the samplicate process and it will go to both destinations.

Can I get clarity on how the list is processed?

FreeBSD 11 now requires the -s 0, -S no longer working

On FreeBSD 11-RELEASE-p1 with samplicator 1.3.8.rc1

Running samplicator with the following syntax no longer works. No flows are captured
$ samplicator -p <port> <target>/<port>

This however, does work
$ samplicator -s 0 -p <port> <target>/<port>

Samplicator is also no longer able to spoof source address. This seems to be due to a change in FreeBSD 11 to no longer convert incoming packet headers ip_len and ip_off fields to host order. They now remain in network order with the rest of the packet header

support to EXCLUDE source hosts/networks?

Is it possible to EXCLUDE source hosts/networks such that their messages are NOT forwarded?

Example:
I have some hosts (firewalls) that send GBs of log messages every day and I don't want to forward those messages to our Solarwinds server, because it simply can't handle the load.

$data _center_network: 10.1.0.0/255.255.0.0
$firewall: 10.1.1.10
$solarwinds: 10.1.2.3

So, I'd like to be able to keep my configuration very simple by having something like this in my config:

$data _center_network:$solarwinds/514, !$firewall

I realize that I could just define every host I want logs forwarded from in the samplicator config, but that means I have to edit that config every time I have a new or changed router or switch.

Portability regression

PR #48 introduced a construct that isn't understood by all C compilers:

  for (int j=0; j<argc; j++)

I should have reviewed this better!

Replicating packages to the same port, on the same host

From [email protected] on August 08, 2011 10:58:32

I'm trying to use samplicate to get around the following issue:

We have one host that receives UDP data on one port, where lots of clients reports back to a specified address and specified port.

We don't have control over the applications that sends the data or the one that receives it, but we host the server that host the receiving application, and have root access to this.

Another application has been developed that also needs to listen to the same data. This could run on the same machine, or, preferrably on a different host.

I've tried running samplicate on the receiving machine, using the arguments; samplicate -p 1707 -S 192.168.x.x/1707 127.0.0.1/1707, where the 192.168.x.x-adddress is the host for the new app.

It forwards the packets nicely to the 192.168-host, but the local application does not receive any packets when using Samplicator in this manner.

Is it possible to forward packets on the same host and port that it receives UDP data on?

Original issue: http://code.google.com/p/samplicator/issues/detail?id=6

Ipv6 default bind

Hello,
By default, samplicate intercept all on ipv4 interface ( 0.0.0.0) but not for ipv6.
So with ipv6 and multiple address, we must start multiple samplicate instance with -s

Regards,
Nicolas

Compiling on Windows

Hello,

I'm having a very hard time compiling Samplicator for Windows. I'm using Cygwin. Is there any Documentation or some Help file anywhere explaining how to Compile for Windows? If there is this would be a much appreciated.

Delaying (-x) doesn't work as expected

From [email protected] on September 19, 2012 05:22:21

What steps will reproduce the problem?

  1. run: samplicate -p 1234 -x 10000000 127.0.0.1/5678
  2. run in another shell: ffmpeg -i ... udp://127.0.0.1:1234
  3. run in another shell: vlc udp://@:5678

What is the expected output?

What do you see instead?

Expected is to see in vlc the image that ffmpeg is sending, but delayed for 10 seconds. What I see instead is a lot of dropped packets, without anything useful received at the vlc's side. If I lover the -x param for samplicator, to some really low value, like -x 1000 or such, then it works as expected. What version of the product are you using? On what operating system? 1.3.6 stable on debian squeeze (stable)/linux Please provide any additional information below. I'll provide if needed, but I just need to delay the udp stream for 10 seconds, that's all.

Original issue: http://code.google.com/p/samplicator/issues/detail?id=26

High CPU utilization

Hi there! First, thank you for sharing this utility, it's really great!

I have noticed on a couple different VMs that I've tested this on (all building from latest source) that CPU utilization for a single core from the samplicate process is consuming ~90% CPU at all times, even with no data being sent to the service. Is this expected behavior? I haven't tried sending data to load it up yet.

Here are the flags I'm using:

/usr/local/bin/samplicate -S -f -p 2055 -c /etc/samplicator/samplicator.conf

And the config file:

# source/mask:destination/port
0.0.0.0/0.0.0.0:127.0.0.1/2055 10.5.2.39/2055 10.250.38.17/2055

Config file not supporting spoofing parameters available via CLI.

From [email protected] on August 30, 2012 01:33:00

What steps will reproduce the problem? 1. Running samplicator in spoofing mode while trying to use the config file instead of the CLI. What is the expected output? What do you see instead? The expected behavior is to have a config file option to allow spoofing which presently doesn't seem to exist. The current behavior is insufficient because this mode can only be enabled via the CLI which causes a discrepancy between CLI versus config file modes of use. What version of the product are you using? On what operating system? 1.3.6 and 1.3.7 beta

OS: Redhat Linux Please provide any additional information below. We have provided a tentative patch to address this issue.

Original issue: http://code.google.com/p/samplicator/issues/detail?id=25

Ability to replicate packets with the original source IP and the samplicator server IP at the same time.

Hello there, I would like to know if it is possible redirect packets keeping the original IP and an alternate source IP at the same time with this tool.

I'm currently running samplicator 1.3.6.

i.e:
source Net: 10.0.0.X / 25
Samplicator Server: 10.2.0.1
Destination Server 1 - IP: 172.27.0.1
Destination Server 2 - IP 10.1.0.1

FWD packets to server 1 with an alternate Source IP. (preferable the IP of the server where samplicator tool is running)
FWD packets to server 2 with the original Source IP.

Thanks!

Samplicator has an issue with forwarding fragmented UDP packets.

From [email protected] on October 15, 2011 22:41:29

What steps will reproduce the problem? 1. Samplicator is running on a relay server with the following options.
./samplicate -p 162 -S -f -n x.x.x.x/162 where x.x.x.x is the final trap receiver's IP address.

  1. Device sends a large size of SNMP trap (UDP over IP) to the relay server. Due to the size of the IP packet, it is split up into multiple IP fragments during transportation across network. What is the expected output? What do you see instead? Samplicator should forward all fragmented IP packets(UDP) to the destination models. However I saw Samplicator only forward 2 packets, and the forwarded UDP packets lost the identification of SNMP protocol. What version of the product are you using? On what operating system? 1.3.6. Please provide any additional information below. Please refer the attached snoop file. The first packet had small size of SNMP trap PDU and it was forwarded to the remote host sucessfully. However the second packet which was fragmented to packets 3 to 7 did not get forwarded correctly. The SNMP protocol idenditification also got lost after forwarding.

Attachment: baltlmcss02_ivms2.pcap

Original issue: http://code.google.com/p/samplicator/issues/detail?id=11

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.