Possible performance improvement idea? about gosnmp HOT 7 CLOSED

UDP is connectionless, so we should be able to fire off tons of requests and wait for the responses whenever, in theory?

Heck, Zmap can scan the internet @ 1.4 million packets per second... Looking into the architecture there...

Zmap:

• Raw ethernet socket (See: https://code.google.com/p/go/issues/detail?id=8432)
• Cache stuff that won't change (ethernet header, minus checksum)
  • By generating and
    caching the Ethernet layer packet, we prevent the Linux
    kernel from performing a routing lookup, an arpcache
    lookup, and netfilter checks for every packet.
• Don't handle responses - just use libpcap to grab (more or less) everything.

from gosnmp.

My experience writing a large snmp polling app for a large telco. I setup channels in my calling (golang) app to queue my requests, then started up many workers (each worker a goroutine that did a gosnmp call by pulling off the queue). Then an "aggregator" in another goroutine to aggregate my results from a "results" queue. That is, one master to many workers to one aggregator.

I found that golang/gosnmp wasn't slowing me down, rather slow devices (and network latency/capacity). In fact I setup a tunable "kill interval" in my app, because some devices were just so d!mn slow. If some devices hadn't answered within the time period, remaining workers would be killed and the next round of polling would start.

The interval was two minutes, so a few milliseconds of extra performance in the library doesn't interest me :-)

from gosnmp.

I agree with sonia's view. Raw sockets would add quite a layout of complexity. The kernel should be pretty efficient here anyway. Nathan: Is it device discovery or reading values that you're after? If it's device discovery, then maybe you can use a raw socket to scan, and then gosnmp to then read the values once you've confirmed the device exists.

from gosnmp.

@soniah
I have a similar design, I have a list of IPs stored in Redis, these get sent to NSQ (https://github.com/bitly/nsq), where I have a bunch of servers as consumers (well, each poller has multiple NSQ consumers). These consumers pull off messages from NSQ, and feed them to >10,000 goroutine workers. The results are then sent back to Redis (just 1 OID right now).

This maxes out at about 32,000/s.

I've not looked into adding a worker kill timeout, as I thought the SNMP timeout would do...

My devices are >20ms away (ping wise), but none should be more than 150ms away, even given that, when I need to poll 10s of millions of devices, I should be able to just send requests as quickly as possible and then just sit back and wait for them to respond (essentially performing an SNMP DDoS on myself...)

@codedance
No device discovery, and I have some other ideas other than raw sockets, which I'm going to try and prototype... I really dont want to be hand crafting packets, thats above the complexity I want as well.

from gosnmp.

Ok, so I had a try with this idea:

• Create a new function GenPacket(community string, version SnmpVersion, reqType PDUType, oids []string) which returns a []byte
• Create an empty UDPConn (net.ListenUDP("udp", &net.UDPAddr{IP: net.IPv4zero, Port: 0}))
• Turn the host addresses into net.UDPAddr's
• Write the returned packet using net.WriteToUDP([]byte, *net.UDPAddr)
• Figure out how to deal with the responses later

I can now do 137,000+ per second...

See my fork for the code specifics...

from gosnmp.

Cleaning up. Are you still working on this?

from gosnmp.

Possibly, haven't come back to it yet.

from gosnmp.