fact finding for asyncio support about puresnmp HOT 15 CLOSED

Do you have an example use case?

from puresnmp.

I'd like to make requests to multiple hosts simultaneously via asyncio.gather() and also make requests from an async web server. I'm trying to revamp some old internal tooling that uses pysnmp. Right now everything is synchronous. I was hoping that I could find some gains by using pysnmp's asyncio support but the newer versions of pysnmp are significantly slower than the older versions in my tests. Puresnmp provides a pleasant API and is more than 2x faster in my simple tests. While searching the web I found a few posts/docs that vaguely suggest that asyncio support might be something that the puresnmp project isn't entirely opposed to. So, I would like to find out what your thoughts and plans are and see if I can get enough guidance to help make it happen.

from puresnmp.

I was hoping that I could find some gains by using pysnmp's asyncio support but the newer versions of pysnmp are significantly slower than the older versions in my tests.

This is a tad bit surprising as recent pyasn1 (which is the basis of pysnmp message serialization) is actually faster. You may consider raising an issue with pysnmp, along with a reproducer, to help pinpointing and hopefully fixing the slow piece.

from puresnmp.

Have you found pysnmp to be faster than puresnmp?

I would be happy to give up the flexibility and power of pysnmp for a simpler api like that of puresnmp, but I was hoping to keep asyncio support.

from puresnmp.

I'm not terribly familiar with asyncio. What would be needed for puresnmp to support it?

from puresnmp.

I'm not sure exactly how the functionality should be integrated into puresnmp, e.g. where to put files, what to name them, etc. But I think we would need to hit three spots and make them async:

• send() in transport.py
• all of api/raw.py
• all of api/pythonic.py

And I think it might be implemented a little like this.

from puresnmp.

Please note, I'm happy to do the work for this effort as long as you are willing to advise me of how you want the code structured. I just don't have the vision of the project that you probably do. If you aren't interested. I could probably just put the async layer right in my project. It is relatively self-contained thanks to the way you've laid things out.

I did an incredibly informal test where I collected the mac tables from 40-50 switches. Async pysnmp took about 50-60 second. Sync puresnmp took 20-25 seconds. And the async version of pure snmp was able to pull the data back in around 10 seconds.

from puresnmp.

Sorry for the late reply guys.

One of the reasons why I started development of puresnmp was in fact to learn asyncio. But I wanted to wrap my head around the SNMP protocol first. Now that I have a working implementation I am absolutely in favour of adding asyncio support. This was also a reason why this was initially Python 3 only.

One thing to be aware of is that #32 has now introduced Python 2 support. I accepted the PR for the time being, but by adding asyncio, Python 2 support must be dropped anyway again.

I was thinking of going as far as completely reverting that PR as it introduced quite a lot of churn due to the fact that type-hints are unsupported in Py2 and because byte-handling is very different in Py2 as well.

Reverting it would move us back to a cleaner code-base. For Python 2 compatibility a fork might make more sense. I believe it's possible to keep the bulk of the logic almost identical, so keeping the two forks in sync should not be too hard. Essentially the only I/O which is made in the library is in the transport.py module.

However, if transport.send() is made async, then this - as far as I can tell - will force all functions calling send would need to be async as well in order to expose this to the outer layer of the API such that clients can leverage asyncio properly.

I have not yet written anything based on asyncio, but I've done my homework and went over the asyncio docs. Any help is absolutely welcome and appreciated. I will only learn from the experience 😄

from puresnmp.

I've glanced over the code linked by @acspike and I like the changes quite a lot:

• It used a Protocol for marshalling
• It still keeps the old non-async functions accessible, which may even make it possible to keep this Py2 compatible (it should only crash once one of the aio modules is imported).

It is a bit unfortunate that it introduces a lot of duplication between both the aio and non-aio modules. But I don't see a way around that. I also see that the code-example in the docstrings are not yet updated, but that's an easy fix.

from puresnmp.

Yes, I didn't want to work on the docstrings until I knew you were interested. This much allowed me to test the functionality.

The duplication is a bit unfortunate, but I'm not sure how to eliminate it. I suppose pieces of some of the more complicated functions could be extracted, but it might just make it more difficult to understand. Asyncio seems to be somewhat infectious this way.

I think the solution to Py2 compatibility could be as easy as not importing the async functionality unless specifically called for.

Please let me know if there are other changes you would like made. Is there a better way to name the files? Better place to put them? Should I fix the docstrings and make this a pull request?

from puresnmp.

@etingof I'm ashamed to admit that the old pysnmp version I am comparing with is 2.0.9 (comparing against 4.4.4). I'd be happy to work help improve performance, but I'm really struggling with the API.

from puresnmp.

@acspike I think pysnmp 2 is about a decade old, I doubt asyncio even existed back then...

The docs on asyncio interface to pysnmp are here. You should probably explain in greater details what puzzles you. ;-)

I guess high-level pysnmp API would always be slower that minimalistic SNMPv2c implementations given that pysnmp implements the whole SNMP system meaning: manager/agent, SNMPv3, proxy, VACM, SMI/MIBs etc.

You may try your luck with low-level pysnmp API which operates at roughly similar level as puresnmp or libsnmp implementations.

However, packet serialization library beneath pysnmp (which is pyasn1) is also highly generic, as opposed to implementations, optimized to deal with specific data structures. That may make pyasn1 somewhat slower than specialized codecs.

But you should really benchmark the fresh versions to get realistic estimation.

from puresnmp.

@etingof absolutely a decade old! So asyncore rather than asyncio. I remember being confused about an async version of the api even back then. Wondering, "what good is a function that returns before it has the answer?!?" I have been comparing pysnmp 2.0.9 (sync) to pysnp 4.4.4 (async) to puresnmp(sync and async). Where is the best venue to continue this conversation?

from puresnmp.

Where is the best venue to continue this conver

We can move to pysnmp issues not to invade puresnmp space. ;-)

from puresnmp.

Closing this in favor of #48

from puresnmp.