That's great and thanks for responding so quickly.
Yeah with two versions it should be good for folks who want to use Python2.7.
Let me know if you need help with testing.

from ioc-finder.

Hey, @fhightower are there any plans to do this?. Was looking into how to speed up the parsing portion.

from ioc-finder.

Yes, I'll start working on this today and hope to have a solid, working version in a couple of weeks.

Implementation details

As far as implementation details are concerned (this is more for my own thoughts than for anyone else), I'm planning on using asyncio. This is pretty easy (I've done similar things before); the only downside is that we will lose python2.7 support (which doesn't have much time remaining anyway: https://pythonclock.org/).

This change will be a major version (v2.x.x). Version 1.x of ioc-finder will be compatible with python2.x+ . Version 2.x of ioc-finder will be comptabile with python3.4+ .

from ioc-finder.

I just shipped version 2.0.0 to pypi which uses https://docs.python.org/3.7/library/multiprocessing.html to run many of the parsing operations concurrently. Let me know if you have any questions!

Implementation Details (or: Lessons Learned)

I had originally planned on using asyncio. I converted the library to use asyncio and it actually made it slower. After doing some research, I found this article which was very helpful: https://realpython.com/python-concurrency/. Basically, a program can run slowly because of I/O latency or because of CPU usage. asyncio is good for solutions involving I/O latency and multiprocessing is good when CPU usage is the limiting factor.

For this library (ioc-finder), the limitation is the CPU usage, so I went with multiprocessing.

from ioc-finder.