Scanner dependency problem about domain-scan HOT 3 OPEN

This makes perfect sense. Though:

This goes against the intent of the splitting of requirements.txt in #224.

Splitting the requirements that way was meant (for me, anyway) to make packaging Lambda functions lighter by eschewing local-only reqs, but not necessarily to make local dependencies lighter by eschewing Lambda-only reqs.

I don't think there's as much of a need to do the latter, since there usually aren't the same kinds of constraints on network/size/etc locally as in Lambda. But maybe I'm not thinking about broader use cases. Would it help your use case to split this out further?

from domain-scan.

I don't think we need to split the requirements files out further. Ideally I'd like it if you only needed to install requirements.txt and the scanners you want to use locally. Right now that doesn't work because of the imports at the top of the files in scanners/, but I suspect we could get there if we moved some imports around in the scanner classes that @tadhg-ohiggins is working on (like scanners/trustymail.py is doing, although for a different reason).

Since my scanning kicks off the Lambda processes from Docker it's nice if that container doesn't have to actually install the actual scanning libraries. It's not a huge deal, but it does make the Docker images smaller. And it removes what are really unnecessary dependencies.

from domain-scan.

That makes sense, and is something I wasn't intuiting because I'm not currently using Docker in my environment.

The ideal is probably to have a standard way of conditionally importing dependencies.

One complication, though - for the pshtt scanner, pshtt is needed even locally anyway, because it uses pshtt.load_suffix_list() in the init method to manage caching the PSL locally. You're currently handling all of that in scan, using a locally packaged PSL, and don't bother with that optimization.

It seems very possible that other scanners may make use of third party deps in their init() or init_domain() or to_rows() functions, not just their scan() functions. That may make it more complicated/annoying to do that kind of separation.

from domain-scan.