New version 0.9.2 breaks loading of joblib pickled model about httpimport HOT 5 CLOSED

@operatorequals thank you for the update. I had thought that your design will make security architects scream and I am glad you are not disappointed me there with your story. There is no need for redesign - I will use import from GitHub from private repos with authentication so your current design suite me better and I will be rewriting my side of pipelines.

from httpimport.

I am aware that some issue was introduced in 0.9.2. I am looking into it, as the unit tests run well on both, so there should be a coverage issue!

Thanks a lot for your verbose report!

from httpimport.

Hello again @AlexMikhalev!

So, I have drilled down to the issue you are having and I prepared a brief explanation of why it is happening and a way to use v0.9.3:

Prior 0.9.0 version, it was mandatory to use an argument in most httpimport functions that had to be either a str or list indicating what packages/modules where expected to be loadable from the given URL. So using ['utils'] as your first argument indicated that (only) utils module can be loaded from your URL (and maybe you can see where this is going by now).
Every import statement in the clause was then checked against this list (or str) in the Finder function and if it existed (or str matched), the loader was called and THEN the actual HTTP/S call was happening, trying to bring the content of the module.
This was a deliberate design decision taken way back around 2017, as this module started as a Python stager for Python based malware (I was working as a Security Engineer / Red Teamer / Penetration Tester back then and I happened to need such a tool). This meant that httpimport had to do as little traffic as possible and double check if a request was needed to be made. The encrypted .zip loading feature was also a decision taken back when httpimport was effectively a "malware".

With the 0.9.0 release I figured out that, as now httpimport is used by Data Analysts and a very different type of beasts than malware devs, it does not need to be so restrictive on the traffic it does. So I removed the str or list argument that indicated what can be loaded and httpimport now tries ALL modules stated in an import statement inside its clause (by "clause" I mean the tabbed lines under a remote_repo call). Yet, the code trying the module is still placed in the Importer's Loader method, assuming that the module is certainly there (as the Finder phase has finished successfully). Failing if the module is not there. E.g nt module is not available to be loaded, and the importer fails.

For a proper fix of this bug, more or less a redesign needs to be done which can finally get a real v1.0.0 version released. But right now I am a bit busy having a life. Yet, I want to do that, and eventually I will!

TL; DR

If you want to use the v0.9.3 version, you can deliberately load your utils module as below:

>>> utils = httpimport.load("utils", "https://raw.githubusercontent.com/applied-knowledge-systems/the-pattern-automata/main/automata/")
>>> dir (utils)
['Automata', '__builtins__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'find_matches', 'loadAutomata']

Tada! Loaded!

If you have any questions/suggestions/remarks on the matter, please answer below! Thanks again for pointing out this old design issue that came as a regression bug!

from httpimport.

@operatorequals thank you for the update. I had thought that your design will make security architects scream and I am glad you are not disappointed me there with your story. There is no need for redesign - I will use import from GitHub from private repos with authentication so your current design suite me better and I will be rewriting my side of pipelines.

Great! Feel free to open a PR should you create changes that you find generic and/or useful in the module!

from httpimport.

Way easier than I thought! Already passes unit tests (and should work for your case as well).
https://github.com/operatorequals/httpimport/blob/rewrite/httpimport2.py#L23

from httpimport.