This is heavily related to #28, though I believe they were half way incorrect when they said importing a second time would result in a working execution. Based on what I've seen, when the import fails it ends up in a half-way state where the import isn't shown in globals() or dir(), but can be seen from sys.modules(). This breaks some stuff and likely won't result in an actually working function/module.

Took some time to dig into this issue and it's not actually an issue with relative path imports, that's more of a symptom, the actual issue stems from the fact that nested modules don't get the proper value set for their __package__ attribute.

This is caused by line 224 in the current code where, no matter what the path is, it splits it and grabs the first item in the index and uses that as the package. So a for example:

Package1_
         |_Module1
         |_Package2_
                    |_Module2
                    |_Module3

If you have the following file structure and Module3 contains 'import .Module2' that will fail, because python uses the __package__ attribute to determine relative paths and all of these modules have Package1 as their __package__.

unable to get followig imported https://github.com/rosomri/atlassian-python-api imported via

with httpimport.github_repo('rosomri', 'atlassian-python-api', ref='confluence_bugfix_export_pdf'):
import atlassian

Traceback (most recent call last):
File "C:\develop\git\scripts\collectors\confluence.py", line 213, in
confluence = connect_to_confluence(args.cf_url, args.cf_user, args.cf_token)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\develop\git\scripts\collectors\confluence.py", line 48, in connect_to_confluence
return atlassian.Confluence(
^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'atlassian' has no attribute 'Confluence'

What i am doing wrong?

This is quite self explanatory.
It is quite useful to have a log of all the requests made by this module, maybe by a different logger than the httpimport one:

req_logger = logging.getLogger("httpimport.requests")

so it can have all kinds of controls like, log to file, colors, etc.

When debugging, or when working with network issues this could be a useful tool.

Thanks for this wonderful package. Until recently, I could use the following code to load a module from a gist:

url = "https://gist.githubusercontent.com/LukasWallrich/42dea3211f0bde452781dd9b69c8199a/raw/"

with httpimport.remote_repo(["Gmodel"], url):
    from Gmodel import GProblem

After updating various modules, it now it fails with AttributeError: 'list' object has no attribute 'endswith' - not sure what changed and how I can work around this ...

When we want to import cv2 library It shows module not found error.
I know that we need to install it by opencv-python but we can't use the statement "import opencv-python".
How to import the libraries that have a different name in usage and installation. Which one to use.
If we can set it in the profile settings how to do it?

For my case, the codes below also work though I didn't specify modules

import httpimport
url = "https://gist.githubusercontent.com/zhqu1148980644/435304b90d1511099b070414c760e968/raw/e388a5ba1e3580eb9f4cb9e3061ab1f47210cf94"
with httpimport.remote_repo([], url):
       import task
task.task1("asdasd")

I can't seem to get remote_repo to work. I've tried accessing modules in my private Gitlab instance and an example module in Github. I keep getting "No module named x" error. I assume it has something to do with my code but I've tried a few different iterations with no luck. It would be nice to see a working example in the readme that pulls from a publicly available python package. The load() method also can't seem to find the module.

Here's a snippet.

#!/usr/bin/env python3
from httpimport import remote_repo
import httpimport
with remote_repo(['myPackage'], 'https://github.com/BillMills/python-package-example/blob/master/myPackage/'):
    import somePython

I want to load this file and call the main function: https://raw.githubusercontent.com/dmlc/xgboost/d3f06467797acd61e80e87be38fb9812a016a64e/demo/guide-python/categorical.py

How do I do this? I see your example that loads a file from a gist that contains a single function, but I can't get this technique to work with the file above.

Given that the URLs of the raw files in a BitBucket repo follows the format:

https://bitbucket.org/{user}/{repo_name}/raw/{commit_hash/branch_name}/{directory/}*{file_name}

a bitbucket_repo() context can be created, similar to already implemented github_repo().
Arguments can be exactly the same as in github_repo():

• username
• repo name
• branch/commit

The Wheel Python distributable format is superior to Egg for multiple reasons. It is also the Python Standard (for the time being) backed by a PEP.

It is a good idea to upgrade to wheel packaging.

A comprehensive list of differences.
https://packaging.python.org/discussions/wheel-vs-egg/

Hi, I ran the code:

import httpimport
profile = """
[github]
headers:
  Authorization: token ghp_WjewI0mwxxxxxxxxxxxxxxxxez9"""

httpimport.set_profile(profile)

with httpimport.github_repo('myid', 'myrepo', profile='github'):
    import mypkg

but it gave ModuleNotFoundError.
could you please test with this function, and maybe make more instruction for us?

with httpimport.github_repo('operatorequals','covertutils'):
    from covertutils.handlers import BaseHandler

ImportError: cannot import name 'BaseHandler' from 'covertutils.handlers' (https://raw.githubusercontent.com/operatorequals/covertutils/master/covertutils/handlers/__init__.py)

httpimport version: 1.3.0

Given that Python has a generic interface on opening tar files (https://docs.python.org/3/library/tarfile.html), and this interface does not vastly differ from zipfile which is already integrated, it is possible to extend the support from .zip files to all .tar.* files.

This will need an interface change, making the zip keyword arguments deprecated, hence needs a major version update.

Hi! httpimport is ok with GitHub, it would be useful to have also Dropbox support. But it is not fundamental.
Thank you for your support John,
Bye

There can be a function suite that directly search PyPI (through the API) and loaded sdist or wheel packages directly.
Also, versions can be set as function parameters, or even a requirements.txt file.

Since Python 3.4, many methods of the imp package, including imp.new_module, are deprecated.

I've been using imp.new_module myself in a side-project and I was curious if you had solved this problem. new_module is still working, but as it's an implementation detail, it might or might not disappear.

I couldn't help but notice that covertutils is python2 only, is it the same for httpimport ?
Any plan for python3 ?

I am also interested in similar topics, doing distributed software development in python, and I maintain a python 2/3 compatibility package for custom importers, in case it helps : https://github.com/asmodehn/filefinder2

Hi there,
this is very neat, would be fantastic to find a way to also support private gitlab/github maybe via an ssh key?
Cheers!

Starting in python 3.12 the find_module has been removed in favor of find_spec. The load_module function has also been deprecated in favor of the create_module and exec_module functions respectively. In order for this project to continue working for the latest python, these functions will need added. I have implemented them in my remote import hook package I wrote from the ground up if you want some ideas as to how this could be implemented. My repo is at od_import.

Coming from this StackOverflow question, how can I import a hello() function from this gist:
https://gist.github.com/kolibril13/fc84c6940a8aeab8071c5d54d4df3a7b ?
My idea would be something like:

import httpimport
with httpimport.load("https://gist.githubusercontent.com/kolibril13/fc84c6940a8aeab8071c5d54d4df3a7b/raw/bedba99860d301fdc20edac5291b52359a46d1e3/"):
    import hello

I am grateful for any help 👍🏼

Given that the URLs of the raw files in a GitLab repo follows the format:

https://gitlab.com/{user}/{repo_name}/raw/{commit_hash/branch_name}/{directory/}*{file_name}

a gitlab_repo() context can be created, similar to already implemented github_repo().
Arguments can be exactly the same as in github_repo():

• username
• repo name
• branch/commit

The context can be tested with the below Python project:
https://gitlab.com/BananenBernd/basictools

As of now, httpimport works for only public repositories.

For developers in companies, enterprise environments, and generally people that work in private repositories a method is needed to remotely load packages/modules from private repos.

So this ticket addresses the integration of such methods (as per design and implementation) into the [github|gitlab|bitbucket]_repo methods:

• GitHub

  • Design
  • Implementation

• BitBucket

  • Design
  • Implementation

• GitLab

  • Design
  • Implementation

The ticket will be updated for Design decisions and for accepted implementations for each of the platforms, with mentions to answers and commits.

It is highly possible, that a package/module contains the .pyc (python compiled) files in the HTTP/S repo/directory. This could happen, if the package/module has been run/imported, at least once, locally.
Yet the httpimport module, as is, looks only for .py files.

There can be a modification/flag/optional argument, that enables the Importer class to look for .pyc files and strip their contents to make them executable. This can be done, as these files include the output of compile() Python function, and the output of compile() can be used with exec function/statement.

This will probably enhance loading time!

The modules parameter was used to declare what packages/modules can be imported from a certain URL.
This parameter was being used so the importer wouldn't try requests for modules that didn't exist at the "find_module" phase. This was done to not generate useless traffic. Minimal traffic was a functional requirement of this project in previous versions, as it was a RAT stager

Now, it is possible to try requests for commands in dynamic ways, hence the whole positional parameter can be omitted in a version change that changes the interface of:

HttpImporter
remote_repo
add_remote_repo

Reading through your documentation you say you've disable HTTP_PROXY for pypi because it's an HTTPS request. I may have missed the code change, but if you're still using ProxyHandler, that isn't related to the HTTP_PROXY and HTTPS_PROXY variables which I believe you're referencing in your README. ProxyHandler with work with proxied requests to both http and https sites.

Starting a Python process with preinstalled remote repos would be useful. Something like the following:

$ python -m httpimport --insecure github://operatorequals/covertutils http://somerepo:8000/python-packages
[...]
>>> import covertutils
>>>

For this issue to be implemented, Issue 21 has to be implemented first, to not declare what packages should be implemented from given URLs.

If it's possible to get the source code of the imported class/function

This is great!
Thank you for your work.

I have a private repo that I would like to import from.
How do I do that?

Hello,
I am not sure what I am doing wrong but this should work right?v

with httpimport.github_repo('priamai','stixorm',ref='master'):
  import stixorm
  from stixorm.module.typedb import TypeDBSink,TypeDBSource

I am getting an:

ImportError: cannot import name 'TypeDBSink' from 'stixorm.module.typedb' (https://raw.githubusercontent.com/priamai/stixorm/master/stixorm/module/typedb.py)


---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.

Hi! First of all thank you for your library.
John can you please confirm that httpimport works with remote libraries and packages only if:

• these remote libraries/packages are full (inside the folder 'master' user finds all the external required libraries) or

• all the dependencies (external required libraries) are already installed properly in the python environment?

An other question is: if I want to import a remote library/package named "A" that requires some external packages not available inside the 'master' folder of "A" (for example the one I sent you, fipy), can I create an alternative library/package named "A_full" with inside all the external required libraries/packages (pure-python or not-pure python like *.dll, *.so, *.a, *.lib, ... ) in order to be able to import all without doing 'pip install ...' before using httpimport with lib "A"?

A simpler example: if remote lib "A" requires a remote lib "B" with a different url from "A", httpimport can't import "A" because httpimport can't know where to search for "B" in internet. But if I add lib "B" inside the master folder of "A" creating a github clone of the lib "A" + lib "B", is it possible to use httpimport with the cloned lib "A"?

I do mostly scientific computation with Python and sometimes I use a remote server that offers a full python environment with a lot of useful math packages (https://sagecell.sagemath.org/). Sometimes I need some packages not installed in the remote server, so it would be very useful to import temporary these packages with a remote importer via web (user can't install with pip any library in the server, user has only RAM and CPU time for each calculation).

Thank you!
Regards
Matteo

It would be great to support zip'd Python packages. There is already a built-in functionality with ZipImporter, but it needs the .zip file to be on disk (listable and with a filename).

Support for .zip has to be in-memory only.

It is possible that the example code below will be useful:

https://stackoverflow.com/questions/39135750/python-load-zip-with-modules-from-memory

It seems as though the package is imported once, and continues to import from temp files without fetching the newest version from github.

My first use case for httpimport was a logging library common to several scripts:

# toconsole.py

import logging
import os

log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s %(message)s')
handler_console = logging.StreamHandler()
level = logging.DEBUG if 'DEV' in os.environ else logging.INFO
handler_console.setLevel(level)
handler_console.setFormatter(formatter)
log.addHandler(handler_console)

# disable httpimport logging except for errors+
logging.getLogger('httpimport').setLevel(logging.ERROR)

A simple usage such as

import httpimport

httpimport.INSECURE = True
with httpimport.remote_repo(['githublogging'], 'http://localhost:8000/') :
    from toconsole import log

log.info('yay!')

gives the following output

[!] Using non HTTPS URLs ('http://localhost:8000//') can be a security hazard!
2019-08-25 13:56:48,671 yay!
yay!

The second (bare) yay! must be coming from httpimport, namely from its logging setup.

How can I disable httpimport logging, or better - raise its level so that only errors+ are logged? The logging.getLogger('httpimport').setLevel(logging.ERROR) I tried did not work.

Gitea is a self-hostable Modern Git Service. It shares many features with Gitlab, Github and the rest, such as HTTP raw file download. Thous it can be supported by httpimport

A gitea_repo can be implemented by formatting URLs such as the one below (taken from Gitea's Swagger page):

https://try.gitea.io/test/test/raw/branch/master/README.md

I have tried to rewrite this module as a class but with no luck. But somehow i believe it would be possible.
If the module can be converted to a class it would more awesome.
E. G.
a = httpimport()
b = a.import(foo)
c = b(new_func)

I have run that code but i have error in importing

this is my repo link

and this is my code

with httpimport.github_repo('AKHACKER-program4hack', 'searcher', branch='master'):
    import searcher.forsearch
    s = forsearch.Searcher()
    s.printuser()

And i am getting this error:

Traceback (most recent call last):
  File "/home/ak/Desktop/development/pythonwork/testing/httpimport/testing.py", line 8, in <module>
    s = forsearch.Searcher()
NameError: name 'forsearch' is not defined

Can anyone tell me what's the problem

If you attempt to do the following in a httpimport'd file

from .file import classname

it results in the error

ImportError: cannot import name 'classname' from package.file

File "/usr/local/lib/python3.7/dist-packages/httpimport.py", line 254, in load_module
    exec(final_src, mod.__dict__)
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.7/dist-packages/httpimport.py", line 254, in load_module
    exec(final_src, mod.__dict__)
  File "<string>", line 26, in <module>
  File "/usr/local/lib/python3.7/dist-packages/httpimport.py", line 254, in load_module
    exec(final_src, mod.__dict__)
  File "<string>", line 1, in <module>

https://www.python.org/dev/peps/pep-0328/

Yet....

if you import again right after it fails... it imports correctly....

I am unable to determine whether the library in my gitlab repo is getting picked up; Its not throwing any errors. But when I try to access the function in from the imported module its saying that attribute does not exist. Any way for me to access the httpimporter class variables to check whats going on?

Test.py

import httpimport
with httpimport.remote_repo("https://gitlab.mycompany.com/dz/co_library/"):
    import passman

print(dir(passman))
print(passman.__url__)
print(passman.list_profiles())

when I run the script to test it, im not seeing any of the function inside of passman.py and I get the following error:

['__builtins__', '__doc__', '__file__', '__loader__', '__package__', '__path__', '__spec__', '__url__']
https://gitlab.mycompany.com/dz/co_library/passman.py
Traceback (most recent call last):
   File "Test.py", line 6, in <module>
     print(passman.list_profiles()
AttributeError: module 'passman' has no attribute 'list_profiles'

init.py from co_library

from . import *

setup.py from co_library

from setuptools import setup, find_packages

setup(name="co_library", 
    version="0.1.0",
    packages=["passman",],
    py_modules=["passman",],
    )

Firstly, thank you for the project - I think it's amazing and shall be advertised wider. It's part of my secret sauce to deploy AI/ML models into distributed systems.
I used version 0.7.2 to load a pickled pre-trained Automata (Trie structure)

import httpimport
with httpimport.remote_repo(['utils'], "https://raw.githubusercontent.com/applied-knowledge-systems/the-pattern-automata/main/automata/"):
    import utils
from utils import loadAutomata, find_matches

with cryptic stack trace:

[!] 'nt' not found in HTTP repository. Moving to next Finder.
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/home/alex/anaconda3/envs/thepattern_3.7/lib/python3.7/site-packages/httpimport.py", line 232, in load_module
    exec(module_src, mod.__dict__)
  File "<string>", line 36, in <module>
  File "<string>", line 5, in loadAutomata
  File "/home/alex/anaconda3/envs/thepattern_3.7/lib/python3.7/site-packages/joblib/__init__.py", line 113, in <module>
    from .memory import Memory, MemorizedResult, register_store_backend
  File "/home/alex/anaconda3/envs/thepattern_3.7/lib/python3.7/site-packages/joblib/memory.py", line 15, in <module>
    import pathlib
  File "/home/alex/anaconda3/envs/thepattern_3.7/lib/python3.7/pathlib.py", line 4, in <module>
    import ntpath
  File "/home/alex/anaconda3/envs/thepattern_3.7/lib/python3.7/ntpath.py", line 257, in <module>
    from nt import _getvolumepathname
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 640, in _load_backward_compatible
KeyError: 'nt'

I went on a path debugging joblib, but actual loading works:

import ahocorasick 
import joblib
Automata=joblib.load("automata_fresh_semantic.pkl.lzma")

full code for loading https://github.com/applied-knowledge-systems/the-pattern-automata/blob/main/automata/utils.py`
Code for creating automata is simple and hosted in the same repo: https://github.com/applied-knowledge-systems/the-pattern-automata

I am reverting to 0.7.2, but any suggestions for uplifting are welcome.

nvm

Hi, I am trying to import from gist in a google collab:

import httpimport
url = "https://gist.githubusercontent.com/ricardofrantz/96ee9d9fc253c2af30b0b9d6405f35ef/raw/2de95ca2d5b7b51aa2a3cfc6f248bc46eb9980d7"
with httpimport.remote_repo(url):
    import diff_matrix as df
L = df.diff_matrix()
print(L)

it seems there is some blockage in the ambient... is there a known way to bypass it?

WARNING:httpimport:[!] Using non HTTPS URLs ('http://localhost:8000//') can be a security hazard!
DEBUG:httpimport:FINDER=================
DEBUG:httpimport:[!] Searching diff_matrix
DEBUG:httpimport:[!] Path is None
INFO:httpimport:[@] Checking if in declared remote module names >
INFO:httpimport:[-] Not found!
DEBUG:httpimport:FINDER=================
DEBUG:httpimport:[!] Searching diff_matrix
DEBUG:httpimport:[!] Path is None
INFO:httpimport:[@] Checking if in declared remote module names >
INFO:httpimport:[-] Not found!
DEBUG:httpimport:FINDER=================
DEBUG:httpimport:[!] Searching diff_matrix
DEBUG:httpimport:[!] Path is None
INFO:httpimport:[@] Checking if in declared remote module names >
INFO:httpimport:[-] Not found!
DEBUG:httpimport:FINDER=================
DEBUG:httpimport:[!] Searching diff_matrix
DEBUG:httpimport:[!] Path is None
INFO:httpimport:[@] Checking if in declared remote module names >
INFO:httpimport:[-] Not found!
DEBUG:httpimport:FINDER=================
DEBUG:httpimport:[!] Searching diff_matrix
DEBUG:httpimport:[!] Path is None
INFO:httpimport:[@] Checking if in declared remote module names >
INFO:httpimport:[-] Not found!
DEBUG:httpimport:FINDER=================
DEBUG:httpimport:[!] Searching diff_matrix
DEBUG:httpimport:[!] Path is None
INFO:httpimport:[@] Checking if in declared remote module names >
INFO:httpimport:[-] Not found!
DEBUG:httpimport:FINDER=================
DEBUG:httpimport:[!] Searching diff_matrix
DEBUG:httpimport:[!] Path is None
INFO:httpimport:[@] Checking if in declared remote module names >
INFO:httpimport:[-] Not found!
DEBUG:httpimport:FINDER=================
DEBUG:httpimport:[!] Searching diff_matrix
DEBUG:httpimport:[!] Path is None
INFO:httpimport:[@] Checking if in declared remote module names >
INFO:httpimport:[-] Not found!
DEBUG:httpimport:FINDER=================

ModuleNotFoundError Traceback (most recent call last)
in
2 url = "https://gist.githubusercontent.com/ricardofrantz/96ee9d9fc253c2af30b0b9d6405f35ef/raw/2de95ca2d5b7b51aa2a3cfc6f248bc46eb9980d7"
3 with httpimport.remote_repo(url):
----> 4 import diff_matrix as df
5 L = df.diff_matrix()
6 print(L)

ModuleNotFoundError: No module named 'diff_matrix'

---

NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.

Thanks for your help,
Ricardo

I have a large number of non-package(i.e. single file) libraries to pull in. The documentation seems to focus a lot on packages. Is there a way to pull them in without converting them to packages?

HTTP/S Proxy Support might be useful in case a host cannot directly connect to the Web.
It could be implemented by just passing a proxy list to a Global Variable PROXY_LIST like follows:

import httpimport
httpimport.PROXY_LIST=[
  ('127.0.0.1', 8080),    # Some Burp Installation?
  ('randomhttpproxy.org',80),
 ]

Not sure if this is something that's feasible, but there are multiple packages that required .so files to work, such as psutil. Is there a possibility of supporting packages like that? I understand there are implications with matching architecture and what not, but unless I'm missing something (entirely possible), there's no way to import a package like that currently.

Given that the module will gain support for tar.* files (Issue #16), a function can be implemented that directly fetches tarballs from pypi.org, visiting the imported module's page, such as https://pypi.org/project//#files.
This page contains download links for all versions of module, that are not statically constructed. These links can be followed to download the tarball and later import it.

So a function pip_load can have the below signature:

def pip_load(module, version="latest"):
[...]

Additionally, a context can be created, named pip() or pip_repo, directly importing to memory from PyPI, with the below usage:

with pip():
  import module_in_pip

Finally, a function can be created that can parse requirements.txt files and directly load the dependencies using calls to pip_load or translating to pip() context commands.