Comments (9)
Thanks for the reports! Almost certainly something to do with #919, except all our tests and CI pass (eg: https://app.circleci.com/pipelines/github/pyinvoke/invoke/288/workflows/9aa4cffc-8199-4158-ae94-cf29e1dd761d). I think the crux is:
I have a directory /tasks/*.py
I can confirm that this breaks in my one (personal) project using a tasks/ directory, but works everywhere else, so it's clearly specific to the package-not-module variant. We do have tests for that though, see eg this test and the package it should be loading.
@kuwv you got time to hunt this down? If not I will try to take a look early next week.
Anyone afflicted should have no problem temporarily pinning to invoke<2.1
, 2.1 itself doesn't have anything super juicy in it.
from invoke.
Dug some (but also got sniped by adjacent issue that throws irritating test errors, now fixed). Already kind of cross as I blew half my freakin Saturday getting turned in circles with all this, but the upshot is:
- Issue may only be reproducible under Python 3.7 and up, certain versions of it don't error on 3.6
- Issue may be related to how the import machinery "looks" from inside a package's
__init__.py
since the actual proximate symptom istasks/__init__.py
trying to do egfrom . import submodule
.- IOW, simply having a
tasks/__init__.py
holding its own self contained tasks, and not doing any from-self imports, does not reproduce the error - though it's possible I saw this during the 3.6 vs 3.7 confusion so it needs a double check.
- IOW, simply having a
My offhand guess is either:
- something
imp
did supports this pattern, in a way that the new use ofimportlib
does not, eg maybeimp
was somehow performing path manipulation under the hood. - something related to the upgrade incidentally broke this specific case (eg the logic being subtly reorganized within
Loader.load
around thesys.path.insert
call, but I just triplechecked that and it looks safe; def needs some debug printouts to be sure though)
I will try taking a closer look early next week.
from invoke.
Unfortunately, I'm currently traveling at the moment. Soonest I'll be able get to it might be Wednesday.
from invoke.
We're seeing the same problem
inv --list
Traceback (most recent call last):
File "/Users/rberger/.pyenv/versions/3.11.2/bin/inv", line 8, in <module>
sys.exit(program.run())
^^^^^^^^^^^^^
File "/Users/rberger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/invoke/program.py", line 387, in run
self.parse_collection()
File "/Users/rberger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/invoke/program.py", line 479, in parse_collection
self.load_collection()
File "/Users/rberger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/invoke/program.py", line 716, in load_collection
module, parent = loader.load(coll_name)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/rberger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/invoke/loader.py", line 80, in load
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/code/tasks/__init__.py", line 2, in <module>
from . import install, utils
ModuleNotFoundError: No module named 'tasks'
from invoke.
Moar dig:
- it's reproducible under 3.6, the main difference under 3.7 might be the exception class changed (which I think is mentioned somewhere in importlib docs). so hopefully just ignore that from above.
- sys.path munging is happening seemingly correctly (the 'task package' is being injected to 0th position)
- importlib is all new to me (not that I was an expert with
imp
either) but at a glance, the derived module spec seems accurate wrt itssubmodule_search_locations
(it's a singleton list containing thetasks/
directory, as we would expect) and itsorigin
(tasks/__init__.py
) - Annoyingly, trying to examine the issue with ipdb results in it not happening: both setting breakpoint just above the
from . import module
and callingnext
, and manually doing the import call via the ipdb REPL, work fine with no error. - At this point in execution (both inside and outside ipdb),
sys.path
continues to look like it did earlier (which was from viewpoint of the loader).
So this is pretty weird. Will continue reading importlib's docs and/or seeing if this is somehow a stdlib bug (would be pretty damn weird if so, to exist for so long & in such a critical spot, people do from .
imports all the time these days).
from invoke.
sys.modules
also not helping here, it does not contain the localtasks
package before the import in any scenario.- when I can get the import to work under ipdb, both
tasks
andtasks.module
(now renamed totasks.wtf
btw 😇) appear afterwards in sys.modules, but again, neither appear before then.
- when I can get the import to work under ipdb, both
- Reverting from
exec_module
to the "legacy"load_module
works great! so I guess that's the fallback plan for now...yes, it's deprecated, but it's still importlib instead ofimp
? 🙃- The resulting module object still looks the "same" as what we had before, eg its
__path__
is identical to the 'new style' one'ssubmodule_search_locations
.
- The resulting module object still looks the "same" as what we had before, eg its
- The docs strongly imply that as long as your loader calls
create_module
followed byexec_module
that "the import machinery does everything for you" but clearly there is some kind of gap here.- #919 leverages
spec_from_file_location
to get a spec, then stuffs that intomodule_from_spec
, which is described as usingcreate_module
...so we should be compliant here?
- #919 leverages
from invoke.
https://docs.python.org/3/library/importlib.html#approximating-importlib-import-module implies we might be missing a step (or two) here, such as manually manipulating sys.modules
. This does seem to work!
(As long as I don't fuck up and do that manipulation after the exec_module
call instead of before...wasn't reading the example closely enough, but in hindsight it makes perfect sense, execution of the module-level code in our tasks file is what needs to 'see' the module?)
I'm a bit salty that this "new improved" importlib API still requires the user to do this, but I guess I also shouldn't fault other folks for "here's our new much more flexible API and also oops you still have to do some manual labor in corner cases that wasn't required before, sorry"...
So it looks like @kuwv missed this one spot when adapting that importlib example snippet to Invoke's loader classes. I'm going to put out a bugfix on this basis, assuming all the other tests pass for me afterwards...
from invoke.
In trying to get the unit test for this breaking properly, I found:
- the unit test wasn't doing any imports in its
__init__.py
(ie the only useful code of any sort is inpackage.module
), so that's awkward. was that ever supposed to work? I don't think so... - even with that fixed, the test still passes if I comment out the core fix.
- strong implications that something in pytest land is papering over the issue somehow, even tho no other tests are loading up the
package
task tree.
- strong implications that something in pytest land is papering over the issue somehow, even tho no other tests are loading up the
- oho: no, if I put literal garbage in the submodule, a bunch of other tests also blow up, so something they are doing is incidentally importing
package.module
. so that's almost surely what is making this test incidentally pass (by accidentally warming up the cache in sys.modules), but is its own mystery.- Sure enough if I use
pytest -k
to select just the package test, it fails as we expect. God bless testing...
- Sure enough if I use
- aha! I was ready to blame pytest's assertion rewriting, but that is only proximately at fault - we're reusing this test fixture from inside another (more commonly used) one that tests namespacing. ugh. if I make a separate 'package test tree' so these two don't overlap, I bet we'll be good.
from invoke.
Thank you!
from invoke.
Related Issues (20)
- Change default shell without a config file or Context? HOT 3
- Task decorator removes docstring HOT 1
- Run tasks relative to `tasks.py` HOT 2
- How to manually set short flags? HOT 1
- context.run(): Expose API to pass command and separate args instead of a single string HOT 2
- Specify a config file per Collection... possible?
- Collection mix-up when cross importing invoke tasks
- Running post tasks even if the main task fail
- `--help` after the command treats `--help` as positional argument HOT 2
- Is there support for making invoke.yaml context settings cli flags?
- EncodeWarning's when running on python >= 3.11
- @task(pre=[call(setup, 'qwe')]) fails with "NameError: name 'call' is not defined" HOT 1
- Config run should handle shell paths with spaces HOT 2
- Is it possible to only mock one command and run the rest with MockContext?
- autoprint generators correctly
- Generate help infomation of task args from function docstring.
- Importing Python modules from a scripts directory beside the tasks directory
- Recommended way to forward arguments to commands HOT 2
- Printing Promise objects from asynchronous Runner.run() raises Attribute Error
- Sudo showing password in clear
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from invoke.