pantsbuild / pants-jupyter-plugin Goto Github PK

View Code? Open in Web Editor NEW

13.0 7.0 10.0 84 KB

License: Apache License 2.0

Python 100.00%

pants-jupyter-plugin's Introduction

Overview

This repo contains a set of Jupyter Notebook plugins that enable interop with pex and pants.

Installation

To install this plugin, simply invoke pip install like so:

pip install pants-jupyter-plugin

Plugin Loading

To load the plugin in your Jupyter Notebook environment, use the %load_ext command in a Notebook cell like so:

%load_ext pants_jupyter_plugin

Magics

This package provides 3 primary magics for python environment loading from pex and/or pants targets. All of these will scrub and replace the existing sys.path/environment so it's best to do this before you import any modules to avoid cached imports.

%pex_load

This magic allows you to load an existing pex file that exists on disk into the python environment.

Usage:

%pex_load <path to pex file>

%requirements_load

This magic allows you to load a transitive closure of arbitrary requirements into the environment using pex(1).

Usage:

%requirements_load <requirements>

%pants_load

This magic allows you to build and load a pants python_binary target into the environment. It works in conjunction with the %pants_repo magic, which is how you point the plugin to a given pants-compatible repo.

%pants_repo <path to pants repo>
...
%pants_load <pants target>

Development

This repo uses tox for testing and project automation. To test your changes before sending them off for review just invoke tox:

$ tox

This will auto-format code and run tests. Tests accept passthough args and pytest-xdist is installed, so you could run the full test suite with maximum parallelism via:

$ tox -p -epy3{6,7,8,9} -- -n auto

Here you run tests against all interpreters the project supports (assuming you have these all installed on your machine and on the $PATH) in parallel (the tox -p flag) and for each parallel run of tox you run the individual test methods in parallel (via passthrough args to pytest-xdist: -- -n auto).

To find out all available tox environments use tox -a or inspect tox.ini.

pants-jupyter-plugin's People

Contributors

Stargazers

Watchers

Forkers

kwlzn stuhood jsirois rahul-theorem ebenami qpc-github sarvex verdude

pants-jupyter-plugin's Issues

Document extension auto-loading

Document alternative loading for pre-installed environments re: https://ipython.readthedocs.io/en/stable/config/extensions/#using-extensions

Improve version check for pants v1 vs v2

currently, the v1 vs v2 check does a:

is_pants_v1 = pants_repo.joinpath('pants.ini').exists()

per Stu's comment here: #1 (comment)

we should make this more robust. one idea is to run ./pants --version on the initial %pants_repo repo definition such that there's a one-time cost.

Setup tests

Establish a basic tox setup
Have tox setup and install papermill
Use integration test notebooks executed by papermill runner as test coverage
Test Coverage: %requirements_load
Test Coverage: %pex_load
Test Coverage: %pants_load (and %pants_repo)

Is this extension maintained? It does not work with at least pants>=2.17

I recently started using pants v2. I tried using this extension to load some of the code and it doesn't even detect the codebase.

Looking into the code of the plugin, this is happening because this line is looking for a directory called pants, which doesn't exist. It also runs the binary directly (./pants instead of using the one in the path). The widgets don't seem to be working, hiding the error messages.

I cloned it locally and changed the code to get it to build, but still wasn't able to import it yet. However, it seems achievable with a few hours of work.

My question is: was this code abandoned by pants, or should I expect an update some time soon?

Document requirement for ipywidgets for UI support

Document requirements for ipywidgets installation in the Jupyter Notebook/Lab environment w/ linkages
This should be as easy as e.g. pip install jupyterlab_widgets in JupyterLab 3.x but should be validated.

ref: https://ipywidgets.readthedocs.io/en/latest/user_install.html#installing-in-jupyterlab-3-0

N.B. Colab has ipywidgets support built-in - and most preinstalled notebook envs will, but for fresh installs this is a requirement.

Pin pex invocations to kernel interpreter

when invoking pex(1), we should ensure that we pass:

--python {sys.executable}

so that the build-time python interpreter and kernel interpreter align 1:1 for uniform build-and-load.

context: #1 (comment)

additionally, if a pants repo target specifies interpreter constraints that do not match the kernel's interpreter we should fail in an elegant and helpful way.

context: #10 (comment)

Improve `%pex_load` error reporting.

The issue over in #37 was more confusing than need be because full stderr output of a failed venv creation in a PEX load was not presented in the notebook.

Pants jupyter plugin fails to load pex file, and error message is unhelpful

Started seeing this error intermittently since yesterday (tried using the 0.0.3 version and still see the same error):

running this line
%pex_load tmp/ml_training_env.pex gives:

HTML(value='<style>.nb-console-output-BCoZQ { background-color: black;} .nb-console-output-BCoZQ pre { color: …
Accordion(children=(Output(layout=Layout(height='300px', overflow_y='scroll'), outputs=({'output_type': 'displ…
Scrubbing sys.path and sys.modules in preparation for pex bootstrap
sys.path contains 7 items, sys.modules contains 1213 keys
sys.path now contains 7 items, sys.modules now contains 1213 keys
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
~/anaconda3/envs/mxnet_latest_p37/lib/python3.7/site-packages/pants_jupyter_plugin/plugin.py in _accordion_widget(self, title, height, collapsed)
    147         # Capture the output context.
    148         with outputter:
--> 149             yield expand, collapse, set_output_glyph
    150 
    151     def _stream_binary_build_with_output(

~/anaconda3/envs/mxnet_latest_p37/lib/python3.7/site-packages/pants_jupyter_plugin/plugin.py in _bootstrap_pex(self, pex_path)
    295 
    296                     # Bootstrap pex.
--> 297                     for path in self._pex_manager.mount(pex_path):
    298                         self._display_line(f"added sys.path entry {path}\n")
    299             except Exception:

~/anaconda3/envs/mxnet_latest_p37/lib/python3.7/site-packages/pants_jupyter_plugin/pex.py in mount(self, pex_to_mount)
    170                     )
    171                 venv_tmp = venv.parent / f"{venv.name}.{uuid4().hex}"
--> 172                 run_pex_tool(args=["venv", str(venv_tmp)])
    173                 venv_tmp.rename(venv)
    174 

~/anaconda3/envs/mxnet_latest_p37/lib/python3.7/site-packages/pants_jupyter_plugin/pex.py in run_pex_tool(args, **subprocess_args)
    142                             env=env.create(PEX_INTERPRETER=1, PEX_PYTHON_PATH=sys.executable),
    143                             check=True,
--> 144                             **subprocess_args,
    145                         ).stdout
    146                         or b""

~/anaconda3/envs/mxnet_latest_p37/lib/python3.7/subprocess.py in run(input, capture_output, timeout, check, *popenargs, **kwargs)
    510         if check and retcode:
    511             raise CalledProcessError(retcode, process.args,
--> 512                                      output=stdout, stderr=stderr)
    513     return CompletedProcess(process.args, retcode, stdout, stderr)
    514 

CalledProcessError: Command '['/home/ec2-user/anaconda3/envs/mxnet_latest_p37/bin/python', '/home/ec2-user/.cache/pants_jupyter_plugin/pex/exes/pex-2.1.56.pex', '-m', 'pex.tools', 'tmp/ml_training_env.pex', 'venv', '/home/ec2-user/.cache/pants_jupyter_plugin/pex/venvs/foo/ml_training_env.pex.bar']' returned non-zero exit status 1.

Add (TOX) config for mypy

Repair PEX_ROOT checking to consider PEX-INFO sources

context: #1 (comment)

Collisions block PEX loading inscrutably

Right now such a collision shows in a notebook like so:

✓ [Build] ./pants package /root/xxx-experimental/src/python/lib/xxx:pex_binary

$ cd /root/xxx-experimental && ./pants --pants-distdir='/root/xxxx-experimental/dist/tmpammo36v6' package /root/media-analyzer-experimental/src/python/lib/xxx:pex_binary
08:01:04.42 [INFO] Initialization options changed: reinitializing scheduler...
08:01:06.83 [INFO] Scheduler initialized.
08:01:18.21 [INFO] Wrote dist/tmpammo36v6/src.python.lib.xxx/pex_binary.pex

Successfully built /root/xxx-experimental/dist/tmpammo36v6/src.python.lib.media_analyzer/pex_binary.pex

✗ [Bootstrap] pex_binary.pex

Scrubbing sys.path and sys.modules in preparation for pex bootstrap
sys.path contains 8 items, sys.modules contains 987 keys
sys.path now contains 8 items, sys.modules now contains 987 keys
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
/usr/local/lib/python3.8/site-packages/pants_jupyter_plugin/plugin.py in _accordion_widget(self, title, height, collapsed)
    147         # Capture the output context.
    148         with outputter:
--> 149             yield expand, collapse, set_output_glyph
    150 
    151     def _stream_binary_build_with_output(

/usr/local/lib/python3.8/site-packages/pants_jupyter_plugin/plugin.py in _bootstrap_pex(self, pex_path)
    295 
    296                     # Bootstrap pex.
--> 297                     for path in self._pex_manager.mount(pex_path):
    298                         self._display_line(f"added sys.path entry {path}\n")
    299             except Exception:

/usr/local/lib/python3.8/site-packages/pants_jupyter_plugin/pex.py in mount(self, pex_to_mount)
    170                     )
    171                 venv_tmp = venv.parent / f"{venv.name}.{uuid4().hex}"
--> 172                 run_pex_tool(args=["venv", str(venv_tmp)])
    173                 venv_tmp.rename(venv)
    174 

/usr/local/lib/python3.8/site-packages/pants_jupyter_plugin/pex.py in run_pex_tool(args, **subprocess_args)
    131                 def run_pex_tool(args: Iterable[str], **subprocess_args: Any) -> bytes:
    132                     return (
--> 133                         subprocess.run(
    134                             args=[
    135                                 sys.executable,

/usr/local/lib/python3.8/subprocess.py in run(input, capture_output, timeout, check, *popenargs, **kwargs)
    510         retcode = process.poll()
    511         if check and retcode:
--> 512             raise CalledProcessError(retcode, process.args,
    513                                      output=stdout, stderr=stderr)
    514     return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command '['/usr/local/bin/python', '/root/.cache/pants_jupyter_plugin/pex/exes/pex-2.1.56.pex', '-m', 'pex.tools', '/root/xxx-experimental/dist/tmpammo36v6/src.python.lib.xxx/pex_binary.pex', 'venv', '/root/.cache/pants_jupyter_plugin/pex/venvs/e64be65ccbc3df/pex_binary.pex.80bcffde']' returned non-zero exit status 1.

Running that failing command manually shows more details:

!PEX_INTERPRETER=1 PEX_PYTHON_PATH=/usr/local/bin/python /usr/local/bin/python /root/.cache/pants_jupyter_plugin/pex/exes/pex-2.1.56.pex -m pex.tools /root/xxx-experimental/dist/tmpammo36v6/src.python.lib.xxx/pex_binary.pex venv /root/.cache/pants_jupyter_plugin/pex/venvs/e64be65ccbc3df/pex_binary.pex.test

Traceback (most recent call last):
  File "/root/.pex/unzipped_pexes/47ec9dea46acc4e6373d07558f9eb2ad1c131f91/.bootstrap/pex/pex.py", line 476, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/root/.pex/unzipped_pexes/47ec9dea46acc4e6373d07558f9eb2ad1c131f91/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
    return runner(*args)
  File "/root/.pex/unzipped_pexes/47ec9dea46acc4e6373d07558f9eb2ad1c131f91/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
    return runner(*args)
  File "/root/.pex/unzipped_pexes/47ec9dea46acc4e6373d07558f9eb2ad1c131f91/.bootstrap/pex/pex.py", line 517, in _execute
    return self.execute_interpreter()
  File "/root/.pex/unzipped_pexes/47ec9dea46acc4e6373d07558f9eb2ad1c131f91/.bootstrap/pex/pex.py", line 579, in execute_interpreter
    return self.execute_module(module, alter_sys=True)
  File "/root/.pex/unzipped_pexes/47ec9dea46acc4e6373d07558f9eb2ad1c131f91/.bootstrap/pex/pex.py", line 691, in execute_module
    runpy.run_module(module_name, run_name="__main__", alter_sys=alter_sys)
  File "/usr/local/lib/python3.8/runpy.py", line 206, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/local/lib/python3.8/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/local/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/tools/__main__.py", line 10, in <module>
    sys.exit(tools())
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/tools/main.py", line 90, in main
    result = catch(pex_command.run, pex)
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/commands/command.py", line 130, in catch
    return func(*args, **kwargs)
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/tools/commands/venv.py", line 416, in run
    populate_venv_with_pex(
  File "/root/.pex/installed_wheels/dae57b1c3877fc0567acc0103c6437da021bcfdf/pex-2.1.56-py2.py3-none-any.whl/pex/tools/commands/venv.py", line 136, in populate_venv_with_pex
    raise CollisionError(message)
pex.tools.commands.venv.CollisionError: Encountered collision building venv at /root/.cache/pants_jupyter_plugin/pex/venvs/e64be65ccbc3df/pex_binary.pex.test from /root/media-analyzer-experimental/dist/tmpammo36v6/src.python.lib.media_analyzer/pex_binary.pex:
1. /root/.cache/pants_jupyter_plugin/pex/venvs/e64be65ccbc3df/pex_binary.pex.test/bin/tensorboard was provided by:
	/root/.pex/installed_wheels/807998e8c36dd0abcbfc1c121fabb56c0bcea942/tensorflow-2.4.1-cp38-cp38-manylinux2010_x86_64.whl/bin/tensorboard
	/root/.pex/installed_wheels/2c63db6945005bcefe2b55eddd09560ae5e96c4f/tensorboard-2.8.0-py3-none-any.whl/bin/tensorboard

So tensorflow and tensorboard both provide a tensorboard console script.

Setup dist build and publish to PyPI

Setup dist build (pyproject.toml)
Setup publish tooling
Establish pypi package namespace and user mappings

Replace pex API usage with PEX_TOOLS CLI

context: #1 (comment)

Setup papermill integration tests

Have tox setup and install papermill
Use integration test notebooks executed by papermill runner as test coverage
Test Coverage: %requirements_load
Test Coverage: %pex_load
Test Coverage: %pants_load (and %pants_repo)

Document quickstart examples using Colab

Link to colab instances that auto-load the example notebooks from the repo in the repos README.md for quckstarts

e.g. https://colab.research.google.com/github/pantsbuild/pants-jupyter-plugin/blob/main/examples/pants_load.ipynb

EnvManager updates sys.path in incorrect order

If there is already a python package on the machine then it will be used instead of the package in the pex
sys.path.append(str(path_entry)) should be replaced with sys.insert(0, str(path_entry))
example:
import numpy as np
print(np.file)

output should be:
'/root/.cache/pants_jupyter_plugin/pex/venvs/af9ad1d89fc2fa245423d7cf55e8918e657a1e3b/5e00e7661f4f5acffe541d6d332efbd8d39ea6b1.pex/lib/python3.7/site-packages/numpy/init.py'
but is instead:
'/usr/local/lib/python3.7/site-packages/numpy/init.py'

pantsbuild / pants-jupyter-plugin Goto Github PK

pants-jupyter-plugin's Introduction

Overview

Installation

Plugin Loading

Magics

%pex_load

%requirements_load

%pants_load

Development

pants-jupyter-plugin's People

Contributors

Stargazers

Watchers

Forkers

pants-jupyter-plugin's Issues

Right now such a collision shows in a notebook like so:

Recommend Projects

Recommend Topics

Recommend Org