parrt / tensor-sensor Goto Github PK

The goal of this library is to generate more helpful exception messages for matrix algebra expressions for numpy, pytorch, jax, tensorflow, keras, fastai.

Home Page: https://github.com/parrt/tensor-sensor

License: MIT License

Python 10.65% Jupyter Notebook 89.35%

numpy pytorch matrix vector debugging tracing deep-learning python tensorflow jax

tensor-sensor's Introduction

Tensor Sensor

See article Clarifying exceptions and visualizing tensor operations in deep learning code and TensorSensor implementation slides (PDF).

(As of September 2021, M1 macs experience illegal instructions in many of the tensor libraries installed via Anaconda, so you should expect TensorSensor to work only on Intel-based Macs at the moment. PyTorch appears to work.)

One of the biggest challenges when writing code to implement deep learning networks, particularly for us newbies, is getting all of the tensor (matrix and vector) dimensions to line up properly. It's really easy to lose track of tensor dimensionality in complicated expressions involving multiple tensors and tensor operations. Even when just feeding data into predefined Tensorflow network layers, we still need to get the dimensions right. When you ask for improper computations, you're going to run into some less than helpful exception messages.

To help myself and other programmers debug tensor code, I built this library. TensorSensor clarifies exceptions by augmenting messages and visualizing Python code to indicate the shape of tensor variables (see figure to the right for a teaser). It works with Tensorflow, PyTorch, JAX, and Numpy, as well as higher-level libraries like Keras and fastai.

TensorSensor is currently at 1.0 (December 2021).

Visualizations

For more, see examples.ipynb at colab. (The github rendering does not show images for some reason: examples.ipynb.)

import numpy as np

n = 200         # number of instances
d = 764         # number of instance features
n_neurons = 100 # how many neurons in this layer?

W = np.random.rand(d,n_neurons)
b = np.random.rand(n_neurons,1)
X = np.random.rand(n,d)
with tsensor.clarify() as c:
    Y = W @ X.T + b

Displays this in a jupyter notebook or separate window:

Instead of the following default exception message:

ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 764 is different from 100)

TensorSensor augments the message with more information about which operator caused the problem and includes the shape of the operands:

Cause: @ on tensor operand W w/shape (764, 100) and operand X.T w/shape (764, 200)

You can also get the full computation graph for an expression that includes all of the sub-expression shapes.

W = torch.rand(size=(2000,2000), dtype=torch.float64)
b = torch.rand(size=(2000,1), dtype=torch.float64)
h = torch.zeros(size=(1_000_000,), dtype=int)
x = torch.rand(size=(2000,1))
z = torch.rand(size=(2000,1), dtype=torch.complex64)

tsensor.astviz("b = W@b + (h+3).dot(h) + z", sys._getframe())

yields the following abstract syntax tree with shapes:

Install

pip install tensor-sensor             # This will only install the library for you
pip install tensor-sensor[torch]      # install pytorch related dependency
pip install tensor-sensor[tensorflow] # install tensorflow related dependency
pip install tensor-sensor[jax]        # install jax, jaxlib
pip install tensor-sensor[all]        # install tensorflow, pytorch, jax

which gives you module tsensor. I developed and tested with the following versions

$ pip list | grep -i flow
tensorflow                         2.5.0
tensorflow-estimator               2.5.0
$ pip list | grep -i numpy
numpy                              1.19.5
numpydoc                           1.1.0
$ pip list | grep -i torch
torch                              1.10.0
torchvision                        0.10.0
$ pip list | grep -i jax
jax                                0.2.20
jaxlib                             0.1.71

Graphviz for tsensor.astviz()

For displaying abstract syntax trees (ASTs) with tsensor.astviz(...), you need the dot executable from graphviz, not just the python library.

On Mac, do this before or after tensor-sensor install:

brew install graphviz

On Windows, apparently you need

conda install python-graphviz  # Do this first; get's dot executable and py lib
pip install tensor-sensor      # Or one of the other installs

Limitations

I rely on parsing lines that are assignments or expressions only so the clarify and explain routines do not handle methods expressed like:

def bar(): b + x * 3

Instead, use

def bar():
	b + x * 3

watch out for side effects! I don't do assignments, but any functions you call with side effects will be done while I reevaluate statements.

Can't handle \ continuations.

With Python threading package, don't use multiple threads calling clarify(). multiprocessing package should be fine.

Also note: I've built my own parser to handle just the assignments / expressions tsensor can handle.

Deploy (parrt's use)

$ python setup.py sdist upload

Or download and install locally

$ cd ~/github/tensor-sensor
$ pip install .

TODO

can i call pyviz in debugger?

tensor-sensor's People

Contributors

Stargazers

Watchers

tensor-sensor's Issues

Default font's spacing on mac incorrect

Compare with courier new

Relevant code:

with ts.explain(fontname="Courier New"):
    y = np.multiply.outer(x, np.arange(2.0))

various enhancements

great works!

debug these math expression code is a very big problem

not only scalar vector matrix , high rank tensor support

Visualizing 3D tensors and beyond:

TensorDim is contain : left to right [N,C,H,W] N->C->H->W
TensorDim like flatten Onions,cabbage box in box represent dim
TensorDim has real represention information (N,C,H,W etc.)

namedtensor support

plaidml dsl
Types Tensor TensorDim TensorIndex

represent actual data:
audio, 1D plot, text : vector
image ,2D plot : matrix
etc.

expression graph:

different color input variable ,leaf parameters/variable
with temporal variable / activation : "Road width"
elementwise leaf / elementwise or slice or other connection(edge) print
AST forward & backward mode

animation:

expression step computation mode (debug)
slice ,reshape, .T() other manipulate N-d array TensorDim/TensorIndex operators
example : matmul = m@v
m@v = (v.squeeze(0).expand_as(w) * w).sum(1,keepdim=True).unsqueeze(1)

more ... conv2d etc

interactive:

interactive build block(visual programing)
reverse interactive(debug) : selected tensor elements and follow it in expression graph
selected element : muti view , like convolution-visualizer Input(Inputgrad) Weight(Weightgrad) Output(Outputgrad) views

NN support:

NN module visualization (conv2d)
bigger computation graph : pytorchrec
multi computation graph visualization and live debug

other useful link

SimpNet memory map

Visualize the virtual address space of a Windows process on a Hilbert curve.

nvidia ppt

tensor network

einops

Window Support?

Can this library run on Window? I am getting this error when I use a notebook to run example

OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a
single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_L
IB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.
[I 11:04:32.509 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports
kernel 7aec0344-7569-4f1d-bb6e-7cdd592802b5 restarted

Not an issue - livepython fyi

I know notebooks are most popular for ml - but consider there's some that prefer to drop in to command line and run python code.
In this case - there's this library that is quite amazing that could potentially be plugged in.
https://github.com/agermanidis/livepython

Suppress visualisation of () as operator in tree

Hello.

I am using tensor-sensor to visualise DAGs of the domain specific language Slate, which is used in a code generation framework for Finite element methods called Firedrake. Slate expresses linear algebra operations on tensors. I am using tensor-sensor to visualise the DAG before and after an optimisation pass. An example would be the following:

Before optimisation:
triplemul_beforeopt.pdf
After optimisation:
tripleopt_afteropt.pdf

While the visualisation of the tree is correct in both cases, I would quite like to suppress the node for the brackets (i.e. for the SubExpr node) to avoid confusion about the amount of temporaries generated. Is there already a way of controlling this as a user and if not would there be interest in supporting it?

Best wishes and thanks in advance,
Sophia

explain() likely affects PyTorch autograd operations

PyTorch: use with torch.no_grad() to prevent messing up autograd computations.

Not available to install

Hi, i am getting an error when trying to install. I am using windows and tensorflow 2.3.0

C:\WINDOWS\system32>pip install tensor-sensor
Processing c:\users\rafar\appdata\local\pip\cache\wheels\77\93\25\cda0532b890faccb41674971ad0ce0d93618c51c0a86effcd1\tensor_sensor-0.1b1-py3-none-any.whl
Collecting torch
  Using cached torch-0.1.2.post2.tar.gz (128 kB)
Requirement already satisfied: IPython in c:\program files\python37\lib\site-packages (from tensor-sensor) (7.8.0)
Collecting tensorflow
  Using cached tensorflow-2.3.1-cp37-cp37m-win_amd64.whl (342.5 MB)
Requirement already satisfied: graphviz>=0.14.1 in c:\program files\python37\lib\site-packages (from tensor-sensor) (0.14.2)
Requirement already satisfied: matplotlib in c:\program files\python37\lib\site-packages (from tensor-sensor) (3.1.1)
Requirement already satisfied: numpy in c:\program files\python37\lib\site-packages (from tensor-sensor) (1.17.0)
Requirement already satisfied: pyyaml in c:\program files\python37\lib\site-packages\pyyaml-5.1.2-py3.7-win-amd64.egg (from torch->tensor-sensor) (5.1.2)
Requirement already satisfied: pickleshare in c:\program files\python37\lib\site-packages (from IPython->tensor-sensor) (0.7.5)
Requirement already satisfied: backcall in c:\program files\python37\lib\site-packages (from IPython->tensor-sensor) (0.1.0)
Requirement already satisfied: colorama; sys_platform == "win32" in c:\program files\python37\lib\site-packages (from IPython->tensor-sensor) (0.4.1)
Requirement already satisfied: prompt-toolkit<2.1.0,>=2.0.0 in c:\program files\python37\lib\site-packages (from IPython->tensor-sensor) (2.0.9)
Requirement already satisfied: decorator in c:\program files\python37\lib\site-packages (from IPython->tensor-sensor) (4.4.0)
Requirement already satisfied: setuptools>=18.5 in c:\program files\python37\lib\site-packages (from IPython->tensor-sensor) (41.2.0)
Requirement already satisfied: traitlets>=4.2 in c:\program files\python37\lib\site-packages (from IPython->tensor-sensor) (4.3.2)
Requirement already satisfied: jedi>=0.10 in c:\program files\python37\lib\site-packages (from IPython->tensor-sensor) (0.15.1)
Requirement already satisfied: pygments in c:\program files\python37\lib\site-packages (from IPython->tensor-sensor) (2.4.2)
Requirement already satisfied: absl-py>=0.7.0 in c:\program files\python37\lib\site-packages (from tensorflow->tensor-sensor) (0.7.1)
Requirement already satisfied: gast==0.3.3 in c:\program files\python37\lib\site-packages (from tensorflow->tensor-sensor) (0.3.3)
Requirement already satisfied: termcolor>=1.1.0 in c:\program files\python37\lib\site-packages (from tensorflow->tensor-sensor) (1.1.0)
Requirement already satisfied: astunparse==1.6.3 in c:\program files\python37\lib\site-packages (from tensorflow->tensor-sensor) (1.6.3)
Requirement already satisfied: google-pasta>=0.1.8 in c:\program files\python37\lib\site-packages (from tensorflow->tensor-sensor) (0.2.0)
Requirement already satisfied: six>=1.12.0 in c:\program files\python37\lib\site-packages (from tensorflow->tensor-sensor) (1.12.0)
Requirement already satisfied: protobuf>=3.9.2 in c:\program files\python37\lib\site-packages (from tensorflow->tensor-sensor) (3.13.0)
Requirement already satisfied: wheel>=0.26 in c:\program files\python37\lib\site-packages (from tensorflow->tensor-sensor) (0.33.6)
Requirement already satisfied: wrapt>=1.11.1 in c:\program files\python37\lib\site-packages (from tensorflow->tensor-sensor) (1.11.2)
Requirement already satisfied: grpcio>=1.8.6 in c:\program files\python37\lib\site-packages (from tensorflow->tensor-sensor) (1.23.0)
Collecting tensorflow-estimator<2.4.0,>=2.3.0
  Using cached tensorflow_estimator-2.3.0-py2.py3-none-any.whl (459 kB)
Requirement already satisfied: tensorboard<3,>=2.3.0 in c:\program files\python37\lib\site-packages (from tensorflow->tensor-sensor) (2.3.0)
Requirement already satisfied: keras-preprocessing<1.2,>=1.1.1 in c:\program files\python37\lib\site-packages (from tensorflow->tensor-sensor) (1.1.2)
Requirement already satisfied: opt-einsum>=2.3.2 in c:\program files\python37\lib\site-packages (from tensorflow->tensor-sensor) (3.0.1)
Requirement already satisfied: h5py<2.11.0,>=2.10.0 in c:\program files\python37\lib\site-packages (from tensorflow->tensor-sensor) (2.10.0)
Requirement already satisfied: kiwisolver>=1.0.1 in c:\program files\python37\lib\site-packages (from matplotlib->tensor-sensor) (1.1.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in c:\program files\python37\lib\site-packages (from matplotlib->tensor-sensor) (2.4.2)
Requirement already satisfied: python-dateutil>=2.1 in c:\program files\python37\lib\site-packages (from matplotlib->tensor-sensor) (2.8.0)
Requirement already satisfied: cycler>=0.10 in c:\program files\python37\lib\site-packages (from matplotlib->tensor-sensor) (0.10.0)
Requirement already satisfied: wcwidth in c:\program files\python37\lib\site-packages (from prompt-toolkit<2.1.0,>=2.0.0->IPython->tensor-sensor) (0.1.7)
Requirement already satisfied: ipython-genutils in c:\program files\python37\lib\site-packages (from traitlets>=4.2->IPython->tensor-sensor) (0.2.0)
Requirement already satisfied: parso>=0.5.0 in c:\program files\python37\lib\site-packages (from jedi>=0.10->IPython->tensor-sensor) (0.5.1)
Requirement already satisfied: requests<3,>=2.21.0 in c:\program files\python37\lib\site-packages (from tensorboard<3,>=2.3.0->tensorflow->tensor-sensor) (2.22.0)
Requirement already satisfied: google-auth<2,>=1.6.3 in c:\program files\python37\lib\site-packages (from tensorboard<3,>=2.3.0->tensorflow->tensor-sensor) (1.21.3)
Requirement already satisfied: markdown>=2.6.8 in c:\program files\python37\lib\site-packages (from tensorboard<3,>=2.3.0->tensorflow->tensor-sensor) (3.1.1)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in c:\program files\python37\lib\site-packages (from tensorboard<3,>=2.3.0->tensorflow->tensor-sensor) (1.7.0)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in c:\program files\python37\lib\site-packages (from tensorboard<3,>=2.3.0->tensorflow->tensor-sensor) (0.4.1)
Requirement already satisfied: werkzeug>=0.11.15 in c:\program files\python37\lib\site-packages (from tensorboard<3,>=2.3.0->tensorflow->tensor-sensor) (0.15.5)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\program files\python37\lib\site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow->tensor-sensor) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in c:\program files\python37\lib\site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow->tensor-sensor) (2019.9.11)
Requirement already satisfied: idna<2.9,>=2.5 in c:\program files\python37\lib\site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow->tensor-sensor) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\program files\python37\lib\site-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow->tensor-sensor) (1.25.3)
Requirement already satisfied: pyasn1-modules>=0.2.1 in c:\program files\python37\lib\site-packages (from google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow->tensor-sensor) (0.2.8)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in c:\program files\python37\lib\site-packages (from google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow->tensor-sensor) (4.1.1)
Requirement already satisfied: rsa<5,>=3.1.4; python_version >= "3.5" in c:\program files\python37\lib\site-packages (from google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow->tensor-sensor) (4.6)
Requirement already satisfied: requests-oauthlib>=0.7.0 in c:\program files\python37\lib\site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<3,>=2.3.0->tensorflow->tensor-sensor) (1.3.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in c:\program files\python37\lib\site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow->tensor-sensor) (0.4.8)
Requirement already satisfied: oauthlib>=3.0.0 in c:\program files\python37\lib\site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<3,>=2.3.0->tensorflow->tensor-sensor) (3.1.0)
**Building wheels for collected packages: torch
  Building wheel for torch (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'C:\Program Files\Python37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\rafar\\AppData\\Local\\Temp\\pip-install-xzyrqv53\\torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\rafar\\AppData\\Local\\Temp\\pip-install-xzyrqv53\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\rafar\AppData\Local\Temp\pip-wheel-u8ylwmm7'
       cwd: C:\Users\rafar\AppData\Local\Temp\pip-install-xzyrqv53\torch\
  Complete output (30 lines):
  running bdist_wheel
  running build
  running build_deps
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "C:\Users\rafar\AppData\Local\Temp\pip-install-xzyrqv53\torch\setup.py", line 265, in <module>
      description="Tensors and Dynamic neural networks in Python with strong GPU acceleration",
    File "C:\Program Files\Python37\lib\site-packages\setuptools\__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "C:\Program Files\Python37\lib\distutils\core.py", line 148, in setup
      dist.run_commands()
    File "C:\Program Files\Python37\lib\distutils\dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "C:\Program Files\Python37\lib\distutils\dist.py", line 985, in run_command
      cmd_obj.run()
    File "C:\Program Files\Python37\lib\site-packages\wheel\bdist_wheel.py", line 192, in run
      self.run_command('build')
    File "C:\Program Files\Python37\lib\distutils\cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "C:\Program Files\Python37\lib\distutils\dist.py", line 985, in run_command
      cmd_obj.run()
    File "C:\Program Files\Python37\lib\distutils\command\build.py", line 135, in run
      self.run_command(cmd_name)
    File "C:\Program Files\Python37\lib\distutils\cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "C:\Program Files\Python37\lib\distutils\dist.py", line 985, in run_command
      cmd_obj.run()
    File "C:\Users\rafar\AppData\Local\Temp\pip-install-xzyrqv53\torch\setup.py", line 51, in run
      from tools.nnwrap import generate_wrappers as generate_nn_wrappers
  ModuleNotFoundError: No module named 'tools.nnwrap'
  ----------------------------------------
  ERROR: Failed building wheel for torch
  Running setup.py clean for torch
  ERROR: Command errored out with exit status 1:
   command: 'C:\Program Files\Python37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\rafar\\AppData\\Local\\Temp\\pip-install-xzyrqv53\\torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\rafar\\AppData\\Local\\Temp\\pip-install-xzyrqv53\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' clean --all
       cwd: C:\Users\rafar\AppData\Local\Temp\pip-install-xzyrqv53\torch
  Complete output (2 lines):
  running clean
  error: [Errno 2] No such file or directory: '.gitignore'
  ----------------------------------------
  ERROR: Failed cleaning build dir for torch
Failed to build torch
Installing collected packages: torch, tensorflow-estimator, tensorflow, tensor-sensor
    Running setup.py install for torch ... error
    ERROR: Command errored out with exit status 1:
     command: 'C:\Program Files\Python37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\rafar\\AppData\\Local\\Temp\\pip-install-xzyrqv53\\torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\rafar\\AppData\\Local\\Temp\\pip-install-xzyrqv53\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\rafar\AppData\Local\Temp\pip-record-upwgi7o7\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\Program Files\Python37\Include\torch'
         cwd: C:\Users\rafar\AppData\Local\Temp\pip-install-xzyrqv53\torch\
    Complete output (23 lines):
    running install
    running build_deps
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\rafar\AppData\Local\Temp\pip-install-xzyrqv53\torch\setup.py", line 265, in <module>
        description="Tensors and Dynamic neural networks in Python with strong GPU acceleration",
      File "C:\Program Files\Python37\lib\site-packages\setuptools\__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "C:\Program Files\Python37\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "C:\Program Files\Python37\lib\distutils\dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "C:\Program Files\Python37\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\rafar\AppData\Local\Temp\pip-install-xzyrqv53\torch\setup.py", line 99, in run
        self.run_command('build_deps')
      File "C:\Program Files\Python37\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Program Files\Python37\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\rafar\AppData\Local\Temp\pip-install-xzyrqv53\torch\setup.py", line 51, in run
        from tools.nnwrap import generate_wrappers as generate_nn_wrappers
    ModuleNotFoundError: No module named 'tools.nnwrap'
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'C:\Program Files\Python37\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\rafar\\AppData\\Local\\Temp\\pip-install-xzyrqv53\\torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\rafar\\AppData\\Local\\Temp\\pip-install-xzyrqv53\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\rafar\AppData\Local\Temp\pip-record-upwgi7o7\install-record.txt' --single-version-externally-managed --compile --install-headers 'C:\Program Files\Python37\Include\torch' Check the logs for full command output.**

Can't visualize operations in pytorch modules

I'd like to see what is happening with my torch module, and it's submodules. But I only get the toplevel:

Ideas for 3D+ shapes

https://www.tensorflow.org/guide/tensor

Seem a problem with np.ones() function

Hi! I really thank u for your brilliant work of tsensor which help me to debug more effectively.

But recently, when I run this code in Jupyter or Pycharm, it always leads to a KeyError:

code:
with ts.explain():
a = np.ones(3)

KeyError report:
KeyError Traceback (most recent call last)
in ()
1 with ts.explain():
----> 2 a = np.ones(3)
3

F:\anaconda_file2\envs\test\lib\site-packages\numpy\core\numeric.py in ones(shape, dtype, order)
206 """
207 a = empty(shape, dtype, order)
--> 208 multiarray.copyto(a, 1, casting='unsafe')
209 return a
210

<array_function internals> in copyto(*args, **kwargs)

F:\anaconda_file2\envs\test\lib\site-packages\tsensor\analysis.py in listener(self, frame, event, arg)
266
267 def listener(self, frame, event, arg):
--> 268 module = frame.f_globals['name']
269 info = inspect.getframeinfo(frame)
270 filename, line = info.filename, info.lineno

KeyError: 'name'

Is there anything I can do to fix this problem? Grateful to gain any feedback!

Contribution Guidelines

I've a feeling that at some point many(including me) would like to contribute to this library and it would be great if it had some contribution guidelines.

Identify nested clarify() calls and ignored nested ones.

This is one of the issues at #15

Remove hard torch dependecies for keras/tensorflow user

Currently, in the analysis.py _shape method it always try to check if torch.Size exist. So for a keras user, if they don't have torch install, it will throw an error since analysis.py is importing it.

  File "/home/shawley/Downloads/tensor-sensor/tsensor/analysis.py", line 27, in <module>
    import torch
ModuleNotFoundError: No module named 'torch'

Related #8

Unhandled statements cause exceptions (Was: Nested calls to clarify can raise stacked Exceptions)

Hello,

I created a decorator to call clarify around the forward function of my custom Pytorch models (derived from torch.nn.Module).

Said decorator looks like this:

def clarify(function: callable) -> callable:
    """ Clarify decorator."""

    def call_clarify(*args, **kwargs):
        with tsensor.clarify(fontname="DejaVu Sans"):
            return function(*args, **kwargs)

    return call_clarify

When doing machine learning using Pytorch, models (derived from torch.nn.Module) can sometimes be "stacked". In a translation task, an EncoderDecoder's forward will call its Decoder's forward, itself calling the forward of an Attention module, for example.

In such a case, this results in nested clarify calls, which raise a succession of Exceptions, because some of the topmost clarify function do not exit correctly. To be more specific, l.124 of analysis.py, self.view can be None, which then raises an Exception on self.view.show().

A quick fix (that I did in local) was adding a check line 131:

                if self.view:
                    if self.show=='viz':
                        self.view.show()
                    augment_exception(exc_value, self.view.offending_expr)

However, I am not sure this would be the best fix possible, as I am not sure whether that is a common problem or not and how/if this is intended to be fixed. What do you think?

Supporting JAX

Hi,

Thanks for the awesome library! This has really made my debugging life much easier.

Just a question. Is there any plan to support JAX? I think this can be similarly supported since the API of JAX almost looks identical to NumPy.

Improvement: See into nn.Sequential models

The following exception not only generates a huge stack trace but also TensorSensor gives an error message augmentation indicating that Y = model(X) is the issue because it does not descend into tensor library code. It would be better to allow it to see inside the model pipeline. so that it can notice that the error is actually here:

nn.Linear(10, n_neurons)

which should be

nn.Linear(n_neurons, 10)

Here's the full example:

from torch import nn
n = 20
n_neurons = 50
model = nn.Sequential(
    nn.Linear(784, n_neurons), # 28x28 flattened image
    nn.ReLU(),
    nn.Linear(10, n_neurons),  # 10 output classes (0-9) <---- ooops! reverse those
    nn.Softmax(dim=1)
)
X = torch.rand(n,784) # n instances of feature vectors with 784 pixels
with tsensor.clarify():
    Y = model(X)

The error message we get is here:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-32-203c7ad8d609> in <module>
      1 with tsensor.clarify():
----> 2     Y = model(X)

~/opt/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

~/opt/anaconda3/lib/python3.8/site-packages/torch/nn/modules/container.py in forward(self, input)
    137     def forward(self, input):
    138         for module in self:
--> 139             input = module(input)
    140         return input
    141 

~/opt/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

~/opt/anaconda3/lib/python3.8/site-packages/torch/nn/modules/linear.py in forward(self, input)
     94 
     95     def forward(self, input: Tensor) -> Tensor:
---> 96         return F.linear(input, self.weight, self.bias)
     97 
     98     def extra_repr(self) -> str:

~/opt/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1845     if has_torch_function_variadic(input, weight):
   1846         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)
-> 1847     return torch._C._nn.linear(input, weight, bias)
   1848 
   1849 

RuntimeError: mat1 and mat2 shapes cannot be multiplied (20x50 and 10x50)
Cause: model(X) tensor arg X w/shape [20, 784]

Boxes for operands packed too tightly

There is some overlap with these boxes:

import torch
import tsensor

n = 200         # number of instances
d = 764         # number of instance features
nhidden = 256

Whh = torch.eye(nhidden, nhidden)  # Identity matrix
Uxh = torch.randn(nhidden, d)
bh  = torch.zeros(nhidden, 1)
h = torch.randn(nhidden, 1)         # fake previous hidden state h
r = torch.randn(nhidden, 3)         # fake this computation
X = torch.rand(n,d)                 # fake input

with tsensor.explain(savefig):
    r*h

executing and pure_eval

Hi! I stumbled across this library and noticed I could help. I've written a couple of libraries that are great for this stuff:

Here is a demo of how you could use it for this kind of project:

import ast

import executing
import pure_eval
import sys


def explain_error():
    ex = executing.Source.executing(sys.exc_info()[2])
    if not (ex.node and isinstance(ex.node, ast.BinOp)):
        return

    evaluator = pure_eval.Evaluator.from_frame(ex.frame)
    atok = ex.source.asttokens()

    try:
        print(f"Cannot add "
              f"{atok.get_text(ex.node.left)} = {evaluator[ex.node.left]!r} and "
              f"{atok.get_text(ex.node.right)} = {evaluator[ex.node.right]!r}")
    except pure_eval.CannotEval:
        print(f"Cannot safely evaluate operands of {ex.text()}. Extract them into variables.")


a = ["abc", 3]

try:
    print(a[0] + a[1])
except:
    explain_error()

try:
    print("only print once") + 3
except:
    explain_error()

To run this you will need to pip install executing pure_eval asttokens.

This should improve the parsing and such significantly. For example this will handle line continuations just fine. pure_eval will only evaluate simple expressions to avoid accidentally triggering side effects.

This uses the ast module from the standard library. Is there a reason you wrote your own parser? The best place to learn about ast is here: https://greentreesnakes.readthedocs.io/en/latest/

I'll let you integrate it into your code yourself, but let me know if you have questions.

Optional dependencies not working properly

Issue:
For some reason both
pip install tensor-sensor
pip install tensor-sensor[torch]
attempt to install Tensorflow too.
Environment:
- win10 latest (10.10.2020)
- conda 4.8.3 virtual env
- pytorch 1.6.0 installed via conda (the official way)
- no tensorflow
Workaround:
pip install tensor-sensor --no-deps
pip install graphviz

Add tensor element type info

From @sbrugman:

Our concrete issue was with Pytorch (unexpectedly) converting tensors with only integers to float, which later in the program resulted in an error because it could not be used as an index.
Another issue was changing size from 32 to 64 bit floats.

It's indeed the element type of the matrix.

There are multiple somewhat related issues:
https://discuss.pytorch.org/t/problems-with-target-arrays-of-int-int32-types-in-loss-functions/140/2
https://discuss.pytorch.org/t/why-pytorch-is-giving-me-hard-time-with-float-long-double-tensor/14678/6

The common denominator between dimensionality debugging is that both type and dimensionality are practically hidden from the user:

import numpy as np
import tsensor as ts

x = np.arange(6, dtype=np.float32)

with ts.explain(savefig="types.pdf"):
    print(x.dtype)
    print((x*x).dtype)
    print((np.sin(x)).dtype)
    print((x + np.arange(6)).dtype)
    print((np.multiply.outer(x, np.arange(2.0))).dtype)
    print((np.outer(x, np.arange(2.0))).dtype)

Graphviz errors for abstract syntax tree (AST) vizualisation

Issue:
Calls of tsensor.astviz() produce no vizualisation, but error:

FileNotFoundError: [WinError 2]

Environment:
- win10 latest (11.10.2020)
- conda 4.8.5 virtual env
- conda-installed graphviz 0.14.2 XOR pip-installed graphviz 0.14.2
Notes:
- Issue seems to be about graphviz on windows in conda env, as per #1666.
A working user-side fix (as per #1666 comment, also tested by me) is:
if using conda env, install graphviz only like conda install python-graphviz, instead of pip install graphviz

So the proper installation process of the current release for conda users would be like:
conda install python-graphviz
pip install tensor-sensor

Power `**` operator not handled

Showing too many matrices for complicated operands

The following code generates an exception but instead of showing the result of the operand subexpressions, it shows all bits of it:

import torch
import tsensor

n = 200         # number of instances
d = 764         # number of instance features
nhidden = 256

Whh = torch.eye(nhidden, nhidden)  # Identity matrix
Uxh = torch.randn(nhidden, d)
bh  = torch.zeros(nhidden, 1)
h = torch.randn(nhidden, 1)         # fake previous hidden state h
# r = torch.randn(nhidden, 1)         # fake this computation
r = torch.randn(nhidden, 3)         # fake this computation
X = torch.rand(n,d)                 # fake input

# Following code raises an exception
with tsensor.clarify():
    h = torch.tanh(Whh @ (r*h) + Uxh @ X.T + bh)  # state vector update equation