technion-kishony-lab / quibbler Goto Github PK

Your data - interactive!

License: MIT License

Python 69.82% Jupyter Notebook 28.97% Makefile 0.04% Batchfile 0.05% TypeScript 0.97% CSS 0.01% JavaScript 0.01% HTML 0.14%

data-analysis data-science data-visualization declarative graphics gui interactive jupyter matplotlib python

quibbler's Introduction

Quibbler: your data - interactive!

Interactive, reproducible and efficient data analytics

With Quibbler, your data analysis is automatically live and interactive.

quibbler_promo.mp4

What is it?

Quibbler is a toolset for building highly interactive, yet reproducible, transparent and efficient data analysis pipelines. Quibbler allows using standard Python syntax to process data through any series of analysis steps, while automatically maintaining connectivity between downstream results and upstream raw data sources. Quibbler facilitates and embraces human interventions as an inherent part of the analysis pipeline: input parameters, as well as exceptions and overrides, can be specified and adjusted either programmatically, or by interacting with live graphics, and all such interventions are automatically recorded in well-documented human-machine readable files. Changes to such parameters propagate downstream, pinpointing which specific data items, or even specific elements thereof, are affected, thereby vastly saving unnecessary recalculations. Quibbler, therefore, facilitates hands-on interactions with data in ways that are not only flexible, fun and interactive, but also traceable, reproducible, and computationally efficient.

Main Features

Interactivity
- Creating interactive graphics is as simple as calling standard Matplotlib graphics functions with arguments that represent your parameter values.
- Any data presented graphically is automatically live and interactive (no need for the tedious programming of callback functions).
Traceability and Reproducibility
- Trace which specific data items and analysis parameters affect focal downstream results (see dependency graph).
- Inherent undo/redo functionality.
- Save/load parameter values as human-readable records (either as external text files, or inside Jupyter notebook).
Computational efficiency
- Upon parameter changes, Quibbler pinpoints and only recalculates the specifically affected array elements of downstream analysis steps (here).
Very little to learn: your standard-syntax code automatically comes to life.
- To get started with Quibbler, you do not need to learn any new syntax or new functions. Just encapsulate your input parameters with iquib and your analysis and graphics automatically become live and interactive.
- Quibbler supports standard coding syntax with all Python operators, slicing, getitem, Numpy functions, Matplotlib graphics functions, Matplotlib widgets, and ipywidgets. It further provides an easy way to incorporate any user functions or functions from any other non-graphics packages (here). Support for other graphics packages, besides Matplotlib, will be offered in future releases.

Minimal app

from pyquibbler import initialize_quibbler, iquib
initialize_quibbler()
import matplotlib.pyplot as plt

x = iquib(0.5)
y = 1 - x
plt.plot([0, 1], [1, 0], '-')
plt.plot([0, x, x], [y, y, 0], '--', marker='D')
plt.title(x, fontsize=20)

Documentation and Examples

For complete documentation and a getting-started tour, see readthedocs.

For simple demos and small apps, see our Examples.

Installation

We recommend installing Quibbler in a new virtual environment (see creating a new environment).

To install run:

pip install pyquibbler

If you are using Quibbler within Jupyter lab, you can also add the pyquibbler Jupyter Lab extension:

pip install pyquibbler_labextension

Development

To install for developers, see our guide here.

Credit

Quibbler was created by Roy Kishony, initially implemented as a Matlab toolbox.

The first release of Quibbler for Python, pyquibbler, was developed at the Kishony lab, Technion - Israel Institute of Technology, by Maor Kern, Maor Kleinberger and Roy Kishony.

We very much welcome any thoughts, suggestions and ideas and of course welcome PR contributions (for some proposed directions, see our pending issues).

Related packages

quibbler's People

Contributors

Stargazers

Watchers

Forkers

digitwolf shalevy1 psychedel tensornetics shallmore lavanya1533 ariewjy

quibbler's Issues

bug in dragging of plot(x,y) where either x or y are matrices

in the following example, dragging a marker leads to an assignment that changes also other markers:

plt.figure(figsize=[10, 7])
ax = plt.gca()
x = iquib(np.arange(12).reshape(4,3))
y = iquib(np.arange(4).reshape(4,1))
ax.plot(x, y, marker='o', markersize=12, markerfacecolor='y',
        linestyle='None', picker=True, pickradius=15)

bug after plt.clf()

See code below.
I get an error "Text(2.0, 11.0, 'Average is 11.0') is not in list"

from pyquibbler import iquib, override_all, q
import matplotlib.pyplot as plt
import numpy as np
override_all()
v = iquib(np.array([5, 3, 1, 2, 4]))
vsquared = np.square(v)
vmean = np.average(vsquared)
%matplotlib widget
x_range = [-0.5, len(vsquared) - 0.5]
plt.text(np.average(x_range), vmean, q("Average is {}".format, vmean), horizontalalignment="center",  verticalalignment="bottom", fontsize=16)
plt.clf()
v[0] = 2

bug in inverse assignment into indexing quibs

The following code produces a bug:

A = iquib(np.arange(6).reshape(2,3))
B = A[:,:]
B[:,:] = 0
A.get_value()

ValueError: shape mismatch: value array of shape (2,3)  could not be broadcast to indexing result of shape (6,)

axs.set_xxx(Q) not refreshing upon drag

in the following code, dragging any of the triangles moves the other (so the quibs are working as expected), but the set_title is not updating. It will though update if we issue any calculation of a cell in the notebook (not related to a quib).

Commit: b257e65

from pyquibbler import iquib, override_all, q
import matplotlib.pyplot as plt
from matplotlib import widgets
import numpy as np
%matplotlib tk
override_all()

X0 = iquib(0.5);

fig = plt.figure()
axs = fig.add_axes([0.15,0.4,0.8,0.5])
axs.axis([0,3,0,1])
axs.set_title(q('{}'.format,X0))
axs.plot(1,  X0,marker='>',markerfacecolor='r', markersize=16, picker=True);
axs.plot(2,1-X0,marker='<',markerfacecolor='r', markersize=16, picker=True);

weird bug in a simple direct assignment to an iquib

took me some time to purify this weird-looking bug:
commit: 12300f3
on master: 7f0c676

from pyquibbler import iquib, override_all, q, quibbler_user_function
override_all()
import numpy as np
import matplotlib.pyplot as plt

d = iquib({'prop':    0})
xy = q(lambda x: np.array([0]), d)
d['prop'] = 5  # this assignment works fine
plt.plot(xy[0]);
d['prop'] = 10 # this assignment fails

IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices

bug when indexing a quib with comma-separated quib indices

commit: 679779e

a = iquib(np.arange(6).reshape(2,3))
i1 = iquib(1)
i2 = iquib(3)
a[1,i1:i2].get_value()

yields:

TypeError: slice indices must be integers or None or have an __index__ method

I guess we need deep replacement in the indexing object?

pyimageXX still persists

I am on
03452b6
(after the Solve cla bug in graphicsfunctionquibs patch)

but I still get some pyimageXX errors.

i think I have purified it to this code (on jupyter):

cell 1:

plt.figure()
ax = plt.gca()
x = iquib([1.,3.,2.,4.])
ax.plot(x)
plt.close()

cell 2:

x[1] = 4

it fails if you manually run the cells one by one

error:


---------------------------------------------------------------------------
TclError                                  Traceback (most recent call last)
~/.conda/envs/pyquibbler/lib/python3.9/site-packages/matplotlib/backends/_backend_tk.py in blit(photoimage, aggimage, offsets, bbox)
    112     try:
--> 113         photoimage.tk.call(_blit_tcl_name, argsid)
    114     except tk.TclError as e:

TclError: invalid command name "mpl_blit_0b2900df46064f79be3c668c2788255e"

During handling of the above exception, another exception occurred:

TclError                                  Traceback (most recent call last)
/var/folders/90/frsch0qx4nb6qc5hzm_jrvvr0000gn/T/ipykernel_19277/3481453627.py in <module>
      2 ax = plt.gca()
      3 ax.plot(x,'.-', markersize=20, picker=True)
----> 4 x[1] = 4

~/Git/pyquibbler/pyquibbler/quib/quib.py in __setitem__(self, key, value)
    212     def __setitem__(self, key, value):
    213         from .assignment.assignment import PathComponent
--> 214         self.assign(Assignment(value=value, path=[PathComponent(component=key, indexed_cls=self.get_type())]))
    215 
    216     def pretty_repr(self):

~/Git/pyquibbler/pyquibbler/quib/quib.py in assign(self, assignment)
    184         assignment's value
    185         """
--> 186         self.override(assignment, allow_overriding_from_now_on=False)
    187 
    188     def assign_value(self, value: Any) -> None:

~/Git/pyquibbler/pyquibbler/quib/quib.py in override(self, assignment, allow_overriding_from_now_on)
    170         self._overrider.add_assignment(assignment)
    171 
--> 172         self.invalidate_and_redraw_at_path(assignment.path)
    173 
    174     def remove_override(self, path: List[PathComponent]):

~/Git/pyquibbler/pyquibbler/quib/quib.py in invalidate_and_redraw_at_path(self, path)
    106             path = []
    107         self._invalidate_children_at_path(path)
--> 108         self.__redraw()
    109 
    110     def _invalidate_children_at_path(self, path: List[PathComponent]) -> None:

~/Git/pyquibbler/pyquibbler/quib/quib.py in __redraw(self)
     96                 axeses.add(axes)
     97         for axes in axeses:
---> 98             redraw_axes(axes)
     99 
    100     def invalidate_and_redraw_at_path(self, path: Optional[List[PathComponent]] = None) -> None:

~/Git/pyquibbler/pyquibbler/quib/graphics/redraw.py in redraw_axes(axes, force)
     31     else:
     32         with timer(name="redraw"):
---> 33             axes.figure.canvas.draw()

~/.conda/envs/pyquibbler/lib/python3.9/site-packages/matplotlib/backends/backend_tkagg.py in draw(self)
      8     def draw(self):
      9         super().draw()
---> 10         self.blit()
     11 
     12     def blit(self, bbox=None):

~/.conda/envs/pyquibbler/lib/python3.9/site-packages/matplotlib/backends/backend_tkagg.py in blit(self, bbox)
     11 
     12     def blit(self, bbox=None):
---> 13         _backend_tk.blit(self._tkphoto, self.renderer.buffer_rgba(),
     14                          (0, 1, 2, 3), bbox=bbox)
     15 

~/.conda/envs/pyquibbler/lib/python3.9/site-packages/matplotlib/backends/_backend_tk.py in blit(photoimage, aggimage, offsets, bbox)
    116             raise
    117         photoimage.tk.createcommand(_blit_tcl_name, _blit)
--> 118         photoimage.tk.call(_blit_tcl_name, argsid)
    119 
    120 

TclError: invalid command name "pyimage10"

ERROR:tornado.application:Exception in callback functools.partial(<function Kernel.enter_eventloop.<locals>.advance_eventloop at 0x7f8c3b68de50>)
Traceback (most recent call last):
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/site-packages/matplotlib/backends/_backend_tk.py", line 113, in blit
    photoimage.tk.call(_blit_tcl_name, argsid)
_tkinter.TclError: invalid command name "mpl_blit_0b2900df46064f79be3c668c2788255e"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 402, in advance_eventloop
    eventloop(self)
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/site-packages/ipykernel/eventloops.py", line 242, in loop_tk
    app.mainloop()
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/tkinter/__init__.py", line 1429, in mainloop
    self.tk.mainloop(n)
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/site-packages/matplotlib/backends/_backend_tk.py", line 65, in _blit
    photoimage.blank()
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/tkinter/__init__.py", line 4068, in blank
    self.tk.call(self.name, 'blank')
_tkinter.TclError: invalid command name "pyimage10"

set_cache_behavior('on'): bug?

a = iquib(10)
q = a+1
q.get_value()
Out: 11
q.set_cache_behavior('on')
q.get_value()

AssertionError: self._cache_behavior has unexpected value: "on"

plt.plot(Q) seem to return artist/list. I think should return a functional quib

I think plt.plot(Q) where Q is a quib should return a functional quib
just like np.sin(Q) returns a functional quib

We can have an "IsGraphics" attribute and then I guess plt.plot(Q) should be a functional quib with an IsGraphics=True

example of graphics which always get stuck upon dragging in Jupiter-tk

try running:
quibdemo_making_the_quib_icon and drag the little black circles (in principle, it should be draggable on a curve)
it almost always leads to a crash with a stucked kernel and the error below. (with pyimage50 sometimes replaced with pyimage10, pyimage20, etc)
commit: c0be027

Traceback (most recent call last):
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/site-packages/matplotlib/backends/_backend_tk.py", line 113, in blit
    photoimage.tk.call(_blit_tcl_name, argsid)
_tkinter.TclError: invalid command name "pyimage50"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/site-packages/matplotlib/cbook/__init__.py", line 270, in process
    func(*args, **kwargs)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/graphics/event_handling/canvas_event_handler.py", line 81, in _handle_motion_notify
    self._inverse_assign_graphics(self.current_pick_event.artist, mouse_event)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/graphics/event_handling/canvas_event_handler.py", line 55, in _inverse_assign_graphics
    graphics_inverse_assigner.inverse_assign_drawing_func(drawing_func=drawing_func,
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/contextlib.py", line 126, in __exit__
    next(self.gen)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/graphics/redraw.py", line 21, in aggregate_redraw_mode
    redraw_axes(axes)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/graphics/redraw.py", line 33, in redraw_axes
    axes.figure.canvas.draw()
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/site-packages/matplotlib/backends/backend_tkagg.py", line 10, in draw
    self.blit()
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/site-packages/matplotlib/backends/backend_tkagg.py", line 13, in blit
    _backend_tk.blit(self._tkphoto, self.renderer.buffer_rgba(),
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/site-packages/matplotlib/backends/_backend_tk.py", line 118, in blit
    photoimage.tk.call(_blit_tcl_name, argsid)
_tkinter.TclError
ERROR:tornado.application:Exception in callback functools.partial(<function Kernel.enter_eventloop.<locals>.advance_eventloop at 0x7fea1002ee50>)
Traceback (most recent call last):
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/site-packages/matplotlib/backends/_backend_tk.py", line 113, in blit
    photoimage.tk.call(_blit_tcl_name, argsid)
_tkinter.TclError: invalid command name "pyimage50"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/site-packages/ipykernel/kernelbase.py", line 402, in advance_eventloop
    eventloop(self)
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/site-packages/ipykernel/eventloops.py", line 242, in loop_tk
    app.mainloop()
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/tkinter/__init__.py", line 1429, in mainloop
    self.tk.mainloop(n)
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/site-packages/matplotlib/backends/_backend_tk.py", line 63, in _blit
    photoimage, dataptr, offsets, bboxptr, blank = _blit_args.pop(argsid)
KeyError: '140643035831408'

plot linewidth=quib - not refreshing

when using plot(..., linewidth=quib) the artist is not refreshing when the quib changes
[on commit: f107eda]

from matplotlib import pyplot as plt

from pyquibbler import iquib, q, override_all
import numpy as np

%matplotlib widget

override_all()

fig = plt.figure(1)

fig.clf()
axs = fig.add_subplot(1,1,1)
lw = iquib([10])
axs.plot([1,2,3],[1,3,2],'r-',linewidth=lw[0])

lw[0] = 1 # value changes but graphics is not updated

add .config method

allow users to config quibs within the creation line, like:

Q = iquib(5).config(assignment_template = ..., allow_override = ..., ...)

functional quibs that call a function with quib arguments

We talked about the possibility of a new operator, say qq, whereqf = qq(fcn,q1,q2,...) creates a functional quib qf that implements the function fcn by calling it with quib arguments (unlike q that calls the function with the output of the quib arguments).

add support of interactive rectangle (for choosing ROI)

pls take 'URGENT' with a gain of salt here :)

override_mask for deep quibs: bug?

a = iquib([10,[21,22],30])
a[1][1] = 222
a.get_override_mask().get_value()

says:

VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  return np.array(value).shape

matplotlib has a really annoying bug in widgets.RectangleSelector

widgets.RectangleSelector has a weird and annoying bug:
when it calls the changed_callback function it sends extents vector of 4 values
these values are correct when the rectangle is dragged but are incorrectly swapped when the rectangle is resized

see the code here and note the colors of the markers as you drag or reshape the rectangle:

plt.axis([0, 10, 0 , 10])
roi = iquib(np.array([2.,5.,3.,7.]))
r = q(widgets.RectangleSelector, plt.gca(), extents=roi)
plt.plot(roi[0],0.2,'r^', markersize=18)
plt.plot(roi[1],0.2,'gs', markersize=18)
plt.plot(0.2,roi[2],'r>', markersize=18)
plt.plot(0.2,roi[3],'gs', markersize=18)
plt.show()

I tried to solve it by adding
extents = self.get_value().extents
as the first line of RectangleSelectorGraphicsFunctionQuib._on_changed
it corrects the problem but somehow prevents continuous drag

what to do???

inverse assignment into Q-Q[0]

this code works as expected

Q = iquib(np.array([100]));
dQ = Q - Q
dQ[0] = 20
a.get_value()  ->  120

but for some reason, when I change the second argument to Q[0] or even to Q[[0]] the inverse-assignment yields an error:

Q = iquib(np.array([100]));
dQ = Q - Q[0]
dQ[0] = 20
a.get_value()   ->   AssignmentNotPossibleException

Marker dragging doesn't work when not all of the markers are draggable

In the following code, the first marker should be undraggable while the second should be draggable. But pyquibbler won't let drag any of them.

Commit: b257e65

import matplotlib.pyplot as plt
from matplotlib import widgets
import numpy as np
%matplotlib tk
override_all()

X0 = iquib(0.5);
fig = plt.figure()
axs = fig.add_subplot(1,1,1)
axs.axis([0,3,0,1])
axs.plot([1,2],  [0.5, X0],marker='>',markerfacecolor='r', markersize=16, picker=True);

inverse assignment for dual argument operators (like plus,...)

In dual argument operators of a quib and non-quib (like Q2=2+Q), the inverse assignment should go to the quib.

currently, the following code is working ok:

A = iquib(np.array([1]))
B = A + 3
B[0] = 10;

but this code is not performing inverse assignment:

A = iquib(np.array([1]))
B = 3 + A
B[0] = 10;

inverse assignment yields a warning

commit: 95eebec

from pyquibbler import iquib, override_all, q
import numpy as np
override_all()

q = iquib(np.array([1,2,3]))
a = q + 1
a12 = a[1:3]
a12.get_value()
a12[1] = 400
q.get_value()

I get:

/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/assignment/overrider.py:37: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  new_element[path] = last_element

implement inverse-assignment on type casting

currently, str(), int(), etc are not implemented
this means that we have to use q, like Qstr = q(str,Qint)
this works ok in forward calculations, but can we then implement inverse assignment on such casting operations?

inverse assignment of Q[:]=value should not break into individual indices

when inverting element-wise functions, need to preserve colon indexing. This is important because of three reasons:

currently we get a long and thereby incomprehensible override_list
speed
we can get unexpected behaviors. see here:

N = iquib([4,3])
A = np.random.randint(0,10,N)
A.allow_overriding = True
B = A + 10

B[:,1] = 0
print(B.get_value())

N[0] = 6
print(B.get_value())

A.get_override_list()

now:

print(B.get_value())

yeilds:

[[16  0 17]
 [16  0 19]
 [13  0 11]
 [16  0 14]]

but then:

N[0] = 6
print(B.get_value())

yields

[[10  0 14]
 [18  0 10]
 [17  0 13]
 [16  0 17]
 [13 15 12]
 [13 12 12]]

Old graphics not deleted when number of artists changes

In the following code the number of artists produced by a plot command changes.
observed bug: only one of the old artists gets deleted.
For some weird reason, this behavior is not fully reproducible upon kernel restart

from pyquibbler import iquib, override_all, q
import matplotlib.pyplot as plt
import numpy as np
override_all()
%matplotlib widget
n = iquib([2, 3])
z = np.random.rand(n[0],n[1])
plt.figure(1)
plt.clf()
h = plt.plot(z)
n[1] = 4 # plot refreshes but only deleting one of the 3 old artists

implement dragging in plt.plot of a list containing quibs

In the following code, the right marker should be movable. but currently pyquibbler won't allow moving it. is this easy to implement?

import matplotlib.pyplot as plt
from matplotlib import widgets
import numpy as np
%matplotlib tk
override_all()

X0 = iquib(0.5);
fig = plt.figure()
axs = fig.add_subplot(1,1,1)
axs.axis([0,3,0,1])
axs.plot([1,2],  [0.5, X0],marker='>',markerfacecolor='r', markersize=16, picker=True);

assignments into iquibs should be implemented in _value rather than as overrides

upon-release versus upon-drag graphics updating

I think what we said is:

when graphics get dragged assignments and invalidation occur immediately during drag
each graphic quib should have an attribute defining whether it gets updated immediately during drag (of itsown or any other quib) or only upon release.

quibs that set axis attribute should be unique

quibs that set axis attributes like xlim, ylim, xlabel, ylable, title, fontsize, etc
should be unique.

So when performing:

A1 = plt.xlabel(Q1)
A2 = plt.xlabel(Q2)

A1 should get invalidated

remove overridden overrides from override_list

When we assign and then re-assign on the same indices of a given quib, need to have an option to remove old overridden assignments.
This is particularly true when we drag an object in continuous mode: only the last assignment should be saved upon the release of the mouse. Otherwise we can get a very long override_list

See here:

A = iquib(np.array([1.,2.,3.]))
B = iquib(np.array([10.,20.,30.]))
plt.plot(A,B,'o',picker=True)

after a simple drag trajectory we can end up with a very long list when we do:
A.get_override_list()

add eq (==) to COMPARISON_OVERRIDES ?

I tried adding __eq__ to COMPARISON_OVERRIDES but it generated some weird infinite recursion.
your thoughts?

implement matplotlib.pyplot.scatter

this is similar to plot but allows specifying different sizes/colors for the markers.
quite useful to have

Graphics fails on Jupiter upon update

this code works perfect from pycharm but fails in jupiter.

from pyquibbler import iquib, override_all, q
import matplotlib.pyplot as plt
from matplotlib import widgets
import numpy as np
override_all()

# Figure setup:
fig = plt.figure(1,figsize=(4,2))
axs = fig.add_axes([0.2,0.3,0.6,0.2])
n = iquib(1)
sldr = widgets.Slider(ax=axs, label='label', valmin=0, valmax=9, valstep=1, valinit=n)
axs.set_title(q(str,n))
axs.set_xlabel(q(str,sldr.val))

I am running with option %matplotlib tk and once you click the slider you get:

Traceback (most recent call last):
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/site-packages/matplotlib/cbook/__init__.py", line 270, in process
    func(*args, **kwargs)
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/site-packages/matplotlib/widgets.py", line 530, in <lambda>
    return self._observers.connect('changed', lambda val: func(val))
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/graphics/elements/slider_graphics_function_quib.py", line 14, in _on_change
    val.assign(Assignment(value=new_value, paths=[...]))
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/quib.py", line 123, in assign
    self._override(assignment)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/quib.py", line 118, in _override
    self.invalidate_and_redraw()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/quib.py", line 90, in invalidate_and_redraw
    self.__redraw()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/quib.py", line 76, in __redraw
    graphics_function_quib.get_value()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/quib.py", line 180, in get_value
    return self._overrider.override(self._get_inner_value(), self._assignment_template)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/function_quibs/default_function_quib.py", line 62, in _get_inner_value
    result = self._call_func()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/graphics/graphics_function_quib.py", line 248, in _call_func
    return self._create_new_artists(axeses_to_array_names_to_indices_and_artists)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/graphics/graphics_function_quib.py", line 192, in _create_new_artists
    func_res = call_func_with_quib_values(self.func, self.args, self.kwargs)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 269, in call_func_with_quib_values
    new_args, new_kwargs = convert_args(args, kwargs)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 248, in convert_args
    return (tuple(copy_and_replace_quibs_with_vals(arg) for arg in args),
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 248, in <genexpr>
    return (tuple(copy_and_replace_quibs_with_vals(arg) for arg in args),
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 144, in copy_and_replace_quibs_with_vals
    result = shallow_copy_and_replace_quibs_with_vals(obj)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 140, in shallow_copy_and_replace_quibs_with_vals
    return deep_copy_and_replace_quibs_with_vals(obj, SHALLOW_MAX_DEPTH, SHALLOW_MAX_LENGTH)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 132, in deep_copy_and_replace_quibs_with_vals
    return recursively_run_func_on_object(func=replace_with_value_if_quib_or_copy, max_depth=max_depth,
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 96, in recursively_run_func_on_object
    return func(obj)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 127, in replace_with_value_if_quib_or_copy
    return o.get_value()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/quib.py", line 180, in get_value
    return self._overrider.override(self._get_inner_value(), self._assignment_template)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/function_quibs/default_function_quib.py", line 62, in _get_inner_value
    result = self._call_func()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/function_quibs/function_quib.py", line 151, in _call_func
    return call_func_with_quib_values(self.func, self.args, self.kwargs)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 269, in call_func_with_quib_values
    new_args, new_kwargs = convert_args(args, kwargs)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 248, in convert_args
    return (tuple(copy_and_replace_quibs_with_vals(arg) for arg in args),
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 248, in <genexpr>
    return (tuple(copy_and_replace_quibs_with_vals(arg) for arg in args),
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 144, in copy_and_replace_quibs_with_vals
    result = shallow_copy_and_replace_quibs_with_vals(obj)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 140, in shallow_copy_and_replace_quibs_with_vals
    return deep_copy_and_replace_quibs_with_vals(obj, SHALLOW_MAX_DEPTH, SHALLOW_MAX_LENGTH)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 132, in deep_copy_and_replace_quibs_with_vals
    return recursively_run_func_on_object(func=replace_with_value_if_quib_or_copy, max_depth=max_depth,
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 96, in recursively_run_func_on_object
    return func(obj)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 127, in replace_with_value_if_quib_or_copy
    return o.get_value()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/quib.py", line 180, in get_value
    return self._overrider.override(self._get_inner_value(), self._assignment_template)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/function_quibs/default_function_quib.py", line 62, in _get_inner_value
    result = self._call_func()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/function_quibs/function_quib.py", line 151, in _call_func
    return call_func_with_quib_values(self.func, self.args, self.kwargs)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 271, in call_func_with_quib_values
    return func(*new_args, **new_kwargs)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/graphics/elements/slider_graphics_function_quib.py", line 29, in val
    return self.get_value().val
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/quib.py", line 180, in get_value
    return self._overrider.override(self._get_inner_value(), self._assignment_template)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/function_quibs/default_function_quib.py", line 62, in _get_inner_value
    result = self._call_func()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/graphics/elements/slider_graphics_function_quib.py", line 19, in _call_func
    slider = super(SliderGraphicsFunctionQuib, self)._call_func()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/graphics/graphics_function_quib.py", line 246, in _call_func
    axeses_to_array_names_to_indices_and_artists = self._get_axeses_to_array_names_to_starting_indices_and_artists()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/graphics/graphics_function_quib.py", line 227, in _get_axeses_to_array_names_to_starting_indices_and_artists
    array_names_to_indices_and_artists[array_name] = (array.index(exemplifying_artist), artists)
ValueError: <matplotlib.patches.Polygon object at 0x7f9cf11b8520> is not in list
Traceback (most recent call last):
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/site-packages/matplotlib/cbook/__init__.py", line 270, in process
    func(*args, **kwargs)
  File "/Users/roeekishony/.conda/envs/pyquibbler/lib/python3.9/site-packages/matplotlib/widgets.py", line 530, in <lambda>
    return self._observers.connect('changed', lambda val: func(val))
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/graphics/elements/slider_graphics_function_quib.py", line 14, in _on_change
    val.assign(Assignment(value=new_value, paths=[...]))
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/quib.py", line 123, in assign
    self._override(assignment)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/quib.py", line 118, in _override
    self.invalidate_and_redraw()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/quib.py", line 90, in invalidate_and_redraw
    self.__redraw()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/quib.py", line 76, in __redraw
    graphics_function_quib.get_value()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/quib.py", line 180, in get_value
    return self._overrider.override(self._get_inner_value(), self._assignment_template)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/function_quibs/default_function_quib.py", line 62, in _get_inner_value
    result = self._call_func()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/graphics/graphics_function_quib.py", line 248, in _call_func
    return self._create_new_artists(axeses_to_array_names_to_indices_and_artists)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/graphics/graphics_function_quib.py", line 192, in _create_new_artists
    func_res = call_func_with_quib_values(self.func, self.args, self.kwargs)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 269, in call_func_with_quib_values
    new_args, new_kwargs = convert_args(args, kwargs)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 248, in convert_args
    return (tuple(copy_and_replace_quibs_with_vals(arg) for arg in args),
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 248, in <genexpr>
    return (tuple(copy_and_replace_quibs_with_vals(arg) for arg in args),
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 144, in copy_and_replace_quibs_with_vals
    result = shallow_copy_and_replace_quibs_with_vals(obj)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 140, in shallow_copy_and_replace_quibs_with_vals
    return deep_copy_and_replace_quibs_with_vals(obj, SHALLOW_MAX_DEPTH, SHALLOW_MAX_LENGTH)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 132, in deep_copy_and_replace_quibs_with_vals
    return recursively_run_func_on_object(func=replace_with_value_if_quib_or_copy, max_depth=max_depth,
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 96, in recursively_run_func_on_object
    return func(obj)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 127, in replace_with_value_if_quib_or_copy
    return o.get_value()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/quib.py", line 180, in get_value
    return self._overrider.override(self._get_inner_value(), self._assignment_template)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/function_quibs/default_function_quib.py", line 62, in _get_inner_value
    result = self._call_func()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/function_quibs/function_quib.py", line 151, in _call_func
    return call_func_with_quib_values(self.func, self.args, self.kwargs)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 269, in call_func_with_quib_values
    new_args, new_kwargs = convert_args(args, kwargs)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 248, in convert_args
    return (tuple(copy_and_replace_quibs_with_vals(arg) for arg in args),
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 248, in <genexpr>
    return (tuple(copy_and_replace_quibs_with_vals(arg) for arg in args),
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 144, in copy_and_replace_quibs_with_vals
    result = shallow_copy_and_replace_quibs_with_vals(obj)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 140, in shallow_copy_and_replace_quibs_with_vals
    return deep_copy_and_replace_quibs_with_vals(obj, SHALLOW_MAX_DEPTH, SHALLOW_MAX_LENGTH)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 132, in deep_copy_and_replace_quibs_with_vals
    return recursively_run_func_on_object(func=replace_with_value_if_quib_or_copy, max_depth=max_depth,
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 96, in recursively_run_func_on_object
    return func(obj)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 127, in replace_with_value_if_quib_or_copy
    return o.get_value()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/quib.py", line 180, in get_value
    return self._overrider.override(self._get_inner_value(), self._assignment_template)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/function_quibs/default_function_quib.py", line 62, in _get_inner_value
    result = self._call_func()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/function_quibs/function_quib.py", line 151, in _call_func
    return call_func_with_quib_values(self.func, self.args, self.kwargs)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/utils.py", line 271, in call_func_with_quib_values
    return func(*new_args, **new_kwargs)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/graphics/elements/slider_graphics_function_quib.py", line 29, in val
    return self.get_value().val
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/quib.py", line 180, in get_value
    return self._overrider.override(self._get_inner_value(), self._assignment_template)
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/function_quibs/default_function_quib.py", line 62, in _get_inner_value
    result = self._call_func()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/graphics/elements/slider_graphics_function_quib.py", line 19, in _call_func
    slider = super(SliderGraphicsFunctionQuib, self)._call_func()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/graphics/graphics_function_quib.py", line 246, in _call_func
    axeses_to_array_names_to_indices_and_artists = self._get_axeses_to_array_names_to_starting_indices_and_artists()
  File "/Users/roeekishony/Git/pyquibbler/pyquibbler/quib/graphics/graphics_function_quib.py", line 227, in _get_axeses_to_array_names_to_starting_indices_and_artists
    array_names_to_indices_and_artists[array_name] = (array.index(exemplifying_artist), artists)
ValueError: <matplotlib.patches.Polygon object at 0x7f9cf11b8520> is not in list

assignment of Default to get back to default functional value of a function quib

should be a way to remove overrides at specific indices.

a clean syntax could be to define a 'Default' object and assign:

Q[...] = Default

iquib assignment should change value not add override

currently an assignment to iquib is still an override:

>> Q = iquib([1,2])
>> Q[1] = 10

>> print(Q.get_override_list())
dict_values([Assignment(value=10, path=[PathComponent(indexed_cls=<class 'list'>, component=1)])])

>> print(Q._value)
[1, 2]

implement obj2quib

need a way to turn any object (including quib containing objects) to a quib.

say Q is a quib and we want to create a functional quib that represents the list[1,Q]. We should be able to do something like: L = obj2quib([1,Q])

currently, I think we only have cumbersome ways like L = q(lambda x: [1,x], Q)

drag is incorrectly restricted to one axis

in the following code the marker for some reason is only draggable in the x-direction:

from pyquibbler import iquib, override_all, q
override_all()
import matplotlib.pyplot as plt
from matplotlib import widgets
import numpy as np
%matplotlib tk

fig = plt.figure(figsize=(4,8))
axs = fig.gca()
axs.axis([0,9,0,9])
XY = iquib(np.array([4.,4.]))
axs.plot(XY[0],XY[1],marker='o',markersize=18,markerfacecolor='r',picker=True)

Commit: 7f0c676

ignore out-of-bound overrides

We need to think about what happens with out-of-bound overrides

These can be generated fairly commonly in some applications.
For example, consider the polyfit demo:
say we start with num_points=iquib(10), then move (override) the last point, and then change num_points to 9.

currently, this will yield an out-of-bound error
I think better to keep these overrides but ignore them as long as they are out-of-bound.
but this may bring another complication an assignment could also be partially out-of-bound.
your thoughts?

notebook stuck and get released only when adding a new cell

try running quibdemo_same_data_in_many_forms
it gets stuck with [*] on the last notebook cell.
can be released and works fine if another cell is executed while being stuck

commit: 49740e1

inconsistent choice of inverse assignment target in two-quib operators

the following code leads to inconsistent behavior, sometimes assigning to a sometimes to b:
(on Jupyter)

a = iquib(np.arange(3).reshape(3,1))
b = iquib(np.arange(4).reshape(1,4))*10
ab = a + b
ab.get_value()
ab[1,2] = 121
print(a.get_value())
print(b.get_value())
print(ab.get_value())

for axis with axs.set_xlim(Q), invokes inverse assignment when the user pans the axis view

say we set the xlim of two axes to the same quib:

Q = iquib([1,4])
axs1 = plt.add_subplot(2,1,1)
axs2 = plt.add_subplot(2,1,2)
axs1.set_xlim(Q)
axs2.set_xlim(Q)

it will be quite powerful if when the user pan/zoom the view area of any of the two axes it will lead to inverse assignment so that the other axis also changes.

fix refresh of RadioButtons

see quibdemo_fft

divergence

thinking more, I think we have a conceptual problem with the scheme for divergence that we discussed.
the idea that quibs that are np.fcns are not converged is wrong if these quibs are coming AFTER a diverged np.vectorize quib.

consider the following fake example:

filenames = iquib(...) # array of file names of N images
imgs = np.vectorize(my_read_file,filenames,...) # reading each file -> output is an array shape=(N,H,D)
norm_imgs = imgs / 255 # normalizing (or any other function on all images)
plt.image(norm_imgs[7,:,:]) # show image number 7

the problem is that while imgs is diverged, norm_imgs is not, so 'norm_imgs[7,:,:].get_value()' will load all the images!

need GUI to allow assigning Default to a graphical functional quib

override special methods of specific classes

we decided to have a dict specifying for each class special methods, like .T and .flat that are implemented

qT = q.T creates a functional quib (without evaluation)
qT.get_value() checks that .T is defined for the class of the output of q.

indexing bug in inverse assignment

There are several (maybe related?) issues with indexing of inverse assignment. here are two examples:

a = iquib(np.arange(3).reshape(3,1))
b = np.arange(4).reshape(1,4)*10
ab = a + b
ab.get_value()
ab[1,2] = 121
print(a.get_value())

this should only change a[1,0], but it yields:
a =

[[101]
 [101]
 [  2]]

a = iquib(np.arange(4).reshape(1,4))
b = iquib(np.arange(4).reshape(1,4))
ab = a + b
ab.get_value()
ab[0,1] = 3
ab.get_value()

yields an error:

     36                 new_element = np.array(new_element)
---> 37             new_element[path] = last_element
     38         last_element = new_element
     39 

IndexError: index 1 is out of bounds for axis 0 with size 1

unexpected behavior of graphics-driven assignment

The following code plots x-y of two functional quibs x and y.
Both quibs x and y are with allow_overriding=True, but dragging a point only leads to overriding of x not of y.

It looks like this behavior is related to y being dependent of x, but I think if we specify allow_overriding for y then overriding should be allowed.

from pyquibbler import iquib, override_all, q
import matplotlib.pyplot as plt
from matplotlib import widgets
import numpy as np
override_all()
x = q(np.arange,0., 5)
y = q(np.power, x, 2)
x.allow_overriding = True
y.allow_overriding = True
plt.figure(figsize=[10, 7])
plt.plot(x,y,'o',markersize=18,picker=True)
plt.show()

aq not equal qa

95eebec

multiplying ndarray*quib yields an ndarray rather than a quib.

>> q = iquib(10)
>> a = np.array([1,2,3])

>> type(q*a)
pyquibbler.quib.function_quibs.default_function_quib.DefaultFunctionQuib

>> type(a*q)
numpy.ndarray

How to implement matplotlib.Patches ?

Are matplotlib.Patches implemented?
For now, I tried something like this (below), but it didn't quite work. advice?

y = iquib(0.8)
Q_patch = q(patches.Rectangle, (0,0), 0.5, y)
Q_addpatch = q(axs.add_patch, Q_patch)

error when dragging coinciding points

Consider the following code which plots two coinciding markers:

from pyquibbler import iquib, override_all, q
import matplotlib.pyplot as plt
override_all()
fig = plt.figure(figsize=(4, 3))
x = iquib([1., 1.])
y = iquib([1., 1.])
axs = fig.add_subplot(1, 1, 1)
axs.plot(x, y, 'o-', markersize=17, picker=True)
plt.show()

Attempting to drag the marker(s) yields an error:

IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

the assign method - bug?

a = iquib(10)
a.assign(11)
a.get_value()

produces an error:

AttributeError: 'int' object has no attribute 'value'

graphics not correctly updating in multi-plot function

the following script makes a plot with two lines:

from pyquibbler import iquib, override_all, q
import matplotlib.pyplot as plt
import numpy as np
override_all()
%matplotlib widget

v = iquib(np.array([[1, 2],[3, 5],[4, 7]]))
plt.figure(1)
plt.clf()
h = plt.plot(v)

v[0,0] = 1.1 # BUG: colors are swapped
v[0,0] = 2 # BUG: old line is not deleted

Upon changes to the quib argument of the plot, several issues are observed:

the resulting graphics is NOT consistent upon repeated run (Kernel restart. then run all). See attached several observed outputs.
colors of lines get swaped
old graphics is not always deleted upon update (see example figure)