Idea: deferred errors about dask HOT 5 OPEN

Hrm, so the very easy thing to do is to assert the result of a python function onto each block without the barrier. Although in multi-threaded contexts this might get weird.

x = da.assert_(x, lambda block: block > 0, ValueError('x must be positive'))

Doing general full-array assertions is also doable (with a bit more complex graph magic) but would probably fill up cache space with the intermediate variables.

from dask.

@eric-czech brought this up today in a call. CC'ing him here.

@shoyer do you have any thoughts on how we would do this today? FWIW I think taht @eric-czech is operating on Dask under Xarray.

from dask.

👍

Yep, using Xarray over Dask. I'd love to be able to use lazy, runtime checks like that.

from dask.

Can you say a bit more about your needs @eric-czech ? Do you mostly need elementwise checks, or something more complicated? How would you like to spell these checks?

from dask.

Elementwise checks would cover the majority of cases I can think of as being useful. Checks on reductions would also be nice (i.e. sums along an axis equal 1), but not critical. I can see some value in making the assertion a terminal task as well, e.g.:

data = da.array(..., dtype=int)
mask = da.array(..., dtype=bool)
da.assertion(
  data[mask].min(), 
  lambda v: v >= 0, 
  lambda v: ValueError(f'Data values must be >= 0 (found min value {v})')
) 
# do stuff with data and mask but not data[mask]

as opposed to:

res = da.assertion(
  data[mask].min(), 
  lambda v: v >= 0, 
  lambda v: ValueError(f'Data values must be >= 0 (found min value {v})')
) 
# now I need to use `res` elsewhere in the graph for the 
# assertion to fire, but I don't necessarily want to

from dask.