Fork of Streamz
Home Page: https://xpdacq.github.io/rapidz
License: BSD 3-Clause "New" or "Revised" License
A Zip node which errors when it buffers more that one piece of data
CC @tacaswell
Hi @sbillinge . I need the maintainer right of this project so that I can release a new version with my bug fix.
Newly cloned, and running pytest results in
16 failed, 979 passed, 496 skipped, 23 xfailed, 112 xpassed, 115 warnings, 102 error
This is with pytest 3.10 and pytest-tornado installed. Full versions below. First, a sample failure:
_______________________________________________________________________ test_filter_zip _______________________________________________________________________
def test_func():
del _global_workers[:]
_global_clients.clear()
active_threads_start = set(threading._active)
reset_config()
dask.config.set({'distributed.comm.timeouts.connect': '5s'})
# Restore default logging levels
# XXX use pytest hooks/fixtures instead?
for name, level in logging_levels.items():
logging.getLogger(name).setLevel(level)
result = None
workers = []
with pristine_loop() as loop:
with check_active_rpc(loop, active_rpc_timeout):
@gen.coroutine
def coro():
with dask.config.set(config):
s = False
for i in range(5):
try:
s, ws = yield start_cluster(
ncores, scheduler, loop, security=security,
Worker=Worker, scheduler_kwargs=scheduler_kwargs,
worker_kwargs=worker_kwargs)
except Exception as e:
logger.error("Failed to start gen_cluster, retryng", exc_info=True)
else:
workers[:] = ws
args = [s] + workers
break
if s is False:
raise Exception("Could not start cluster")
if client:
c = yield Client(s.address, loop=loop, security=security,
asynchronous=True, **client_kwargs)
args = [c] + args
try:
future = func(*args)
if timeout:
future = gen.with_timeout(timedelta(seconds=timeout),
future)
result = yield future
if s.validate:
s.validate_state()
finally:
if client:
yield c._close(fast=s.status == 'closed')
yield end_cluster(s, workers)
yield gen.with_timeout(timedelta(seconds=1),
cleanup_global_workers())
try:
c = yield default_client()
except ValueError:
pass
else:
yield c._close(fast=True)
raise gen.Return(result)
result = loop.run_sync(coro, timeout=timeout * 2 if timeout else timeout)
for w in workers:
if getattr(w, 'data', None):
try:
w.data.clear()
except EnvironmentError:
# zict backends can fail if their storage directory
# was already removed
pass
del w.data
DequeHandler.clear_all_instances()
for w in _global_workers:
w = w()
w._close(report=False, executor_wait=False)
if w.status == 'running':
w.close()
del _global_workers[:]
if PY3 and not WINDOWS and check_new_threads:
start = time()
while True:
bad = [t for t, v in threading._active.items()
if t not in active_threads_start and
"Threaded" not in v.name and
"watch message" not in v.name]
if not bad:
break
else:
sleep(0.01)
if time() > start + 5:
from distributed import profile
tid = bad[0]
thread = threading._active[tid]
call_stacks = profile.call_stack(sys._current_frames()[tid])
> assert False, (thread, call_stacks)
E AssertionError: (<Thread(Profile, started daemon 123145584852992)>, [' File "/Users/mdurant/anaconda/envs/py36/lib/python3.6/threadin...ant/anaconda/envs/py36/lib/python3.6/site-packages/distributed/profile.py", line 249, in _watch
E \tsleep(interval)
E '])
E assert False
../../anaconda/envs/py36/lib/python3.6/site-packages/distributed/utils_test.py:865: AssertionError
-------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:62582
distributed.worker - INFO - Start worker at: tcp://127.0.0.1:62583
distributed.worker - INFO - Listening to: tcp://127.0.0.1:62583
distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:62582
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 1
distributed.worker - INFO - Memory: 17.18 GB
distributed.worker - INFO - Local Directory: /Users/mdurant/code/rapidz/dask-worker-space/worker-te39lu4r
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Start worker at: tcp://127.0.0.1:62584
distributed.worker - INFO - Listening to: tcp://127.0.0.1:62584
distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:62582
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Threads: 2
distributed.worker - INFO - Memory: 17.18 GB
distributed.worker - INFO - Local Directory: /Users/mdurant/code/rapidz/dask-worker-space/worker-suzejl65
distributed.worker - INFO - -------------------------------------------------
distributed.scheduler - INFO - Register tcp://127.0.0.1:62583
distributed.scheduler - INFO - Register tcp://127.0.0.1:62584
distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:62583
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:62584
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://127.0.0.1:62582
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.worker - INFO - Registered to: tcp://127.0.0.1:62582
distributed.worker - INFO - -------------------------------------------------
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Receive client connection: Client-fbdd58e6-1424-11e9-be60-acbc32a9ff1d
distributed.core - INFO - Starting established connection
distributed.scheduler - INFO - Remove client Client-fbdd58e6-1424-11e9-be60-acbc32a9ff1d
distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:62583
distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:62584
distributed.scheduler - INFO - Scheduler closing...
distributed.scheduler - INFO - Scheduler closing all comms
distributed.worker - INFO - Connection to scheduler broken. Reconnecting...
distributed.worker - INFO - Connection to scheduler broken. Reconnecting...
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
Versions:
# packages in environment at /Users/mdurant/anaconda/envs/py36:
#
# Name Version Build Channel
_nb_ext_conf 0.4.0 py36_1
affine 2.2.0 py36_0
alabaster 0.7.10 py36h174008c_0
anaconda-client 1.6.14 py36_0
aplus 0.11.0 py_1 conda-forge
appdirs 1.4.3 <pip>
appdirs 1.4.3 py36_0
appnope 0.1.0 py36hf537a9a_0
arrow-cpp 0.10.0 py36h70250a7_0 conda-forge
asciitree 0.3.3 py36_1 conda-forge
asn1crypto 0.24.0 py36_0
astroid 1.6.2 py36_0
astropy 3.0.1 py36h1de35cc_1
atomicwrites 1.2.1 py36_0
attrs 17.4.0 py36_0
autopep8 1.3.4 py_0 conda-forge
awkward 0.0.3 <pip>
aws-xray-sdk 0.95 <pip>
aws-xray-sdk 0.95 py_0 conda-forge
babel 2.5.3 py36_0
backcall 0.1.0 py36_0
backports 1.0 py_2 conda-forge
backports.tempfile 1.0 py_0 conda-forge
backports.weakref 1.0.post1 py36_1000 conda-forge
bcolz 1.2.1 py36h1702cab_0
bcrypt 3.1.4 py36h1de35cc_0
beautifulsoup4 4.6.0 py36h72d3c9f_1
blas 1.0 mkl
bleach 2.1.3 py36_0
blinker 1.4 py36_0
blosc 1.14.3 hd9629dc_0
bokeh 1.0.1 py36_0
boost-cpp 1.67.0 h3a22d5f_0 conda-forge
boto 2.48.0 py36hdbc59ac_1
boto3 1.7.84 py_0 conda-forge
botocore 1.10.84 py_0 conda-forge
bottleneck 1.2.1 py36hbd380ad_0
bqplot 0.11.0 py36_0 conda-forge
brotli 0.6.0 1 conda-forge
bzip2 1.0.6 hd86a083_4
ca-certificates 2018.03.07 0
cachetools 2.0.1 py_0 conda-forge
cachetools 2.0.1 <pip>
cairo 1.14.12 he6fea26_5 conda-forge
certifi 2018.11.29 py36_0
cffi 1.11.5 py36h342bebf_0
cftime 1.0.0b1 py36h1d22016_0
chardet 3.0.4 <pip>
chardet 3.0.4 py36h96c241c_1
cityhash 0.2.3.post9 <pip>
click 6.7 py36hec950be_0
click-plugins 1.0.3 py36hb58b8ed_0
cligj 0.4.0 py36h479156e_0
cloudpickle 0.5.3 py36_0
clyent 1.2.2 py36hae3ad88_0
colorcet 1.0.0 py36_0
coloredlogs 9.0 py36_0
conda 4.5.12 py36_0
conda-build 3.8.0 py36_0
conda-env 2.6.0 h36134e3_0
conda-verify 2.0.0 py36he837df3_0
cookies 2.2.1 py36_0
coverage 4.5.1 py36h1de35cc_0
cryptography 2.2.2 py36h1de35cc_0
curl 7.61.0 ha441bb4_0
cyberpandas 1.1.1 py36_0 conda-forge
cycler 0.10.0 py36hfc81398_0
cython 0.29 py36h0a44026_0
cytoolz 0.9.0.1 py36h1de35cc_1
dask 0.18.2+45.gfe57472 <pip>
dask 0.19.0 py36_0
dask-core 0.19.0 py36_0
dask-glm 0.1.0 py36hd478735_0
dask-ml 0.4.1 py36h917ab60_0
dask-searchcv 0.2.0 py36hf54d424_0
data-us-states 1.0.0 1 local
datashader 0.6.5 py36_0
datashape 0.5.4 py36hfb22df8_0
dbus 1.13.0 h3a4f0e9_0 conda-forge
decorator 4.1.2 py36h69a1b52_0
deprecation 1.0.1 <pip>
dicttoxml 1.7.4 py36_0 conda-forge
distributed 1.21.6+8.g362c169.dirty <pip>
distributed 1.22.0+43.g55b933c <pip>
distributed 1.23.0 py36_0
docker 3.2.1 <pip>
docker-py 3.6.0 py36_1000 conda-forge
docker-pycreds 0.2.2 <pip>
docker-pycreds 0.3.0 py_1 conda-forge
docopt 0.6.2 py36_0
docutils 0.14 py36hbfde631_0
ecdsa 0.13 py_0 conda-forge
elasticsearch 6.2.0 py36_0
entrypoints 0.2.3 py36hd81d71f_2
expat 2.2.5 hb8e80ba_0
fastavro 0.21.4 py36h1de35cc_0
fastcache 1.0.2 py36h1de35cc_2
fasteners 0.14.1 py36_2 conda-forge
fastparquet 0.2.0 <pip>
fastparquet 0.1.3 py36hb84e16e_0
filelock 3.0.4 py36_0
flake8 2.5.5 <pip>
flask 0.12.2 py36h5658096_0
fletcher 0.1.2 py_0 conda-forge
fontconfig 2.13.1 hce039c3_0 conda-forge
freetype 2.9.1 h6debe1e_4 conda-forge
freexl 1.0.5 h1de35cc_0
fs 2.0.23 <pip>
fs 2.0.20 py36_0 conda-forge
fsspec 0.1.2 <pip>
funcsigs 1.0.2 py36_0
fusepy 2.0.4 py36_0 bioconda
future 0.16.0 py36_1
gcsfs 0.0.6 <pip>
gcsfs 0.2.0 <pip>
gdal 2.2.4 py36hb00a9d7_9 conda-forge
geos 3.6.2 h5470d99_2
geotiff 1.4.2 h700e5ad_4 conda-forge
gettext 0.19.8.1 h15daf44_3
giflib 5.1.4 h1035379_1
glib 2.55.0 h464dc38_2 conda-forge
glob2 0.6 py36h94c9186_0
gmp 6.1.2 hb37e062_1
gmpy2 2.0.8 py36h6ef4df4_2
google-auth 1.4.1 <pip>
google-auth 1.5.0 py_0 conda-forge
google-auth-oauthlib 0.1.0 py36_0 conda-forge
google-auth-oauthlib 0.2.0 <pip>
graphviz 2.38.0 hd4f1ed2_9 conda-forge
gsw 3.2.1 py36h470a237_2 conda-forge
gunicorn 19.9.0 py36_0
h5py 2.8.0 py36h3c9e6ae_0
hacking 1.0.0 <pip>
hadoop-test-cluster 0.0.4 <pip>
happybase 1.1.0 py36_2 conda-forge
hdf4 4.2.13 h39711bb_2
hdf5 1.10.2 hfa1e0ec_1
heapdict 1.0.0 py36h27a9ac6_0
holoviews 1.9.5 <pip>
holoviews 1.10.9 py_0 pyviz
html5lib 1.0.1 py36h2f9c1c0_0
httpretty 0.9.6 py36_0
humanfriendly 4.10 py36_0
hvplot 0.2.1 py_0 pyviz
hypothesis 3.38.5 py36hcba0581_0
icu 58.2 h4b95b61_1
idna 2.6 <pip>
idna 2.6 py36h8628d0a_1
imageio 2.3.0 py36_0
imagesize 1.0.0 py36_0
import-profiler 0.0.3 <pip>
intake 0.2.9 py_0 conda-forge
intake 0.3.0 <pip>
intake-accumulo 0.1.1 <pip>
intake-astro 0.1.1 <pip>
intake-avro 0.1.0 0 intake
intake-avro 0.2.0 <pip>
intake-github 0.0.1 <pip>
intake-mnist-data 0.0.1 py_0 local
intake-mongo 0.1.0 <pip>
intake-parquet 0.2.1 <pip>
intake-solr 0.0.4+2.g0f3030d <pip>
intake-spark 0.1.1 <pip>
intake-splunk 0.0.4 <pip>
intake-sql 0.2.0 <pip>
intake-xarray 0.0.1 <pip>
intel-openmp 2018.0.0 h8158457_8
ipykernel 4.8.2 py36_0
ipyleaflet 0.9.0 py36_0 conda-forge
ipympl 0.2.1 py36_0 conda-forge
ipython 6.3.1 py36_0
ipython_genutils 0.2.0 py36h241746c_0
ipyvolume 0.4.6 py_1 conda-forge
ipywebrtc 0.3.0 py_1 conda-forge
ipywidgets 7.2.1 py36_0
isodate 0.6.0 py36h339218c_0
isort 4.3.4 py36_0
itk 4.13.0 py36_0 conda-forge
itkwidgets 0.9.0 <pip>
itsdangerous 0.24 py36h49fbb8d_1
jedi 0.11.1 py36_0
jinja2 2.10 py36hd36f9c5_0
jmespath 0.9.3 py36h767a2d6_0
joblib 0.11 py36_0
jpeg 9c h470a237_1 conda-forge
json-c 0.12.1 h470a237_1 conda-forge
jsondiff 1.1.1 py_0 conda-forge
jsondiff 1.1.1 <pip>
jsonpickle 0.9.6 <pip>
jsonpickle 0.9.6 py_1 conda-forge
jsonschema 2.6.0 py36hb385e00_0
jupyter 1.0.0 py36_4
jupyter_client 5.2.3 py36_0
jupyter_console 5.2.0 py36hccf5b1c_1
jupyter_core 4.4.0 py36h79cf704_0
jupyterlab 0.34.3 py36_0
jupyterlab-thredds 0.1.0 <pip>
jupyterlab_launcher 0.13.1 py36_0
kapteyn 2.3 np111py36_1 conda-forge
kealib 1.4.9 h0bee7d0_2 conda-forge
keepalive 0.5 py36_0 conda-forge
keras 2.2.0 0
keras-applications 1.0.2 py36_0
keras-base 2.2.0 py36_0
keras-preprocessing 1.0.1 py36_0
keyring 15.1.0 py36_0
kiwisolver 1.0.1 py36h792292d_0
krb5 1.14.2 h9a779f2_6
lazy-object-proxy 1.3.1 py36h2fbbe47_0
libboost 1.65.1 hcc95346_4
libcurl 7.61.0 hf30b1f0_0
libcxx 4.0.1 h579ed51_0
libcxxabi 4.0.1 hebd6815_0
libdap4 3.19.1 h18059cb_1 conda-forge
libedit 3.1.20170329 hb402a30_2
libffi 3.2.1 h475c297_4
libgdal 2.2.4 hdb23ba1_9 conda-forge
libgfortran 3.0.1 h93005f0_2
libiconv 1.15 hdd342a3_7
libkml 1.3.0 hc0cae1e_3
libnetcdf 4.6.1 h148f0bd_0
libntlm 1.4 h1de35cc_2
libpcap 1.8.1 ha9f8877_0 intake
libpng 1.6.34 he12f830_0
libpq 10.3 hf30b1f0_0
libprotobuf 3.5.2 h2cd40f5_0
librdkafka 0.11.0 0 conda-forge
libsodium 1.0.16 h3efe00b_0
libspatialite 4.3.0a h3b29d86_23 conda-forge
libssh2 1.8.0 h322a93b_4
libtiff 4.0.9 h0dac147_0
libuuid 1.0.3 h6bb4b03_2
libxml2 2.9.8 h2866646_0
libxslt 1.1.32 hb819dd2_0
line_profiler 2.1.2 py36h1de35cc_0
llvmlite 0.26.0 py36_0 numba
locket 0.2.0 py36hca03003_1
lxml 4.2.4 py36hef8c89e_0
lz4 0.10.1 py36_0
lz4 0.19.1 <pip>
lz4-c 1.8.0 h8e33030_0
lzo 2.10 h362108e_2
mako 1.0.7 py36_0
markdown 3.0.1 py36_0
markupsafe 1.0 py36h3a1e703_1
marshmallow 3.0.0b6 <pip>
matplotlib 2.2.3 py36h0e0179f_0 conda-forge
mccabe 0.6.1 py36hdaeb55d_0
mccabe 0.2.1 <pip>
mechanicalsoup 0.11.0 py_0 conda-forge
memory_profiler 0.52.0 py36_0
mistune 0.8.3 py36_0
mkl 2018.0.3 1
mkl_fft 1.0.4 py36h5d10147_1
mkl_random 1.0.1 py36h5d10147_1
mmh3 2.5.1 <pip>
mock 2.0.0 py36hd48142b_0
mock 2.0.0 <pip>
mongo-driver 0.7.1 0
mongoadapter 2.0.0 py36_0 file:///Users/mdurant/anaconda/envs/py36/conda-bld
monotonic 1.4 py36_0 conda-forge
more-itertools 4.3.0 py36_0
moto 1.3.6 py_0 conda-forge
moto 1.3.3 <pip>
mpc 1.1.0 h6ef4df4_1
mpfr 4.0.1 h3018a27_3
mpmath 1.0.0 py36_2
msgpack 0.5.6 <pip>
msgpack-numpy 0.4.3 py36_0
msgpack-python 0.5.6 py36h04f5b5a_0
multipledispatch 0.5.0 py36_0
nb_anacondacloud 1.4.0 py36_0
nb_conda 2.2.1 py36h349edbb_0
nb_conda_kernels 2.1.0 py36_0
nbconvert 5.3.1 py36h810822e_0
nbformat 4.4.0 py36h827af21_0
nbpresent 3.0.2 py36hef4c988_1
nbsphinx 0.3.3 py_0 conda-forge
ncurses 6.1 hfc679d8_1 conda-forge
netcdf4 1.4.0 py36h08833f9_1
networkx 2.1 py36_0
ninja 1.8.2 py36h04f5b5a_1
notebook 5.4.1 py36_0
nufft 0.0.1.dev0 <pip>
numba 0.41.0 np115py36h0a44026_0 numba
numcodecs 0.5.5 py36_0 conda-forge
numexpr 2.6.5 py36h057f876_0
numpy 1.15.3 py36h6a91979_0
numpy-base 1.15.3 py36h8a80b8c_0
numpydoc 0.8.0 py36_0
oauthlib 2.1.0 py36_0
oauthlib 2.0.7 <pip>
olefile 0.45.1 py36_0
openjpeg 2.3.0 h9b27163_0
openssl 1.0.2p h1de35cc_0
owslib 0.16.0 py36_0
packaging 17.1 py36_0
pandas 0.23.4 py36h6440ff4_0
pandas-datareader 0.6.0 py36_0
pandoc 1.19.2.1 ha5e8f32_1
pandocfilters 1.4.2 py36h3b0b094_1
panel 0.1.2 py_0 pyviz
param 1.8.1 py_0 pyviz
paramiko 2.4.2 py36_0
paramnb 2.0.4 py_0 pyviz
parquet-cpp 1.5.0.pre h83d4a3d_0 conda-forge
parso 0.1.1 py36hc90e01c_0
partd 0.3.8 py36hf5c4cb8_0
patsy 0.5.0 py36_0
pbr 4.0.1 <pip>
pbr 4.0.2 py36_0
pbr 0.11.1 <pip>
pcapy 0.11.1 py36ha52361c_0 intake
pcre 8.42 h378b8a2_0
pep8 1.5.7 <pip>
pexpect 4.3.1 py36_0
pickle5 0.0.4 py36h470a237_0 conda-forge
pickleshare 0.7.4 py36hf512f8e_0
pillow 5.2.0 py36hb68e598_0
pims 0.4.1 py_1 conda-forge
pip 18.1 py36_0
pixman 0.34.0 hca0a616_3
pkginfo 1.4.2 py36_1
pluggy 0.8.0 py36_0
ply 3.11 py36_0
poppler 0.67.0 h4d7e492_3 conda-forge
poppler-data 0.4.8 hf2eda46_0
postgresadapter 2.0.1 py36_0 intake
postgresql 10.3 hf30b1f0_0
profimp 0.0.3 <pip>
progressbar2 3.37.1 py36_0
proj4 4.9.3 h470a237_8 conda-forge
prompt_toolkit 1.0.15 py36haeda067_0
protobuf 3.5.2 py36h0a44026_0
psutil 5.4.3 py36h1de35cc_0
psycopg2 2.7.4 py36hdbc3d79_0
ptyprocess 0.5.2 py36he6521c3_0
py 1.5.2 py36ha69170d_0
py4j 0.10.7 py36_0
pyaccumulo 1.5.0.9 py_0 local
pyaml 17.12.1 py_0 conda-forge
pyarrow 0.10.0 py36hfc679d8_0 conda-forge
pyasn1 0.4.3 py36_0
pyasn1 0.4.2 <pip>
pyasn1-modules 0.2.1 py36h1776605_0
pyasn1-modules 0.2.1 <pip>
pybind11 2.2.1 <pip>
pycodestyle 2.3.1 py36h83e8646_0
pycosat 0.6.3 py36hee92d8f_0
pycparser 2.18 py36h724b2fc_1
pycryptodome 3.7.1 py36h16a7912_0 conda-forge
pydap 3.2.1 py36_0 conda-forge
pydicom 1.1.0 py_0 conda-forge
pyflakes 0.8.1 <pip>
pyflakes 1.6.0 py36hea45e83_0
pyftpdlib 1.5.4 py_1 conda-forge
pygithub 1.39 py36_0 conda-forge
pygments 2.2.0 py36h240cd3f_0
pyjwt 1.6.4 py36_0
pykerberos 1.2.1 py36h1de35cc_0
pylint 1.8.3 py36_0
pymongo 3.4.0 py36_0
pympler 0.5 py36hc2752d1_0
pynacl 1.3.0 py36h1de35cc_0
pyopengl 3.1.1a1 py36_0
pyopenssl 17.5.0 py36h51e4350_0
pyparsing 2.2.0 py36hb281f35_0
pypdf2 1.26.0 py36_1 conda-forge
pyproj 1.9.5.1 py36h508ed2a_5 conda-forge
pyqt 5.9.2 py36h11d3b92_0
pysocks 1.6.8 py36_0
pysolr 3.6.0 py36_0 conda-forge
pyspark 2.4.0 py36_0
pytables 3.4.4 py36h13cba08_0
pytest 3.10.1 py36_0
pytest-arraydiff 0.2 py36_0
pytest-astropy 0.2.1 py36_0
pytest-cov 2.5.1 py36hf4510af_0
pytest-doctestplus 0.1.2 py36_0
pytest-openfiles 0.2.0 py36_0
pytest-remotedata 0.2.0 py36_0
pytest-runner 4.2 py36_0
pytest-tornado 0.5.0 py36_0
python 3.6.4 hc167b69_0
python-blosc 1.5.1 py36h8366757_1
python-confluent-kafka 0.11.4 py36_0 conda-forge
python-dateutil 2.6.1 py36h86d2abb_1
python-graphviz 0.8.3 py36_0 conda-forge
python-jose 2.0.2 py_0 conda-forge
python-snappy 0.5.2 py36h0a44026_0
python-utils 2.3.0 py36_0
python.app 2 py36_8
pytz 2017.3 py36hf0bf824_0
pyviz_comms 0.6.0 py_0 pyviz
pywavelets 0.5.2 py36h2710a04_0
pyyaml 3.12 py36h2ba1e63_1
pyzmq 17.0.0 py36h1de35cc_1
qt 5.9.6 h45cd832_2
qtawesome 0.4.4 py36h468c6fb_0
qtconsole 4.3.1 py36hd96c0ff_0
qtpy 1.4.0 py36_0
rasterio 0.36.0 py36h01fd17b_1
rdflib 4.2.2 py36_0 conda-forge
readline 7.0 hc1231fa_4
reikna 0.6.8 py36_0 conda-forge
requests 2.18.4 <pip>
requests 2.20.1 py36_0
requests-file 1.4.3 py36_0
requests-ftp 0.3.1 py36_0
requests-kerberos 0.12.0 py36_0
requests-oauthlib 0.8.0 <pip>
requests-oauthlib 0.8.0 py36_1
requests-toolbelt 0.8.0 py36_0
responses 0.9.0 py36_0
rise 5.2.0 py36_0 damianavila82
rope 0.10.7 py36h68959ac_0
rsa 3.4.2 py36_0 conda-forge
rsa 3.4.2 <pip>
ruamel.yaml 0.15.37 py36_0 conda-forge
ruamel_yaml 0.15.35 py36h1de35cc_1
s3fs 0.2.0 <pip>
s3fs 0.1.4 py36_0
s3transfer 0.1.13 py36_0
scikit-image 0.13.0 py36h398857d_1
scikit-learn 0.19.1 py36hffbff8c_0
scipy 1.0.0 py36h1de22e9_0
seaborn 0.8.1 py36h595ecd9_0
seaice_data 1 0 file:///Users/mdurant/anaconda/envs/py36/conda-bld
selenium 3.11.0 py36_0
send2trash 1.5.0 py36_0
setproctitle 1.1.10 py36_0 conda-forge
setuptools 40.6.3 py36_0
shapely 1.6.2 py36hed20685_0
sidecar 0.2.0 <pip>
simplegeneric 0.8.1 py36he5b5b09_0
simplejson 3.13.2 py36_0
singledispatch 3.4.0.3 py36hf20db9d_0
sip 4.19.8 py36h0a44026_0
six 1.11.0 py36h0e22d5e_1
slicerator 0.9.8 py_1 conda-forge
snakeviz 0.4.2 py36h2529b96_0
snappy 1.1.7 he62c110_3
snowballstemmer 1.2.1 py36h6c7b616_0
snuggs 1.4.1 py36hc1acd09_0
sortedcontainers 1.5.9 py36_0
sparqlwrapper 1.8.0 py36_0 conda-forge
sphinx 1.7.2 py36_0
sphinx_rtd_theme 0.2.4 py36_0
sphinxcontrib 1.0 py36h9364dc8_1
sphinxcontrib-websupport 1.0.1 py36h92f4a7a_1
spyder 4.0.0b1 py36_1 spyder-ide
spyder-kernels 1.1.0 py_0 spyder-ide
spyder-line-profiler 0.1.1 py_1 spyder-ide
spyder-memory-profiler 0.1.0 py36_1 spyder-ide
spyder-notebook 0.1.3 py36_0 spyder-ide
spyder-terminal 0.2.4 py36_0 spyder-ide
sqlalchemy 1.2.6 py36h1de35cc_0
sqlite 3.24.0 ha441bb4_0
statsmodels 0.8.0 py36h9c68fc9_0
streamz 0.3.0 <pip>
sympy 1.3 py36_0
tabulate 0.8.2 <pip>
tbb 2018.0.5 h04f5b5a_0
tblib 1.3.2 py36hda67792_0
tensorflow 1.1.0 py36_0 conda-forge
terminado 0.8.1 py36_1
testpath 0.3.1 py36h625a49b_0
thredds-crawler 1.5.4 <pip>
thrift 0.11.0 <pip>
thrift 0.10.0 py36h160bd8a_0
thrift-cpp 0.11.0 2 conda-forge
thriftpy 0.3.9 py36h1de35cc_1
tifffile 0.14.0 py36_1 conda-forge
tk 8.6.7 h35a86e2_3
toolz 0.9.0 py36_0
tornado 5.0.1 py36_1
tqdm 4.25.0 py36h28b3542_0
traitlets 4.3.2 py36h65bd3ce_0
traittypes 0.2.1 py_1 conda-forge
turbodbc 2.7.0 py36h6dcb523_1 conda-forge
twine 1.11.0 py36_0
typing 3.6.4 py36_0
uavro 0.0.3 py36_0 conda-forge
unixodbc 2.3.7 h09ba92c_0 conda-forge
urllib3 1.22 py36h68b9469_0
urllib3 1.22 <pip>
us_crime 0.1.0 1 local
vaex 1.0.0b7 py_0 conda-forge
vaex-astro 0.2.0 py36_0 conda-forge
vaex-core 0.4.0 py36_0 conda-forge
vaex-distributed 0.2.0 py_0 conda-forge
vaex-hdf5 0.2.1 py_0 conda-forge
vaex-jupyter 0.2.1 py_0 conda-forge
vaex-server 0.2.1 py_0 conda-forge
vaex-ui 0.2.0 py_0 conda-forge
vaex-viz 0.3.0 py_0 conda-forge
vcrpy 1.11.1 py36_0 conda-forge
vispy 0.5.3 py36hf951d6b_0
voc 0.1.5 <pip>
wcwidth 0.1.7 py36h8c6ec74_0
webencodings 0.5.1 py36h3b9701d_1
webob 1.8.2 py36_0
websocket-client 0.54.0 py36_1000 conda-forge
websocket-client 0.47.0 <pip>
Werkzeug 0.14.1 <pip>
werkzeug 0.14.1 py36_0
wheel 0.31.1 py36_0
widgetsnbextension 3.2.0 py36_0
wrapt 1.10.11 <pip>
wrapt 1.10.11 py36hc29e774_0
xarray 0.10.7 py36_0
xarray 0.10.9+12.gf9c4169 <pip>
xerces-c 3.2.0 h5d6a6da_2 conda-forge
xmltodict 0.11.0 py36_0 conda-forge
xmltodict 0.11.0 <pip>
xonsh 0.6.2 py36_0 conda-forge
xz 5.2.4 h1de35cc_4
yaml 0.1.7 hc338f04_2
zarr 2.2.1.dev10 <pip>
zarr 2.2.0 py_1 conda-forge
zeromq 4.2.5 h378b8a2_0
zict 0.1.3 py36h71da714_0
zlib 1.2.11 hf3cbc9b_2
zope 1.0 py36_0
zope.interface 4.4.3 py36h74e83c9_0
zstandard 0.8.1 py36_0 conda-forge
zstandard 0.9.0 <pip>
We can update the asyc docs with the parallel backend
This buffer would not preserve data ordering.
Most likely implementation would be to check if any of the existing futures in the buffer are done yet, if so send those downstream, else send the next in the queue.
CC: @tacaswell
This would allow accumulate to update its state when a new value of state_stream came down, allowing us to reset the accumulate state as needed.
The idea would be to have the nodes have a representation of themselves which reference their characteristic variables (their name, the name/module of any incoming functions, args, kwargs, etc.) plus their order in the execution and their parent. All of this will be hashed. Thus to check if a graph is different from another we can check if any of the nodes has a different characteristic hash.
Note that these hashes may not be unique, since source nodes would all look the same, but that may be ok. As a consequence we won't be able to use these as node names for networkx.
From @tacaswell: this way we can do sink().sink().sink().sink().
The null parallel backend would cause the system to run in series. This way we could turn on/off parallelism with the flip of a switch
@CJ-Wright There is no information about how to install this program in document.
xref: xpdAcq/xpdtools#62 (comment)
This produces an interesting pair of problems for the linker.
Since we have the qoi arg the pipeline the linker will look for a qoi upstream node to subscribe to. This node might not exist, or worse we may want to use this on multiple upstream nodes. I'm not certain how to handle this properly currently. One approach would be to modify the namespace dict before adding the tomography nodes. This would allow us to make a temporary qoi node which is the same as an existing node.
Sine we may use the tomo pipeline chunk multiple times we will need to modify the linker to make unique names for all the output names. This way we can keep them in the namespace without clashing.
def abs_chunk(img):
abs = img.map(np.sum)
return locals()
def tomo_chunk(qoi, ...):
tomo = qoi.map(...)
return locals()
def mutating_linker(func, namespace, input_lut, output_lut):
# takes valid node name to qoi (abs -> qoi)
for k, v in input_lut.items():
namespace[k] = namespace[v]
loc = func(**namespace)
# takes output name and changes it (tomo -> abs_tomo)
for k, v in output_lut.items():
loc[v] = loc.pop(k)
namespace.update(loc)
# Note that this means that input_lut keys can't be a valid output node name
# (qoi -> N/A)
for k, v in input_lut.items():
namespace.pop(k)We need a better name.
@sbillinge
Parallel filter causes a whole host of problems:
Currently we have half of a graph solver via the linker function. The linker function takes in a namespace hands it to the pipeline chunk function and then appends the output to the current namespace, and repeats. This allows us to solve the graph, attaching all the nodes together in the correct order, if we have the correct order for the links to be done. If the order is incorrect then we run into problems, as we will try to use parts of the namespace which don't exist yet. This is sub-optimal as it requires that we know the order at any point in time, which might not be true (eg. if we start composing pipelines dynamically we might not know the order ahead of time). The proper solution to this requires having for each pipeline chunk the required inputs and the outputs. The inputs are easy, they are the args to the function so we can go and look them up. The outputs are less so. Since we use the locals() function to avoid writing the explicit outputs we don't have any actual access to them until the function is run.
A potential solution to this problem is to spoof the namespace. By passing in dummy Stream objects we can see exactly what the output of the chunk is allowing us to get the output part of the chunk. With both the chunk inputs (via inspect.getargspec) and the outputs we can then get the correct order in which the nodes need to be created via chunks.
eg
def inspect_chunk(chunk)
args = inspect.getargspec(chunk)[0]
ns = chunk(**{k: Stream() for k in args})
streams = [k for k in ns if isinstance(ns[k], Stream) and k not in args]
return args, streamsNote that one could potentially call these chunks metanodes.
Note that this means each input/output node needs to have a unique name, which might not be a bad thing. (This will produce problems for the tomo system, but we were going to have problems on that front anyway, since it really is a metanode factory-like thing.
This could work? We might need to scatter the list to the executor first?
def f(x, l):
xr = x.result()
for y in l:
if xr == y.result():
return "~~NULL_COMPUTE~~"
return xr
def update(x, who=None):
future = self.executor.submit(f, x, self.list)
self.list.append(x)
yield self._emit(future)Satellite positions: https://www.n2yo.com/api/
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.