navis-org / fafbseg-py Goto Github PK

View Code? Open in Web Editor NEW

12.0 4.0 5.0 98.21 MB

Tools to work with the FlyWire connectome. Fully interoperable with navis.

Home Page: https://fafbseg-py.readthedocs.io

License: GNU General Public License v3.0

Python 100.00%

fafb flywire navis segmentation fly brain google synapses annotation cave

fafbseg-py's People

Contributors

Stargazers

Watchers

Forkers

braincircuitsio tomka bjm5164 bocklab

fafbseg-py's Issues

graceful failures in l2_skeleton

Via R but I guess it will be the same in python. 720575940452114071 is a very small body (2 supervoxels). This is annoying when you are skeletonising 200 neurons in a single call and one fails ...

> fp$flywire$l2_skeleton("720575940452114071")
Error in py_call_impl(callable, dots$args, dots$keywords) : 
  AxisError: axis 1 is out of bounds for array of dimension 1

Detailed traceback:
  File "/Users/jefferis/Library/r-miniconda/envs/r-reticulate/lib/python3.8/site-packages/fafbseg/flywire/l2.py", line 237, in l2_skeleton
    l2_eg = np.unique(np.sort(l2_eg, axis=1), axis=0)
  File "<__array_function__ internals>", line 180, in sort
  File "/Users/jefferis/Library/r-miniconda/envs/r-reticulate/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 1004, in sort
    a.sort(axis=axis, kind=kind, order=order)

OverflowError: Python int too large to convert to C long

when I read the tutorials to fetche the neuron as mesh and then skeletonizes it in https://fafbseg-py.readthedocs.io/en/latest/source/tutorials/flywire_neurons.html, but the program reported an error OverflowError: Python int too large to convert to C long ，The code is all from the official website tutorial. I remember a few months ago when I wanted to implement the same code to obtain the skeleton, it was very smooth. I don't know what happened now, and I sincerely request your help

import navis
from fafbseg import flywire
sk = flywire.skeletonize_neuron(720575940617774213, progress=False)

Skeletonizing: 100%|██████████| 232440/232440 [00:01<00:00, 214595.25it/s]
Traceback (most recent call last):
  File "D:/Codes/230308-point-visualize/0417_get_flywire_swc.py", line 14, in <module>
    sk = flywire.skeletonize_neuron(720575940634710629)
  File "C:\Users\Administrator\miniconda3\lib\site-packages\fafbseg\flywire\skeletonize.py", line 204, in skeletonize_neuron
    if is_materialized_root(id):
  File "C:\Users\Administrator\miniconda3\lib\site-packages\fafbseg\flywire\utils.py", line 388, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\Administrator\miniconda3\lib\site-packages\fafbseg\flywire\annotations.py", line 140, in is_materialized_root
    id = make_iterable(id, force_type=int)
  File "C:\Users\Administrator\miniconda3\lib\site-packages\fafbseg\utils.py", line 124, in make_iterable
    return np.asarray(x, dtype=force_type)
OverflowError: Python int too large to convert to C long

The same ID was okay when I obtained the mesh

n = flywire.get_mesh_neuron(720575940617774213)
print(n)

it succeeded

type            navis.MeshNeuron
name                        None
id            720575940617774213
units                1 nanometer
n_vertices                265599
n_faces                   532059
dtype: object

Outsource some of the FlyWire functions to a separate package

Maybe navis-graphene? That way we can more easily reuse code for FANC.

Alternatively, we could widen the scope of fafbseg and rename it to e.g. navis-emdatasets with modules for flywire, fafb, fanc and hemibrain.

navis-flyem, navis-fly, navis-flydata

Can not pull synapses, update IDs... perform any 'intensive'

Rewrite neuroglancer scenes as classes

use __add__ method to combine scenes
use __repr__ to format as URL
subclass to FlyWireScene

l2 dotprops seems to fail when pca attributes missing for some points

From R, but I think this is general now that not all PCAs are calculate https://flywire-forum.slack.com/archives/C01M4LP2Y2D/p1646955614385879

> read_l2dp('720575940619695860')
Fetching L2 vectors:   0%|          | 0/233 [00:00<?, ?it/s]/Users/jefferis/Library/r-miniconda/envs/r-reticulate/lib/python3.8/site-packages/caveclient/l2cache.py:78: UserWarning: L2Cache is in an experimental stage
  warnings.warn("L2Cache is in an experimental stage", UserWarning)
                                                                       Error in py_call_impl(callable, dots$args, dots$keywords) : 
  KeyError: 'pca'

Detailed traceback:
  File "/Users/jefferis/Library/r-miniconda/envs/r-reticulate/lib/python3.8/site-packages/fafbseg/flywire/l2.py", line 494, in l2_dotprops
    vec = np.vstack([i['pca'][0] for i in this_info])
  File "/Users/jefferis/Library/r-miniconda/envs/r-reticulate/lib/python3.8/site-packages/fafbseg/flywire/l2.py", line 494, in <listcomp>
    vec = np.vstack([i['pca'][0] for i in this_info])

Failure to complete significant API calls, e.g. update 3+ neuron ID at same time or flywire.get_synapses()

Background
Hi. I have been at this for weeks now, without resolving the issue. I have tried multiple python versions, IDEs, networks/ethernet, generated new flywire tokens, conda environments... everything I can think of. I am on Mac OS X 14.2.1 (Sonoma) on a MacBook Pro M3 Max, using Pycharm to develop in a Python 3.10 virtual miniconda environment, with fafbseg 3.0.2, CAVE client 5.17.2, When using the newest python/fafbseg, I'd had some quirky issues with installation dependencies due to using a Mac with a Silicon chip, and so I downgraded to the versions my lab colleague uses successfully for flywire. I have an account with flywire & the codex, but do not seem to have production dataset access, but I just set the default database to public for now.

Goal
Anyway, I have been attempting to write a script which gets a synapse list for a specific neuron ID, and then compares it to lists of neurons I have by cell type, in order to produce a basic graph of what cell types this neuron projects to. Eventually I will do a ton of other stuff, but I have been stuck at just getting the synapse list.

What works and what doesn't
I am able to successfully update the ID for 1-4/5 neurons at one time with the update_ids() function. However, if I try to update more than 5 at once using a list, I get the attached timeout error:
ERROR_update_ids.txt

I am also able to use the. get_mesh_neuron() function successfully, though I don't actually need the function.

I am entirely unable to get a response for the get_synapses(), get_connectivity/synapses.fetch_connectivity(), get_transmitter_predictions(), or get_synapse_counts() functions. I have attached the typical error I get when attempting to run the following line (regardless of any combination of parameters):
ERROR_get_synapses.txt

fafbseg.flywire.get_synapses($neuronID, downstream=False, min_score=30)

Since the two errors are extremely similar, I'm thinking the error has something to do with connectivity..? Please let me know if anyone has seen this error before and/or what I could do about it. I have also attached a trimmed down version of my script below. Thanks!

script_for_help.txt

-- Mason Weinstock

minimum python version is now >=3.7.1

was trying to install under python 3.6 but pandas >=1.2.0 => python >=3.7.1 https://pandas.pydata.org/pandas-docs/stable/whatsnew/v1.2.0.html#increased-minimum-version-for-python

Ideas for new features

Just to keep a running list of potential new features:

annotations:
- pre-populate a type column (cell_type back-filled with hemibrain_type)
connectivity queries:
- optionally add annotations (type, side, etc)
- optionally collapse neurons into types
- add a get_graph function (this could include a grow parameter to grow the graph by n hops from the query IDs)
- add a function to download the entire graph - I never checked how long that would even take these days
- for large queries add a sparse parameter to get_adjacencies which would return either the edge list or a sparse matrix

module 'fafbseg.flywire' has no attribute 'set_default_dataset'

According to the documentation, setting the dataset to public requires fafbseg.set_default_dataset("public"). However, when I run it, I receive the error: module 'fafbseg.flywire' has no attribute 'set_default_dataset'. Here is my code:

from fafbseg import flywire
flywire.set_default_dataset("public")

Attribute error in `flywire.fetch_connectivity`

python version = 3.9
fafbseg version = 2.0.2

Code:

from fafbseg import flywire

a = flywire.fetch_connectivity(720575940623472716, clean = True,
                               min_score = 50, upstream = True, 
                               downstream = False, neuropils = True,
                              )

Error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[88], line 1
----> 1 a = flywire.fetch_connectivity(720575940623472716, clean = True,
      2                                min_score = 50, upstream = True, 
      3                                downstream = False, neuropils = True,
      4                               )

File ~/opt/miniconda3/envs/py39/lib/python3.9/site-packages/fafbseg/flywire/utils.py:120, in inject_dataset.<locals>.outer.<locals>.inner(*args, **kwargs)
    118 if disallowed and ds in disallowed:
    119     raise ValueError(f'Dataset "{ds}" not allowed for function {func}.')
--> 120 return func(*args, **kwargs)

File ~/opt/miniconda3/envs/py39/lib/python3.9/site-packages/fafbseg/flywire/synapses.py:1083, in fetch_connectivity(x, clean, style, upstream, downstream, proofread_only, transmitters, neuropils, filtered, min_score, batch_size, mat, progress, dataset)
   1080 neuropils = make_iterable(neuropils)
   1082 if len(neuropils):
-> 1083     filter_in = [n for n in neuropils if not n.startswith("~")]
   1084     filter_out = [n[1:] for n in neuropils if n.startswith("~")]
   1086     syn["neuropil"] = get_synapse_areas(syn["id"].values)

File ~/opt/miniconda3/envs/py39/lib/python3.9/site-packages/fafbseg/flywire/synapses.py:1083, in <listcomp>(.0)
   1080 neuropils = make_iterable(neuropils)
   1082 if len(neuropils):
-> 1083     filter_in = [n for n in neuropils if not n.startswith("~")]
   1084     filter_out = [n[1:] for n in neuropils if n.startswith("~")]
   1086     syn["neuropil"] = get_synapse_areas(syn["id"].values)

AttributeError: 'numpy.bool_' object has no attribute 'startswith'

flywire/skeletonize.py possibly outdated

The use of the skeletonize_neuron() function in flywire/merge.py does not match its definition in flywire/skeletonize.py . The function definition has the input variable remove_soma_hairball and returns one variable, while the function usage in merge.py (line 88) has the input variable drop_soma_hairball and expects 3 variables returned.

fafbseg.flywire.synapses.fetch_adjacency doesn't recognize lists as inputs

In fafbseg/utils.py, we have the following function:
def make_iterable(x, force_type = None)And when passing a list, utils.make_iterable is called in `fafbseg/flywire/utils.py:


 if isinstance(x, navis.BaseNeuron):
    ids = [x.id]
 elif isinstance(x, navis.NeuronList):
    ids = x.id
 elif isinstance(x, (int, [np.int](http://np.int/))):
    ids = [x]
 else:
    ids = utils.make_iterable(x, dtype=np.int64)

But an error occurs, because the parameters do not match the function declaration.