brainglobe / cellfinder Goto Github PK

View Code? Open in Web Editor NEW

170.0 15.0 38.0 171.41 MB

Automated 3D cell detection in very large images

Home Page: https://brainglobe.info/documentation/cellfinder/index.html

License: BSD 3-Clause "New" or "Revised" License

Python 100.00%

microscopy cell-detection deep-learning image-analysis neuroscience python neuroanatomy imaging

cellfinder's Introduction

cellfinder

cellfinder is software for automated 3D cell detection in very large 3D images (e.g., serial two-photon or lightsheet volumes of whole mouse brains). There are three different ways to interact and use it, each with different user interfaces and objectives in mind. For more details, head over to the documentation website.

At a glance:

There is a command-line interface called brainmapper that integrates with brainreg for automated cell detection and classification. You can install it through brainglobe-workflows.
There is a napari plugin for interacting graphically with the cellfinder tool.
There is a Python API to allow users to integrate BrainGlobe tools into their custom workflows.

Installation

You can find the installation instructions on the BrainGlobe website, which will go into more detail about the installation process if you want to minimise your installation to suit your needs. However, we recommend that users install cellfinder either through installing BrainGlobe version 1, or (if you also want the command-line interface) installing brainglobe-workflows.

# If you want to install all BrainGlobe tools, including cellfinder, in a consistent manner with one command:
pip install brainglobe>=1.0.0
# If you want to install the brainmapper CLI tool as well:
pip install brainglobe-workflows>=1.0.0

If you only want the cellfinder package by itself, you can pip install it alone:

pip install cellfinder>=1.0.0

Be sure to specify a version greater than version v1.0.0 - prior to this version the cellfinder package had a very different structure that is incompatible with BrainGlobe version 1 and the other tools in the BrainGlobe suite. See our blog posts for more information on the release of BrainGlobe version 1.

Contributing

If you have encountered a bug whilst using cellfinder, please open an issue on GitHub.

If you are interested in contributing to cellfinder (thank you!) - please head over to our developer documentation.

cellfinder's People

Contributors

Stargazers

Watchers

cellfinder's Issues

[BUG] memory usage gets very high during cube extraction

Describe the bug
When running cellfinder_curate when running cube extraction memory usage gradually increases until I have to terminate it (at 48Gb).

To Reproduce
ran cellfinder_curate with two channels reclassified ~100 cells manually, saved and then ran extraction

Expected behavior
expected less memory to be used

Log file
Please attach the cellfinder log file (e.g. output/cellfinder_2019-12-17_13-29-46.log)
no log file

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

OS: [e.g. Linux, Windows, MacOS]
Version [e.g. 18.04]
Hardware specs [e.g. 128GB RAM, Xeon 4114 CPU]
Linux 4.15.0-76-generic x86_64
48Gb ram

Additional context
Add any other context about the problem here.

[Docs] Add help for DLL load failed error

Looks like as of tensorflow 2.1, on Windows you need "Microsoft Visual C++ Redistributable for Visual Studio 2015, 2017 and 2019" (see here). This should be added to the docs here and probably also here.

[Feature] GUI for training data: toggle flip all selected cells into same category in one step

Is your feature request related to a problem? Please describe.
When selecting many cells/non-cells in same area (maybe dense spacing), a multi-select does not mean application of the same category assignment, but will result in each individual selection being toggled into the inverse. It is not possible to select a group that should be classified as cells or non-cells and do so in one step. - It takes very long to toggle one category individually (item by item) and then confirm the area instead of all at once.

Describe the solution you'd like
It would be convenient to select a group and on toggle move them all into one label category, so that at most two flips are required to move all of them into the desired label.

Describe alternatives you've considered
Allow toggle only category-specific, e.g. when "cells" is selected, "toggle" will only act on these.

Additional context
.

[DOCS] cellfinder_curate missing the pixel size/metadata requirement

[BUG] pandas version incompatible with allensdk

Error when running pip install cellfinder:

ERROR: allensdk 1.5.1 has requirement pandas<=0.25.3,>=0.25.1, but you'll have pandas 1.0.1 which is incompatible.

Test with tensorflow 2.2.0

This apparently fixes some of the memory issues (tensorflow/tensorflow#35030 (comment)).

Test inference
Test training

[Feature] Simple (non training) curation interface

cellfinder_curate should have an option to not just save the toggled/confirmed cells, but edit (or make a new) cell_classification.xml. In case the classification is broadly correct, but users want to manually edit a couple of cells.

[Feature] cellfinder training save iterations by default and reduce epoch number

Is your feature request related to a problem? Please describe.

I sometimes accidentally run cellfinder without saving intermediate stages and it takes a long time without saving outputs, means I have to cancel and restart.

Describe the solution you'd like

I think it would be useful to change the default behaviour of cellfinder train to a lower number of epochs and/or to save intermediate models by default

[BUG] Cellfinder 2D viewer not working on windows

Describe the bug
The 2D viewer to inspect the results of cellfinder doesn't start on windows [gets stuck while loading the data].

[Docs] Link to neuro & amap docs

Make web docs coherent by linking to the neuro & amap docs.

[Feature] Allow training weights to be saved every N epochs

Currently, they can only be saved every epoch, which is often unnecessary.

[Docs] Add a more practical contribution guide.

Similar to neuro , a practical guide (e.g. to fork, submit pull request etc) would be useful.

[BUG] More than 1024 cells in the cell viewer can cause crashes.

This is a napari bug, fixed, but not yet released (napari/napari#932).

If a new napari release is a way off, maybe install napari from github? (or just add to troubleshooting).

Add PR template

There is some guidance but potential contributors aren't psychic. It would be nice if there was a PR template to ensure that everything required was complete (formatting with black etc.).

Document all command line options

Currently not all command line options are in the documentation. This should be improved.

[Feature] Deploy tagged commits with Travis

Deploy via Travis (will need manylinux deployment with docker for cython extensions).

[BUG] Running Cellfinder with two signal channels

Describe the bug
I can't find a way to run cell finder with two signal channels.

To Reproduce
This is the cellfinder.sh script I've used to run cellfinder on our cluster:

#! /bin/bash

#SBATCH -p gpu # partition (queue)
#SBATCH -N 1   # number of nodes
#SBATCH --mem 60G # memory pool for all cores
#SBATCH --gres=gpu:1
#SBATCH -n 10
#SBATCH -t 2-0:0 # time
#SBATCH -o cellfinder.out
#SBATCH -e cellfinder.err
#SBATCH --mail-type=ALL
#SBATCH [email protected]

starter_cells_file='/nfs/winstor/branco/Federico/Anatomy/AY_246_4/stitchedImages_100/3'
cells_file ='/nfs/winstor/branco/Federico/Anatomy/AY_246_4/stitchedImages_100/2'
background_file='/nfs/winstor/branco/Federico/Anatomy/AY_246_4/stitchedImages_100/4'
output_dir='/nfs/winstor/branco/Federico/Anatomy/AY_246_4/cellfinder_2ch'

echo "Loading conda env"
module load miniconda
conda activate cellfinder

echo "Running cellfinder"
cellfinder -s $starter_cells_file $cells_file -b $background_file -o $output_dir --summarise -x 2.231 -y 2.231 -z 5.0

So in theory it should use two sets of images as signal planes.
However it only analysed one of the two and the reason why is evident from an inspection of the log file.



Command: /nfs/nhome/live/federicoc/.conda/envs/cellfinder/bin/cellfinder 
Input arguments: ['-s', '/nfs/winstor/branco/Federico/Anatomy/AY_246_4/stitchedImages_100/3', '-b', '/nfs/winstor/branco/Federico/Anatomy/AY_246_4/stitchedImages_100/4', '-o', '/nfs/winstor/branco/Federico/Anatomy/AY_246_4/cellfinder_2ch', '--summarise', '-x', '2.231', '-y', '2.231', '-z', '5.0']

Namespace:
signal_planes_paths: ['/nfs/winstor/branco/Federico/Anatomy/AY_246_4/stitchedImages_100/3']
background_planes_path: ['/nfs/winstor/branco/Federico/Anatomy/AY_246_4/stitchedImages_100/4']
output_dir: /nfs/winstor/branco/Federico/Anatomy/AY_246_4/cellfinder_2ch

Expected behavior
It should be possible to pass multiple paths to be used as signal planes.

Log file
(https://github.com/SainsburyWellcomeCentre/cellfinder/files/4081794/cellfinder_2020-01-18_11-48-11.log)

Desktop (please complete the following information):

OS: HPC Linus server

Additional context
I've tried passing $starter_cells_file $cells_file as $starter_cells_file$cells_file and $starter_cells_file,$cells_file but these resulted in syntax errors.
[cellfinder_2020-01-18_11-48-11.log]

Provide sample data

Considering how different whole-brain data can be (serial2p, fmost, clarity, idisco etc.) it would be good to provide some sample data that works well with cellfinder and amap.

cellfinder_sample could download data from https://gin.g-node.org/cellfinder/data and extract to a specified location.

[Feature] Cells data extraction

Is your feature request related to a problem? Please describe.
I know that there are a few ways to export data about the detected cells (e.g. cellfinder_cells_to_brainrender), but it would be good to be able to export the following as a table, for each cell:

x,y,z and radius in raw data coordinates
x,y,z and radius in standard space
brain region according to the Allen atlas. Perhaps even the whole hierarchy of brain structures the region is in (e.g. Gray\isocortex\MOs\MOs5b)
hemisphere

This would make it a lot easier to import this data into other analysis and visualisation pipelines.
In fact I suggest this should be done automatically once the analysis is completed.

Does it work on individual brain slices?

Needs testing on Windows

Only tested on Linux. #11 might causes issues with installation from pypi.

If it works, upload wheels to pypi.

[Feature] Need an intelligent way of picking batch size.

The default batch size for classification (32) was chosen so it would run on pretty much any good GPU but as @larsrollik has found, this can be increased a lot, reducing interference time (128 works with the default resnet on a RTX 2080Ti).

Maybe use tf to get available GPU memory and pick batch size based on that? Allow users to specify max GPU memory to use?

[Feature] Train from xml file and raw data

Need to be able to train from an xml file of cell candidate positions, and the raw data.

Compare loading on the fly vs extracting, and then loading (and augmenting) for speed, RAM usage and disk space needed.

[BUG] Cell detection fails on Windows

Cell detection on windows fails with:

C:\Users\adam.GARFIELD-WIN\.conda\envs\cellfinder\python.exe C:/Users/adam.GARFIELD-WIN/projects/cellfinder/cellfinder/main.py -s C:\Users\adam.GARFIELD-WIN\data\cellfinder\CT_BY_70_2_stitched\stitched\ch00 -b C:\Users\adam.GARFIELD-WIN\data\cellfinder\CT_BY_70_2_stitched\stitched\ch01 -o C:\Users\adam.GARFIELD-WIN\data\output\cellfindertest2 -x 2 -y 2 -z 5 --start-plane 1288 --end-plane 1305
2020-02-11 13:25:14 PM - INFO - MainProcess fancylog.py:265 - Starting logging
2020-02-11 13:25:14 PM - INFO - MainProcess fancylog.py:267 - Multiprocessing-logging module found. Logging from all processes
2020-02-11 13:25:14 PM - INFO - MainProcess main.py:64 - Skipping registration
2020-02-11 13:25:14 PM - INFO - MainProcess main.py:110 - Detecting cell candidates
Processing planes:   0%|          | 0/17 [00:00<?, ?it/s]Traceback (most recent call last):
  File "C:/Users/adam.GARFIELD-WIN/projects/cellfinder/cellfinder/main.py", line 158, in <module>
    main()
  File "C:/Users/adam.GARFIELD-WIN/projects/cellfinder/cellfinder/main.py", line 91, in main
    run_all(args, what_to_run)
  File "C:/Users/adam.GARFIELD-WIN/projects/cellfinder/cellfinder/main.py", line 112, in run_all
    detect.main(args)
  File "C:\Users\adam.GARFIELD-WIN\projects\cellfinder\cellfinder\detect\detect.py", line 114, in main
    bf_process.start()  # needs to be started before the loop
  File "C:\Users\adam.GARFIELD-WIN\.conda\envs\cellfinder\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "C:\Users\adam.GARFIELD-WIN\.conda\envs\cellfinder\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\adam.GARFIELD-WIN\.conda\envs\cellfinder\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\adam.GARFIELD-WIN\.conda\envs\cellfinder\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\adam.GARFIELD-WIN\.conda\envs\cellfinder\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "stringsource", line 2, in View.MemoryView._memoryviewslice.__reduce_cython__
TypeError: no default __reduce__ due to non-trivial __cinit__
Processing planes:   0%|          | 0/17 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\adam.GARFIELD-WIN\.conda\envs\cellfinder\lib\multiprocessing\spawn.py", line 99, in spawn_main
    new_handle = reduction.steal_handle(parent_pid, pipe_handle)
  File "C:\Users\adam.GARFIELD-WIN\.conda\envs\cellfinder\lib\multiprocessing\reduction.py", line 87, in steal_handle
    _winapi.DUPLICATE_SAME_ACCESS | _winapi.DUPLICATE_CLOSE_SOURCE)
PermissionError: [WinError 5] Access is denied

[Feature] Add formatting tests to CI with black.

Should be able to add black ./ -l 79 --check to travis. The build will fail if the codebase isn't formatted correctly.

Not sure if we want travis to actually apply the formatting though.

Needs registration testing

Registration is taken care of with amap, but we still need to test that it works within cellfinder.

[DOCUMENTATION] cellfinder_view_cells documentation is incorrect

Order of the args is incorrect.

Update curation to deal with selected_points being a set (rather than list).

See napari/napari#1126

Not sure if this will cause any issues, but need to check the behaviour of cellfinder.train.curate.main with the napari master branch.

[Feature] GUI/tool for assessing the quality of cellfinder steps and choosing the right model

Is your feature request related to a problem? Please describe.

I am trying to establish a workflow for cellfinder and I find it can be difficult to have an overview of the quality of cell classification and even though cellfinder is fast it can still take a long time to assess how well a given set of parameters performs.

Currently I run cellfinder using the defaults, then I curate the outcome and retrain and rerun and then find out that my new network was not sufficient). This can take a long time.

Describe the solution you'd like
I think it would be useful to have a GUI that allows for comparison of outcomes on the fly on reduced datasets giving the user control of pre-preprocessing steps and allowing on the fly classification using different trained models for classification.

[Feature] Save training weights regularly to interrupt without data loss

Is your feature request related to a problem? Please describe.
No. Just convenience.

Describe the solution you'd like
Save intermediate network weights after each epoch cycle in training, so that training can be manually stopped once loss plateaus.

Describe alternatives you've considered
None.

Additional context
.

Installation failing on MacOS [BUG]

Describe the bug
Trying to install the software for testing it on a macOS catalina, conda 4.7.
As per installation guide, I made an environment for it and then run pip install cellfinder.
I get the error log below:

ERROR: Command errored out with exit status 1:
     command: /Users/luigipetrucco/anaconda3/envs/cellfinder_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/2_/dvscqkzj7rj7chs9sk8ysjfm0000gn/T/pip-install-q09rxn54/cellfinder/setup.py'"'"'; __file__='"'"'/private/var/folders/2_/dvscqkzj7rj7chs9sk8ysjfm0000gn/T/pip-install-q09rxn54/cellfinder/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/2_/dvscqkzj7rj7chs9sk8ysjfm0000gn/T/pip-record-5lo4tmgu/install-record.txt --single-version-externally-managed --compile --install-headers /Users/luigipetrucco/anaconda3/envs/cellfinder_env/include/python3.7m/cellfinder
         cwd: /private/var/folders/2_/dvscqkzj7rj7chs9sk8ysjfm0000gn/T/pip-install-q09rxn54/cellfinder/
    Complete output (130 lines):
    running install
    running build
    running build_py
    package init file 'pip-egg-info/__init__.py' not found (or not a regular file)
    creating build
    creating build/lib.macosx-10.9-x86_64-3.7
    creating build/lib.macosx-10.9-x86_64-3.7/cellfinder
    copying cellfinder/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder
    copying cellfinder/main.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder
    package init file 'tests/tests/__init__.py' not found (or not a regular file)
    package init file 'tests/tests/test_integration/__init__.py' not found (or not a regular file)
    creating build/lib.macosx-10.9-x86_64-3.7/tests
    creating build/lib.macosx-10.9-x86_64-3.7/tests/tests
    creating build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_integration
    copying tests/tests/test_integration/test_train.py -> build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_integration
    copying tests/tests/test_integration/test_detection.py -> build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_integration
    copying tests/tests/test_integration/test_extract.py -> build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_integration
    package init file 'tests/tests/test_unit/__init__.py' not found (or not a regular file)
    package init file 'tests/tests/test_integration/test_utils/__init__.py' not found (or not a regular file)
    creating build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_integration/test_utils
    copying tests/tests/test_integration/test_utils/test_xml_scale.py -> build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_integration/test_utils
    package init file 'tests/tests/test_unit/test_tools/__init__.py' not found (or not a regular file)
    creating build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_unit
    creating build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_unit/test_tools
    copying tests/tests/test_unit/test_tools/test_image_processing.py -> build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_unit/test_tools
    copying tests/tests/test_unit/test_tools/test_tools_general.py -> build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_unit/test_tools
    copying tests/tests/test_unit/test_tools/test_system.py -> build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_unit/test_tools
    copying tests/tests/test_unit/test_tools/test_prep.py -> build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_unit/test_tools
    copying tests/tests/test_unit/test_tools/test_geometry.py -> build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_unit/test_tools
    package init file 'tests/tests/test_unit/test_detect/__init__.py' not found (or not a regular file)
    package init file 'tests/tests/test_unit/test_analyse/__init__.py' not found (or not a regular file)
    package init file 'tests/tests/test_unit/test_detect/test_filters/__init__.py' not found (or not a regular file)
    package init file 'tests/tests/test_unit/test_detect/test_filters/test_volume_filters/__init__.py' not found (or not a regular file)
    creating build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_unit/test_detect
    creating build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_unit/test_detect/test_filters
    creating build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_unit/test_detect/test_filters/test_volume_filters
    copying tests/tests/test_unit/test_detect/test_filters/test_volume_filters/test_structure_detection.py -> build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_unit/test_detect/test_filters/test_volume_filters
    package init file 'tests/tests/test_unit/test_analyse/test_group/__init__.py' not found (or not a regular file)
    creating build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_unit/test_analyse
    creating build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_unit/test_analyse/test_group
    copying tests/tests/test_unit/test_analyse/test_group/test_region_summary.py -> build/lib.macosx-10.9-x86_64-3.7/tests/tests/test_unit/test_analyse/test_group
    creating build/lib.macosx-10.9-x86_64-3.7/cellfinder/detect
    copying cellfinder/detect/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/detect
    copying cellfinder/detect/detect.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/detect
    creating build/lib.macosx-10.9-x86_64-3.7/cellfinder/tools
    copying cellfinder/tools/system.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/tools
    copying cellfinder/tools/tf.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/tools
    copying cellfinder/tools/tiff.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/tools
    copying cellfinder/tools/array_operations.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/tools
    copying cellfinder/tools/tools.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/tools
    copying cellfinder/tools/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/tools
    copying cellfinder/tools/parser.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/tools
    copying cellfinder/tools/prep.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/tools
    copying cellfinder/tools/geometry.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/tools
    copying cellfinder/tools/image_processing.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/tools
    creating build/lib.macosx-10.9-x86_64-3.7/cellfinder/classify
    copying cellfinder/classify/classify.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/classify
    copying cellfinder/classify/tools.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/classify
    copying cellfinder/classify/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/classify
    copying cellfinder/classify/augment.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/classify
    copying cellfinder/classify/resnet.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/classify
    copying cellfinder/classify/cube_generator.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/classify
    creating build/lib.macosx-10.9-x86_64-3.7/cellfinder/config
    copying cellfinder/config/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/config
    creating build/lib.macosx-10.9-x86_64-3.7/cellfinder/standard_space
    copying cellfinder/standard_space/cells_to_standard_space.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/standard_space
    copying cellfinder/standard_space/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/standard_space
    creating build/lib.macosx-10.9-x86_64-3.7/cellfinder/utils
    copying cellfinder/utils/xml_crop.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/utils
    copying cellfinder/utils/roi_transform.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/utils
    copying cellfinder/utils/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/utils
    copying cellfinder/utils/generate_region_volume.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/utils
    copying cellfinder/utils/xml_scale.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/utils
    creating build/lib.macosx-10.9-x86_64-3.7/cellfinder/summarise
    copying cellfinder/summarise/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/summarise
    copying cellfinder/summarise/count_summary.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/summarise
    creating build/lib.macosx-10.9-x86_64-3.7/cellfinder/viewer
    copying cellfinder/viewer/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/viewer
    copying cellfinder/viewer/three_dimensional.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/viewer
    copying cellfinder/viewer/two_dimensional.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/viewer
    creating build/lib.macosx-10.9-x86_64-3.7/cellfinder/download
    copying cellfinder/download/models.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/download
    copying cellfinder/download/download.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/download
    copying cellfinder/download/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/download
    copying cellfinder/download/cli.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/download
    creating build/lib.macosx-10.9-x86_64-3.7/cellfinder/analyse
    copying cellfinder/analyse/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/analyse
    creating build/lib.macosx-10.9-x86_64-3.7/cellfinder/figures
    copying cellfinder/figures/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/figures
    copying cellfinder/figures/cli.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/figures
    copying cellfinder/figures/figures.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/figures
    creating build/lib.macosx-10.9-x86_64-3.7/cellfinder/train
    copying cellfinder/train/curation.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/train
    copying cellfinder/train/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/train
    copying cellfinder/train/train_yml.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/train
    creating build/lib.macosx-10.9-x86_64-3.7/cellfinder/extract
    copying cellfinder/extract/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/extract
    copying cellfinder/extract/extract_cubes.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/extract
    creating build/lib.macosx-10.9-x86_64-3.7/cellfinder/detect/filters
    copying cellfinder/detect/filters/setup_filters.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/detect/filters
    copying cellfinder/detect/filters/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/detect/filters
    creating build/lib.macosx-10.9-x86_64-3.7/cellfinder/detect/filters/volume_filters
    copying cellfinder/detect/filters/volume_filters/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/detect/filters/volume_filters
    copying cellfinder/detect/filters/volume_filters/structure_splitting.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/detect/filters/volume_filters
    copying cellfinder/detect/filters/volume_filters/multiprocessing.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/detect/filters/volume_filters
    creating build/lib.macosx-10.9-x86_64-3.7/cellfinder/detect/filters/plane_filters
    copying cellfinder/detect/filters/plane_filters/classical_filter.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/detect/filters/plane_filters
    copying cellfinder/detect/filters/plane_filters/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/detect/filters/plane_filters
    copying cellfinder/detect/filters/plane_filters/multiprocessing.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/detect/filters/plane_filters
    copying cellfinder/detect/filters/plane_filters/tile_walker.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/detect/filters/plane_filters
    creating build/lib.macosx-10.9-x86_64-3.7/cellfinder/analyse/group
    copying cellfinder/analyse/group/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/analyse/group
    copying cellfinder/analyse/group/region_summary.py -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/analyse/group
    running egg_info
    writing cellfinder.egg-info/PKG-INFO
    writing dependency_links to cellfinder.egg-info/dependency_links.txt
    writing entry points to cellfinder.egg-info/entry_points.txt
    writing requirements to cellfinder.egg-info/requires.txt
    writing top-level names to cellfinder.egg-info/top_level.txt
    reading manifest file 'cellfinder.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'cellfinder.egg-info/SOURCES.txt'
    copying cellfinder/config/cellfinder.conf -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/config
    copying cellfinder/detect/filters/typedefs.pxd -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/detect/filters
    copying cellfinder/detect/filters/volume_filters/ball_filter.c -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/detect/filters/volume_filters
    copying cellfinder/detect/filters/volume_filters/structure_detection.cpp -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/detect/filters/volume_filters
    copying cellfinder/detect/filters/plane_filters/base_tile_filter.cpp -> build/lib.macosx-10.9-x86_64-3.7/cellfinder/detect/filters/plane_filters
    running build_ext
    cythoning cellfinder/detect/filters/volume_filters/ball_filter.pyx to cellfinder/detect/filters/volume_filters/ball_filter.c
    error: /private/var/folders/2_/dvscqkzj7rj7chs9sk8ysjfm0000gn/T/pip-install-q09rxn54/cellfinder/cellfinder/detect/filters/volume_filters/ball_filter.pyx


ERROR: Command errored out with exit status 1: /Users/luigipetrucco/anaconda3/envs/cellfinder_env/bin/python -u -c 'import sys, setuptools, tokenize;
 sys.argv[0] = '"'"'/private/var/folders/2_/dvscqkzj7rj7chs9sk8ysjfm0000gn/T/pip-install-q09rxn54/cellfinder/setup.py'"'"'; 
__file__='"'"'/private/var/folders/2_/dvscqkzj7rj7chs9sk8ysjfm0000gn/T/pip-install-q09rxn54/cellfinder/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);
code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/2_/dvscqkzj7rj7chs9sk8ysjfm0000gn/T/pip-record-5lo4tmgu/install-record.txt --single-version-externally-managed --compile --install-headers /Users/luigipetrucco/anaconda3/envs/cellfinder_env/include/python3.7m/cellfinder

If required I can give a more complete error log

Test with napari 0.30

See: https://napari.org/docs/release/release_0_3_0.html

Looks like there might be some breaking changes, and we should take advantage of the new plugin features.

[Feature] Smarter contrast limits for viewer

Linked to #26

Is there a smarter way to calculate contrast limits without calculating the histogram for lots of data (defeating the point of doing it in advance)?

Could also have the contrast limits as a CLI parameter

cc @FedeClaudi

[BUG] Cellfinder cell viewer - point size

Describe the bug
Changing the point size in Cellfinder cell viewer only affects points that are subsequently added.

Expected behavior
I would have expected the point size to change for all points that already exist in a given category (i.e. Cells / Non-Cells), and for the ones that are subsequently added. This would make inspection of cells/candidates easier as they are - depending on cell size - mostly covered by the symbol indicating the point location.

Desktop (please complete the following information):

OS: Windows 7

[Feature] Curation interface additions

Two ideas for the cell curation interface, cc @chrytsi

Option to only view cells/non cells
In addition to the toggle/confirm interface, add modes such as "clicking marks as cells", "clicking marks as correct", "clicking toggles" etc.

[Feature] Integrated cellfinder/amap/neuro viewer

Once a standardised output directory structure is decided upon, it would be useful to have a viewer that loads any of the following that have been created:

Raw data (virtually)
Downsampled data
Registration outlines
Registration annotations
Detected cells & artefacts
Heatmap
Any structures segmented in neuro

(Only make one visible by default to avoid using crazy amounts of RAM).

[BUG] pip instalation from sdist causes cython errors

Installation using prebuilt wheels works fine, but the sdist fails (at least on linux).

[DOCS] Need a docs github issue template

[Docs] Pixel sizes in the example are incorrect.

In https://sainsburywellcomecentre.github.io/cellfinder/main/user_guide/run.html

Pixel sizes should be in um, not mm.

[BUG] Import statement in xml_crop not updated.

Describe the bug
Import broken after moving function from submodule to imlib package.

To Reproduce
Run cellfinder_xml_crop with standard parameters.
Current behaviour:

Traceback (most recent call last):
  File "/opt/anaconda/envs/cellfinder/bin/cellfinder_xml_crop", line 5, in <module>
    from cellfinder.utils.xml_crop import main
  File "/opt/anaconda/envs/cellfinder/lib/python3.7/site-packages/cellfinder/utils/xml_crop.py", line 12, in <module>
    from cellfinder.summarise.structures.structures_tree import (
ModuleNotFoundError: No module named 'cellfinder.summarise.structures'

Expected behavior
Should be:

from imlib.IO.structures import load_structures_as_df

[Feature] Change default registration behaviour

Currently, registration, region summaries and figures are not run by default. I think it makes sense to have everything as an opt out (to make usage simpler for new users).

[Feature] Warning messages for possibly incorrect pixel sizes

Supplying incorrect pixel sizes can not just cause problems for registration and cell detection, but in the case of supplying incorrectly higher pixel sizes can use so much RAM for registration that crashes can occur. E.g. supplying -x 2 -y 2 -z 5 rather than -x 1 -y 2 -z 5 could (incorrectly) require ~120GB RAM for registration.

This should be avoided by:

Highlighting the importance of correct pixel sizes to be passed.
Encouraging the use of --metadata over specifying the pixel sizes (and adding as many formats as possible to micrometa.
Add warning messages, e.g.:

if atlas_type == "mouse" and physical_size_image > 2e12: # um^3
    raise SyntaxWarning(f"Based on pixel sizes, the physical size of your 
           image is {round(physical_image_size / 1e9)} mm^3. This is very large
            for a sample of type: {atlas_type}. Please check the pixel sizes supplied 
            via -x, -y and -z or via --metadata.If your sample is simply very large, 
            please rerun with '--force-image-size' ")

[Feature] Freeze upper layers of the network for retraining.

Linked to #354. Allowing upper layers to be frozen when retraining should make slight tweaks easier

[Docs] Cell candidate detection parameters for non-standard voxel size

This is rather a question on how to set up cell candidate detection if my data has a voxel size very different from the default values.

The default voxel size, as I understand, is 2 um in xy and 5 um in z. My data has 0.875 in xy and 25 um in z. As a result, a lot of big cells are not detected at the stage of cell candidate detection. I have tried to play around with the size of the ball filter, but it has not changed a lot.

@adamltyson do you have any advice on how to choose the size of the ball filter depending on the size of the cells that I want to detect? (I have not yet figured out what exactly it does). Are there other parameters that could influence cell detection? Or would simply downsampling the images be the best solution?

Thanks in advance for any help with this.

[BUG] IndexError: list index out of range while Loading raw image data

Describe the bug
Cellfinder yields an IndexError while loading raw images from a list of file paths. [While running on the HPC cluster and fetching data from winstor]

From the Log file:

**************  COMMAND LINE ARGUMENTS  **************

Command: /nfs/nhome/live/federicoc/.conda/envs/cellfinder/bin/cellfinder
Input arguments: ['-s', '/nfs/winstor/branco/BrainSaw/AY_246_4/stitchedImages_100/3', '/nfs/winstor/branco/BrainSaw/AY_246_4/stitchedImages_100/2', '-b', '/nfs/winstor/branco/BrainSaw/AY_246_4/stitchedImages$

...


**************  LOGGING  **************

2020-01-29 18:17:50 PM - INFO - MainProcess fancylog.py:257 - Starting logging
2020-01-29 18:17:50 PM - INFO - MainProcess fancylog.py:259 - Multiprocessing-logging module found. Logging from all processes
2020-01-29 18:17:50 PM - INFO - MainProcess main.py:32 - Registering to atlas
2020-01-29 18:17:50 PM - INFO - MainProcess prep.py:398 - Checking whether the atlas exists
2020-01-29 18:17:50 PM - DEBUG - MainProcess prep.py:417 - Making registration directory
2020-01-29 18:17:50 PM - DEBUG - MainProcess system.py:111 - Determining the maximum number of CPU cores to use
2020-01-29 18:17:50 PM - DEBUG - MainProcess system.py:120 - Number of CPU cores available is: 8
2020-01-29 18:17:50 PM - DEBUG - MainProcess system.py:145 - Setting number of processes to: 8
2020-01-29 18:17:50 PM - INFO - MainProcess main.py:85 - Preprocessing data for registration
2020-01-29 18:17:50 PM - INFO - MainProcess main.py:86 - Loading data
2020-01-29 18:17:50 PM - INFO - MainProcess batteryrunners.py:274 - pixdim[0] (qfac) should be 1 (default) or -1; setting qfac to 1
2020-01-29 18:17:50 PM - INFO - MainProcess brain_processor.py:79 - Loading raw image data
2020-01-29 18:17:50 PM - DEBUG - MainProcess brainio.py:72 - Data type is: directory of files
2020-01-29 18:17:53 PM - DEBUG - MainProcess utils.py:104 - Determining the maximum number of CPU cores to use
2020-01-29 18:17:53 PM - DEBUG - MainProcess utils.py:116 - Setting number of processes to: 8

Complete error message:

"""
Traceback (most recent call last):
  File "/nfs/nhome/live/federicoc/.conda/envs/cellfinder/lib/python3.7/concurrent/futures/process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/nfs/nhome/live/federicoc/.conda/envs/cellfinder/lib/python3.7/site-packages/brainio/brainio.py", line 304, in load_from_paths_sequence
    img = tifffile.imread(p)
  File "/nfs/nhome/live/federicoc/.conda/envs/cellfinder/lib/python3.7/site-packages/tifffile/tifffile.py", line 716, in imread
    return tif.asarray(**kwargs)
  File "/nfs/nhome/live/federicoc/.conda/envs/cellfinder/lib/python3.7/site-packages/tifffile/tifffile.py", line 2373, in asarray
    series = self.series[series]
IndexError: list index out of range
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nfs/nhome/live/federicoc/.conda/envs/cellfinder/bin/cellfinder", line 8, in <module>
    sys.exit(main())
  File "/nfs/nhome/live/federicoc/.conda/envs/cellfinder/lib/python3.7/site-packages/cellfinder/main.py", line 61, in main
    debug=args.debug,
  File "/nfs/nhome/live/federicoc/.conda/envs/cellfinder/lib/python3.7/site-packages/amap/main.py", line 98, in main
    n_free_cpus=n_free_cpus,
  File "/nfs/nhome/live/federicoc/.conda/envs/cellfinder/lib/python3.7/site-packages/amap/register/brain_processor.py", line 87, in __init__
    n_free_cpus=n_free_cpus,
  File "/nfs/nhome/live/federicoc/.conda/envs/cellfinder/lib/python3.7/site-packages/brainio/brainio.py", line 79, in load_any
    n_free_cpus=n_free_cpus,
  File "/nfs/nhome/live/federicoc/.conda/envs/cellfinder/lib/python3.7/site-packages/brainio/brainio.py", line 190, in load_from_folder
    paths, x_scaling_factor, y_scaling_factor, n_free_cpus=n_free_cpus
  File "/nfs/nhome/live/federicoc/.conda/envs/cellfinder/lib/python3.7/site-packages/brainio/brainio.py", line 279, in threaded_load_from_sequence
    stack = np.dstack([s.result() for s in stacks])
  File "/nfs/nhome/live/federicoc/.conda/envs/cellfinder/lib/python3.7/site-packages/brainio/brainio.py", line 279, in <listcomp>
    stack = np.dstack([s.result() for s in stacks])
  File "/nfs/nhome/live/federicoc/.conda/envs/cellfinder/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/nfs/nhome/live/federicoc/.conda/envs/cellfinder/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
IndexError: list index out of range

[BUG] Curation interface fails to save cubes as "cells" dir doesn't exist

From @FedeClaudi.

This may be windows specific. Why isn't the directory being created?

[Feature] Reduce overfitting when training

Users are generating small amounts of data, and the network seems to be over fitting. They're having to manually get existing data to constrain their models. This could be improved by:

Providing old training data via GIN
Allowing upper layers to be frozen

Interested to know what aspects of training have helped @FedeClaudi @larsrollik

[Feature] 2D cell detection

There are a number of users who have 3D data, but require 2D cell detection (i.e. low Z sampling).

Theoretically this should be possible to add within the cellfinder framework.

cc @amedyukhina

[BUG] Incorrect import in cellfinder_roi_transform

In cellfinder.utils.roi_tranform, get_niftyreg_binaries() must be imported from amap.tools.source_files not cellfinder.tools.source_files.

[Docs] Provide instructions for different OSs

All the instructions [here](https://sainsburywellcomecentre.github.io/cellfinder/main/user_guide/install.html] assume a Linux environment. Although the instructions are very similar, it would be useful to explicitly state them all for each OS.

brainglobe / cellfinder Goto Github PK

cellfinder's Introduction

cellfinder

Installation

Contributing

cellfinder's People

Contributors

Stargazers

Watchers

Forkers

cellfinder's Issues

Recommend Projects

Recommend Topics

Recommend Org