This is the repository for the Images to Graphs Pipeline, the first automated pipeline to estimate graphs from high-resolution serial section electron microscopy tissue.

Leveraging the OCP services, it provides a set of interfaces and tools to estimate and assess automatically generated graphs.

Membrane Detection:

• Theano CNN implementation (based on Ciresan 2012)
• Cafe CNN implementation (based on Ciresan 2012)

Vesicle Detection:

• Template Matching

Neuron Segmentation:

• Gala
• Rhoana
• Watershed

Synapse Detection:

• Context Synapse Detection
• Carlos Becker's method
• Ilastik classifier (not yet implemented)

Ilastik and the deep learning frameworks are general tools that may be adapted for other purposes.

For each package, we provide:

• A LONI module
• Relevant functions and scripts
• An example script, with validation (e.g., unit tests) [Still in progress]
• A separate method for training, when applicable
• Required dependencies and installation instructions
• Package install validation (where available)

These packages are designed to be used in a LONI pipelining framework, with data derivatives stored using the RAMON data standard. We provide an example integrated with a standalone LONI client for simplicity, although large scale runs are managed by a LONI server.

Workflows are also provided for two cases: Images to Graphs (aligned image volume through graph estimation), and a parameter tuning workflow, which focuses only on optimizing segmentation and synapse detection methods as an example (following the manuscript).

Workflows should run "out of the box," although assume that the code is installed in /mnt/pipeline/tools/CAJAL3D. Modules will need to be updated (or batch edited) to reflect the appropriate root directory.

• Centos 6.x (6.6 recommended)
• MATLAB
• Anaconda (Python 2.7) - this takes care of most of the python dependencies
• LONI (client or server)
• OCP MATLAB API
• I2G Repo

• Gala
• Rhoana -- CPLEX
• Becker CCBoost
• Ilastik
• Caffe
• Theano
• VLFeat
• 3D SLIC code

Outstanding TODOs:

• Ilastik issue with glib 2.14
• How to point to python2.7
• Deep learning not integrated
• Matlab compiler setup

Configure a basic Centos installation, along with a non-root user account (sudo privileges)

Configure and install MATLAB (this is a required dependency - your install process will vary)

Get OCP Matlab

  wget https://github.com/openconnectome/ocpMatlab/archive/master.zip

Unzip to: /mnt/pipeline/tools/CAJAL3D

Update all packages:

  sudo yum -y update  

Download and Install LONI (Client) into home directory

  http://www.loni.usc.edu/Software/Pipeline

TODO: Add code to unzip and move

Download CC Boost

  cd /mnt/pipeline/tools/CAJAL3D/external/
 
  wget https://documents.epfl.ch/groups/c/cv/cvlab-unit/www/data/synapses/ccboost-precompiled-v0.2.tar bz2

  tar -xvf ccboost-precompiled-v0.2.tar

Download VL Feat

  wget http://www.vlfeat.org/download/vlfeat-0.9.20-bin.tar.gz

  tar -xvf vlfeat-0.9.20-bin.tar.gz

  rm vlfeat-0.9.20-bin.tar.gz

TODO: Need to run('VLFEATROOT/toolbox/vl_setup')

Download Ilastik

  wget http://files.ilastik.org/ilastik-1.1.5-Linux.tar.gz

  tar -xvf ilastik-1.1.5-Linux.tar.gz
  
  rm ilastik-1.1.5-Linux.tar.gz

Install Anaconda

  wget http://09c8d0b2229f813c1b93-c95ac804525aac4b6dba79b00b39d1d3.r79.cf1.rackcdn.com/Anaconda-2.1.0-Linux-x86_64.sh

  bash Anaconda-2.1.0-Linux-x86_64.sh

TODO: at prompt accept license TODO: at prompt install in: /usr/bin/local/anaconda

export path in ~/.bashrc file

  export PATH="/usr/local/bin/anaconda/bin:$PATH"

Gala

When using anaconda, just need the gala package itself

  wget https://github.com/janelia-flyem/gala/archive/master.zip

Install manually to link to anaconda

  unzip master.zip
  
  mv gala-master /mnt/pipeline/tools/CAJAL3D/external/gala
  
  cd /mnt/pipeline/tools/CAJAL3D/external/gala
  
  python /usr/local/bin/anaconda/bin/python/setup.py install 

Download Viridis

  wget https://github.com/jni/viridis/archive/master.zip

  unzip master.zip
  
  cd viridis-master
  
  /usr/local/bin/anaconda/bin/python setup.py install

Rhoana

Original Rhoana package (not needed, because we wrapped it) Also need mahotas, pymaxflow, fast64counter, cplex

Mahotas

  wget https://pypi.python.org/packages/source/m/mahotas/mahotas-1.2.4.tar.gz  

  tar -xvf mahotas-1.2.4.tar.gz  

  cd mahotas-1.2.4.tar.gz  

  sudo /usr/local/bin/anaconda/bin/python setup.py install  

Download pymaxflow from github.com/Rhoana/pymaxflow

  wget https://github.com/Rhoana/pymaxflow/archive/master.zip

  unzip master.zip #make sure we cleanup each time

  cd pymaxflow-master

  sudo /usr/local/bin/anaconda/bin/python setup.py install

  rm ~/Downloads/master.zip  #make sure we download to this location

Download fast64counter from github.com/Rhoana/fast64counter

  wget https://github.com/Rhoana/fast64counter/archive/master.zip

  unzip master.zip #make sure we cleanup each time

  cd fast64counter

  sudo /usr/local/bin/anaconda/bin/python setup.py install

  rm ~/Downloads/master.zip

Download cplex bin file

  TODO - for now hosted privately - or get from IBM site
  
  sudo sh cplex_studio126.linux-x86-64.bin
  (interactive prompts - need to configure for non-interactive mode)

Python bindings:

  cd /opt/ibm/ILOG/CPLEX_Studio126/cplex/python/x86-64_linux
  
  sudo python setup.py install

Clone repo

cd /mnt/pipeline/tools/CAJAL3D/

git clone https://github.com/openconnectome/i2g.git