Includes:
- Loading of coordinate data and stage timing data
- Loading and caching of cropped videos
- Contour computation and storage
- Midline computation and storage/caching, including head and tail detection
The analysis pipeline was tested with julia version 1.9.3 on Windows, and also with remote workers on a Linux machine for the contours and midline computation.
To import experiment data, after running the MATLAB analysis scripts, several Julia scripts need to be run
on (copies of) the MATLAB analysis output directories.
These scripts are currently designed to be run inside VSCode, by manually editing input argument assignments
(and possibly some code) at the top of the script, similarly to the MATLAB scripts.
They are also divided into cells (separated by ##
), so they can be run one cell at a time if desired to examine
intermediate results.
The import scripts are at scripts/experiment_import
, and have their own Julia environment which should be activated
first in a fresh julia REPL.
After installing vscode and the vscode julia extension, start a Julia REPL inside VSCode (Ctrl+Shift+p
for the command palette
then choose "Julia: Start REPL", or Alt+j,Alt+o
).
To activate the environment you can run ]activate scripts/experiment_import
at the REPL running in the Elegans
project directory
(hitting ]
in the REPL enters "pkg
mode", where you type the rest of the command). Alternatively, in VSCode you can right-click
the directory and select "Julia: activate this environment" (this option is only available when a Julia REPL has been started).
When doing this for the first time on given machine, the required packages are probably not installed yet, so you need to also run
]instantiate
.
The scripts should generally be run in the following order:
-
denest_wells.jl
:MATLAB coordinate analysis output from each experiment is sometimes divided into subdirs when it was run in several batches. This scripts flatten the directory structure to so each well is at
<experiment>/<well>
.Modify the assigment to
root
at the top of the script. Comment out the line at the bottom to do a real run rather than just listing which files are moved. -
save_to_single_file.jl
Saves coordinates, video file frame-number ranges, and size information to a single file per well, named
coords_and_size.jld2
.Modify the assigment to
root
and the regular expression for experiment names (or the whole condition) in the following line. Below that, you may list wells to skip in thebadwells
variable. -
import_stages.jl
This script operates on the "analyzed" experiment data (the output of the three MATLAB scripts starting with
separate_to_developmental_stages.m
), as well as coordinate and video data. It imports stage data (which frame each stage starts with for each well) from MATLAB output into thestages.toml
file stored inside theElegans
repository.Modify
coord_data_root
to point to coordinate and video data andanalyzed_root
to analyzed data, and theexpname
function logic according to how experiment directories are named in this analysis (this function should extract experiment name from dirname).
After importing a set of experiments, the contours and midlines are computed by scripts at scripts/contours_and_midlines
.
Both scripts are configured by editing scripts/contours_and_midlines/args.toml
Modify the paths listed thereunder [paths]
to point to the experiments, stage data, and desired output paths.
For running the analysis locally, no changes are needed under the [dist]
section (the default dist.n_remote = 0
means analysis runs locally. To run analysis remotely, see additional setup below), but you may want to modify the number of worker processes, according to available memory, by setting the value of n_local
.
After configuring args.toml
, running the script scripts/contours_and_midlines/contour-store.jl
will install required packages and
start the contour computation, parallelized among workers. e.g.,
julia> include("scripts/contours_and_midlines/contour-store.jl")
[...]
Progress available at localhost:8108 (copied to clipboard)
Visit the address listed on the last line to view the progress of workers.
Contours are computed with two different sets of parameters: one for L1, the other for L2-adulthood. Each worker processes one well
with one set of parameters at a time.
Computed contours are stored at the path defined by paths.contours_dir
at scripts/contours_and_midlines/args.toml
. There should
be two contour files per well โ one for each parameter set.
Midlines computation works similarly, by running the script scripts/contours_and_midlines/midline-store.jl
. It shares the same configuration file args.toml
. Computed midpoints are stored at the path defined by paths.midpoints_dir
in args.toml
.
To use remote workers, some setup steps are required on the server and local machine.
The server and local machine should work on the same directories through a network share.
Setup a shared directory on the server accessible on the local machine, where all input file are available. List its path on the remote in args.toml
under dist.remote_root
, and its path on the local machine under paths.local_root
. List the remote's address under dist.remote
. E.g., if /home/user/
on server
is shared and mapped locally to U:
, your args.toml
should read
[paths]
local_root = "U:/"
# ...
[dist]
remote = "server"
remote_root = "/home/user/"
Julia should be available on the remote server at $remote_root/julia-$VERSION/bin/julia
.
Clone or download the repository https://github.com/yha/Elegans/ to the remote. The environment on the remote machine need to be instantiated in advance of running the scripts using ]activate Elegans
followed by ]instantiate
, e.g.:
elegans-pipeline> git clone https://github.com/yha/Elegans.git
Cloning into 'Elegans'...
[...]
elegans-pipeline> ls
Elegans
elegans-pipeline> ~/julia-1.9.3/bin/julia
[~] ~/julia-1.9.3/bin/julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.9.3 (2023-08-24)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
(@v1.9) pkg> activate Elegans/
Activating project at `~/elegans-pipeline/Elegans`
(Elegans) pkg> instantiate
On the local machine, list the location of the repository under dist.remote_project_root
in args.toml
, and set n_remote
to the number of remote workers requires:
[dist]
# ...
n_remote = 4
# ...
remote_project_root = "path/to/Elegans" # resolved relative to `dist.root`
You should also set up password-less ssh access from the local machine to the remote.