scratcharchive / mountainsort_examples Goto Github PK

I am trying to run the bash examples, but when I try to run "synthesize_dataset.sh", I get a few errors (shown below) and the dataset directory lacks the raw.dat.prv file (see the very end for the directory contents). What am I doing wrong?

$ ➜ ./synthesize_dataset.sh
[ Getting processor spec... ]
[ Checking inputs and substituting prvs ... ]
[ Computing process signature ... ]
Process signature: 6505449d72ace76d3541da1f4309c55c8e25bb64
[ Checking outputs... ]
{"waveforms_out":"dataset/waveforms_true.mda.prv","geometry_out":"dataset/geom.csv"}
Processing ouput - dataset/waveforms_true.mda.prv
false
{"waveforms_out":"/tmp/mountainlab-tmp/output_6505449d72ace76d3541da1f4309c55c8e25bb64_waveforms_out.mda","geometry_out":"dataset/geom.csv"}
Processing ouput - dataset/geom.csv
false
{"waveforms_out":"/tmp/mountainlab-tmp/output_6505449d72ace76d3541da1f4309c55c8e25bb64_waveforms_out.mda","geometry_out":"/data/multimem/code/mountainsort_examples/bash_examples/001_ms4_bash_example/dataset/geom.csv"}
[ Checking process cache ... ]
[ Creating temporary directory ... ]
[ Creating links to input files... ]
[ Preparing temporary outputs... ]
Processing ouput - /tmp/mountainlab-tmp/output_6505449d72ace76d3541da1f4309c55c8e25bb64_waveforms_out.mda
false
Processing ouput - /data/multimem/code/mountainsort_examples/bash_examples/001_ms4_bash_example/dataset/geom.csv
false
[ Initializing process ... ]
[ Running ... ] /data/code/python/miniconda3-cdh/envs/mtnsort/bin/python3 /data/code/python/miniconda3-cdh/envs/mtnsort/etc/mountainlab/packages/ml_ephys/synthesis/synthesis.py.mp ephys.synthesize_random_waveforms --_tempdir=/tmp/mountainlab-tmp/tempdir_6505449d72_HT0zui --waveforms_out=/tmp/mountainlab-tmp/tempdir_6505449d72_HT0zui/output_waveforms_out.mda --geometry_out=/tmp/mountainlab-tmp/tempdir_6505449d72_HT0zui/output_geometry_out.csv --upsamplefac=13 --M=4 --average_peak_amplitude=100
Elapsed time for processor ephys.synthesize_random_waveforms: 0.905 sec
Finalizing output waveforms_out
Finalizing output geometry_out
This is only a warning: Unable to rename file /tmp/mountainlab-tmp/tempdir_6505449d72_HT0zui/output_geometry_out.csv -> /data/multimem/code/mountainsort_examples/bash_examples/001_ms4_bash_example/dataset/geom.csv. Perhaps temporary directory is not on the same device as the output file directory. Copying instead.
[ Creating output prv for waveforms_out ... ]
[ Saving to process cache ... ]
(node:20063) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'original_checksum' of undefined
    at /data/code/python/miniconda3-cdh/envs/mtnsort/lib/node_modules/mountainlab/mlproc/prv_utils.js:202:24
    at /data/code/python/miniconda3-cdh/envs/mtnsort/lib/node_modules/mountainlab/mlproc/prv_utils.js:192:7
(node:20063) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:20063) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
[ Getting processor spec... ]
[ Checking inputs and substituting prvs ... ]
[ Computing process signature ... ]
Process signature: 19f60d5d6dfeb2a24e5866112ed0472a994d4733
[ Checking outputs... ]
{"firings_out":"dataset/firings_true.mda.prv"}
Processing ouput - dataset/firings_true.mda.prv
false
{"firings_out":"/tmp/mountainlab-tmp/output_19f60d5d6dfeb2a24e5866112ed0472a994d4733_firings_out.mda"}
[ Checking process cache ... ]
[ Creating temporary directory ... ]
[ Creating links to input files... ]
[ Preparing temporary outputs... ]
Processing ouput - /tmp/mountainlab-tmp/output_19f60d5d6dfeb2a24e5866112ed0472a994d4733_firings_out.mda
false
[ Initializing process ... ]
[ Running ... ] /data/code/python/miniconda3-cdh/envs/mtnsort/bin/python3 /data/code/python/miniconda3-cdh/envs/mtnsort/etc/mountainlab/packages/ml_ephys/synthesis/synthesis.py.mp ephys.synthesize_random_firings --_tempdir=/tmp/mountainlab-tmp/tempdir_19f60d5d6d_qC7yF4 --firings_out=/tmp/mountainlab-tmp/tempdir_19f60d5d6d_qC7yF4/output_firings_out.mda --duration=600
Elapsed time for processor ephys.synthesize_random_firings: 0.836 sec
Finalizing output firings_out
[ Creating output prv for firings_out ... ]
[ Saving to process cache ... ]
(node:20264) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'original_checksum' of undefined
    at /data/code/python/miniconda3-cdh/envs/mtnsort/lib/node_modules/mountainlab/mlproc/prv_utils.js:202:24
    at /data/code/python/miniconda3-cdh/envs/mtnsort/lib/node_modules/mountainlab/mlproc/prv_utils.js:192:7
(node:20264) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:20264) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
[ Getting processor spec... ]
[ Checking inputs and substituting prvs ... ]
Error in input firings (/data/multimem/code/mountainsort_examples/bash_examples/001_ms4_bash_example/dataset/firings_true.mda.prv): Error: Error reading .prv file: /data/multimem/code/mountainsort_examples/bash_examples/001_ms4_bash_example/dataset/firings_true.mda.prv

$ ➜ ls dataset
firings_true.mda.prv  geom.csv  waveforms_true.mda.prv

Have run into a problem when verifying my installation of the mountainsort package.
I'm using a Windows 10 system running WSL2 to create a Linux (Ubuntu 20.0+) environment.
I've tried a couple of times to uninstall/reinstall anaconda3 and can create new envs but have not found a solution to the following "async function main() {..." error below when calling basic ml package commands to verify the install.

I am trying to install everything per the README so I can work with the Jupyter examples.

When I get to the step where I am supposed to install spikeforestwidgets, I get a PackagesNotFoundError.

What am I doing wrong?

conda install -c flatiron -c conda-forge spikeforestwidgets
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - spikeforestwidgets

Current channels:

  - https://conda.anaconda.org/flatiron/osx-64
  - https://conda.anaconda.org/flatiron/noarch
  - https://conda.anaconda.org/conda-forge/osx-64
  - https://conda.anaconda.org/conda-forge/noarch
  - https://repo.anaconda.com/pkgs/main/osx-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/osx-64
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

I'm trying to work with this project again, and am immediately hitting a block. I just installed everything via conda and cloned the "mountainsort_examples" repo to my computer. I am trying to run a python script to sort a toy data file. The code is written as this so far:

import os
import sys

sys.path.append('/PATH/TO/MOUNTAINSORT/EXAMPLES/CLONE/python')
from mountainlab_pytools import mlproc as mlp
import mountainsort4_1_0 as ms4

Pipeline = mlp.initPipeline()

For clarity, "/PATH/TO/MOUNTAINSORT/EXAMPLES/CLONE" is where I cloned the repo.

Anyways, if I run this (which should ONLY init the pipeline) I get the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/code/python/miniconda3-cdh/envs/mtsort/lib/python3.6/site-packages/mountainlab_pytools/mlproc/mlproc_impl.py", line 41, in initPipeline
    P.client().displayJobMonitor()
  File "/data/code/python/miniconda3-cdh/envs/mtsort/lib/python3.6/site-packages/mountainlab_pytools/mlproc/mlclient.py", line 42, in displayJobMonitor
    self._job_monitor.display()
  File "/data/code/python/miniconda3-cdh/envs/mtsort/lib/python3.6/site-packages/mountainlab_pytools/mlproc/mljobmonitor.py", line 12, in display
    display(self._W)
NameError: name 'display' is not defined

Can someone help me figure this out?

Hi,
I used old MountainSort for some time and I planed to switch to the "new" one.
I used spikeIterface to launch it.

Howerver when I analysed datas obtained it appear that with the same params results are better with the "old" one. Is there change in the way the analysis is done that could explain that?

I was able to pip install ml_ms4alg and pip3 install ml_ephys, but when I ran pip3 install ml_pyms, I received the error:

Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://pypi.org/simple/ml-pyms/

Hello,

I'm running into an index error when running MountainSort. This seems to only happen with a few specific data sets and not others, and I'm not clear on where the error is coming from. Here is a screenshot of the error:

.. and this is the bash script I'm using for sorting:

(this isn't the whole script, but it crashes at the masking step, anyway).

I've used the exact same script on a larger raw.mda file, so it doesn't seem to be an issue of file size. The algorithm works fine on the first third of my data, but not the subsequent two portions.

Anyone have any idea what might be happening here? Any insight would be helpful. Please let me know if I can provide anymore information (first time I've had to post a problem myself, so sorry if some information is lacking!).

Any idea what could be going on here? I can't seem to get this working due to a lot of widget related issues.

Exception Traceback (most recent call last)
in
4 output_dir=output_base_dir+'/ms4'
5 with Pipeline:
----> 6 ms4_sort_dataset(dataset_dir=dsdir,output_dir=output_dir,adjacency_radius=-1,detect_threshold=3)
7 #A=validate_sorting_results(dataset_dir=dsdir,sorting_output_dir=output_dir,output_dir=output_dir)
8 #amplitudes_true=A['amplitudes_true']

~/anaconda3/envs/mountainlab/lib/python3.6/site-packages/mountainlab_pytools/mlproc/mlproc_impl.py in exit(self, type, value, traceback)
33 Global['current_client']=self._client
34 def exit(self, type, value, traceback):
---> 35 self._client.run()
36 Global['current_client']=None
37

~/anaconda3/envs/mountainlab/lib/python3.6/site-packages/mountainlab_pytools/mlproc/mlclient.py in run(self)
195
196 while True:
--> 197 ret=self.next_iteration()
198 if not ret:
199 break

~/anaconda3/envs/mountainlab/lib/python3.6/site-packages/mountainlab_pytools/mlproc/mlclient.py in next_iteration(self)
162 except Exception as e:
163 self._stop_everything()
--> 164 raise e
165
166 def _stop_everything(self):

~/anaconda3/envs/mountainlab/lib/python3.6/site-packages/mountainlab_pytools/mlproc/mlclient.py in next_iteration(self)
135 str='[Click to expand] Error running process {}: {}'.format(job['processor_name'],job['error'])
136 display(vdom.details(vdom.summary(str),vdom.pre(job['console_output'])))
--> 137 raise Exception(str)
138 elif job['status'] == 'finished':
139 num_finished=num_finished+1

Exception: [Click to expand] Error running process ephys.whiten: Process returned with non-zero error code (255)

Hi,
I installed newest version of mountainlab in miniconda and I think I have followed every instructions in the readme files. But when I tried to run the first example file "001_ms4_jupyter_example" it throws error in the following cell block, which returned: "Exception: Error running process ms4alg.sort: Process returned with non-zero error code (255)".
Do anyone knows how to solve it?
Thanks!

Can you please add a step-by-step example with instructions for own data (jupyter example)?

According to issue #18 set a path to mountainsort4_1_0 using
sys.path.append('.../mountainsort_examples-master/python')
Is this step necessary or there are other ways to fix it?

According to .mda format descriptions generate for own data
raw.mda (easy to do with mdaio for Matlab)
geom.csv
params.json

To solve 'widjets' problem use
conda install -c tjd2002 'mountainlab_pytools=0.7.4'

All other initialization instructions according to ms4_jupyter_example.ipynb
and just call ms4_sort_dataset

What '.prv' is actually, when it necessary and when is not?

Hi,
I try to use mountainSort with spikeInterface and I have an issue, there is no way to change tmp/ directory location and mine is too small.

I manage to set it with this line: os.environ['TEMPDIR'] = "/tmp"
It could be nice to have this params in mountainsort4 function.

Best,
Victor

Hey,

I'm wondering how I can set the number of threads available to ensure I don't use too much RAM.

I brought this up in the old version - can't figure out how to do it within the jupyter examples here

We synthesized data once, and when we ran the pipeline multiple times (clearing the output folder and tmp files just to sure), we always got different clustering results (i.e. numbers) when we looked at firings_uncurated.mda and as a result in the final firings.mda .

Hi! I have set up mountainsort and ephys-viz as per https://github.com/flatironinstitute/mountainsort_examples. I then made the sample dataset with "synthesize_dataset.sh", but when I try to look at the data with ev-dataset dataset, an error is raised:

/data/code/python/miniconda3-chuck/envs/mount2/lib/node_modules/ephys-viz/node_modules/electron/dist/electron: symbol lookup error: /data/code/python/miniconda3-chuck/envs/mount2/lib/node_modules/ephys-viz/node_modules/electron/dist/../../../../../libfontconfig.so.1: undefined symbol: FT_Done_MM_Var

Any idea what the issue is?
Thanks

Installed today (thanks!), but ran into trouble in first jupyter example, at this line:

from mountainsort4_1_0 import sort_dataset as ms4_sort_dataset # MountainSort spike sorting

Error:

ModuleNotFoundError Traceback (most recent call last)
in
23 # imports from this repo
24 append_to_path(os.getcwd()+'/../../python')
---> 25 from mountainsort4_1_0 import sort_dataset as ms4_sort_dataset # MountainSort spike sorting
26 from validate_sorting_results import validate_sorting_results # Validation processors
27 from synthesize_dataset import synthesize_dataset # Synthesize a test dataset

ModuleNotFoundError: No module named 'mountainsort4_1_0'

I tried to locate the method, but can't find it. Guessing the module changed, or the method moved to another module?

Hey,

I successfully ran mountainsort on my data and it works well. However, I can't export or save the images it says I should contact you for help in the area. How could you help me with save images? Thanks alot. Thanks for releasing mountainsort version 4.

I am not sure should I write about it here, but I found that using import spikeforestwidgets leads to following changes in top panel of Jupyter notebook (in Jupyter lab - everything is ok)

Is it a local bug or known problem?
Ubuntu 18.04.1. Reproducable in firefox and chromium

Hi
where can I find Details on how to set the params.json file.?

Hi, I'm trying to run the view_datasets example but am getting the following error with the kbucket link. How do I fix this?

My OS is Ubuntu 16.04.6 LTS (GNU/Linux 4.4.0-134-generic x86_64) and I'm running the program on Python 3.6.8. I've attached a screenshot of the mountainlab installations from my conda environment.

Thanks!

I've converted my dataset to mda format, generated a probe map according to the instructions, and ran the jupyter notebook example with my geom.csv and raw.mda files in the datasets folder. The automatically generated data runs fine in the notebook, but the real data stall on the whiten step and eventually the pipeline spits out the error pictured below. It's not terribly specific and I've not had much success debugging it. I tried going in and eliminating the whitening step (and the filtering step) but that leads to new errors downstream.

All files (raw.mda, geom.csv, params.json, and the mountainsort notebook are all in a repository if needed (https://github.com/haleyspeed/willow).

Recording is 250 ms from 4 channels for testing. Actual data will be 2-5 minutes for 256-1024 channels.

Screenshot of error:

1. How should I set adjacency_radius parameter in mountainsort4_1_0.sort_dataset function?
   I am using a linear probe (Neuronexus "A1x32-5mm-25-177"), tetrode (Neuronexus "A4x2-tet-5mm-150-200-121") and polytrode (Neuronexus "A1x32-Poly3-5mm-25s-177").

2. In my recording setting, channel # 8 is devoted for recording the external TTL pulse, not for recording the signals from the silicon probe. In that case, how should I set the coordinate of that channel in the geom.csv file?

See flatironinstitute/mountainlab-js#71

Hello, I am very new to MountainSort. I ran the sorting program on my own dataset (in vivo recording using a Neuronexus 32-ch linear probe) following ms4_jupyter_example.ipynb, and firing.mda was successfully generated. However, I have several questions on the output files since there seem no documents for the interpretation of the outputs (except for https://mountainsort.readthedocs.io/en/latest/first_sort.html ).

1. Does the third row of firings.mda indicate the cluster IDs? i.e. if there are 5 kinds of numbers (e.g. {50, 51, 56, 25, 32}), does it mean 5 clusters were identified after curation?
2. How can I get the time course of each identified clusters?

Thank you in advance.

Hi Jeremy,

Do you plan to add support for detected spikes? I know is not the best way to test things, but reading about the algorithm I really want to test things like interpolating before aligning the clips, etc.

Hi,
I've been trying to test the mountainsort package but at the first try to run the examples I get error with mountainlab_pytools.mlproc attributes, e.g. when running ms4_jupyter_example , I got the AttributeError: module 'mountainlab_pytools.mlproc' has no attribute 'initPipeline' Error. I checked and all the required packages have been installed as per the instruction.
any help would appreciated.
BR

In some of the identified clusters after running the whole pipeline, the timestamps for spikes do not represent the peak amplitude of the spikes, but seem shifted. This happens to all spikes within that cluster. See below where we plot a window 20 timepoints before and after the timestamp.

When we expanded the time window further back, we see that the timestamps do correspond to a real cell as shown below.

Thanks in advance.

Hello, I'm excited to try this out, but I'm getting a 'No module' error for mountainsort4_1_0 in this example. Is this package deprecated or, more likely, am I missing something? I also tried importing \ml_ms4alg instead, with no luck. Thanks!

I created the .mda file, geom.csv and params.json.
My data has only 1 channel.
The contents of my geom.csv file: 0,0
The contents of my params.json file: {"samplerate": 10000}

While running the following, I got the error.

with Pipeline:
ms4_sort_dataset(dataset_dir=dsdir,output_dir=output_dir,adjacency_radius=-1,detect_threshold=3)

Hello,
I installed everything according to the ReadMe and tryed to run the example for Jupyter Lab but faced with this:

Pipeline=mlp.initPipeline()
Error displaying widget

with Pipeline:
synthesize_dataset(dsdir,M=4,duration=600,average_snr=8)

185 #self._job_monitor.widget().clearJobs()
186
--> 187 self._status_out=widgets.Output()
188 display(self._status_out)
189 self._last_status_string=''

NameError: name 'widgets' is not defined

As I understand is something wrong with setting up widgets, but there was no errors during install.
Why it happens?
Bash example works perfectly.

Hello, I have very little experience with Python and am having trouble figuring out how to change clip size when sorting. So far I have been using a jupyter notebook that I modeled after the example provided (ms4_jupyter_example.ipynb) and it has been working. I would like to change the clip_size parameter used in sorting from the default 50 to a different value. Is there a simple way to do this, based on the code I'm using below? Thank you!

#Python module imports + initialization
%load_ext autoreload
%autoreload 2
def append_to_path(dir0):
if dir0 not in sys.path:
sys.path.append(dir0)
import os, sys, json
import numpy as np
from matplotlib import pyplot as plt
from mountainlab_pytools import mlproc as mlp
from mountainlab_pytools import mdaio
import spikeforestwidgets as SFW
append_to_path(os.getcwd()+'/../python')
from mountainsort4_1_0 import sort_dataset as ms4_sort_dataset

#Initialize the pipeline object and job monitor widget
Pipeline=mlp.initPipeline()

#Set dataset + output directories
dsdir=os.getcwd()+'/Gy146_344'
output_dir=os.getcwd()+'/Gy146_344_results'
if not os.path.exists(output_dir):
os.mkdir(output_dir)

#Run the spike sorting
with Pipeline:
ms4_sort_dataset(dataset_dir=dsdir,output_dir=output_dir,freq_min=300,freq_max=5000,adjacency_radius=-1,detect_threshold=3)

I have met the same problem as this one: #10 (comment).

I am using mac os. Did this cause the problem? Since I did not find the relative import of initipeline in the init.py file in the folder of mlproc (in mountainlab_pytools 0.2.7 version). But I did find one in the linux version (mountainlab_pytools 0.7.3). But if I install the linux version, I found some problem like this one #14. What should I do?

Based on the MS3 docs it seems the curate flag needs to be set to true to run automatic curation. Is there documentation on how to do this with ms4alg.sort?

Hi,

I have problems running the Python example synthesize_dataset.py from the mountainsort_examples.
I use Anaconda with the Spyder environment setup and run the example file with the command
synthesize_dataset('/home/chris/Documents/dataset0000/',M=4,duration=60,average_snr=7,K=20)
While debugging, Python enters the function, and runs line 12ff
mlp.addProcess('ephys.synthesize_random_waveforms', ...
Then, in line 27ff,
mlp.addProcess('ephys.synthesize_random_firings', ...
the debugger stops with the error message
AttributeError: 'NoneType' object has no attribute 'addProcess'

Thus, the problem might be an wrongly installed package. Here is how I did the installation:

1_ Install MountainLab as indicated in
https://github.com/flatironinstitute/mountainlab-js
using the conda commands in the Ubuntu terminal.

2_ Install MountainSort as indicated in
https://github.com/flatironinstitute/mountainsort_examples/blob/master/README.md
using the conda commands in the Ubuntu terminal as well.

These installations seemed to run smoothly.

2_ Additionally, I installed from
https://github.com/flatironinstitute
/ qt-mountainview
/ mountainsort_examples
and
/ ephys-viz

Every package seemed to be installed fine.

Maybe someone has some ideas. I think, that it is an easy problem, if you are experienced with Ubuntu or Linux in general.

Best,

Chris