There is a difference between the way timestamps are outputted between load_open_ephys_data_faster.m and Openephys.py. Namely, the former outputs a set of timestamps equivalent to the length of the data in the channel. My reading of the code suggests this is not stored within the .continuous file itself, but is generated in the matlab script. Is there anyway to duplicate the same output in OpenEphys.py in future versions?

Hi,

I am trying to align the continuous data with the spike data. I have several questions.

• What is the y scale in the continuous and the spike data (40 samples per detected spike) ? How do I convert the data to microvolts ?
• How do I make sure that the channel of the continuous data corresponds to the right electrode ?During the recording I configured a channel mapping module and positioned it just after the FPGA module. I assumed that the mapping would apply to all the following modules in the signal chain. Is this the case ?
• Is there any available matlab example code available ?

Thx for your help

Karin

Is there a trivial way to convert between flat binary to the open-ephys formats? I see there is a tool to convert between open-ephys and kwik. Has this been generalized?

Thanks!

My example file is here: https://iu.box.com/s/68nel6xs7g73z6i732dgp4guintjsnyr

The faster function skips chunks of data over 10001 samples or more long

[~, ts]=load_open_ephys_data_faster('100_CH19.continuous');
db_fast=diff(ts);
db_fast(db_fast>100)

returns three instances:

 10001
 10001
111245 

They are located somewhere in the middle:

find(db_fast>100)
ans =
     4659200
     8309760
    50656256 
length(db_fast)
ans =
   266704895

Wheres load_open_ephys_data.m does not:

[~, ts]=load_open_ephys_data('100_CH19.continuous');db=diff(ts);find(db>100)

ans =
   Empty matrix: 0-by-1

Hi, opening event file from my last recording is returning an error

Undefined function or variable 'header'

I am not sure what the two lines

    hdr = fread(fid, NUM_HEADER_BYTES, 'char*1');
    eval(char(hdr'));

do in load_open_ephys_data (lines 83,84) but they appear to not be working this time. Previous recordings are ok. Last recording was done in Linux, OE compiled from the latest code on github. Events were generated using TTL. Event file attached here https://drive.google.com/file/d/0ByTsMM7VWmWKUG5nbVRCMTVDOWM/view?usp=sharing Thanks.

Hello,

I’m trying to import data from open ephys to neuroexplorer 4.

I first import data in matlab using load_open_ephys_data.m or load_open_ephys_data_faster.m.
Then, to bypass the well known import issues, I import data in neuroexplore by means of nex script GetMatlabVectorAsContVar.nsc for continuous variables and GetMatlabTimestampVector.nsc for events variables.

Everything works fine for .continuous data, but I have problems when trying to import .events.

In particular, I can import in Matlab only 1 column of data. In this column each number represents the number of channel-line changing phase in time (e.g. if I switch on dig 2, then dig 3 and then off 3 and 2 I’ll read 2332). However I cannot import timestamps and the flag (1 or 0, on/off).
Note that importing data with the demo version of neuroexplorer 5 and sending to Matlab, I have as a result a matrix of 5 columns: 1 timestmps; 2 event type; 3 processor ID; 4 event ID (i.e. 1 or 0 of the dig. Channel); 5 event channel, thus I believe all data are present in the .event file.

Do you think it is a problem in our Open-ephys configuration?
Did someone else had the same problem? Any solution or suggestion?

Thanks in advance for help

Hi,

I recently had a problem with one of my recordins. At around 1/3 of the recording the mice chewed the cable returning basically noise and increasing the spike file size a lot. I intended to go through the data and find the initial point of disconnection but realized that loading file in matlab for big files is a problem: my .spikes files were ~800Mb and the load_open_ephys_data.m got stuck at around 40%.

I think it would be useful for en users to have a loader that would allow for:

1. only extract portion of the data (using Nspikes or timestamps range)
2. allow to segment or merge two .spike/continuous files

I don't feel confident doing it in a clean way so i am suggesting it as an improvement of the analysis tool for matlab/python.It could be great to have that in the near future.

Hello everyone,

We are using a 64 channel probe for recording and the data is stored in .continuous format. Each channel has a separate file. For spike sorting, I would like to convert the .continuous files to flat binary .dat for use in Klusta. Is there any script for such a purpose?

I would appreciate if you can give me instruction on how to proceed.

My recording usually lasts for three days, but I want to pull out the data either from LFP, or Spikes for a specific hour (hour before and after the injection time). Can anyone help me with the code ? I also use MATLAB

I want to get the data from specific channels in binary data like the way that open_ephys format lists the channels. Can anyone help me with that ?

Hi,

I noticed a bug in the messages.events timestamps.

During my experiment, I send a TCP message containing a timestamp to open ephys. Once I opened my data I realised that the open ephys timestamps do not all correspond to the sender program timestamps.
I hope this plot makes sense.

Is there anyone who could solve the problem ?

Thx

Karin

The m file and spike, continuous and event file is in same dir. ( same error with continuous and event file)

I already tried with this commands:
load_open_ephys_data("C:\Open Ephys\2018-06-01_19-23-51\100_CH31.continuous");

anything else I need to do? I copied (https://github.com/open-ephys/analysis-tools/blob/master/load_open_ephys_data.m) this in new script file and named load_open_ephys_data.m

I'm a new user of OpenEphy (and Matlab...), when I try to compile openephy data to plexon data it appear error in Matlab2014b:
ephys2plx
Error using getFileHeader (line 6)
Error: Unexpected MATLAB expression.

Error in ephysConv (line 6)
FileHeaders{1,1}=getFileHeader([pathName ephysInfo.events_file]);

Error in ephys2plx (line 15)
ephysConv; %get .xml file, and read information

Any idea what should I do ? my laptop is MAC OS X Yosemite, Matlab2014b for mac and Xcode 6.3.1 installed. I follow the steps in https://github.com/open-ephys/analysis-tools/tree/master/ephys2plx and execute mex ephys2plx_mex.c says successful installed complier. Data format is openEphy format stored in one fold with 32 channel continuous data , 8 Tetrode spike data and 8 ADC channel event.

Hello

I've noticed that load_open_ephys_data_faster.m requires a lot more memory to load .continuous files in Matlab 2020a. Specifically, I'm getting a memory error on line 84:

hdr = fread(fid, NUM_HEADER_BYTES, 'char*1');

when I have about 4 GB of memory left for a .continuous file that takes up only about 1 GB of space when it is loaded (and obviously the header is much smaller than the full 1 GB for all the data). In Matlab 2016, this was not a problem.

I found this post on the mathworks site that indicates something changed with fread in Matlab2020a that might explain this behavior:

https://www.mathworks.com/matlabcentral/answers/84385-out-of-memory-error-while-reading-binary-file

I'd appreciate it if any necessary updates could be made to load_open_ephys_data_faster.m so it will work properly with Matlab2020a, or if someone could provide suggestions on how to modify the code. I don't know much about the structure of the .continuous files, so I don't feel confident making this change myself. Thank you for all your help!

~Nick

Hi all,

I have two separate recording sessions corresponding to two different behavioral tasks, and I am interested in following a unit's response across the two tasks.

I think the best way to do this would be to concatenate the .spikes files from the two sessions together so that I can spike sort at once (*I use MATLAB and MClust).
Is there an easy way to do this?
I am new to openephys and am not very familiar with the structure of the files. I apologize in advance if this is a very basic question.

Thank you!

i have loaded the .continuous files in to work space but now i want see the signal which i stored is correct or not. how can i open them and see.

The loadContinuous() function in OpenEphys.py produces several VisibleDeprecationWarnings, for two reasons.

The first (lines 128, 129, 130) is because the constants defined around line 30 are floats instead of integers. This can be fixed by wrapping them in int().

The second (line 157) is because N is an array ([1024]) instead of an integer (1024), because np.fromfile() returns an array even when asked for a single value (line 147).

I'll submit a pull request with what I assume are the correct fixes.

When I import the OpenEphys module I can load the .continous data without issues.
But when I try to load the spikes it gives me the following error:

data = OpenEphys.load(r"T:\Dokumente\Data\test_2019-01-08_13-57-09\STp105.0n5.spikes")
loading spikes...
Traceback (most recent call last):

File "", line 1, in
data = OpenEphys.load(r"T:\Dokumente\Data\test_2019-01-08_13-57-09\STp105.0n5.spikes")

File "c:\anaconda3\lib\OpenEphys.py", line 37, in load
data = loadSpikes(filepath)

File "c:\anaconda3\lib\OpenEphys.py", line 208, in loadSpikes
waveforms = np.fromfile(f, np.dtype('<u2'), numChannels*numSamples)

TypeError: only integer scalar arrays can be converted to a scalar index

Does anybody know what I am doing wrong here? Any help would be appreciated.

Are the time-stamps in seconds, maybe I am wrong but that doesn't seems to be the case for me.
Please could you double check.
Thank you.

When trying to load a continuous file (that I can open using other loaders) I get the following error:

In [8]: data = OpenEphys.load(path)
Loading continuous data...
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-8-79b386c02a12> in <module>()
----> 1 data = OpenEphys.load(path)

/home/galen/Documents/Code/analysis-tools/Python3/OpenEphys.py in load(filepath)
     37     # redirects to code for individual file types
     38     if 'continuous' in filepath:
---> 39         data = loadContinuous(filepath)
     40     elif 'spikes' in filepath:
     41         data = loadSpikes(filepath)

/home/galen/Documents/Code/analysis-tools/Python3/OpenEphys.py in loadContinuous(filepath, dtype)
    159             data = np.fromfile(f,np.dtype('>i2'),N)  # big-endian 16-bit signed integer
    160         try:
--> 161             samples[indices[recordNumber]:indices[recordNumber+1]] = data
    162         except ValueError:
    163             raise

ValueError: could not broadcast input array from shape (1024) into shape (256)

Normally this error is not raised due to index not being defined in the except block (#43), but once I remove the print statement this is the error that is raised.

When converting time stamps from loadSpikes in OpenEphys.py by dividing by the sample rate I get spike times that are larger than the recording duration e.g.

Spike time 1011.3367666666667 s exceeds duration 983.1765333333333 s

Am I converting time stamps wrongly?

settings.xml doesn't always (at least when I'm running it) have the processor name as the second attribute (line 38 in the code). Not sure if this is an issue with the code writing the settings file or the way Matlab reads it. I wrote a quick loop to step through the attributes to identify the "name" and tried to submit it for revision but I'm new to working with this kind of collaborative system and am not sure I did it right. I changed lines 37 & 38:

             nodeId = str2num(char(xProcessor.getAttributes.item(0).getValue));
             processorName = char(xProcessor.getAttributes.item(2).getValue);

into the following loop:

            for procAtt = 0:xProcessor.getAttributes.getLength-1
                if strcmp(xProcessor.getAttributes.item(procAtt).getNodeName, 'NodeId')
                    nodeId = str2num(char(xProcessor.getAttributes.item(procAtt).getValue));
                elseif strcmp(xProcessor.getAttributes.item(procAtt).getNodeName, 'name')
                    processorName = char(xProcessor.getAttributes.item(procAtt).getValue);
                end
            end

With the previous code the later code using strcmp to check for the 'Filters/Spike Detector' node (line 44 I believe):

            if strcmp(processorName, 'Filters/Spike Detector')

was not working because it wasn't correctly pulling the "name" attribute. Additionally, the output from the file was not correctly including the processor names as strings in the second column of the output "info."

Edit: code blocks weren't being formatted properly.

I recently put a Julia package for loading Open Ephys .continuous data files on github. I want to keep it as a separate repository so that Julia's package manager can use it, but it might be useful here as well.

Hello,
I'm trying to use the analysis tools to load my OpenEphys data in Jupyter notebook, but am not having any success. Has anyone done this and could guide me through?
Ben

Hi all,

I recently started to use a new openephys acquisition board, and I usually acquire raw data and filtered data for LFP and unit analysis.
I recently noticed that the size of filter data is smaller than the raw data, and sometimes I got a few channels that have intact recordings, but sometimes none of the channels has intact recording, though I did not change the configuration file.

Here is my configuration:

Does anyone have any idea what is wrong in my recording?

Many thanks!
Yu

These tools are simple and therefore are easy targets for an automated test suite. I think it would be great to have this...

There's a discontinuity in the length of the array of timestamps that are outputted by load_open_ephys_faster and OpenEphys.py run independently on the same analog continuous channel. For whatever reason it seems that OpenEphys.py is only keeping every 512th entry in the timestamp matrix given by load_open_ephys_faster run in matlab.

I’m new user to Open Ephys. When I try to compile the OE data to plexon, the ephys2plx script fails at ephys2plx_mex.c

Error using mex

I have tried the script in two versions of matlab (2013a and 2015a) and get the same failure.

Any suggestions as to why the complier is failing. Im using windows 7. Data format is in Openephys format.

When I try to create a .dat file from a folder of data saved in Open Ephys format using the pack function of OpenEphys.py I get the following error:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Anaconda\lib\site-packages\OpenEphys\OpenEphys.py", line 316, in pack
ch = data[''.join((source,'_CH',str(order[j]).replace(source+'_CH','')))]['data']
KeyError: '100_CH100_AUX2'

Changing line 316 from:

ch = data[''.join((source,'_CH',str(order[j]).replace(source+'_CH','')))]['data']

to:

ch = data[order[j]]['data']

prevents this error, and the program then outputs data files that appear to be correct.

I noticed that load_open_ephys_data_faster.m currently outputs continuous data with lower resolution than load_open_ephys_data.m, which I don't think is intentional. Specifically, one change made in pull request #26 (line 150 in the current file) causes the data values read in from the continuous file to be stored in an int16 matlab array (which is fine so far, since they are integers). Then, when they are scaled by info.header.bitVolts in line 120, the result is automatically rounded to another int16 array. In contrast, load_open_ephys_data.m reads the data in as doubles, so they are not rounded after multiplying. In my case, the bitVolts value was about 0.2, so reading data with this function decreased the resolution by about 5 times.

Is there a good reason for writing to an int16 array in the fread function? Otherwise, that could just be reverted.

-Ethan

line 190
info.header = header;
because header is a matlab native function it doesn't anymore.

I replaced the function getHeader with

function info = getHeader(hdr)
header='';
eval(char(hdr'));
info.header = header;
end

Hi,

I have the following error in Matlab when I use the file: convert_open_ephys_to_kwik.m.

Error using h5create>create_dataset (line 154)
The dataset '/kwik_version' already exists.

Error in h5create (line 64)
create_dataset(options)

Error in convert_open_ephys_to_kwik (line 128)
h5create(kwdfile, '/kwik_version', [1 1],'Datatype', 'int16');

To implement, I have copied and renamed the file load_open_ephys_data_faster.m as load_open_ephys_data.m.

I think that the problem is because I have several repetitions (blocks) in the same experiment. That is, in a given .continuous file I have three recording sections. So, I have three experiments joint in one.

Do you know how I can fix this problem?

Cheers,

nc

When I try to use loadEvents from OpenEphys.py to import event files into python I get the following error:

File "C:\Anaconda\lib\site-packages\OpenEphys\OpenEphys.py", line 241, in loadEvents
    nodeId[index] = np.fromfile(f, np.dtype('<ui'), 1)

TypeError: data type "<ui" not understood

Changing line 241 to:

nodeId[index] = np.fromfile(f, np.dtype('<u1'), 1)

resolves the problem.

Is there a way to load only parts of the recording? For example if I record over 3h and and I want to analyse a 2-3 min fraction in the middle of the recording. At the moment I would have to load the full 3h which is time consuming with 32 channels.

I am using OpenEphys.py

Hi there,

It seems that the recorded sampling rate in *.spikes files (30kHz) is different from set sampling rate during acquisition with the GUI (20kHz).

Does it mean that the spikes files sampling rate is by default 30 kHz or is it just a bug ?

Thx in advance for your help.

Karin

Hello!

I'm trying to load open-ephys files in python3. After 2to3 script conversion, when I try to load a .continuous file, I got this error:

In [1]: import OpenEphys

In [2]: data = OpenEphys.load('100_CH1.continuous')
Loading continuous data...
Traceback (most recent call last):

  File "<ipython-input-2-e99a956d2fe5>", line 1, in <module>
    data = OpenEphys.load('100_CH1.continuous')

  File "/home/malfatti/Documents/SciScripts/Python3/Deps/OpenEphys/OpenEphys.py", line 39, in load
    data = loadContinuous(filepath)

  File "/home/malfatti/Documents/SciScripts/Python3/Deps/OpenEphys/OpenEphys.py", line 136, in loadContinuous
    header = readHeader(f)

  File "/home/malfatti/Documents/SciScripts/Python3/Deps/OpenEphys/OpenEphys.py", line 294, in readHeader
    h = f.read(1024).replace('\n','').replace('header.','')

TypeError: 'str' does not support the buffer interface

In line 134, f is defined as byte coding:

f = open(filepath,'rb')

so I'm gessing that something in the readHeader function is not byte-encoded. I tried to use .encode() method in some strings (in line 294) but it didn't work.

Could you help me?

Cheers,
T Malfatti

I acquired external analog signal with open-ephys format and load the continuous data with the "load_open_ephys_data.mat". I noticed that the timestamps extracted for the analog have lots of jitters.
For example, I acquired the analog with 30K, and loaded the continuous data, and directly ploted "timestamps" generated by the code. The graph is not a perfect line. Is it the problem of my data acquisition or the problem of the code?

x-axis is each sample, y-axis is the time.

Hello,

Im interested in the spike sorting feature of Open Ephys for upcoming neuro phys experiments -

Currently, Ive been able to setup a practice signal chain of the following:
File read (cortex file) > Bandpass (300-6000hz) > Spike Sorter to get the hang of using the OpenEphys software.

Im able to successfully set some arbitrary thresholds and record the signal - to then load the '.spikes' file of a particular channel into matlab.

Problem: Im attempting to visualize/analyze/plot the spike data and have no idea how to proceed. Upon loading the data, a 30x40x2 array is created. Im unsure how to perceive the contents of this array, what the values correspond to, and how to plot them in a manner that allows me to see the spikes as well as any other data that is usually acquired via the spike sorter.

Any information/elaboration/tutorial would be tremendous as I am a beginner attempting to learn this software.

Thank you very much.

Hello,

I saved data in flat binary format, but found out that the number of samples is not correct. Each my recording is 2 second long sampled at 20kHz, so 40000 samples should be in the continuous.dat file for each channel in each recording. However, when I load the data with the function in Binary.py and checked the data dimension, it turned out to be either more or less than 40000 samples for each recording.

Is there a way to fix this? How do I align the data from each recording in this case?

Appreciate it for any help,
Yu

index is undefined on line 163

Hi everyone;
newbee here.
i have some problems with load_open_ephys_data function.
I started in this days to use Open Ephys GUI, trying to record some samples and convert in Matfile to test the system. I'have dowloaded the analysis_tools and used the function, but some syntax problems are aroused from the fist time.

Example:
"Error: File: load_open_ephys_data.m Line: 140 Column: 6
Illegal use of reserved keyword "elseif".

line 140- elseif strcmp(filetype, 'continuous')
141
142- disp(['Loading ' filename '...']);

line 140: Parse error at ELSEIF: usage might be invalid in Matlab syntax
line 140: Parse error at ')': usage might be invalid in Matlab syntax"

load_open_ephys_data_faster does not give this issue.
Can you give me some help or suggestion to resolve this problems?
Thanks!

Matlab version: 2016a

i have an issue running TT openephys data recorded in binary to Mclust.

i can open data using load_open_ephys_binary (both spikes and continuous) but then when i want to cut them in Mclust using LoadTT_openephys then it doesnt work.

any one got similar issue? any idea how to do it

thanks

Hi.
I'm using open-ephys GUI and neuropixels.
I recorded 50 min of LFP and SPK.

1. 'Continuous.dat' file of the LFP is around 3G. I have 48G RAM on my computer.
   According to the advice of using memmapfile, I tried to use it. But still, I have trouble with importing LFP data to Matlab.
   The error message indicates 'out of memory.'
   I can't figure out how to map memory in Matlab.
   Could you help me out?
2. Without slitting recording time, Is it possible to select a certain set of electrodes out of 384 total electrodes?
   I've been trying to select a certain set of electrodes using Recording Node, but it doesn't help.

Thanks for your help in advance.

I was analyzing my continuous LFP data and trying to line-up manual timestamps that I had placed during recording (via comments entered in the GUI and pressing 'save',) when I noticed that clear events on the recording were not lining up with the timestamp. For a number of my recordings, I had hit the play button prior to hitting the record button in the GUI to make sure that I had a good signal before recording. I was wondering whether the timestamp on the messages.events file related to the sample number on the actual recording or the overall sample number (including non-recorded time). I did a test where I did a number of timestamps, paused the recording, and did additional timestamps. From that data, it looks like the timestamp includes non-recorded time (which means that the int64 stamp doesn't actually correspond to sample number in the recording.) I have a lot of data that hinges on accurate timestamps and is already recorded- is there any way to determine how these timestamps line up? It looks like the load_open_ephys.m reads the int64 timestamp in the .continuous files, which is what I believe is in the messages.events file, but I'm not clear how to correlate everything. Any help appreciated.

What is the license of this repository?

Hi!

I'd like to make some changes to this repo, but I just wanted to get some feedback first.

I would like to reorganize the python code into a python module. This allows people to install the tools as a python package. The downside is that is adds a bit of complexity to the structure. Also, I'm not sure the name "analysis-tools" is going to work as a python module. The pep8 has some guidance on how to name a python package here, I propose "openephys", or maybe "pyopenephys" (or "pyoephys"?) to distinguish it from the matlab code

The new structure of the repo would then be:

analysis-tools/
    setup.py <- for installing the package
    [py]openephys/
        kwik.py
        (etc).py
    matlab/
        *.m

Would love to hear what you think, possibly we could create a new branch for these changes?

Does anyone know of a tool to convert .continuous files to the SpikeGLX binary format? Alternatively, if there isn't one, is there a good spec for the .continuous file so that I might build my own converter?

Hello,

I'm using "convert_open_ephys_to_kwik.m" to read 32-channel recording files in the OpenEphys GUI, however, it looks like the script only includes the 32 channels and not the ADC and AUX channels, which I also need to visualize. Is there a way to do this?

Thanks,
Jack

When using load_open_ephys_data, we have an issue of timestamps missing and being out of order. They can skip ahead, which results in losing some samples or they can skip backwards, which makes them misaligned. This happens frequently throughout recording. I'm using the latest open ephys version compiled on linux computer.

https://github.com/open-ephys/analysis-tools/blob/master/Python3/OpenEphys.py#L222

Should be wv = np.reshape(waveforms, (numChannels, numSamples)).T