Giter Club home page Giter Club logo

ibllib's Introduction

IBL Python Libraries

Coverage badge Tests status badge Tests status badge

Description

Library used to implement the International Brain Laboratory data pipeline. Currently in active development. The library is currently 2 main modules:

  • brainbox: neuroscience analysis oriented library
  • ibllib: general purpose library containing I/O, signal processing and IBL data pipelines utilities.

Release Notes here

Requirements

OS: Only tested on Linux. Windows and Mac may work, but are not supported.

Python Module: Python 3.8 or higher

Installation, documentation and examples

https://docs.internationalbrainlab.org

Contribution and development practices

See https://int-brain-lab.github.io/iblenv/07_contribution.html

We use gitflow and Semantic Versioning.

Before committing to your branch:

  • run tests
  • flake8 This is also enforced by continuous integration.

Matlab Library

The Matlab library has moved to its own repository here: https://github.com/int-brain-lab/ibllib-matlab/

ibllib's People

Contributors

alejandropan avatar anne-urai avatar bensonbrandon avatar berkgercek avatar bimac avatar chris-langfield avatar cskrasniak avatar dbirman avatar gaellechapuis avatar guidomeijer avatar jeanpaulnc avatar jhdlee avatar juhuntenburg avatar k1o0 avatar kdharris101 avatar kushbanga avatar mayofaulkner avatar micheleangelofabbri avatar mschart avatar nbonacchi avatar njmiska avatar noamroth avatar nsteinme avatar oliche avatar rossant avatar sebastianbruijns avatar shenshan avatar stevenjwest avatar themattinthehatt avatar yeebc avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ibllib's Issues

logging error in alf extraction

2019-02-05 20:59:19.887 INFO     [extract_session.py:64] Extracting D:\globus_server\Subjects\IBL_1\2019-02-05\001
2019-02-05 21:01:14.7 INFO     [raw_data_loaders.py:290] extracting_ibl_trials.feedbackType.npy
2019-02-05 21:01:14.23 INFO     [raw_data_loaders.py:290] extracting_ibl_trials.contrastLeft.npy
2019-02-05 21:01:14.29 INFO     [raw_data_loaders.py:290] extracting_ibl_trials.contrastRight.npy
2019-02-05 21:01:14.35 INFO     [raw_data_loaders.py:290] extracting_ibl_trials.probabilityLeft.npy
2019-02-05 21:01:14.41 INFO     [raw_data_loaders.py:290] extracting_ibl_trials.choice.npy
2019-02-05 21:01:14.46 INFO     [raw_data_loaders.py:290] extracting_ibl_trials.repNum.npy
2019-02-05 21:01:14.52 INFO     [raw_data_loaders.py:290] extracting_ibl_trials.rewardVolume.npy
2019-02-05 21:01:14.59 INFO     [raw_data_loaders.py:290] extracting_ibl_trials.feedback_times.npy
--- Logging error ---
Traceback (most recent call last):
  File "C:\Users\IBLuser\Anaconda3\envs\iblenv\lib\logging\__init__.py", line 1034, in emit
    msg = self.format(record)
  File "C:\Users\IBLuser\Anaconda3\envs\iblenv\lib\logging\__init__.py", line 880, in format
    return fmt.format(record)
  File "C:\Users\IBLuser\Anaconda3\envs\iblenv\lib\logging\__init__.py", line 619, in format
    record.message = record.getMessage()
  File "C:\Users\IBLuser\Anaconda3\envs\iblenv\lib\logging\__init__.py", line 380, in getMessage
    msg = msg % self.args
TypeError: %i format: a number is required, not WindowsPath
Call stack:
  File "C:\ibllib\python\alf\one_iblrig.py", line 99, in <module>
    extract(args.folder, dry=args.dry)
  File "C:\ibllib\python\alf\one_iblrig.py", line 25, in extract
    extract_session.bulk(root_data_folder, dry=dry)
  File "c:\ibllib\python\alf\extract_session.py", line 84, in bulk
    from_path(p.parent, force=True, save=save)
  File "c:\ibllib\python\alf\extract_session.py", line 30, in func_wrapper
    f = func(sessionpath, *args, **kwargs)
  File "c:\ibllib\python\alf\extract_session.py", line 70, in from_path
    training_trials.extract_all(session_path, save=save)
  File "c:\ibllib\python\alf\extractors\training_trials.py", line 529, in extract_all
    stimOn_times = get_stimOn_times(session_path, save=save, data=data)
  File "c:\ibllib\python\alf\extractors\training_trials.py", line 322, in get_stimOn_times
    count_missing, session_path)
Message: '%s: Missing BNC1 stimulus on %i trials'
Arguments: (77, WindowsPath('D:/globus_server/Subjects/IBL_1/2019-02-05/001'))
--- Logging error ---
Traceback (most recent call last):
  File "C:\Users\IBLuser\Anaconda3\envs\iblenv\lib\logging\__init__.py", line 1034, in emit
    msg = self.format(record)
  File "C:\Users\IBLuser\Anaconda3\envs\iblenv\lib\logging\__init__.py", line 880, in format
    return fmt.format(record)
  File "C:\Users\IBLuser\Anaconda3\envs\iblenv\lib\site-packages\colorlog\colorlog.py", line 119, in format
    message = super(ColoredFormatter, self).format(record)
  File "C:\Users\IBLuser\Anaconda3\envs\iblenv\lib\logging\__init__.py", line 619, in format
    record.message = record.getMessage()
  File "C:\Users\IBLuser\Anaconda3\envs\iblenv\lib\logging\__init__.py", line 380, in getMessage
    msg = msg % self.args
TypeError: %i format: a number is required, not WindowsPath
Call stack:
  File "C:\ibllib\python\alf\one_iblrig.py", line 99, in <module>
    extract(args.folder, dry=args.dry)
  File "C:\ibllib\python\alf\one_iblrig.py", line 25, in extract
    extract_session.bulk(root_data_folder, dry=dry)
  File "c:\ibllib\python\alf\extract_session.py", line 84, in bulk
    from_path(p.parent, force=True, save=save)
  File "c:\ibllib\python\alf\extract_session.py", line 30, in func_wrapper
    f = func(sessionpath, *args, **kwargs)
  File "c:\ibllib\python\alf\extract_session.py", line 70, in from_path
    training_trials.extract_all(session_path, save=save)
  File "c:\ibllib\python\alf\extractors\training_trials.py", line 529, in extract_all
    stimOn_times = get_stimOn_times(session_path, save=save, data=data)
  File "c:\ibllib\python\alf\extractors\training_trials.py", line 322, in get_stimOn_times
    count_missing, session_path)
Message: '%s: Missing BNC1 stimulus on %i trials'
Arguments: (77, WindowsPath('D:/globus_server/Subjects/IBL_1/2019-02-05/001'))
2019-02-05 21:01:14.265 INFO     [raw_data_loaders.py:290] extracting_ibl_trials.stimOn_times.npy
2019-02-05 21:01:14.270 INFO     [raw_data_loaders.py:290] extracting_ibl_trials.intervals.npy
2019-02-05 21:01:14.279 INFO     [raw_data_loaders.py:290] extracting_ibl_trials.response_times.npy
2019-02-05 21:01:14.672 INFO     [raw_data_loaders.py:290] extracting_ibl_trials.itiDuration.npy
2019-02-05 21:01:14.678 INFO     [raw_data_loaders.py:290] extracting_ibl_trials.included
2019-02-05 21:01:16.6 INFO     [raw_data_loaders.py:290] extracting_ibl_wheel.timestamps.npy
2019-02-05 21:01:16.18 INFO     [raw_data_loaders.py:290] extracting_ibl_wheel.position.npy
2019-02-05 21:01:16.41 INFO     [raw_data_loaders.py:290] extracting_ibl_wheel.velocity.npy
2019-02-05 21:01:16.81 INFO     [extract_session.py:72] session extracted```

WARNING [registration.py:185] No matching dataset type for raw_behavior_data_iblrig_micData and alf_ibl_trials.goCueTrigger_times

During registration I have occurring two warnings for each of registered session:

2019-03-20 17:44:11.265 WARNING [registration.py:185] No matching dataset type for: C:\globus_server\Subjects\KS002\2019-03-20\001\raw_behavior_data_iblrig_micData.raw.wav

2019-03-20 17:44:11.239 WARNING [registration.py:185] No matching dataset type for: C:\globus_server\Subjects\KS002\2019-03-20\001\alf_ibl_trials.goCueTrigger_times.npy

Will this interfere with future analysis?

Documentation work

  • Organize the index of ibllib docs
  • Link the docstrings using sys.addpath in conf.py

Historic weighings in Subjects rest endpoint

Answer to Slack message.

So the subjects endpoint has 2 behaviours:
list, which returns multiple subjects depending on the filter sub = one.alyx.rest('subjects', 'read', 'IBL_11')
details, which returns subjects matching nickname:
sub = one.alyx.rest('subjects', 'read', 'IBL_11')

The details endpoint returns full weighings and water_administrations history for a subject.

from oneibl.one import ONE
one = ONE()

sub = one.alyx.rest('subjects', 'read', 'IBL_11')
for w in sub['weighings']:
    print(w['weight'], w['date_time'][:10])

for wa in sub['water_administrations']:
    print(wa['water_administered'], wa['date_time'][:10])

one.search to return eid's for any() of the dataset types instead of all()

So for example:

one.search(lab='mainenlab', dataset_types=['_ibl_trials.goCue_times', '_ibl_trials.goCueTrigger_times.npy',], any=False)

returns all eIDs from the mainenlab that have _ibl_trials.goCue_times AND _ibl_trials.goCueTrigger_times, recapitulating the current behavior so we can have any=False as default.

but

one.search(lab='mainenlab', dataset_types=['_ibl_trials.goCue_times', '_ibl_trials.goCueTrigger_times.npy',], any=True)

would return all eIDs from the mainenlab that had either _ibl_trials.goCue_times OR _ibl_trials.goCueTrigger_times.

Installation problem

Hi there,

I guess I need to reinstall to get the new version working. (Just cloning it didn't make a difference.)

But I have had a problem with installation. Maybe I just did something dumb. But there seems to be a problem that it has doubled the Cache directory.

Full dump (including installation) below:

(base) A:\git\ibllib\python>python setup.py install
Enter the URL of Alyx Instance [https://test.alyx.internationalbrainlab.org]:
https://test.alyx.internationalbrainlab.org
Enter Alyx username [test_user]:
test_user
Enter the Alyx password for test_user:
Enter the Alyx password (again):
Directory to cache FlatIron downloads [C:\Users\Kenneth\Downloads\FlatIron]:A:\ONEcache
A:\ONEcache
Enter the FlatIron username [iblmember]:
iblmember
Enter the FlatIron password:
Enter the FlatIron password (again):
running install
running bdist_egg
running egg_info
writing ibllib.egg-info\PKG-INFO
writing dependency_links to ibllib.egg-info\dependency_links.txt
writing requirements to ibllib.egg-info\requires.txt
writing top-level names to ibllib.egg-info\top_level.txt
reading manifest file 'ibllib.egg-info\SOURCES.txt'
writing manifest file 'ibllib.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
running build_py
copying oneibl\params.py -> build\lib\oneibl
copying oneibl\params_secret.py -> build\lib\oneibl
creating build\bdist.win-amd64\egg
creating build\bdist.win-amd64\egg\alf
copying build\lib\alf\extractors.py -> build\bdist.win-amd64\egg\alf
copying build\lib\alf\scraper.py -> build\bdist.win-amd64\egg\alf
copying build\lib\alf_init_.py -> build\bdist.win-amd64\egg\alf
creating build\bdist.win-amd64\egg\ibllib
creating build\bdist.win-amd64\egg\ibllib\dsp
copying build\lib\ibllib\dsp\savitzky_golay.py -> build\bdist.win-amd64\egg\ibllib\dsp
copying build\lib\ibllib\dsp\smooth.py -> build\bdist.win-amd64\egg\ibllib\dsp
copying build\lib\ibllib\dsp_init_.py -> build\bdist.win-amd64\egg\ibllib\dsp
creating build\bdist.win-amd64\egg\ibllib\io
copying build\lib\ibllib\io\raw_data_loaders.py -> build\bdist.win-amd64\egg\ibllib\io
copying build\lib\ibllib\io_init_.py -> build\bdist.win-amd64\egg\ibllib\io
creating build\bdist.win-amd64\egg\ibllib\misc
copying build\lib\ibllib\misc\flatten.py -> build\bdist.win-amd64\egg\ibllib\misc
copying build\lib\ibllib\misc\misc.py -> build\bdist.win-amd64\egg\ibllib\misc
copying build\lib\ibllib\misc\timing.py -> build\bdist.win-amd64\egg\ibllib\misc
copying build\lib\ibllib\misc_init_.py -> build\bdist.win-amd64\egg\ibllib\misc
copying build\lib\ibllib\time.py -> build\bdist.win-amd64\egg\ibllib
copying build\lib\ibllib\webclient.py -> build\bdist.win-amd64\egg\ibllib
copying build\lib\ibllib_init_.py -> build\bdist.win-amd64\egg\ibllib
creating build\bdist.win-amd64\egg\oneibl
creating build\bdist.win-amd64\egg\oneibl\examples
copying build\lib\oneibl\examples\tutorial_script.py -> build\bdist.win-amd64\egg\oneibl\examples
copying build\lib\oneibl\examples_init_.py -> build\bdist.win-amd64\egg\oneibl\examples
copying build\lib\oneibl\one.py -> build\bdist.win-amd64\egg\oneibl
copying build\lib\oneibl\params.py -> build\bdist.win-amd64\egg\oneibl
copying build\lib\oneibl\params_secret.py -> build\bdist.win-amd64\egg\oneibl
copying build\lib\oneibl\params_secret_template.py -> build\bdist.win-amd64\egg\oneibl
copying build\lib\oneibl_init_.py -> build\bdist.win-amd64\egg\oneibl
byte-compiling build\bdist.win-amd64\egg\alf\extractors.py to extractors.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\alf\scraper.py to scraper.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\alf_init_.py to init.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\ibllib\dsp\savitzky_golay.py to savitzky_golay.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\ibllib\dsp\smooth.py to smooth.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\ibllib\dsp_init_.py to init.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\ibllib\io\raw_data_loaders.py to raw_data_loaders.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\ibllib\io_init_.py to init.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\ibllib\misc\flatten.py to flatten.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\ibllib\misc\misc.py to misc.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\ibllib\misc\timing.py to timing.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\ibllib\misc_init_.py to init.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\ibllib\time.py to time.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\ibllib\webclient.py to webclient.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\ibllib_init_.py to init.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\oneibl\examples\tutorial_script.py to tutorial_script.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\oneibl\examples_init_.py to init.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\oneibl\one.py to one.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\oneibl\params.py to params.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\oneibl\params_secret.py to params_secret.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\oneibl\params_secret_template.py to params_secret_template.cpython-36.pyc
byte-compiling build\bdist.win-amd64\egg\oneibl_init_.py to init.cpython-36.pyc
creating build\bdist.win-amd64\egg\EGG-INFO
copying ibllib.egg-info\PKG-INFO -> build\bdist.win-amd64\egg\EGG-INFO
copying ibllib.egg-info\SOURCES.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying ibllib.egg-info\dependency_links.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying ibllib.egg-info\requires.txt -> build\bdist.win-amd64\egg\EGG-INFO
copying ibllib.egg-info\top_level.txt -> build\bdist.win-amd64\egg\EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist\ibllib-0.1.2-py3.6.egg' and adding 'build\bdist.win-amd64\egg' to it
removing 'build\bdist.win-amd64\egg' (and everything under it)
Processing ibllib-0.1.2-py3.6.egg
Removing a:\anaconda3\lib\site-packages\ibllib-0.1.2-py3.6.egg
Copying ibllib-0.1.2-py3.6.egg to a:\anaconda3\lib\site-packages
ibllib 0.1.2 is already the active version in easy-install.pth

Installed a:\anaconda3\lib\site-packages\ibllib-0.1.2-py3.6.egg
Processing dependencies for ibllib==0.1.2
Searching for requests==2.14.2
Best match: requests 2.14.2
Adding requests 2.14.2 to easy-install.pth file

Using a:\anaconda3\lib\site-packages
Searching for pandas==0.20.2
Best match: pandas 0.20.2
Adding pandas 0.20.2 to easy-install.pth file

Using a:\anaconda3\lib\site-packages
Searching for numpy==1.12.1
Best match: numpy 1.12.1
Adding numpy 1.12.1 to easy-install.pth file

Using a:\anaconda3\lib\site-packages
Searching for matplotlib==2.2.2
Best match: matplotlib 2.2.2
Adding matplotlib 2.2.2 to easy-install.pth file

Using a:\anaconda3\lib\site-packages
Searching for dataclasses==0.6
Best match: dataclasses 0.6
Adding dataclasses 0.6 to easy-install.pth file

Using a:\anaconda3\lib\site-packages
Searching for pytz==2017.2
Best match: pytz 2017.2
Adding pytz 2017.2 to easy-install.pth file

Using a:\anaconda3\lib\site-packages
Searching for python-dateutil==2.6.0
Best match: python-dateutil 2.6.0
Adding python-dateutil 2.6.0 to easy-install.pth file

Using a:\anaconda3\lib\site-packages
Searching for kiwisolver==1.0.1
Best match: kiwisolver 1.0.1
Adding kiwisolver 1.0.1 to easy-install.pth file

Using a:\anaconda3\lib\site-packages
Searching for six==1.10.0
Best match: six 1.10.0
Adding six 1.10.0 to easy-install.pth file

Using a:\anaconda3\lib\site-packages
Searching for pyparsing==2.1.4
Best match: pyparsing 2.1.4
Adding pyparsing 2.1.4 to easy-install.pth file

Using a:\anaconda3\lib\site-packages
Searching for cycler==0.10.0
Best match: cycler 0.10.0
Adding cycler 0.10.0 to easy-install.pth file

Using a:\anaconda3\lib\site-packages
Searching for setuptools==27.2.0
Best match: setuptools 27.2.0
Processing setuptools-27.2.0-py3.6.egg
setuptools 27.2.0 is already the active version in easy-install.pth
Installing easy_install-script.py script to A:\Anaconda3\Scripts
Installing easy_install.exe script to A:\Anaconda3\Scripts
Installing easy_install-3.6-script.py script to A:\Anaconda3\Scripts
Installing easy_install-3.6.exe script to A:\Anaconda3\Scripts

Using a:\anaconda3\lib\site-packages\setuptools-27.2.0-py3.6.egg
Finished processing dependencies for ibllib==0.1.2

(base) A:\git\ibllib\python>run_tests

(base) A:\git\ibllib\python>cd .\ibllib\tests

(base) A:\git\ibllib\python\ibllib\tests>python -m unittest discover
.Downloading: C:\Users\Kenneth\AppData\Local\Temp\tmphp1n85gh\clusters._phy_annotation.6f3eb5f5-f6e8-4c1a-80e5-88e127a80893.npy Bytes: 8952
8952 [100.00%]
...

Ran 4 tests in 0.853s

OK

(base) A:\git\ibllib\python\ibllib\tests>cd ....\oneibl\tests

(base) A:\git\ibllib\python\oneibl\tests>python -m unittest discover
[
"86e27228-8708-48d8-96ed-9aa61ab951db"
]
data : <class 'list'> , 29 items = []
dataset_id : <class 'list'> , 29 items = 80930c88-0c93-41db-977f-40adcff66df2
local_path : <class 'list'> , 29 items =
dataset_type : <class 'list'> , 29 items = unknown
url : <class 'list'> , 29 items = http://ibl.flatironinstitute.org/mainenlab/Subjects/clns0730/2018-08-24/1/_ibl_wheel.timestamps.80930c88-0c93-41db-977f-40a
dcff66df2.npy
eid : <class 'list'> , 29 items = 86e27228-8708-48d8-96ed-9aa61ab951db

data : <class 'list'> , 29 items = []
dataset_id : <class 'list'> , 29 items = 80930c88-0c93-41db-977f-40adcff66df2
local_path : <class 'list'> , 29 items =
dataset_type : <class 'list'> , 29 items = unknown
url : <class 'list'> , 29 items = http://ibl.flatironinstitute.org/mainenlab/Subjects/clns0730/2018-08-24/1/_ibl_wheel.timestamps.80930c88-0c93-41db-977f-40a
dcff66df2.npy
eid : <class 'list'> , 29 items = 86e27228-8708-48d8-96ed-9aa61ab951db

subject : clns0730
users : ['olivier']
lab : None
type : Base
start_time : 2018-08-24T12:34:56
end_time : 2018-08-24T12:34:57
subject : ['clns0730', 'flowers']
users : [['olivier'], ['olivier']]
lab : [None, None]
type : ['Base', 'Base']
start_time : ['2018-08-24T12:34:56', '2018-07-13T12:34:56']
end_time : ['2018-08-24T12:34:57', '2018-07-13T12:34:57']
..EE.....E

ERROR: test_load (test_one.TestLoad)

Traceback (most recent call last):
File "A:\git\ibllib\python\oneibl\tests\test_one.py", line 47, in test_load
t, cr, cl = myone.load(eid, dataset_types=dataset_types)
File "A:\Anaconda3\lib\site-packages\ibllib-0.1.2-py3.6.egg\oneibl\one.py", line 174, in load
cache_dir = _cache_directory(par.CACHE_DIR, ses)
File "A:\Anaconda3\lib\site-packages\ibllib-0.1.2-py3.6.egg\oneibl\one.py", line 20, in _cache_directory
pathlib.Path(cache_dir).mkdir(parents=True, exist_ok=True)
File "A:\Anaconda3\lib\pathlib.py", line 1233, in mkdir
self._accessor.mkdir(self, mode)
File "A:\Anaconda3\lib\pathlib.py", line 390, in wrapped
return strfunc(str(pathobj), *args)
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'A:\ONEcache\A:\ONEcacheclns0730\2018-08-24\1'

======================================================================
ERROR: test_load_all_data_available (test_one.TestLoad)

Traceback (most recent call last):
File "A:\git\ibllib\python\oneibl\tests\test_one.py", line 77, in test_load_all_data_available
a = myone.load(eid)
File "A:\Anaconda3\lib\site-packages\ibllib-0.1.2-py3.6.egg\oneibl\one.py", line 174, in load
cache_dir = _cache_directory(par.CACHE_DIR, ses)
File "A:\Anaconda3\lib\site-packages\ibllib-0.1.2-py3.6.egg\oneibl\one.py", line 20, in _cache_directory
pathlib.Path(cache_dir).mkdir(parents=True, exist_ok=True)
File "A:\Anaconda3\lib\pathlib.py", line 1233, in mkdir
self._accessor.mkdir(self, mode)
File "A:\Anaconda3\lib\pathlib.py", line 390, in wrapped
return strfunc(str(pathobj), *args)
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'A:\ONEcache\A:\ONEcacheflowers\2018-07-13\1'

======================================================================
ERROR: test_one_tutorial (unittest.loader._FailedTest)

ImportError: Failed to import test module: test_one_tutorial
Traceback (most recent call last):
File "A:\Anaconda3\lib\unittest\loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "A:\Anaconda3\lib\unittest\loader.py", line 369, in _get_module_from_name
import(name)
File "A:\git\ibllib\python\oneibl\tests\test_one_tutorial.py", line 1, in
import oneibl.examples.tutorial_script
File "", line 961, in _find_and_load
File "", line 950, in _find_and_load_unlocked
File "", line 646, in _load_unlocked
File "", line 616, in _load_backward_compatible
File "A:\Anaconda3\lib\site-packages\ibllib-0.1.2-py3.6.egg\oneibl\examples\tutorial_script.py", line 31, in
wf, pr, d = one.load(eid, dataset_types=dataset_types)
File "A:\Anaconda3\lib\site-packages\ibllib-0.1.2-py3.6.egg\oneibl\one.py", line 174, in load
cache_dir = _cache_directory(par.CACHE_DIR, ses)
File "A:\Anaconda3\lib\site-packages\ibllib-0.1.2-py3.6.egg\oneibl\one.py", line 20, in _cache_directory
pathlib.Path(cache_dir).mkdir(parents=True, exist_ok=True)
File "A:\Anaconda3\lib\pathlib.py", line 1233, in mkdir
self._accessor.mkdir(self, mode)
File "A:\Anaconda3\lib\pathlib.py", line 390, in wrapped
return strfunc(str(pathobj), *args)
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'A:\ONEcache\A:\ONEcacheclns0730\2018-08-24\1'


Ran 10 tests in 5.671s

FAILED (errors=3)

(base) A:\git\ibllib\python\oneibl\tests>

Problem with MATLAB version

I haven't been able to get the MATLAB version to install. When I run AlyxClient.setup() it doesn't ask for username or password for the flatiron server. Then it gives an error accessing it.

Is there a specific branch I should be trying?

missed trials: set response time to NaN, and feedbackType to -1

'Missed' (i.e. no response within 1minute) trials are currently coded in Alf in a way that's a bit confusing. When choice == 0, it would be helpful if response_times == NaN, and feedbackType == -1 (since I believe the punishment tone played is the same after a miss?

skip copy/extraction/registration of all test files

It looks like registering _iblrig_test_user fails, whereas _ibl_test_mouse is registered to Alyx when run under a real user. To save time and disk space, could all tests simply be skipped?

Sample command line output

INFO: Registering C:\globus_server\Subjects\_iblrig_test_mouse\2019-01-24\006\raw_video_data\_iblrig_leftCamera.timestamps.ssv
------> [MainProcess:registration.py:199] 2019-01-29T22:13:40.888

INFO: Registering C:\globus_server\Subjects\_iblrig_test_mouse\2019-01-24\006\raw_video_data\_iblrig_videoCodeFiles.raw.zip
------> [MainProcess:registration.py:199] 2019-01-29T22:13:40.892

INFO: registeringC:\globus_server\Subjects\_iblrig_test_mouse\2019-01-24\008
------> [MainProcess:registration.py:60] 2019-01-29T22:13:42.665
https://alyx.internationalbrainlab.org/users/_iblrig_test_user
{"detail":"Not found."}

ERROR: C:\globus_server\Subjects\_iblrig_test_mouse\2019-01-24\008 failed registration
        Subject: _iblrig_test_user doesn't exist in AlyxNoneType: None

------> [MainProcess:registration.py:66] 2019-01-29T22:13:44.333

wheelMoves object

@anne-urai pointed out that the wheel moves ALF object is not extracted on the Python rigs. We want to begin doing wheel movement analysis with these traces. What's the status of this?

If you think this will be difficult to implement it could in theory not be an ALF file but instead be calculated in DataJoint.

allow loading of multiple eid in one call

when a one.search returns a list of eids, it would be great to pass this to one.load (returning for example a list of data, or even better the appended data) rather than having to load each session in a loop

TODO for doc : definition of data dimension

TODO for doc: need to define data dimension (e.g. ‘clusters.templateWaveforms’, ‘clusters.probes’, ‘clusters.depths’).
e.g. ‘clusters.templateWaveforms’: NxMxW where N = number of spikes, M = time, W = shank etc.

Happy to do the doc, but I could not find this info anywhere.

one.list

I had some more thoughts on one.list. (Moving the conversation to git as suggested.) This could make simpler, easier for new users, and hopefully still as powerful for advanced users. The syntax (with default arguments) would be :

term_list = one.list(eids = None, search_term = 'dataset_type')

It would return a list of possible search term values, for the specified search term, and the given eids.

The default argument eids=None means find the terms for all eids. So if you just call one.list(), it will give you all possible dataset types (as now). But if you call one.list(eid), it will list the datasets available for that eid. This is the first thing I wanted to do on the system.

If you want to find all users on the system, you would call one.list(None, 'users'). Etcetera.

automated transfer, extraction and registration of files

Assuming a reasonable duration for file transfer to the servpc, and another reasonable duration for alf extraction and registration, run scripts on all rig pcs and the servpc every night (starting at e.g. 8pm) in time for sync to flatiron

Loading csv files

Hi Olivier,

I found that the loading of .csv files only returns empty lists, for example, the_ibl_wheelMoves.type. Would you please check that? Thanks!

Shan

Alyx registration failed

2019-01-10T15:08:40,751 INFO [extract_session.py:73] Extracting C:\Users\apv2\Documents\IBL\Data\Subjects\ibl_witten_01\2019-01-08\001
2019-01-10T15:08:43,390 WARNING [raw_data_loaders.py:175] _iblrig_encoderPositions.raw.ssv has missing/incomplete records
C:\Users\apv2\Documents\IBL\Data\Subjects\ibl_witten_01\2019-01-08\001\raw_behavior_data_iblrig_encoderPositions.raw.ssv
2019-01-10T15:08:44,861 INFO [extract_session.py:73] Extracting C:\Users\apv2\Documents\IBL\Data\Subjects\ibl_witten_02\2019-01-08\001
2019-01-10T15:08:49,322 INFO [extract_session.py:73] Extracting C:\Users\apv2\Documents\IBL\Data\Subjects\ibl_witten_03\2019-01-08\001
2019-01-10T15:08:50,779 WARNING [training_wheel.py:102] Time-stamp arrays have inconsistent size. Trimming to the smallest size
2019-01-10T15:08:50,854 INFO [extract_session.py:73] Extracting C:\Users\apv2\Documents\IBL\Data\Subjects\ibl_witten_03\2019-01-08\002
Connected to https://alyx.internationalbrainlab.org as alejandro
2019-01-10T15:08:59,121 INFO [registration.py:39] registeringC:\Users\apv2\Documents\IBL\Data\Subjects\ibl_witten_01\2019-01-08\001
2019-01-10T15:09:01,926 INFO [registration.py:100] creating sessionibl_witten_01\2019-01-08\001
2019-01-10T15:09:04,499 INFO [registration.py:39] registeringC:\Users\apv2\Documents\IBL\Data\Subjects\ibl_witten_02\2019-01-08\001
2019-01-10T15:09:09,298 INFO [registration.py:39] registeringC:\Users\apv2\Documents\IBL\Data\Subjects\ibl_witten_03\2019-01-08\001
https://alyx.internationalbrainlab.org/register-file
ValueError at /register-file
A base session for ibl_witten_03 on 2019-01-08 does not exist

Request Method: POST
Request URL: https://alyx.internationalbrainlab.org/register-file
Django Version: 2.1.1
Python Executable: /usr/bin/python3
Python Version: 3.5.2
Python Path: ['/var/www/alyx-main/alyx', '/var/www/alyx-main/venv/lib/python35.zip', '/var/www/alyx-main/venv/lib/python3.5', '/var/www/alyx-main/venv/lib/python3.5/plat-x86_64-linux-gnu', '/var/www/alyx-main/venv/lib/python3.5/lib-dynload', '/usr/lib/python3.5', '/usr/lib/python3.5/plat-x86_64-linux-gnu', '/var/www/alyx-main/venv/lib/python3.5/site-packages']
Server time: Thu, 10 Jan 2019 20:09:11 +0000
Installed Applications:
('test_without_migrations',
'dal',
'dal_select2',
'reversion',
'django_admin_listfilter_dropdown',
'rangefilter',
'subjects',
'django_filters',
'django.contrib.admin',
'django.contrib.contenttypes',
'django.contrib.auth',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'polymorphic',
'rest_framework',
'rest_framework.authtoken',
'rest_framework_docs',
'actions',
'data',
'misc')
Installed Middleware:
('django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'django.middleware.security.SecurityMiddleware',
'alyx.base.QueryPrintingMiddleware')

Traceback:

File "/var/www/alyx-main/venv/lib/python3.5/site-packages/django/core/handlers/exception.py" in inner
34. response = get_response(request)

File "/var/www/alyx-main/venv/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response
126. response = self.process_exception_by_middleware(e, request)

File "/var/www/alyx-main/venv/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response
124. response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/var/www/alyx-main/venv/lib/python3.5/site-packages/django/views/decorators/csrf.py" in wrapped_view
54. return view_func(*args, **kwargs)

File "/var/www/alyx-main/venv/lib/python3.5/site-packages/rest_framework/viewsets.py" in view
103. return self.dispatch(request, *args, **kwargs)

File "/var/www/alyx-main/venv/lib/python3.5/site-packages/rest_framework/views.py" in dispatch
483. response = self.handle_exception(exc)

File "/var/www/alyx-main/venv/lib/python3.5/site-packages/rest_framework/views.py" in handle_exception
443. self.raise_uncaught_exception(exc)

File "/var/www/alyx-main/venv/lib/python3.5/site-packages/rest_framework/views.py" in dispatch
480. response = handler(request, *args, **kwargs)

File "/var/www/alyx-main/alyx/data/views.py" in create
251. subject=subject, date=date, number=session_number, user=user)

File "/var/www/alyx-main/alyx/data/models.py" in _get_session
26. raise ValueError("A base session for %s on %s does not exist" % (subject, date))

Exception Type: ValueError at /register-file
Exception Value: A base session for ibl_witten_03 on 2019-01-08 does not exist
Request information:
USER: alejandro

GET: No GET data

POST: No POST data

FILES: No FILES data

COOKIES: No cookie data

META:
CONTENT_LENGTH = '538'
CONTENT_TYPE = 'application/json'
CONTEXT_DOCUMENT_ROOT = '/var/www/html'
CONTEXT_PREFIX = ''
DOCUMENT_ROOT = '/var/www/html'
GATEWAY_INTERFACE = 'CGI/1.1'
HTTP_ACCEPT = 'application/json'
HTTP_ACCEPT_ENCODING = 'gzip, deflate'
HTTP_AUTHORIZATION = 'Token bd495b7e06cb1dde48ae754dea75c53a75c6d267'
HTTP_CONNECTION = 'keep-alive'
HTTP_HOST = 'alyx.internationalbrainlab.org'
HTTP_USER_AGENT = 'python-requests/2.19.1'
PATH_INFO = '/register-file'
PATH_TRANSLATED = '/var/www/alyx-main/alyx/alyx/wsgi.py/register-file'
QUERY_STRING = ''
REMOTE_ADDR = '128.112.216.239'
REMOTE_PORT = '53196'
REQUEST_METHOD = 'POST'
REQUEST_SCHEME = 'https'
REQUEST_URI = '/register-file'
SCRIPT_FILENAME = '/var/www/alyx-main/alyx/alyx/wsgi.py'
SCRIPT_NAME = ''
SERVER_ADDR = '172.31.1.209'
SERVER_ADMIN = '[email protected]'
SERVER_NAME = 'alyx.internationalbrainlab.org'
SERVER_PORT = '443'
SERVER_PROTOCOL = 'HTTP/1.1'
SERVER_SIGNATURE = '

Apache/2.4.18 (Ubuntu) Server at alyx.internationalbrainlab.org Port 443\n'
SERVER_SOFTWARE = 'Apache/2.4.18 (Ubuntu)'
SSL_TLS_SNI = 'alyx.internationalbrainlab.org'
apache.version = '(2, 4, 18)'
mod_wsgi.application_group = 'alyx.internationalbrainlab.org|'
mod_wsgi.callable_object = 'application'
mod_wsgi.connection_id = 'bc7kJoDKmTk'
mod_wsgi.daemon_connects = '1'
mod_wsgi.daemon_restarts = '0'
mod_wsgi.daemon_start = '1547150951911285'
mod_wsgi.enable_sendfile = '0'
mod_wsgi.handler_script = ''
mod_wsgi.ignore_activity = '0'
mod_wsgi.listener_host = ''
mod_wsgi.listener_port = '443'
mod_wsgi.path_info = '/register-file'
mod_wsgi.process_group = 'alyxmain'
mod_wsgi.queue_start = '1547150951911118'
mod_wsgi.request_handler = 'wsgi-script'
mod_wsgi.request_id = 'HQ7nJoDKmTk'
mod_wsgi.request_start = '1547150951910941'
mod_wsgi.script_name = ''
mod_wsgi.script_reloading = '1'
mod_wsgi.script_start = '1547150951911411'
mod_wsgi.thread_id = 7
mod_wsgi.thread_requests = 281
mod_wsgi.total_requests = 13574
mod_wsgi.version = '(4, 6, 4)'
wsgi.errors = <_io.TextIOWrapper name='<wsgi.errors>' encoding='utf-8'>
wsgi.file_wrapper = ''
wsgi.input = <mod_wsgi.Input object at 0x7f46caa86ea0>
wsgi.input_terminated = True
wsgi.multiprocess = False
wsgi.multithread = True
wsgi.run_once = False
wsgi.url_scheme = 'https'
wsgi.version = '(1, 0)'

Settings:
Using settings module alyx.settings
ABSOLUTE_URL_OVERRIDES = {}
ADMINS = []
ALLOWED_HOSTS = ['localhost', 'alyx.internationalbrainlab.org']
APPEND_SLASH = True
AUTHENTICATION_BACKENDS = ['django.contrib.auth.backends.ModelBackend']
AUTH_PASSWORD_VALIDATORS = ''
AUTH_USER_MODEL = 'misc.LabMember'
BASE_DIR = '/var/www/alyx-main/alyx'
CACHES = {'default': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'}}
CACHE_MIDDLEWARE_ALIAS = 'default'
CACHE_MIDDLEWARE_KEY_PREFIX = '
'
CACHE_MIDDLEWARE_SECONDS = 600
CSRF_COOKIE_AGE = 31449600
CSRF_COOKIE_DOMAIN = None
CSRF_COOKIE_HTTPONLY = False
CSRF_COOKIE_NAME = 'csrftoken'
CSRF_COOKIE_PATH = '/'
CSRF_COOKIE_SAMESITE = 'Lax'
CSRF_COOKIE_SECURE = False
CSRF_FAILURE_VIEW = 'django.views.csrf.csrf_failure'
CSRF_HEADER_NAME = 'HTTP_X_CSRFTOKEN'
CSRF_TRUSTED_ORIGINS = []
CSRF_USE_SESSIONS = False
DATABASES = {'default': {'HOST': '127.0.0.1', 'ENGINE': 'django.db.backends.postgresql_psycopg2', 'PASSWORD': '', 'TIME_ZONE': None, 'NAME': 'ibl', 'TEST': {'CHARSET': None, 'COLLATION': None, 'NAME': None, 'MIRROR': None}, 'ATOMIC_REQUESTS': False, 'OPTIONS': {}, 'CONN_MAX_AGE': 0, 'AUTOCOMMIT': True, 'PORT': '5432', 'USER': 'ibl_dev'}, 'cortexlab': {'HOST': '127.0.0.1', 'ENGINE': 'django.db.backends.postgresql_psycopg2', 'PASSWORD': '', 'TIME_ZONE': None, 'NAME': 'cortexlab', 'TEST': {'CHARSET': None, 'COLLATION': None, 'NAME': None, 'MIRROR': None}, 'ATOMIC_REQUESTS': False, 'OPTIONS': {}, 'CONN_MAX_AGE': 0, 'AUTOCOMMIT': True, 'PORT': '5432', 'USER': 'ibl_dev'}}
DATABASE_ROUTERS = []
DATA_UPLOAD_MAX_MEMORY_SIZE = 2621440
DATA_UPLOAD_MAX_NUMBER_FIELDS = 10000
DATETIME_FORMAT = 'N j, Y, P'
DATETIME_INPUT_FORMATS = ['%Y-%m-%d %H:%M:%S', '%Y-%m-%d %H:%M:%S.%f', '%Y-%m-%d %H:%M', '%Y-%m-%d', '%m/%d/%Y %H:%M:%S', '%m/%d/%Y %H:%M:%S.%f', '%m/%d/%Y %H:%M', '%m/%d/%Y', '%m/%d/%y %H:%M:%S', '%m/%d/%y %H:%M:%S.%f', '%m/%d/%y %H:%M', '%m/%d/%y']
DATE_FORMAT = 'N j, Y'
DATE_INPUT_FORMATS = "('%d/%m/%Y',)"
DEBUG = True
DEBUG_PROPAGATE_EXCEPTIONS = False
DECIMAL_SEPARATOR = '.'
DEFAULT_CHARSET = 'utf-8'
DEFAULT_CONTENT_TYPE = 'text/html'
DEFAULT_EXCEPTION_REPORTER_FILTER = 'django.views.debug.SafeExceptionReporterFilter'
DEFAULT_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage'
DEFAULT_FROM_EMAIL = 'webmaster@localhost'
DEFAULT_INDEX_TABLESPACE = ''
DEFAULT_PROTOCOL = '1'
DEFAULT_SOURCE = 'IBL'
DEFAULT_TABLESPACE = ''
DISALLOWED_USER_AGENTS = []
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = 'mail.gandi.net'
EMAIL_HOST_PASSWORD = ''
EMAIL_HOST_USER = '[email protected]'
EMAIL_PORT = 587
EMAIL_SSL_CERTFILE = None
EMAIL_SSL_KEYFILE = '
'
EMAIL_SUBJECT_PREFIX = '[Django] '
EMAIL_TIMEOUT = None
EMAIL_USE_LOCALTIME = False
EMAIL_USE_SSL = False
EMAIL_USE_TLS = True
FILE_CHARSET = 'utf-8'
FILE_UPLOAD_DIRECTORY_PERMISSIONS = None
FILE_UPLOAD_HANDLERS = ['django.core.files.uploadhandler.MemoryFileUploadHandler', 'django.core.files.uploadhandler.TemporaryFileUploadHandler']
FILE_UPLOAD_MAX_MEMORY_SIZE = 2621440
FILE_UPLOAD_PERMISSIONS = None
FILE_UPLOAD_TEMP_DIR = None
FIRST_DAY_OF_WEEK = 0
FIXTURE_DIRS = []
FORCE_SCRIPT_NAME = None
FORMAT_MODULE_PATH = None
FORM_RENDERER = 'django.forms.renderers.DjangoTemplates'
GLOBUS_CLIENT_ID = '525cc517-8ccb-4d11-8036-af332da5eafd'
IGNORABLE_404_URLS = []
INSTALLED_APPS = "('test_without_migrations', 'dal', 'dal_select2', 'reversion', 'django_admin_listfilter_dropdown', 'rangefilter', 'subjects', 'django_filters', 'django.contrib.admin', 'django.contrib.contenttypes', 'django.contrib.auth', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'polymorphic', 'rest_framework', 'rest_framework.authtoken', 'rest_framework_docs', 'actions', 'data', 'misc')"
INTERNAL_IPS = []
LANGUAGES = [('af', 'Afrikaans'), ('ar', 'Arabic'), ('ast', 'Asturian'), ('az', 'Azerbaijani'), ('bg', 'Bulgarian'), ('be', 'Belarusian'), ('bn', 'Bengali'), ('br', 'Breton'), ('bs', 'Bosnian'), ('ca', 'Catalan'), ('cs', 'Czech'), ('cy', 'Welsh'), ('da', 'Danish'), ('de', 'German'), ('dsb', 'Lower Sorbian'), ('el', 'Greek'), ('en', 'English'), ('en-au', 'Australian English'), ('en-gb', 'British English'), ('eo', 'Esperanto'), ('es', 'Spanish'), ('es-ar', 'Argentinian Spanish'), ('es-co', 'Colombian Spanish'), ('es-mx', 'Mexican Spanish'), ('es-ni', 'Nicaraguan Spanish'), ('es-ve', 'Venezuelan Spanish'), ('et', 'Estonian'), ('eu', 'Basque'), ('fa', 'Persian'), ('fi', 'Finnish'), ('fr', 'French'), ('fy', 'Frisian'), ('ga', 'Irish'), ('gd', 'Scottish Gaelic'), ('gl', 'Galician'), ('he', 'Hebrew'), ('hi', 'Hindi'), ('hr', 'Croatian'), ('hsb', 'Upper Sorbian'), ('hu', 'Hungarian'), ('ia', 'Interlingua'), ('id', 'Indonesian'), ('io', 'Ido'), ('is', 'Icelandic'), ('it', 'Italian'), ('ja', 'Japanese'), ('ka', 'Georgian'), ('kab', 'Kabyle'), ('kk', 'Kazakh'), ('km', 'Khmer'), ('kn', 'Kannada'), ('ko', 'Korean'), ('lb', 'Luxembourgish'), ('lt', 'Lithuanian'), ('lv', 'Latvian'), ('mk', 'Macedonian'), ('ml', 'Malayalam'), ('mn', 'Mongolian'), ('mr', 'Marathi'), ('my', 'Burmese'), ('nb', 'Norwegian Bokmål'), ('ne', 'Nepali'), ('nl', 'Dutch'), ('nn', 'Norwegian Nynorsk'), ('os', 'Ossetic'), ('pa', 'Punjabi'), ('pl', 'Polish'), ('pt', 'Portuguese'), ('pt-br', 'Brazilian Portuguese'), ('ro', 'Romanian'), ('ru', 'Russian'), ('sk', 'Slovak'), ('sl', 'Slovenian'), ('sq', 'Albanian'), ('sr', 'Serbian'), ('sr-latn', 'Serbian Latin'), ('sv', 'Swedish'), ('sw', 'Swahili'), ('ta', 'Tamil'), ('te', 'Telugu'), ('th', 'Thai'), ('tr', 'Turkish'), ('tt', 'Tatar'), ('udm', 'Udmurt'), ('uk', 'Ukrainian'), ('ur', 'Urdu'), ('vi', 'Vietnamese'), ('zh-hans', 'Simplified Chinese'), ('zh-hant', 'Traditional Chinese')]
LANGUAGES_BIDI = ['he', 'ar', 'fa', 'ur']
LANGUAGE_CODE = 'en-us'
LANGUAGE_COOKIE_AGE = None
LANGUAGE_COOKIE_DOMAIN = None
LANGUAGE_COOKIE_NAME = 'django_language'
LANGUAGE_COOKIE_PATH = '/'
LOCALE_PATHS = []
LOGGING = {'handlers': {'console': {'level': 'DEBUG', 'formatter': 'simple', 'class': 'logging.StreamHandler'}, 'file': {'level': 'DEBUG', 'formatter': 'simple', 'class': 'logging.handlers.RotatingFileHandler', 'filename': '/var/log/alyx.log', 'maxBytes': 16777216}}, 'version': 1, 'loggers': {'django': {'level': 'INFO', 'handlers': ['file'], 'propagate': True}}, 'formatters': {'simple': {'()': 'colorlog.ColoredFormatter', 'format': '%(log_color)s%(asctime)s [%(levelname)s] %(message)s', 'log_colors': {'INFO': 'white', 'ERROR': 'red', 'WARNING': 'yellow', 'DEBUG': 'cyan', 'CRITICAL': 'bold_red'}, 'datefmt': '%d/%m %H:%M:%S'}}, 'root': {'level': 'INFO', 'handlers': ['file', 'console'], 'propagate': True}, 'disable_existing_loggers': False}
LOGGING_CONFIG = 'logging.config.dictConfig'
LOGIN_REDIRECT_URL = '/accounts/profile/'
LOGIN_URL = '/accounts/login/'
LOGOUT_REDIRECT_URL = None
MANAGERS = []
MEDIA_ROOT = '/var/www/alyx-main/uploaded'
MEDIA_URL = '/uploaded/'
MESSAGE_STORAGE = 'django.contrib.messages.storage.fallback.FallbackStorage'
MIDDLEWARE = "('django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django.middleware.security.SecurityMiddleware', 'alyx.base.QueryPrintingMiddleware')"
MIGRATION_MODULES = {}
MONTH_DAY_FORMAT = 'F j'
NUMBER_GROUPING = 0
PASSWORD_HASHERS = ''
PASSWORD_RESET_TIMEOUT_DAYS = '
'
PREPEND_WWW = False
REST_FRAMEWORK = {'DEFAULT_FILTER_BACKENDS': ('django_filters.rest_framework.DjangoFilterBackend',), 'DEFAULT_AUTHENTICATION_CLASSES': ('rest_framework.authentication.SessionAuthentication', 'rest_framework.authentication.TokenAuthentication'), 'STRICT_JSON': False}
ROOT_URLCONF = 'alyx.urls'
SECRET_KEY = '********************'
SECURE_BROWSER_XSS_FILTER = False
SECURE_CONTENT_TYPE_NOSNIFF = False
SECURE_HSTS_INCLUDE_SUBDOMAINS = False
SECURE_HSTS_PRELOAD = False
SECURE_HSTS_SECONDS = 0
SECURE_PROXY_SSL_HEADER = None
SECURE_REDIRECT_EXEMPT = []
SECURE_SSL_HOST = None
SECURE_SSL_REDIRECT = False
SERVER_EMAIL = 'root@localhost'
SESSION_CACHE_ALIAS = 'default'
SESSION_COOKIE_AGE = 1209600
SESSION_COOKIE_DOMAIN = None
SESSION_COOKIE_HTTPONLY = True
SESSION_COOKIE_NAME = 'sessionid'
SESSION_COOKIE_PATH = '/'
SESSION_COOKIE_SAMESITE = 'Lax'
SESSION_COOKIE_SECURE = False
SESSION_ENGINE = 'django.contrib.sessions.backends.db'
SESSION_EXPIRE_AT_BROWSER_CLOSE = False
SESSION_FILE_PATH = None
SESSION_SAVE_EVERY_REQUEST = False
SESSION_SERIALIZER = 'django.contrib.sessions.serializers.JSONSerializer'
SETTINGS_MODULE = 'alyx.settings'
SHORT_DATETIME_FORMAT = 'm/d/Y P'
SHORT_DATE_FORMAT = 'm/d/Y'
SIGNING_BACKEND = 'django.core.signing.TimestampSigner'
SILENCED_SYSTEM_CHECKS = []
STATICFILES_DIRS = []
STATICFILES_FINDERS = ['django.contrib.staticfiles.finders.FileSystemFinder', 'django.contrib.staticfiles.finders.AppDirectoriesFinder']
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage'
STATIC_ROOT = '/var/www/alyx-main/alyx/static/'
STATIC_URL = '/static/'
STOCK_MANAGERS = "('olivier',)"
SUBJECT_REQUEST_EMAIL_FROM = '[email protected]'
SUPERUSERS = 'root'
TEMPLATES = [{'APP_DIRS': True, 'DIRS': ['/var/www/alyx-main/alyx/templates'], 'OPTIONS': {'context_processors': ['django.template.context_processors.debug', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages']}, 'BACKEND': 'django.template.backends.django.DjangoTemplates'}]
TEST_NON_SERIALIZED_APPS = []
TEST_RUNNER = 'django.test.runner.DiscoverRunner'
THOUSAND_SEPARATOR = ','
TIME_FORMAT = 'P'
TIME_INPUT_FORMATS = ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M']
TIME_ZONE = 'GB'
UPLOADED_IMAGE_WIDTH = 800
USE_I18N = False
USE_L10N = False
USE_THOUSAND_SEPARATOR = False
USE_TZ = False
USE_X_FORWARDED_HOST = False
USE_X_FORWARDED_PORT = False
WEIGHT_THRESHOLD = 0.75
WSGI_APPLICATION = 'alyx.wsgi.application'
X_FRAME_OPTIONS = 'SAMEORIGIN'
YEAR_MONTH_FORMAT = 'F Y'

You're seeing this error because you have DEBUG = True in your
Django settings file. Change that to False, and Django will
display a standard page generated by the handler for this status code.

Traceback (most recent call last):

File "", line 11, in
rc.register_sync(ROOT_DATA_FOLDER)

File "c:\users\apv2\ibllib\python\oneibl\registration.py", line 40, in register_sync
status_str = self.register_session(flag_file.parent)

File "c:\users\apv2\ibllib\python\oneibl\registration.py", line 21, in func_wrapper
f = func(self, sessionpath, *args, **kwargs)

File "c:\users\apv2\ibllib\python\oneibl\registration.py", line 164, in register_session
self.one.alyx.post('/register-file', data=r_)

File "c:\users\apv2\ibllib\python\oneibl\webclient.py", line 260, in post
return self._generic_request(requests.post, rest_query, data=data)

File "c:\users\apv2\ibllib\python\oneibl\webclient.py", line 176, in _generic_request
raise Exception(r)

Exception: <Response [500]>

Error after command : source run_tests

I followed the information on this page to install ibllib: https://ibllib.readthedocs.io/en/latest/02_installation.html
I am having these errors (Note: I am on Mac).

(gaellework) MBP-de-Gaelle:python gaellechapuis$ source run_tests
EE..         
======================================================================
ERROR: test_download_datasets (ibllib.tests.test_alyxclient.TestDownloadHTTP)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
    raise err
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x10af8ce48>: Failed to establish a new connection: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='test.alyx.internationalbrainlab.org', port=443): Max retries exceeded with url: /auth-token (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10af8ce48>: Failed to establish a new connection: [Errno 60] Operation timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gaellechapuis/Documents/gitClones/ibllib/python/ibllib/tests/test_alyxclient.py", line 15, in setUp
    base_url=par.BASE_URL)
  File "/Users/gaellechapuis/Documents/gitClones/ibllib/python/ibllib/webclient.py", line 151, in __init__
    self.authenticate(**kwargs)
  File "/Users/gaellechapuis/Documents/gitClones/ibllib/python/ibllib/webclient.py", line 167, in authenticate
    data=dict(username=username, password=password))
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/requests/api.py", line 112, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/requests/adapters.py", line 508, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='test.alyx.internationalbrainlab.org', port=443): Max retries exceeded with url: /auth-token (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10af8ce48>: Failed to establish a new connection: [Errno 60] Operation timed out',))

======================================================================
ERROR: test_download_datasets_with_api (ibllib.tests.test_alyxclient.TestDownloadHTTP)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
    raise err
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x10afacc18>: Failed to establish a new connection: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='test.alyx.internationalbrainlab.org', port=443): Max retries exceeded with url: /auth-token (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10afacc18>: Failed to establish a new connection: [Errno 60] Operation timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gaellechapuis/Documents/gitClones/ibllib/python/ibllib/tests/test_alyxclient.py", line 15, in setUp
    base_url=par.BASE_URL)
  File "/Users/gaellechapuis/Documents/gitClones/ibllib/python/ibllib/webclient.py", line 151, in __init__
    self.authenticate(**kwargs)
  File "/Users/gaellechapuis/Documents/gitClones/ibllib/python/ibllib/webclient.py", line 167, in authenticate
    data=dict(username=username, password=password))
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/requests/api.py", line 112, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/requests/adapters.py", line 508, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='test.alyx.internationalbrainlab.org', port=443): Max retries exceeded with url: /auth-token (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10afacc18>: Failed to establish a new connection: [Errno 60] Operation timed out',))

----------------------------------------------------------------------
Ran 4 tests in 150.124s

FAILED (errors=2)
Traceback (most recent call last):
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
    raise err
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x114c53160>: Failed to establish a new connection: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='test.alyx.internationalbrainlab.org', port=443): Max retries exceeded with url: /auth-token (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x114c53160>: Failed to establish a new connection: [Errno 60] Operation timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/unittest/__main__.py", line 18, in <module>
    main(module=None)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/unittest/main.py", line 141, in parseArgs
    self.createTests()
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/unittest/main.py", line 148, in createTests
    self.module)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/unittest/loader.py", line 219, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/unittest/loader.py", line 219, in <listcomp>
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/unittest/loader.py", line 153, in loadTestsFromName
    module = __import__(module_name)
  File "/Users/gaellechapuis/Documents/gitClones/ibllib/python/oneibl/tests/test_one_tutorial.py", line 1, in <module>
    import oneibl.examples.tutorial_script
  File "/Users/gaellechapuis/Documents/gitClones/ibllib/python/oneibl/examples/tutorial_script.py", line 5, in <module>
    password='TapetesBloc18')
  File "/Users/gaellechapuis/Documents/gitClones/ibllib/python/oneibl/one.py", line 67, in __init__
    self._alyxClient = wc.AlyxClient(username=username, password=password, base_url=base_url)
  File "/Users/gaellechapuis/Documents/gitClones/ibllib/python/ibllib/webclient.py", line 151, in __init__
    self.authenticate(**kwargs)
  File "/Users/gaellechapuis/Documents/gitClones/ibllib/python/ibllib/webclient.py", line 167, in authenticate
    data=dict(username=username, password=password))
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/requests/api.py", line 112, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/Users/gaellechapuis/anaconda3/lib/python3.6/site-packages/requests/adapters.py", line 508, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='test.alyx.internationalbrainlab.org', port=443): Max retries exceeded with url: /auth-token (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x114c53160>: Failed to establish a new connection: [Errno 60] Operation timed out',))
(gaellework) MBP-de-Gaelle:python gaellechapuis$ 

one.list keywords and plurals

For one.list it seems the correct syntax is:

one.list(None, keyword='subjects')
but also
one.list(eids, keyword='subject')

This is confusing!!

I also can't do one.list(eids, keyword='dataset_types'), even with any combination of plurals I can find.

Aren't the keywords supposed to be the same as the search terms? In which case keyword could have any of the values ('dataset_types', 'users', 'subjects', 'date_range')?

I realize date_range could present a problem (since it is a range) but other than that can they not be the same?

Also, could keywords=None return a list for all keywords?

session creation: location, object with name does not exist

I just start the training for DY_003 in my third rig. After run the habituation today, I try to transfer the data to local server, but it showed an error like this (and I can transfer DY_002's data in this rig):
`(base) C:\Users\IBLuser>conda activate iblenv

(iblenv) C:\Users\IBLuser>python C:\ibllib\python\alf\one_iblrig.py create C:\iblrig_data\Subjects
Connected to https://alyx.internationalbrainlab.org as feihu
2019-03-09 13:21:06.11 INFO [registration.py:46] creating session for C:\iblrig_data\Subjects\DY_002\2019-03-02\001
2019-03-09 13:21:12.937 ERROR [webclient.py:173] https://alyx.internationalbrainlab.org/sessions
2019-03-09 13:21:12.941 ERROR [webclient.py:174] {"location":["Object with name=iblrig_danlab_behavior_3 does not exist."]}
Traceback (most recent call last):
File "C:\ibllib\python\alf\one_iblrig.py", line 108, in
create(args.folder, dry=args.dry)
File "C:\ibllib\python\alf\one_iblrig.py", line 41, in create
rc.create_sessions(root_data_folder, dry=dry)
File "c:\ibllib\python\oneibl\registration.py", line 48, in create_sessions
status_str = self.register_session(flag_file.parent, file_list=False)
File "c:\ibllib\python\oneibl\registration.py", line 24, in func_wrapper
f = func(self, sessionpath, *args, **kwargs)
File "c:\ibllib\python\oneibl\registration.py", line 151, in register_session
session = self.one.alyx.rest('sessions', 'create', data=ses_)
File "c:\ibllib\python\oneibl\webclient.py", line 347, in rest
return self.post('/' + endpoint, data)
File "c:\ibllib\python\oneibl\webclient.py", line 262, in post
return self._generic_request(requests.post, rest_query, data=data)
File "c:\ibllib\python\oneibl\webclient.py", line 175, in _generic_request
raise Exception(r)
Exception: <Response [400]>`

reduce command line output when extracting Alyx files

When running python C:\ibllib\python\alf\one_iblrig.py extract C:\globus_server\Subjects, a more compact output would be helpful:
• if BNC sync is missing for all trial, only write 'missing for session'
• rather than ------> [MainProcess:raw_data_loaders.py:291] 2019-01-29T21:58:15.184 + whitespace between each extracted file, list them in a smaller block

This would make warnings more salient. The output could even be limited to summary warnings only, plus a 'completed' message for each full session.

If possible, a status bar for the total extraction would be also be great!

Syntax in the tutorial may be outdated

Looks like the syntax used in the tutorial does not work.

eIDs, eInfo = ONE.search(lab='CortexLabUCL', subject='hercules', dataset_types=['spikes.times', 'spikes.clusters','headTracking.xyPos'])

One working version I tried was:

eIDs = ONE().search(lab='CortexLabUCL', subjects='hercules', dataset_types=['spikes.times', 'spikes.clusters','headTracking.xyPos'])

Empty stimOn_times

When using one.load I get empty stimOn_times for all the trials in certain sessions (e.g ibl_witten_02, 2018-01-07)

one=ONE() URL error

This is what I have in python prompt
one = ONE()
Traceback (most recent call last):
File "", line 1, in
File "D:\IBL\Programs\Anaconda3\envs\ibllib\python\oneibl\one.py", line 84, in init
base_url=base_url)
File "D:\IBL\Programs\Anaconda3\envs\ibllib\python\oneibl\webclient.py", line 154, in init
self.authenticate(**kwargs)
File "D:\IBL\Programs\Anaconda3\envs\ibllib\python\oneibl\webclient.py", line 192, in authenticate
data=dict(username=username, password=password))
File "D:\IBL\Programs\Anaconda3\lib\site-packages\requests\api.py", line 112, in post
return request('post', url, data=data, json=json, **kwargs)
File "D:\IBL\Programs\Anaconda3\lib\site-packages\requests\api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "D:\IBL\Programs\Anaconda3\lib\site-packages\requests\sessions.py", line 498, in request
prep = self.prepare_request(req)
File "D:\IBL\Programs\Anaconda3\lib\site-packages\requests\sessions.py", line 441, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "D:\IBL\Programs\Anaconda3\lib\site-packages\requests\models.py", line 309, in prepare
self.prepare_url(url, params)
File "D:\IBL\Programs\Anaconda3\lib\site-packages\requests\models.py", line 383, in prepare_url
raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL 'one /auth-token': No schema supplied. Perhaps you meant http://one /auth-token?

update.py

Provide means to update the code without having to go through the installation again

Help search function for information on data dimension

Create help search function that allows to search on dataset_type and displays information.

Information: as given in dataset_details (see example below), BUT ALSO meaning of dataset dimension.

[dataset_type , dataset_details] = one.list(eid)

ans =

  2×1 cell array
    'clusters.templateWaveforms'
    'eye.area'
% Help function, something like: 
one.help.dataset('clusters.templateWaveforms')

displays

 Dataset_type: clusters.templateWaveforms
 Dimension: NxMxT , where N = ... , M = ..., T = ... 
 Data format: ... % info in dataset_details
 (etc other useful info)

Corrupted session end time in the bpod rig output data

Trying to run TRANSFER_2_SERVER.bat and it gives me this error:

  File "C:\ibllib\python\alf\one_iblrig.py", line 108, in <module>
    create(args.folder, dry=args.dry)
  File "C:\ibllib\python\alf\one_iblrig.py", line 41, in create
    rc.create_sessions(root_data_folder, dry=dry)
  File "c:\ibllib\python\oneibl\registration.py", line 48, in create_sessions
    status_str = self.register_session(flag_file.parent, file_list=False)
  File "c:\ibllib\python\oneibl\registration.py", line 24, in func_wrapper
    f = func(self, sessionpath, *args, **kwargs)
  File "c:\ibllib\python\oneibl\registration.py", line 119, in register_session
    end_time = start_time + datetime.timedelta(seconds=ses_duration_secs)
OverflowError: date value out of range```

list index out of range

When I finish habituation training I have this error message in the end? Is it related to registration of the session?

> Traceback (most recent call last):
>   File "..\..\..\create_session.py", line 33, in <module>
>     main()
>   File "..\..\..\create_session.py", line 21, in main
>     create(IBLRIG_DATA, dry=False)
>   File "c:\ibllib\python\alf\one_iblrig.py", line 41, in create
>     rc.create_sessions(root_data_folder, dry=dry)
>   File "c:\ibllib\python\oneibl\registration.py", line 48, in create_sessions
>     status_str = self.register_session(flag_file.parent, file_list=False)
>   File "c:\ibllib\python\oneibl\registration.py", line 24, in func_wrapper
>     f = func(self, sessionpath, *args, **kwargs)
>   File "c:\ibllib\python\oneibl\registration.py", line 99, in register_session
>     project = self.one.alyx.rest('projects', 'read', pname[0])
> IndexError: list index out of range

Alyx registration error

rc.register_sync(ROOT_DATA_FOLDER)
2019-01-13T01:22:41,601 INFO [extract_session.py:73] Extracting C:\Users\apv2\Documents\IBL\Data\Subjects\ibl_witten_01\2019-01-12\002
2019-01-13T01:22:43,278 ERROR [extract_session.py:80] C:\Users\apv2\Documents\IBL\Data\Subjects\ibl_witten_01\2019-01-12\002 failed extraction
Unexpected end of string while parsing second. Expected 1 more character
2019-01-13T01:22:43,302 INFO [extract_session.py:73] Extracting C:\Users\apv2\Documents\IBL\Data\Subjects\ibl_witten_02\2019-01-12\001
2019-01-13T01:22:50,276 INFO [extract_session.py:73] Extracting C:\Users\apv2\Documents\IBL\Data\Subjects\ibl_witten_03\2019-01-12\001
Connected to https://alyx.internationalbrainlab.org as alejandro
2019-01-13T01:23:07,201 INFO [registration.py:39] registeringC:\Users\apv2\Documents\IBL\Data\Subjects\ibl_witten_02\2019-01-12\001
2019-01-13T01:23:10,408 WARNING [registration.py:139] No matching dataset type for: C:\Users\apv2\Documents\IBL\Data\Subjects\ibl_witten_02\2019-01-12\001\raw_behavior_data_iblrig_taskCodeFiles.raw.zip
2019-01-13T01:23:12,126 INFO [registration.py:39] registeringC:\Users\apv2\Documents\IBL\Data\Subjects\ibl_witten_03\2019-01-12\001
2019-01-13T01:23:17,551 WARNING [registration.py:139] No matching dataset type for: C:\Users\apv2\Documents\IBL\Data\Subjects\ibl_witten_03\2019-01-12\001\raw_behavior_data_iblrig_taskCodeFiles.raw.zip

oneibl_rig report command

right now, the user has to take a few daily steps to make sure data are transferred - and things can go wrong for a few of them, which has led to data not appearing on flatiron (and therefore datajoint) for a while.

it would be great to have a system where these discrepancies are flagged to the user (and the data coordinator?) so that we don't have a data backlog that's too long. since we don't have psychometric functions and many other crucial things plotted during training in pybpod, only seeing the data a few days later can be a problem for diagnosing problems with the animals in a timely manner.

ImportError: Failed to import test module: test_alyxclient / test_one / test_one_tutorial

Karolina and I are running the test from the documentation and we get the same error (see below)

My parameter file is the following:

    "ALYX_LOGIN": "Gaelle",
    "ALYX_PWD": --hidden for github,
    "ALYX_URL": "https://alyx.internationalbrainlab.org",
    "CACHE_DIR": "C:\\Users\\Experiment\\Downloads\\FlatIron",
    "FTP_DATA_SERVER": "ftp://ibl.flatironinstitute.org",
    "FTP_DATA_SERVER_LOGIN": "iblftp",
    "FTP_DATA_SERVER_PWD": --hidden for github
    "HTTP_DATA_SERVER": "http://ibl.flatironinstitute.org",
    "HTTP_DATA_SERVER_LOGIN": "iblmember",
    "HTTP_DATA_SERVER_PWD": --hidden for github
    "GLOBUS_CLIENT_ID": null
(ibllibenv) C:\Users\Experiment\Documents\GitFolder\ibllib\python>call run_tests.bat

(ibllibenv) C:\Users\Experiment\Documents\GitFolder\ibllib\python>cd .\ibllib\tests

(ibllibenv) C:\Users\Experiment\Documents\GitFolder\ibllib\python\ibllib\tests>python -m unittest discover
.EE
======================================================================
ERROR: test_io (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_io
Traceback (most recent call last):
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 434, in _find_test_path
    module = self._get_module_from_name(name)
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 375, in _get_module_from_name
    __import__(name)
  File "C:\Users\Experiment\Documents\GitFolder\ibllib\python\ibllib\tests\test_io.py", line 5, in <module>
    import ibllib.io.params as params
ModuleNotFoundError: No module named 'ibllib'


======================================================================
ERROR: test_time (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_time
Traceback (most recent call last):
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 434, in _find_test_path
    module = self._get_module_from_name(name)
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 375, in _get_module_from_name
    __import__(name)
  File "C:\Users\Experiment\Documents\GitFolder\ibllib\python\ibllib\tests\test_time.py", line 2, in <module>
    import ibllib.time
ModuleNotFoundError: No module named 'ibllib'


----------------------------------------------------------------------
Ran 3 tests in 0.000s

FAILED (errors=2)

(ibllibenv) C:\Users\Experiment\Documents\GitFolder\ibllib\python\ibllib\tests>cd ..\..\oneibl\tests

(ibllibenv) C:\Users\Experiment\Documents\GitFolder\ibllib\python\oneibl\tests>python -m unittest discover
EEE
======================================================================
ERROR: test_alyxclient (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_alyxclient
Traceback (most recent call last):
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 434, in _find_test_path
    module = self._get_module_from_name(name)
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 375, in _get_module_from_name
    __import__(name)
  File "C:\Users\Experiment\Documents\GitFolder\ibllib\python\oneibl\tests\test_alyxclient.py", line 4, in <module>
    import oneibl.webclient as wc
ModuleNotFoundError: No module named 'oneibl'


======================================================================
ERROR: test_one (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_one
Traceback (most recent call last):
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 434, in _find_test_path
    module = self._get_module_from_name(name)
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 375, in _get_module_from_name
    __import__(name)
  File "C:\Users\Experiment\Documents\GitFolder\ibllib\python\oneibl\tests\test_one.py", line 3, in <module>
    from oneibl.one import ONE
ModuleNotFoundError: No module named 'oneibl'


======================================================================
ERROR: test_one_tutorial (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_one_tutorial
Traceback (most recent call last):
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 434, in _find_test_path
    module = self._get_module_from_name(name)
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 375, in _get_module_from_name
    __import__(name)
  File "C:\Users\Experiment\Documents\GitFolder\ibllib\python\oneibl\tests\test_one_tutorial.py", line 1, in <module>
    import oneibl.examples.tutorial_script
ModuleNotFoundError: No module named 'oneibl'


----------------------------------------------------------------------
Ran 3 tests in 0.000s

FAILED (errors=3)

(ibllibenv) C:\Users\Experiment\Documents\GitFolder\ibllib\python\oneibl\tests>cd..

(ibllibenv) C:\Users\Experiment\Documents\GitFolder\ibllib\python\oneibl>cd..

(ibllibenv) C:\Users\Experiment\Documents\GitFolder\ibllib\python>call run_tests.bat

(ibllibenv) C:\Users\Experiment\Documents\GitFolder\ibllib\python>cd .\ibllib\tests

(ibllibenv) C:\Users\Experiment\Documents\GitFolder\ibllib\python\ibllib\tests>python -m unittest discover
.EE
======================================================================
ERROR: test_io (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_io
Traceback (most recent call last):
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 434, in _find_test_path
    module = self._get_module_from_name(name)
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 375, in _get_module_from_name
    __import__(name)
  File "C:\Users\Experiment\Documents\GitFolder\ibllib\python\ibllib\tests\test_io.py", line 5, in <module>
    import ibllib.io.params as params
ModuleNotFoundError: No module named 'ibllib'


======================================================================
ERROR: test_time (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_time
Traceback (most recent call last):
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 434, in _find_test_path
    module = self._get_module_from_name(name)
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 375, in _get_module_from_name
    __import__(name)
  File "C:\Users\Experiment\Documents\GitFolder\ibllib\python\ibllib\tests\test_time.py", line 2, in <module>
    import ibllib.time
ModuleNotFoundError: No module named 'ibllib'


----------------------------------------------------------------------
Ran 3 tests in 0.000s

FAILED (errors=2)

(ibllibenv) C:\Users\Experiment\Documents\GitFolder\ibllib\python\ibllib\tests>cd ..\..\oneibl\tests

(ibllibenv) C:\Users\Experiment\Documents\GitFolder\ibllib\python\oneibl\tests>python -m unittest discover
EEE
======================================================================
ERROR: test_alyxclient (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_alyxclient
Traceback (most recent call last):
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 434, in _find_test_path
    module = self._get_module_from_name(name)
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 375, in _get_module_from_name
    __import__(name)
  File "C:\Users\Experiment\Documents\GitFolder\ibllib\python\oneibl\tests\test_alyxclient.py", line 4, in <module>
    import oneibl.webclient as wc
ModuleNotFoundError: No module named 'oneibl'


======================================================================
ERROR: test_one (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_one
Traceback (most recent call last):
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 434, in _find_test_path
    module = self._get_module_from_name(name)
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 375, in _get_module_from_name
    __import__(name)
  File "C:\Users\Experiment\Documents\GitFolder\ibllib\python\oneibl\tests\test_one.py", line 3, in <module>
    from oneibl.one import ONE
ModuleNotFoundError: No module named 'oneibl'


======================================================================
ERROR: test_one_tutorial (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_one_tutorial
Traceback (most recent call last):
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 434, in _find_test_path
    module = self._get_module_from_name(name)
  File "C:\Users\Experiment\AppData\Local\Continuum\miniconda3\lib\unittest\loader.py", line 375, in _get_module_from_name
    __import__(name)
  File "C:\Users\Experiment\Documents\GitFolder\ibllib\python\oneibl\tests\test_one_tutorial.py", line 1, in <module>
    import oneibl.examples.tutorial_script
ModuleNotFoundError: No module named 'oneibl'


----------------------------------------------------------------------
Ran 3 tests in 0.000s

FAILED (errors=3)

(ibllibenv) C:\Users\Experiment\Documents\GitFolder\ibllib\python\oneibl\tests>


one cache dir

Replicate ALF folder organization on the cache dir

Handle wrong subject labeling in training session

(base) C:\iblrig>EXTRACT_REGISTER_FLATIRON.bat
Connected to https://alyx.internationalbrainlab.org as anneu
2019-02-22 12:08:16.283 INFO     [registration.py:60] registeringD:\globus_server\Subjects\CSHL_004\2019-02-21\001
2019-02-22 12:08:18.236 INFO     [registration.py:162] https://alyx.internationalbrainlab.org/sessions/65be29fb-bcd3-46b6-af1d-08f39d7438ee
2019-02-22 12:08:18.243 WARNING  [registration.py:185] No matching dataset type for: D:\globus_server\Subjects\CSHL_004\2019-02-21\001\raw_behavior_data\_iblrig_micData.raw.wav
2019-02-22 12:08:18.244 INFO     [registration.py:205] Registering D:\globus_server\Subjects\CSHL_004\2019-02-21\001\raw_video_data\_iblrig_leftCamera.raw.mp4
2019-02-22 12:08:18.598 ERROR    [webclient.py:173] https://alyx.internationalbrainlab.org/register-file
2019-02-22 12:08:18.674 ERROR    [webclient.py:174] ValueError at /register-file
A base session for CSHL_004 on 2019-02-21 does not exist

Request Method: POST
Request URL: https://alyx.internationalbrainlab.org/register-file
Django Version: 2.1.1
Python Executable: /usr/bin/python3
Python Version: 3.5.2
Python Path: ['/var/www/alyx-main/alyx', '/var/www/alyx-main/venv/lib/python35.zip', '/var/www/alyx-main/venv/lib/python3.5', '/var/www/alyx-main/venv/lib/python3.5/plat-x86_64-linux-gnu', '/var/www/alyx-main/venv/lib/python3.5/lib-dynload', '/usr/lib/python3.5', '/usr/lib/python3.5/plat-x86_64-linux-gnu', '/var/www/alyx-main/venv/lib/python3.5/site-packages']
Server time: Fri, 22 Feb 2019 17:08:19 +0000
Installed Applications:
('test_without_migrations',
 'dal',
 'dal_select2',
 'reversion',
 'django_admin_listfilter_dropdown',
 'rangefilter',
 'subjects',
 'django_filters',
 'django.contrib.admin',
 'django.contrib.contenttypes',
 'django.contrib.auth',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'polymorphic',
 'rest_framework',
 'rest_framework.authtoken',
 'rest_framework_docs',
 'actions',
 'data',
 'misc')
Installed Middleware:
('django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'alyx.base.QueryPrintingMiddleware')


Traceback:

File "/var/www/alyx-main/venv/lib/python3.5/site-packages/django/core/handlers/exception.py" in inner
  34.             response = get_response(request)

File "/var/www/alyx-main/venv/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response
  126.                 response = self.process_exception_by_middleware(e, request)

File "/var/www/alyx-main/venv/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response
  124.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/var/www/alyx-main/venv/lib/python3.5/site-packages/django/views/decorators/csrf.py" in wrapped_view
  54.         return view_func(*args, **kwargs)

File "/var/www/alyx-main/venv/lib/python3.5/site-packages/rest_framework/viewsets.py" in view
  103.             return self.dispatch(request, *args, **kwargs)

File "/var/www/alyx-main/venv/lib/python3.5/site-packages/rest_framework/views.py" in dispatch
  483.             response = self.handle_exception(exc)

File "/var/www/alyx-main/venv/lib/python3.5/site-packages/rest_framework/views.py" in handle_exception
  443.             self.raise_uncaught_exception(exc)

File "/var/www/alyx-main/venv/lib/python3.5/site-packages/rest_framework/views.py" in dispatch
  480.             response = handler(request, *args, **kwargs)

File "/var/www/alyx-main/alyx/data/views.py" in create
  251.             subject=subject, date=date, number=session_number, user=user)

File "/var/www/alyx-main/alyx/data/models.py" in _get_session
  26.         raise ValueError("A base session for %s on %s does not exist" % (subject, date))

Exception Type: ValueError at /register-file
Exception Value: A base session for CSHL_004 on 2019-02-21 does not exist
Request information:
USER: anneu

GET: No GET data

POST: No POST data

FILES: No FILES data

COOKIES: No cookie data

META:
CONTENT_LENGTH = '150'
CONTENT_TYPE = 'application/json'
CONTEXT_DOCUMENT_ROOT = '/var/www/html'
CONTEXT_PREFIX = ''
DOCUMENT_ROOT = '/var/www/html'
GATEWAY_INTERFACE = 'CGI/1.1'
HTTP_ACCEPT = 'application/json'
HTTP_ACCEPT_ENCODING = 'gzip, deflate'
HTTP_AUTHORIZATION = 'Token e215b99c40073d8ada0fd01d1742c386eab3cc54'
HTTP_CONNECTION = 'keep-alive'
HTTP_HOST = 'alyx.internationalbrainlab.org'
HTTP_USER_AGENT = 'python-requests/2.21.0'
PATH_INFO = '/register-file'
PATH_TRANSLATED = '/var/www/alyx-main/alyx/alyx/wsgi.py/register-file'
QUERY_STRING = ''
REMOTE_ADDR = '143.48.140.98'
REMOTE_PORT = '49622'
REQUEST_METHOD = 'POST'
REQUEST_SCHEME = 'https'
REQUEST_URI = '/register-file'
SCRIPT_FILENAME = '/var/www/alyx-main/alyx/alyx/wsgi.py'
SCRIPT_NAME = ''
SERVER_ADDR = '172.31.1.209'
SERVER_ADMIN = '[email protected]'
SERVER_NAME = 'alyx.internationalbrainlab.org'
SERVER_PORT = '443'
SERVER_PROTOCOL = 'HTTP/1.1'
SERVER_SIGNATURE = '<address>Apache/2.4.18 (Ubuntu) Server at alyx.internationalbrainlab.org Port 443</address>\n'
SERVER_SOFTWARE = 'Apache/2.4.18 (Ubuntu)'
SSL_TLS_SNI = 'alyx.internationalbrainlab.org'
apache.version = '(2, 4, 18)'
mod_wsgi.application_group = 'alyx.internationalbrainlab.org|'
mod_wsgi.callable_object = 'application'
mod_wsgi.connection_id = 'lvI1o96PsB4'
mod_wsgi.daemon_connects = '1'
mod_wsgi.daemon_restarts = '0'
mod_wsgi.daemon_start = '1550855299420641'
mod_wsgi.enable_sendfile = '0'
mod_wsgi.handler_script = ''
mod_wsgi.ignore_activity = '0'
mod_wsgi.listener_host = ''
mod_wsgi.listener_port = '443'
mod_wsgi.path_info = '/register-file'
mod_wsgi.process_group = 'alyxmain'
mod_wsgi.queue_start = '1550855299420498'
mod_wsgi.request_handler = 'wsgi-script'
mod_wsgi.request_id = 'plA4o96PsB4'
mod_wsgi.request_start = '1550855299420326'
mod_wsgi.script_name = ''
mod_wsgi.script_reloading = '1'
mod_wsgi.script_start = '1550855299420771'
mod_wsgi.thread_id = 3
mod_wsgi.thread_requests = 494
mod_wsgi.total_requests = 1553
mod_wsgi.version = '(4, 6, 4)'
wsgi.errors = <_io.TextIOWrapper name='<wsgi.errors>' encoding='utf-8'>
wsgi.file_wrapper = ''
wsgi.input = <mod_wsgi.Input object at 0x7fcddd78bd50>
wsgi.input_terminated = True
wsgi.multiprocess = False
wsgi.multithread = True
wsgi.run_once = False
wsgi.url_scheme = 'https'
wsgi.version = '(1, 0)'

Settings:
Using settings module alyx.settings
ABSOLUTE_URL_OVERRIDES = {}
ADMINS = []
ALLOWED_HOSTS = ['localhost', 'alyx.internationalbrainlab.org']
APPEND_SLASH = True
AUTHENTICATION_BACKENDS = ['django.contrib.auth.backends.ModelBackend']
AUTH_PASSWORD_VALIDATORS = '********************'
AUTH_USER_MODEL = 'misc.LabMember'
BASE_DIR = '/var/www/alyx-main/alyx'
CACHES = {'default': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'}}
CACHE_MIDDLEWARE_ALIAS = 'default'
CACHE_MIDDLEWARE_KEY_PREFIX = '********************'
CACHE_MIDDLEWARE_SECONDS = 600
CSRF_COOKIE_AGE = 31449600
CSRF_COOKIE_DOMAIN = None
CSRF_COOKIE_HTTPONLY = False
CSRF_COOKIE_NAME = 'csrftoken'
CSRF_COOKIE_PATH = '/'
CSRF_COOKIE_SAMESITE = 'Lax'
CSRF_COOKIE_SECURE = False
CSRF_FAILURE_VIEW = 'django.views.csrf.csrf_failure'
CSRF_HEADER_NAME = 'HTTP_X_CSRFTOKEN'
CSRF_TRUSTED_ORIGINS = []
CSRF_USE_SESSIONS = False
DATABASES = {'default': {'OPTIONS': {}, 'ENGINE': 'django.db.backends.postgresql_psycopg2', 'NAME': 'ibl', 'CONN_MAX_AGE': 0, 'HOST': '127.0.0.1', 'TEST': {'MIRROR': None, 'NAME': None, 'CHARSET': None, 'COLLATION': None}, 'ATOMIC_REQUESTS': False, 'USER': 'ibl_dev', 'PASSWORD': '********************', 'PORT': '5432', 'TIME_ZONE': None, 'AUTOCOMMIT': True}, 'cortexlab': {'OPTIONS': {}, 'ENGINE': 'django.db.backends.postgresql_psycopg2', 'NAME': 'cortexlab', 'CONN_MAX_AGE': 0, 'HOST': '127.0.0.1', 'TEST': {'MIRROR': None, 'NAME': None, 'CHARSET': None, 'COLLATION': None}, 'ATOMIC_REQUESTS': False, 'USER': 'ibl_dev', 'PASSWORD': '********************', 'PORT': '5432', 'TIME_ZONE': None, 'AUTOCOMMIT': True}}
DATABASE_ROUTERS = []
DATA_UPLOAD_MAX_MEMORY_SIZE = 2621440
DATA_UPLOAD_MAX_NUMBER_FIELDS = 10000
DATETIME_FORMAT = 'N j, Y, P'
DATETIME_INPUT_FORMATS = ['%Y-%m-%d %H:%M:%S', '%Y-%m-%d %H:%M:%S.%f', '%Y-%m-%d %H:%M', '%Y-%m-%d', '%m/%d/%Y %H:%M:%S', '%m/%d/%Y %H:%M:%S.%f', '%m/%d/%Y %H:%M', '%m/%d/%Y', '%m/%d/%y %H:%M:%S', '%m/%d/%y %H:%M:%S.%f', '%m/%d/%y %H:%M', '%m/%d/%y']
DATE_FORMAT = 'N j, Y'
DATE_INPUT_FORMATS = "('%d/%m/%Y',)"
DEBUG = True
DEBUG_PROPAGATE_EXCEPTIONS = False
DECIMAL_SEPARATOR = '.'
DEFAULT_CHARSET = 'utf-8'
DEFAULT_CONTENT_TYPE = 'text/html'
DEFAULT_EXCEPTION_REPORTER_FILTER = 'django.views.debug.SafeExceptionReporterFilter'
DEFAULT_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage'
DEFAULT_FROM_EMAIL = 'webmaster@localhost'
DEFAULT_INDEX_TABLESPACE = ''
DEFAULT_LAB_NAME = 'cortexlab'
DEFAULT_PROTOCOL = '1'
DEFAULT_SOURCE = 'IBL'
DEFAULT_TABLESPACE = ''
DISALLOWED_USER_AGENTS = []
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = 'smtp.gmail.com'
EMAIL_HOST_PASSWORD = '********************'
EMAIL_HOST_USER = '[email protected]'
EMAIL_PORT = 587
EMAIL_SSL_CERTFILE = None
EMAIL_SSL_KEYFILE = '********************'
EMAIL_SUBJECT_PREFIX = '[Django] '
EMAIL_TIMEOUT = None
EMAIL_USE_LOCALTIME = False
EMAIL_USE_SSL = False
EMAIL_USE_TLS = True
FILE_CHARSET = 'utf-8'
FILE_UPLOAD_DIRECTORY_PERMISSIONS = None
FILE_UPLOAD_HANDLERS = ['django.core.files.uploadhandler.MemoryFileUploadHandler', 'django.core.files.uploadhandler.TemporaryFileUploadHandler']
FILE_UPLOAD_MAX_MEMORY_SIZE = 2621440
FILE_UPLOAD_PERMISSIONS = None
FILE_UPLOAD_TEMP_DIR = None
FIRST_DAY_OF_WEEK = 0
FIXTURE_DIRS = []
FORCE_SCRIPT_NAME = None
FORMAT_MODULE_PATH = None
FORM_RENDERER = 'django.forms.renderers.DjangoTemplates'
GLOBUS_CLIENT_ID = '525cc517-8ccb-4d11-8036-af332da5eafd'
IGNORABLE_404_URLS = []
INSTALLED_APPS = "('test_without_migrations', 'dal', 'dal_select2', 'reversion', 'django_admin_listfilter_dropdown', 'rangefilter', 'subjects', 'django_filters', 'django.contrib.admin', 'django.contrib.contenttypes', 'django.contrib.auth', 'django.contrib.sessions', 'django.contrib.messages', 'django.contrib.staticfiles', 'polymorphic', 'rest_framework', 'rest_framework.authtoken', 'rest_framework_docs', 'actions', 'data', 'misc')"
INTERNAL_IPS = []
LANGUAGES = [('af', 'Afrikaans'), ('ar', 'Arabic'), ('ast', 'Asturian'), ('az', 'Azerbaijani'), ('bg', 'Bulgarian'), ('be', 'Belarusian'), ('bn', 'Bengali'), ('br', 'Breton'), ('bs', 'Bosnian'), ('ca', 'Catalan'), ('cs', 'Czech'), ('cy', 'Welsh'), ('da', 'Danish'), ('de', 'German'), ('dsb', 'Lower Sorbian'), ('el', 'Greek'), ('en', 'English'), ('en-au', 'Australian English'), ('en-gb', 'British English'), ('eo', 'Esperanto'), ('es', 'Spanish'), ('es-ar', 'Argentinian Spanish'), ('es-co', 'Colombian Spanish'), ('es-mx', 'Mexican Spanish'), ('es-ni', 'Nicaraguan Spanish'), ('es-ve', 'Venezuelan Spanish'), ('et', 'Estonian'), ('eu', 'Basque'), ('fa', 'Persian'), ('fi', 'Finnish'), ('fr', 'French'), ('fy', 'Frisian'), ('ga', 'Irish'), ('gd', 'Scottish Gaelic'), ('gl', 'Galician'), ('he', 'Hebrew'), ('hi', 'Hindi'), ('hr', 'Croatian'), ('hsb', 'Upper Sorbian'), ('hu', 'Hungarian'), ('ia', 'Interlingua'), ('id', 'Indonesian'), ('io', 'Ido'), ('is', 'Icelandic'), ('it', 'Italian'), ('ja', 'Japanese'), ('ka', 'Georgian'), ('kab', 'Kabyle'), ('kk', 'Kazakh'), ('km', 'Khmer'), ('kn', 'Kannada'), ('ko', 'Korean'), ('lb', 'Luxembourgish'), ('lt', 'Lithuanian'), ('lv', 'Latvian'), ('mk', 'Macedonian'), ('ml', 'Malayalam'), ('mn', 'Mongolian'), ('mr', 'Marathi'), ('my', 'Burmese'), ('nb', 'Norwegian Bokmål'), ('ne', 'Nepali'), ('nl', 'Dutch'), ('nn', 'Norwegian Nynorsk'), ('os', 'Ossetic'), ('pa', 'Punjabi'), ('pl', 'Polish'), ('pt', 'Portuguese'), ('pt-br', 'Brazilian Portuguese'), ('ro', 'Romanian'), ('ru', 'Russian'), ('sk', 'Slovak'), ('sl', 'Slovenian'), ('sq', 'Albanian'), ('sr', 'Serbian'), ('sr-latn', 'Serbian Latin'), ('sv', 'Swedish'), ('sw', 'Swahili'), ('ta', 'Tamil'), ('te', 'Telugu'), ('th', 'Thai'), ('tr', 'Turkish'), ('tt', 'Tatar'), ('udm', 'Udmurt'), ('uk', 'Ukrainian'), ('ur', 'Urdu'), ('vi', 'Vietnamese'), ('zh-hans', 'Simplified Chinese'), ('zh-hant', 'Traditional Chinese')]
LANGUAGES_BIDI = ['he', 'ar', 'fa', 'ur']
LANGUAGE_CODE = 'en-us'
LANGUAGE_COOKIE_AGE = None
LANGUAGE_COOKIE_DOMAIN = None
LANGUAGE_COOKIE_NAME = 'django_language'
LANGUAGE_COOKIE_PATH = '/'
LOCALE_PATHS = []
LOGGING = {'loggers': {'django': {'level': 'INFO', 'propagate': True, 'handlers': ['file']}}, 'version': 1, 'disable_existing_loggers': False, 'handlers': {'console': {'level': 'DEBUG', 'class': 'logging.StreamHandler', 'formatter': 'simple'}, 'file': {'filename': '/var/log/alyx.log', 'level': 'DEBUG', 'class': 'logging.handlers.RotatingFileHandler', 'maxBytes': 16777216, 'formatter': 'simple'}}, 'formatters': {'simple': {'datefmt': '%d/%m %H:%M:%S', 'log_colors': {'DEBUG': 'cyan', 'WARNING': 'yellow', 'ERROR': 'red', 'CRITICAL': 'bold_red', 'INFO': 'white'}, '()': 'colorlog.ColoredFormatter', 'format': '%(log_color)s%(asctime)s [%(levelname)s] {%(filename)s:%(lineno)s} %(message)s'}}, 'root': {'level': 'INFO', 'propagate': True, 'handlers': ['file', 'console']}}
LOGGING_CONFIG = 'logging.config.dictConfig'
LOGIN_REDIRECT_URL = '/accounts/profile/'
LOGIN_URL = '/accounts/login/'
LOGOUT_REDIRECT_URL = None
MANAGERS = []
MEDIA_ROOT = '/var/www/alyx-main/uploaded'
MEDIA_URL = '/uploaded/'
MESSAGE_STORAGE = 'django.contrib.messages.storage.fallback.FallbackStorage'
MIDDLEWARE = "('django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django.middleware.security.SecurityMiddleware', 'alyx.base.QueryPrintingMiddleware')"
MIGRATION_MODULES = {}
MONTH_DAY_FORMAT = 'F j'
NUMBER_GROUPING = 0
PASSWORD_HASHERS = '********************'
PASSWORD_RESET_TIMEOUT_DAYS = '********************'
PREPEND_WWW = False
REST_FRAMEWORK = {'STRICT_JSON': False, 'DEFAULT_FILTER_BACKENDS': ('django_filters.rest_framework.DjangoFilterBackend',), 'DEFAULT_AUTHENTICATION_CLASSES': ('rest_framework.authentication.SessionAuthentication', 'rest_framework.authentication.TokenAuthentication')}
ROOT_URLCONF = 'alyx.urls'
SECRET_KEY = '********************'
SECURE_BROWSER_XSS_FILTER = False
SECURE_CONTENT_TYPE_NOSNIFF = False
SECURE_HSTS_INCLUDE_SUBDOMAINS = False
SECURE_HSTS_PRELOAD = False
SECURE_HSTS_SECONDS = 0
SECURE_PROXY_SSL_HEADER = None
SECURE_REDIRECT_EXEMPT = []
SECURE_SSL_HOST = None
SECURE_SSL_REDIRECT = False
SERVER_EMAIL = 'root@localhost'
SESSION_CACHE_ALIAS = 'default'
SESSION_COOKIE_AGE = 1209600
SESSION_COOKIE_DOMAIN = None
SESSION_COOKIE_HTTPONLY = True
SESSION_COOKIE_NAME = 'sessionid'
SESSION_COOKIE_PATH = '/'
SESSION_COOKIE_SAMESITE = 'Lax'
SESSION_COOKIE_SECURE = False
SESSION_ENGINE = 'django.contrib.sessions.backends.db'
SESSION_EXPIRE_AT_BROWSER_CLOSE = False
SESSION_FILE_PATH = None
SESSION_SAVE_EVERY_REQUEST = False
SESSION_SERIALIZER = 'django.contrib.sessions.serializers.JSONSerializer'
SETTINGS_MODULE = 'alyx.settings'
SHORT_DATETIME_FORMAT = 'm/d/Y P'
SHORT_DATE_FORMAT = 'm/d/Y'
SIGNING_BACKEND = 'django.core.signing.TimestampSigner'
SILENCED_SYSTEM_CHECKS = []
STATICFILES_DIRS = []
STATICFILES_FINDERS = ['django.contrib.staticfiles.finders.FileSystemFinder', 'django.contrib.staticfiles.finders.AppDirectoriesFinder']
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage'
STATIC_ROOT = '/var/www/alyx-main/alyx/static/'
STATIC_URL = '/static/'
STOCK_MANAGERS = "('olivier',)"
SUBJECT_REQUEST_EMAIL_FROM = '[email protected]'
SUPERUSERS = 'root'
TEMPLATES = [{'APP_DIRS': True, 'OPTIONS': {'context_processors': ['django.template.context_processors.debug', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages']}, 'DIRS': ['/var/www/alyx-main/alyx/templates'], 'BACKEND': 'django.template.backends.django.DjangoTemplates'}]
TEST_NON_SERIALIZED_APPS = []
TEST_RUNNER = 'django.test.runner.DiscoverRunner'
THOUSAND_SEPARATOR = ','
TIME_FORMAT = 'P'
TIME_INPUT_FORMATS = ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M']
TIME_ZONE = 'GB'
UPLOADED_IMAGE_WIDTH = 800
USE_I18N = False
USE_L10N = False
USE_THOUSAND_SEPARATOR = False
USE_TZ = False
USE_X_FORWARDED_HOST = False
USE_X_FORWARDED_PORT = False
WEIGHT_THRESHOLD = 0.75
WSGI_APPLICATION = 'alyx.wsgi.application'
X_FRAME_OPTIONS = 'SAMEORIGIN'
YEAR_MONTH_FORMAT = 'F Y'


You're seeing this error because you have DEBUG = True in your
Django settings file. Change that to False, and Django will
display a standard page generated by the handler for this status code.


Traceback (most recent call last):
  File "C:\ibllib\python\alf\one_iblrig.py", line 106, in <module>
    register(args.folder, dry=args.dry)
  File "C:\ibllib\python\alf\one_iblrig.py", line 33, in register
    rc.register_sync(root_data_folder, dry=dry)
  File "c:\ibllib\python\oneibl\registration.py", line 61, in register_sync
    status_str = self.register_session(flag_file.parent, file_list=file_list)
  File "c:\ibllib\python\oneibl\registration.py", line 24, in func_wrapper
    f = func(self, sessionpath, *args, **kwargs)
  File "c:\ibllib\python\oneibl\registration.py", line 213, in register_session
    self.one.alyx.post('/register-file', data=r_)
  File "c:\ibllib\python\oneibl\webclient.py", line 262, in post
    return self._generic_request(requests.post, rest_query, data=data)
  File "c:\ibllib\python\oneibl\webclient.py", line 175, in _generic_request
    raise Exception(r)
Exception: <Response [500]>

(iblenv) C:\iblrig>




one.search

Thanks Cyrille for that last post. The conversation is now shifting to a different (but related) topic, so i created a new issue.

Could we actually incorporate generic Django queries into one.search? The current example we have in the notebook is

eid, ses = one.search(users='olivier', date_range=['2018-08-24', '2018-08-24'])

If this was a django filter command on the sessions table, the first one (users='olivier') would I think work exactly as intended. The second one would require a different syntax. The most common query - for a list of required dataset types that must all be associated to the session - I don't think can be done by a single django filter command.

But could we have something where there are a list of "reserved words" in the one.search command, which trigger pre-defined searches, and anything else goes through as a django filter command? So for example in

one.search(dataset_types=['spikes.times', 'spikes.clusters'], subject__virus_injection__virus_batch__contains='Pvalb')

the first argument would be intercepted by the one.search code, and the second would go right through as an additional Django filter?

Ambient sensor data access

They are on the Flatiron:

sessions, details = one.search(dataset_types='_iblrig_ambientSensorData.raw', lab='churchlandlab')
asd = one.load(sessions[0], dataset_types='_iblrig_ambientSensorData.raw')

There is one temperature/pressure value per trial.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.