I already implemented the logging module to replace the homemade _verbose module. This is for referencing the commits

Try installing and running on Windows

.travis.yml file is configured to include two tests: one with coverage, one without. This is redundant because the tests are exactly the same.
The file is also not configured for doctests (there is a non-set environment variable for that).

Investigate what happens with BOLD VEM when we use block design. It seems like the HRF is not well estimated.

• Compare to MCMC version.

This is my feedback from using PyHRF. I tried using pyhrf in September-October 2013. I tried to apply it to the dataset described in the paper Kay et al. [1]. I'll describe here the usability issues I encountered with the hope that it will be useful.

• Huge number of conditions. This dataset has 374 per session (and 5 sessions). It is a huge pain to write that by hand to the XML file. I tried generating that with a script, which finally worked after quite some debugging to make the XML conform to what the program was expecting.
• I already had a script that uses nipy to perform the GLM and do some decoding afterwards. Being both Python packages I was hoping that I could just call a function from pyhrf to run pyhrfs's (hrf-estimation) GLM. Instead, (according to pyrhf documentation, more on that later) I had to break the workflow I had and use command line arguments. Not a blocking issue but the extra development time was unnecessary given that everything is Python code.
• Having worked with a pyhrf skilled user, I realized that it is possible to use pyhrf function call to achieve the HRF estimation. However, this is not documented in the website nor could I find it by inspection of the source code (there are many layers between the script and the useful functions).

As you see, all these problems could have easily solved by documented a function API for the package (an example on this API would have been just great!)

Hope that my comments are constructure and my apologies if I've missed something obvious. Looking forward to using pyhrf again in the future.

Best,

Fabian

[1] http://www.ncbi.nlm.nih.gov/pubmed/18322462

This bug comes from setuptools package newer versions.
There is no more finders list on setuptools.command.sdist.

All the non-stable code will move to the labs subfolder until it is stable enough to be in the "main package"

There is a issue related to $DISPLAY environment variable with matplotlib when running the tests from a non graphical terminal (e.g. from ssh or on the cluster).
On travis the issue is bypassed by using a matplotlibrc file setting the backend to Agg but IMHO, it should run without external fixes.

covariance matrix with different D matrices to take into account different levels of smoothness depending on the part of the HRF

I added a function with constraints in h for positivity and norm 1

coveralls.io badge is present but link to a non existing project

Currently pyhrf_view can opne only gzipped files. Extension is required for nii files

using nosetests --with-doctest -v pyhrf.ndarray:xndarray.get_domain, I get :

Doctest: pyhrf.ndarray.get_domain ... FAIL

======================================================================
FAIL: Doctest: pyhrf.ndarray.get_domain
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/doctest.py", line 2226, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for get_domain
  File "/home/tperret/code/pyhrf/python/pyhrf/ndarray.py", line 195, in get_domain

----------------------------------------------------------------------
File "/home/tperret/code/pyhrf/python/pyhrf/ndarray.py", line 202, in get_domain
Failed example:
    c.get_domain('y')
Expected:
    array(['plop', 'plip'],
          dtype='|S4')
Got:
    array(['plop', 'plip'], 
          dtype='|S4')


----------------------------------------------------------------------
Ran 1 test in 0.003s

FAILED (failures=1)

but the Expected and Got sections look identical.
In fact every doctests with numpy array and dtype argument seems to fail. (check nosetests --with-doctest -v pyhrf.ndarray command)

Probably should be two issues: attempt to build fails:

Installed /home/yoh/deb/perspect/pyhrf/pyhrf/INSTALL/lib/python2.7/site-packages/pyhrf-0.3-py2.7-linux-x86_64.egg
Processing dependencies for pyhrf==0.3
Searching for PyXML>=0.8.4
Reading http://pypi.python.org/simple/PyXML/
Best match: PyXML 0.8.4
Downloading http://downloads.sourceforge.net/pyxml/PyXML-0.8.4.tar.gz?modtime=1101741917&big_mirror=0
Processing PyXML-0.8.4.tar.gz
Writing /home/yoh/.tmp/easy_install-UkQJ5M/PyXML-0.8.4/setup.cfg
Running PyXML-0.8.4/setup.py -q bdist_egg --dist-dir /home/yoh/.tmp/easy_install-UkQJ5M/PyXML-0.8.4/egg-dist-tmp-jMvXA6
warning: no files found matching '*.html' under directory 'extensions/expat'
warning: no files found matching '*Makefile' under directory 'extensions/expat'
warning: no files found matching '*.dsp' under directory 'extensions/expat'
warning: no previously-included files matching '*/CVS/*' found anywhere in distribution
extensions/expat/lib/xmlparse.c:75:2: error: #error memmove does not exist on this platform, nor is a substitute available
 #error memmove does not exist on this platform, nor is a substitute available
  ^
error: Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
PYTHONPATH=$PWD/INSTALL/lib/python2.7/site-packages/ python setup.py install   19,97s user 1,41s system 90% cpu 23,592 total

PyXML project is long gone -- http://sourceforge.net/projects/pyxml/?source=directory says that it is no longer maintained, it was removed from Debian quite a while ago etc

I tried to launch a glm estimation using one session from a xml file previously defined:
pyhrf_glm_estim -c glmV4_2sess_MM31.xml
all data can be found here :
Your link: http://bfs.ujf-grenoble.fr/files/20bfddc8410820773cfaf4a6cfd5a485/Syn_MM_31_md.zip

I got warnings and the process is stopped
/Users/mdojat/Soft/pyhrf/python/pyhrf/jde/beta.py:3621: RuntimeWarning: underflow encountered in exp
temp = np.exp(log_P_NewBeta - log_P_CurrentBeta + log_g_cur - log_g_new)
/Users/mdojat/Soft/pyhrf/python/pyhrf/jde/beta.py:3621: RuntimeWarning: overflow encountered in exp
temp = np.exp(log_P_NewBeta - log_P_CurrentBeta + log_g_cur - log_g_new)

the following error trace is provided after CRTL C
File "/Users/mdojat/.local/bin/pyhrf_glm_estim", line 7, in
execfile(file)
File "/Users/mdojat/Soft/pyhrf/bin/pyhrf_glm_estim", line 9, in
'glm.xml', 'glm.prf','GLM')
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/ui/treatment.py", line 797, in run_pyhrf_cmd_treatment
treatment.run()
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/ui/treatment.py", line 219, in run
result = self.execute()
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/ui/treatment.py", line 205, in execute
result = self.analyser.analyse(self.data, self.output_dir)
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/ui/analyser_ui.py", line 137, in analyse
return [self.analyse_roi_wrap(d) for d in explodedData]
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/ui/analyser_ui.py", line 109, in analyse_roi_wrap
res = self.analyse_roi(roiData)
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/ui/jde.py", line 147, in analyse_roi
sampler.runSampling()
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/jde/samplerbase.py", line 232, in runSampling
v.sampleNext(self.variables)
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/jde/noise.py", line 309, in sampleNextInternal
varYtildeTdelta = np.dot(y_tilde[:,i],self.dataInput.delta)
KeyboardInterrupt

Make a specific command line for JDE analysis of ASL functional MRI data.

Fix description of output files in Volume_treatment

The functions that are currently being used will be in

vem_bold.py for the classical ones
vem_bold_constrained.py for the classical modified to add constraints in the computation of H
vem_asl_constrained.py for the functions for ASL data

The small functions of E-steps and M-steps and others are in
vem_tools.py for BOLD
vem_tools_asl.py for ASL

These modifications are already done on my branch cleaning-vem2.

Changes made in asl_physio and files for versions of 1-step and 2-steps considering differet priors

Changing user_mode option has no effect

I've tested tested minimal installation of pyhrf on a new installed Ubuntu 12.04.5 LTS and Fedora 20.
Installing minimal requirements is not sufficient for unit tests using pyhrf_maketests command.
There is an ImportError about joblib:

ERROR: test_cartesian_apply_parallel (pyhrf.test.toolsTest.CartesianTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tperret/.local/lib/python2.7/site-packages/pyhrf-0.3-py2.7-linux-x86_64.egg/pyhrf/test/toolsTest.py", line 173, in test_cartesian_apply_parallel
    nb_parallel_procs=4)
  File "/home/tperret/.local/lib/python2.7/site-packages/pyhrf-0.3-py2.7-linux-x86_64.egg/pyhrf/tools/misc.py", line 349, in cartesian_apply
    from joblib import Parallel, delayed
ImportError: No module named joblib

Different choices opened to discussion:

• Make joblib a required dependency (included as system package in the two distributions)
• Remove it from the tests (not a good idea)
• Test if joblib is available (this would be my choice)
• Add an option to skip the tests requiring joblib in $HOME/.pyhrf/config.cfg (note: there is an option concerning the number of processors to be used)

I tried to launch a glm estimation using one session from a xml file previously defined:
pyhrf_glm_estim -c glmV4_1sess_MM31.xml

I got apparently trouble with my xml file and mask create using pyhrf_mask_from_bold

all data can be found here :
Your link: http://bfs.ujf-grenoble.fr/files/20bfddc8410820773cfaf4a6cfd5a485/Syn_MM_31_md.zip
the following error trace is provided:
Traceback (most recent call last):
File "/Users/mdojat/.local/bin/pyhrf_glm_estim", line 7, in
execfile(file)
File "/Users/mdojat/Soft/pyhrf/bin/pyhrf_glm_estim", line 9, in
'glm.xml', 'glm.prf','GLM')
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/ui/treatment.py", line 742, in run_pyhrf_cmd_treatment
treatment = xmlio.fromXML(sXml)
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/xmlio/xmlbase.py", line 1075, in fromXML
return handler.parseXMLString(s)
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/xmlio/xmlbase.py", line 406, in parseXMLString
return xmlHandler.rootTagDOMReader(walker) # An XML document always starts with the root tag
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/xmlio/xmlbase.py", line 431, in rootTagDOMReader
data[name] = self.readDOMData(walker)
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/xmlio/xmlbase.py", line 472, in readDOMData
paramDict = self.tagDOMReaders[self.TYPE_LABEL_DICT](walker, self)
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/xmlio/xmlbase.py", line 1032, in dictTagDOMReader
result[nn.tagName] = xmlHandler.readDOMData(walker)
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/xmlio/xmlbase.py", line 543, in readDOMData
obj = eval('%s(*_paramDict)' %(func_str))
File "", line 1, in
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/tools/misc.py", line 369, in call
return self.fn(self.cls, *args, *_kwargs)
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/core.py", line 853, in from_vol_ui
m, mmo, mlf, b, ss = load_vol_bold_and_mask(bold_files, mask_file)
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/core.py", line 261, in load_vol_bold_and_mask
raise Exception("Mask is not n-ary (%s)" %mask_file)
Exception: Mask is not n-ary (/Users/mdojat/Data/data_SynesDec2013/MM31/Syn_MM_31_retino/439290030_701_EPI-Localizer-LL_20131205/sraSynes_MM_031_WIP_EPI-Localizer-LL_SENSE.nii)

The python-graph-core project was hosted on google code but since google code has close (and even if the project has migrated on github) the package can't be installed using pip.
Also removing this (optional) dependency is not a bad thing...

Note that the package can still be installed on Ubuntu (at least 14.04) via the python-pygraph.

TODO: check if the function from scipy is equivalent

edit: the graph functions are available from scipy>=0.11.0

Add possibility to run only one test, group of tests, profiling, using nose, using unittest, w/ or w/o doctests...

There are some import errors concerning sklearn (scikit-learn) when running nosetests with --with-doctest option.
IMHO the tests should run (or at least there should not be any errors) with minimal dependencies.

I see that setup.py got version changed from 0.2 to 0.3 and 0.3 is told to be released, but there is no tag, so it is difficult (if not impossible) to figure out what actually the 0.3 release was.

Try to import the paradigm parameters from SPM;mat file generated using SPM12.
We have 2 sessions of 3 events with 24 onsets each.
You have the SPM.MAT file here http://bfs.ujf-grenoble.fr/files/e24baa4dbc5becf323e67b07da6dd053/SPM.mat

cmd: pyhrf mdojat$ pyhrf_jde_buildcfg --spm-mat-file=SPM.mat

error mess:
Traceback (most recent call last):
File "/Users/mdojat/.local/bin/pyhrf_jde_buildcfg", line 7, in
execfile(file)
File "/Users/mdojat/Soft/pyhrf/bin/pyhrf_jde_buildcfg", line 106, in
fdata = ptr.parse_data_options(options) #de treatment.py
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/ui/treatment.py", line 574, in parse_data_options
data_fns[isess]))
IndexError: list index out of range

This depends on #84, #101, #86 and maybe #88

Contrast are not included in the configuration file

• Some of the dependencies check functions seem useless.
• the some_workflow package is badly detected (checks for soma.workflow but the correct name is soma_workflow)

Add physiological prior to VEM ASL

When I run pyhrf_validate pyhrf.validation.valid_beta_estim.ObsField2DTest.test_single_PFES_ML, sometimes the test fails and sometimes it pass:

$ pyhrf_validate pyhrf.validation.valid_beta_estim.ObsField2DTest.test_single_PFES_ML
generating potts ..., beta = 0.4
betaML: 0.5
F
======================================================================
FAIL: test_single_PFES_ML (pyhrf.validation.valid_beta_estim.ObsField2DTest)
PF estimation method : extrapolation scheme. ML on p(label|beta).
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tperret/code/pyhrf/python/pyhrf/validation/valid_beta_estim.py", line 167, in test_single_PFES_ML
    assert abs(be-beta) <= gridPace
AssertionError

----------------------------------------------------------------------
Ran 1 test in 0.048s

FAILED (failures=1)


$ pyhrf_validate pyhrf.validation.valid_beta_estim.ObsField2DTest.test_single_PFES_ML
generating potts ..., beta = 0.4
betaML: 0.4
.
----------------------------------------------------------------------
Ran 1 test in 0.048s

OK

This is a bug due to the wildcard import from pyhrf.core import * at the end of the main __init__.py file

paramiko package is optional but doctest option of nose tries to import it on the grid.py file.
Just adding a try/except should resolve it.

The issue is not visible on travis because I think it is installed by default on the travis-ci workers.

There should be two tests on travis-ci: one with minimal dependencies and one with optional dependencies (this is already done on a branch on my repo with e84b943)

create design matrix and run GLM

To remove PyQt dependencies we need to remove the viewer (pyqt3 and pyqt4 versions) and create a new repository for it.

Run pyhrf_maketests (or nosetests -v pyhrf) and check the /tmp folder

The standard way of organizing a python package is to put python code directly on package root folder.
The python folder is historical and not needed anymore.

Just found a bulk of pyhrf_ files in my /tmp -- please make sure to clean up (tempfile.TemporaryFile is a handy construct for that)

I tried to launch a jde estimation using one session or two sessions from a xml file previously defined:
pyhrf_jde_estim -c jdeV4_1sess_MM31.xml
pyhrf_jde_estim -c jdeV4_2sess_MM31.xml
all data can be found here :
Your link: http://bfs.ujf-grenoble.fr/files/20bfddc8410820773cfaf4a6cfd5a485/Syn_MM_31_md.zip

As suspect I got the similar errors to glm
One session:
Traceback (most recent call last):
File "/Users/mdojat/.local/bin/pyhrf_jde_estim", line 7, in
execfile(file)
File "/Users/mdojat/Soft/pyhrf/bin/pyhrf_jde_estim", line 9, in
DEFAULT_CFG_FILE, PROFILE_FILE,'JDE_MCMC')
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/ui/treatment.py", line 742, in run_pyhrf_cmd_treatment
treatment = xmlio.fromXML(sXml)
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/xmlio/xmlbase.py", line 1075, in fromXML
return handler.parseXMLString(s)
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/xmlio/xmlbase.py", line 406, in parseXMLString
return xmlHandler.rootTagDOMReader(walker) # An XML document always starts with the root tag
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/xmlio/xmlbase.py", line 431, in rootTagDOMReader
data[name] = self.readDOMData(walker)
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/xmlio/xmlbase.py", line 472, in readDOMData
paramDict = self.tagDOMReaders[self.TYPE_LABEL_DICT](walker, self)
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/xmlio/xmlbase.py", line 1032, in dictTagDOMReader
result[nn.tagName] = xmlHandler.readDOMData(walker)
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/xmlio/xmlbase.py", line 543, in readDOMData
obj = eval('%s(*_paramDict)' %(func_str))
File "", line 1, in
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/tools/misc.py", line 369, in call
return self.fn(self.cls, *args, *_kwargs)
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/core.py", line 853, in from_vol_ui
m, mmo, mlf, b, ss = load_vol_bold_and_mask(bold_files, mask_file)
File "/Users/mdojat/Soft/pyhrf/python/pyhrf/core.py", line 261, in load_vol_bold_and_mask
raise Exception("Mask is not n-ary (%s)" %mask_file)
Exception: Mask is not n-ary (/Users/mdojat/Data/data_SynesDec2013/MM31/Syn_MM_31_retino/439290030_701_EPI-Localizer-LL_20131205/sraSynes_MM_031_WIP_EPI-Localizer-LL_SENSE.nii)

Two sessions
/Users/mdojat/Soft/pyhrf/python/pyhrf/jde/beta.py:3621: RuntimeWarning: underflow encountered in exp
temp = np.exp(log_P_NewBeta - log_P_CurrentBeta + log_g_cur - log_g_new)
/Users/mdojat/Soft/pyhrf/python/pyhrf/jde/beta.py:3621: RuntimeWarning: overflow encountered in exp
temp = np.exp(log_P_NewBeta - log_P_CurrentBeta + log_g_cur - log_g_new)

If you perform installation of pyhrf using the following command line:
python setup.py develop --user
Then, you don't need to set the PYTHONPATH env variable because automatic search will be done in your local installation folder (~/.local/lib/pythonXX/site-packages/)

Lots of prints are displayed while running pyhrf_maketests, especially from the sandbox.
Make things quiet.

These scripts are used to make snapshots of the package but are not relevant in the repository. Let's move them to the wiki

csv files can be delimited by several different separators (space, comma, tabulation,…) and the standard python library provides a simple way of detecting it (csv.Sniffer().sniff method)