donders-institute / bidscoin Goto Github PK

To produce this bug:

1. Click on the 'edit' button in the main window of the bidseditor
2. Choose 'fmap' as a BIDS modality
3. Add some text in the 'IntendedFor' cell
4. Click 'OK'
5. Click on the 'edit' button of the edited/fmap sample
6. The IntendedFor cell is empty :-(

When you re-run the bidsmapper with new subjects with new bids modalities then they will be moved to extra data, whereas they should be taken from the template first

Plainly running:

bidscoiner.py [sourcefolder] [bidsfolder]

without first running the bidsmapper gives an error message:

No bidsmap file found: /opt/bidscoin/dev/heuristics/bidsmap.yaml

Which indeed there isn't:

$ ls -l /opt/bidscoin/2.3/heuristics/
total 48
-rw-r--r-- 1 marzwi mrphys 21065 Sep 19 18:53 bidsmap_dccn.yaml
-rw-r--r-- 1 marzwi mrphys 17141 Sep 19 18:53 bidsmap_template.yaml
-rw-r--r-- 1 marzwi mrphys    19 Aug 25 12:12 __init__.py

It would be nice to get an error that suggests to first run the bidsmapper. This error lead me to think that there might be a file internal to the bidsmapper that was missing.

Keep initial order after (multiple) edits.

Hi Marcel,

I've looked into bidscoin earlier today and played a bit with it, and I was positively surprised on how much it has grown and how easy it is to use!

Nonetheless, I noticed that for the data I was trying to sort, I had issues both with the supplied dicomsort and the bidseditor. The fact is that I'm acquired multi-echo MPRAGE data, and the dicomsort doesn't allow me to separate folders for each echo. After fixing that manually, I then tried to organize these data using the bidseditor. When selecting anat, however, I see that the echo variable is not available. Not a big deal again, since I could use acq for that, but I figured I'd reach out and report.

Regards,

Daniel

Add tests for the arguments of the tools in the Option-tab
For instance, clicking a test-button would try to subproces.run -> dcm2niix -h

note2issue

It could help users if they would be pointed to online(?) documentation at a few places. Here are some options:

• Main window
  • A help button that points to the README.md documents (e.g. on github), as show in the 'reset button' issue (#9), or perhaps render the document in a separate tab if it that is not to heavy (see e.g. https://github.com/AndrewRadev/quickmd)
• Edit window
  • A help button that points to the BIDS documentation, e.g. to https://bids-specification.readthedocs.io/en/stable/04-modality-specific-files/01-magnetic-resonance-imaging-data.html#anatomy-imaging-data for 'anat' and to https://bids-specification.readthedocs.io/en/stable/04-modality-specific-files/01-magnetic-resonance-imaging-data.html#diffusion-imaging-data for dwi, etc

It can find the fieldmaps on line 256
for jsonfile in glob.glob(os.path.join(bidsses, 'fmap', bidsname.replace('_run-1_','_run-[0-9]*_').replace(acqlabel,acqlabel+'[CE][0-9]*') + '.json')): # Account for multiple runs and dcm2niix suffixes inserted into the acquisition label

Hello,

When trying to edit the bidsmap in the editor, it crashes. command used:

~$ bidsmapper . -n pt organized/ -i 2

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/bidscoin/bidseditor.py", line 772, in handle_edit_button_clicked
    modality   = self.samples_table.item(rowindex, 2).text()
AttributeError: 'NoneType' object has no attribute 'text'
Abort trap: 6

Mac OSX 10.14.6
Python 3.9.0

I'm converting a lot of ADNI data, which is really variable in terms of sequences/acquisitions (Siemens, Phillips, GE). Getting this error quite a lot:

2020-09-25 09:00:03 - bidscoin - INFO ------------------- Subject 62/713 -------------------
2020-09-25 09:00:03 - bidscoin - INFO Coining session: /90days/uqtshaw/ADNI_DATA/ADNI_FILTERED/sub-006_S_4485/ses-01
2020-09-25 09:00:16 - bidscoin - INFO Processing: /90days/uqtshaw/ADNI_DATA/ADNI_FILTERED/sub-006_S_4485/ses-01/MPRAGE
2020-09-25 09:00:16 - bidscoin - INFO Running: PATH=/data/lfs2/uqtshaw/tools/dcm2niix/build/bin:$PATH;dcm2niix -b y -z y -i n -f "sub-006S4485_ses-01_acq-Sagittal3DAcceleratedMPRAGE_run-1_T1w" -o "/90days/uqtshaw/ADNI_DATA/bids/sub-006S4485/ses-01/anat" "/90days/uqtshaw/ADNI_DATA/ADNI_FILTERED/sub-006_S_4485/ses-01/MPRAGE"
2020-09-25 09:00:22 - bidscoin - INFO Output:
Chris Rorden's dcm2niiX version v1.0.20200331  GCC5.4.0 (64-bit Linux)
Found 212 DICOM file(s)
Philips Scaling Values RS:RI:SS = 3.40928:0:0.00323077 (see PMC3998685)
Convert 1 DICOM as /90days/uqtshaw/ADNI_DATA/bids/sub-006S4485/ses-01/anat/sub-006S4485_ses-01_acq-Sagittal3DAcceleratedMPRAGE_run-1_T1w (256x256x211x1)
Compress: "/usr/bin/pigz" -b 960 -n -f -6 "/90days/uqtshaw/ADNI_DATA/bids/sub-006S4485/ses-01/anat/sub-006S4485_ses-01_acq-Sagittal3DAcceleratedMPRAGE_run-1_T1w.nii"
Slices not stacked: dimensions or bit-depth varies
Using RWVSlope:RWVIntercept = 3.40928:0
 Philips Scaling Values RS:RI:SS = 3.40928:0:0.00323077 (see PMC3998685)
Convert 211 DICOM as /90days/uqtshaw/ADNI_DATA/bids/sub-006S4485/ses-01/anat/sub-006S4485_ses-01_acq-Sagittal3DAcceleratedMPRAGE_run-1_T1wa (256x256x211x1)
Compress: "/usr/bin/pigz" -b 960 -n -f -6 "/90days/uqtshaw/ADNI_DATA/bids/sub-006S4485/ses-01/anat/sub-006S4485_ses-01_acq-Sagittal3DAcceleratedMPRAGE_run-1_T1wa.nii"
Conversion required 6.110341 seconds (0.639448 for core code).

Traceback (most recent call last):
  File "/data/home/uqtshaw/anaconda3/bin/bidscoiner", line 8, in <module>
    sys.exit(main())
  File "/data/home/uqtshaw/anaconda3/lib/python3.7/site-packages/bidscoin/bidscoiner.py", line 623, in main
    sesprefix    = args.sesprefix)
  File "/data/home/uqtshaw/anaconda3/lib/python3.7/site-packages/bidscoin/bidscoiner.py", line 539, in bidscoiner
    coin_data2bids(dataformat, session, bidsmap, bidsfolder, personals, subprefix, sesprefix)
  File "/data/home/uqtshaw/anaconda3/lib/python3.7/site-packages/bidscoin/bidscoiner.py", line 243, in coin_data2bids
    acq_time = dateutil.parser.parse(data['AcquisitionTime'])
  File "/data/home/uqtshaw/anaconda3/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 1374, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/data/home/uqtshaw/anaconda3/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 652, in parse
    raise ParserError("String does not contain a date: %s", timestr)
dateutil.parser._parser.ParserError: String does not contain a date:

Is there a way that I can fix this?
Thanks!

Hi all,

I was tasked to adapt the bidscoin to the needs to my laboratory. As the changes was quite significant,
I transformed bidscoin into more object oriented code (which is in theory more flexible). The readme files and licences are not changed, I will work on it next week.

I've put code in https://github.com/nbeliy/bidscoin, and example dataset
with all needed files in https://github.com/nbeliy/bidscoin_example

1. So, would you be interested in participating in further development of it?
2. Can I use the name bidscoin2? I would like to conserve bidscoin, as it follows the same logic and progression as your code, but interface and structure is changed considerably.

Main changes:

• Main class Module define an interface between bidscoin and different images formats
• Module daughter MRI define global MRI image characteristics (like list of modalities, JSON fields, bids entities etc...)
• Module grand-daughter Nifti_SPM12 defines how nifti files converted by SPM12 should be treated (it is just nii with dumped json file)
• Overall, daughter of Module defines BIDS propriety for specifics types of images, and grand-daughter defines how individual formats of given type are treated
• yaml file mirror this structure
• Individual runs contains json dictionary for additional entries for json sidecar files
• Matching strings for attributes supports regexp (they just passed to re)
• get_dynamic_field with double brackets supports prefixes, for example <<bids:task>> will retrieve value of task entity
• The intermediary step of sorting files now becomes mandatory together with sub and ses prefixes
• Improved support of plugins which is called at various points of coinage, making it more flexible
• The conversion of files is removed: the conversion tools are too versatile and laboratory dependent
• The GUI dissapeared (It took me too much time to adapt it), but is in plans

Cheers,
Nikita

The ruamel.yaml library sometimes writes invalid yaml files. Presumably this is caused by the library getting confused and starts inserting corrupted comment anchors when list items are added or deleted. See also #41 and #43

The ruamel.yaml library sometimes writes invalid YAML. It seems that this is caused by adding / replacing list items, which confuses the comments anchors. See also #41 and #43

If users mess up, they may want to go back to the start-up state. It would therefore be good to have a 'reload' / 'restore' button on the bottom of the main window, next to the 'save' button

Make windows/widgets scale properly.

Currently, run-item matching is performed with fnmatch, which is useful for strings. Add '<' and '>' range matching that is useful for numerical datatypes (e.g. EchoTime: ">80<100")

Hello! I am trying to test bidscoin on my laptop. I've done pip3 install bidscoin on Ubuntu18 and gave it a directory where I have my test dicom files. When I run bidsmapper, it pops up a warning dialog

It then tries to launch bidseditor.py(?) and it runs into the following error message.

hayashis@xps15:/tmp 127 bidsmapper.py test test-bids
2019-11-13 10:46:41 - bidscoin - INFO 
2019-11-13 10:46:41 - bidscoin - INFO -------------- START BIDSmapper ------------
2019-11-13 10:46:41 - bidscoin - INFO No bidsmap file found: /usr/local/lib/python3.6/dist-packages/heuristics/bidsmap.yaml
2019-11-13 10:46:41 - bidscoin - INFO Reading: /usr/local/lib/python3.6/dist-packages/heuristics/bidsmap_template.yaml
2019-11-13 10:46:41 - bidscoin - INFO Writing bidsmap to: /tmp/test-bids/code/bidscoin/bidsmap.yaml
2019-11-13 10:47:12 - bidscoin - INFO Opening the bidseditor
Traceback (most recent call last):
  File "/usr/local/bin/bidsmapper.py", line 350, in <module>
    interactive  = args.interactive)
  File "/usr/local/bin/bidsmapper.py", line 314, in bidsmapper
    gui.setupUi(mainwin, bidsfolder, rawfolder, bidsmapfile, bidsmap_new, copy.deepcopy(bidsmap_new), template, subprefix=subprefix, sesprefix=sesprefix)
  File "/usr/local/lib/python3.6/dist-packages/bidscoin/bidseditor.py", line 203, in setupUi
    self.set_tab_bidsmap()
  File "/usr/local/lib/python3.6/dist-packages/bidscoin/bidseditor.py", line 685, in set_tab_bidsmap
    samples_table.setRowCount(num_files)
UnboundLocalError: local variable 'num_files' referenced before assignment

"Because all the mapping information can be easily edited with the Graphical User Interface (GUI), BIDScoin requires no programming knowledge in order to use it."

GUI link does not work: https://github.com/Donders-Institute/bidscoin/blob/master/screenshots.html

For example,

1. Edit file
2. Select modality fmap
3. Change suffix from epi to phase2
4. Press OK
5. Edit same file once more
6. Suffix should read phase2, but shows epi

I just wanted to try things out and tried to run:

bidscoiner.py -b /opt/bidscoin/2.3/heuristics/bidsmap_template.yaml  [sourcefolder] [bidsfolder]

I had not previsouly run the bidsmapper and link to a template map that has not been trained on my data, so indeed gives an error message:

2019-09-20 11:17:43 - bidscoin - INFO Writing acquisition time data to: /project/3015003.04/Studies/DSC_AttentionStudy/BasePipeline/Subject11/Scans/Functional/NifTI/sub-011/sub-011_scans.tsv
Traceback (most recent call last):
  File "/opt/bidscoin/dev/bidscoin/bidscoiner.py", line 599, in <module>
    sesprefix    = args.sesprefix)
  File "/opt/bidscoin/dev/bidscoin/bidscoiner.py", line 516, in bidscoiner
    coin_dicom(session, bidsmap, bidsfolder, personals, subprefix, sesprefix)
  File "/opt/bidscoin/dev/bidscoin/bidscoiner.py", line 249, in coin_dicom
    bidsname    = bids.get_bidsname(subid, sesid, 'fmap', fieldmap)
  File "/opt/bidscoin/dev/bidscoin/bids.py", line 1045, in get_bidsname
    subid, sesid = get_subid_sesid(run['provenance'], subid, sesid, subprefix, sesprefix)
  File "/opt/bidscoin/dev/bidscoin/bids.py", line 1008, in get_subid_sesid
    dicompath = os.path.dirname(dicomfile)
  File "/opt/bidscoin/lib/python3.6/posixpath.py", line 156, in dirname
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType

The fact that it yields an error is unsurprising, yet it might be nice to give a clearer error of what's going wrong exaclty. At this moment I can't quite see if this might turn up in different circumstances where the tool is used correctly but the data is wrong.

This is in reference to #49

We're attempting to run the workflow on the tutorial data (to see how it compares to our data); however, the documentation for getting the test data seems to be geared towards the Donders Institute system (i.e. module add bidscoin). Is there another way for users to get access to the test data?

Thanks!

BIDS has strict rules about the characters that are allowed, so the user input should be checked against this using bids.py:cleanup_label(label)

The user has to double click to find out which fields are editable, it would be better if that would become clear from the interface, e.g. by making the background or or the font in that field in non-editable fields grey, compared to black/white in editable fields

The bidseditor currently can only show DICOM or PAR/REC run items. But the dataformat could be stored per run, making the editor more generic (the bidsmapper and bidscoiner can already handle mixed dataformats)

Hi and thank you so much for this great tool. I was convinced to use it after a little chat at the OHBM in Roma ;-)

Here is my little problem:
After upgrading to the latest version we have serious problems defining fmri-sequences in the bidsmapper. The reason seems to be that the DicomTag (0018,0088) Spacing Between Slices is slightly different for each functional run. For example one run is 3.600000000471292 the next is 3.6000000877783. So, is there a way to a) round this value or b) to ignore this attribute?

Thank you
Karl

This example bidsmap template (already containing the mappings for the standard DCCN MRI scan-protocols) could be used by the DCCN users and serve as an example for other institutes.

Thanks for helping me with the „SpacingBetweeSlices“ Problem - super fast;-)

I just wanted to give you a short feedback about a few problems I had using the updated version (I found a work-around for me - so I just wanted you to give the info)

1. I tried to bring a gradientfieldmap into the bids-format. Acquired on Siemens the output is magnitude1, magnitude2 and phasediff. Problem was that the EchoTime of the first magnitude was not correctly written into the .json file of the phasediff (TE1= null) I think there might be a problem with indexing the Echotimes? Echtime2 (from magnitude2) was correctly written.
2. I found out that it is a requirement to have the „run-x“ in the fieldmap . If you don’t define „run“ - intendedfor will not work.
3. After using the bidseditor I found that at the very end of the .ymal-file (Section Filesystem) participants: and session: were written in the same line. I had to put „session:“ into the next line and everything went smoothly.

Best,
Karl

Sometimes people don't want source data to be included / converted, even if it is in extra_data. So we would need an 'ignore' option, besides the bids.bidsmodalities and the bids.unknownmodality. The bidsmapper & bidscoiner would then just skip these series.

Add a tab where users can edit the bidsmap Options

When scrolling the edit window you can crash the interface and get this error:

Traceback (most recent call last):
File "/home/mrphys/marzwi/python/bidscoin/bidscoin/bidseditor.py", line 843, in selection_dropdown_change
self.target_suffix = self.allowed_suffixes[self.target_modality][0]
IndexError: list index out of range

Hi,

I used an Anaconda environment to run bidscoin.
I installed dcm2niix via conda and bidscoin via pip install in this environment.
I can run bidsmapper without a problem.
If I go to the obtions to test bidscoin it tells me:
BIDS-version: 1.1.0
BIDScoin-version: 2.2

The dcm2niix test is by default:
module add dcm2niix;

If I test this I the following error:
Failed to run module add dcm2niix;dcm2niix -h (errorcode 0)
I can change the path to "...anaconda3/envs/BIDS/bin/" where the dcm2niix file is located and then it passes the test.
Then I save this "bidsmap.yaml" file and try to run bidscoiner.py and get an error, which then doesn’t work.
The error message is attached.

Is the a possibility to solve this issue?

Thanks for your help,

Marco

bidscoiner_error.txt

Particpant and session label are taken from the foldername unless it is specified in the bidsmap.yaml to take it from a DICOM-fields (some institutes may do so). It is a setting at the session level rather than at the series-level (which are now editable by separate (series-level) edit buttons), so a different interface should be used for this

DICOM:
# --------------------------------------------------------------------------------
# DICOM key-value heuristics (DICOM fields that are mapped to the BIDS labels)
# --------------------------------------------------------------------------------
  participant_label:                              # A <<DICOM attribute>> that is used as participant_label instead of the subject-label from the sourcefolder
  session_label:                                  # A <<DICOM attribute>> that is used as session_label instead of the session-label from the sourcefolder

The value of DICOM key-value pairs needs to be editable since the values can be a list containing wildcards, see def bids.py:get_matching_dicomseries():

# Check if the attribute value matches with the info from the dicomfile
if attrvalue:
    if not dicomvalue:
        match = False
    elif isinstance(attrvalue, list):                     # The user-edited 'wildcard' option
        match = match and any([attrvalue_ in dicomvalue for attrvalue_ in attrvalue])

This is a hard problem that I didn't think about before when I thought about the new bidseditor workflow. It may even be necessary to re-run the bidsmapper after changing the modality :-(

The suffix should be freely editable for extra_data, as these are series not part of BIDS, but the user may want to make them BIDS-like. The opposite is true for the BIDS-modalities, where the suffix should comply to the standard. The possible suffices are first of all defined in the bidsmap_template.yaml, but also in the samples-tree (used by the bidstrainer): ./heuristics/samples/. E.g. for 'anat', the suffix (modality_label) can be 'angio', 'defacemask', etc, but for 'func' it can be 'bold', 'events', etc.

/home/mrphys/marzwi/python/bidscoin/heuristics/samples/
|-- README.txt
|-- anat
|   |-- FLAIR
|   |-- FLASH
|   |-- PD
|   |-- PDT2
|   |-- PDmap
|   |-- SWImagandphase
|   |-- T1map
|   |-- T1rho
|   |-- T1w
|   |-- T2map
|   |-- T2star
|   |-- T2w
|   |-- angio
|   |-- defacemask
|   |-- inplaneT1
|   `-- inplaneT2
|-- beh
|   |-- beh
|   |-- events
|   |-- physio
|   `-- stim
|-- dwi
|   |-- dwi
|   `-- sbref
|-- fmap
|   |-- epi
|   |-- fieldmap
|   |-- magnitude
|   |-- magnitude1
|   |-- magnitude2
|   |-- phase1
|   |-- phase2
|   `-- phasediff
|-- func
|   |-- bold
|   |-- events
|   |-- physio
|   |-- sbref
|   `-- stim
`-- pet
    `-- pet

Now the bidseditor does not handle sub and ses identifiers, e.g. is contains code like this
bids_name = bids.get_bidsname('', '', self.target_modality, self.target_series, run)

However, the bidsmap should be something like:
subid = bidsmap[SOURCE]['participant_label']
sesid = bidsmap[SOURCE]['session_label']
bids_name = bids.get_bidsname(subid, sesid, self.target_modality, self.target_series, run)

This can happen if you add a new protocol after you have done the bidsmapper. The last non-empty entry of the bidsmap is then used during bidscoiner runtime as unknownmodality / extra_data.

Solution:

1. Detect this during bidscoiner runtime, inform the user and use the template or skip the session
2. Always add an empty run at the end of the extra_data / unknownmodality

Hi,

I have data that was collected about five years ago across three sites.
Two of the sites outputted DICOM files and one outputted PAR files. I've managed to convert the DICOM files into BIDS format and bidscoin has been extremely helpful for converting the PAR/REC files.
I do have one issue though that I so far have been unable to solve and was wondering if anyone had any insight/had come across this before.
I have a single PAR/REC file pair for the Fieldmap images. Looking at the PAR header they only have one echo but the PAR file does report two image types (0 = magnitude and 3 = phase diff). I've noticed for the DICOMs you can add in image type which would be helpful for my bids mapping here, is this a functionality for the PAR files as well?

I am currently getting this error when converting my files;

2021-01-28 14:53:36 - bidscoin - WARNING Empty 'IntendedFor' fieldmap value in /Volumes/toshiba/DICOMS/BIDS/sub-3002/ses-1/fmap/sub-3002_ses-1_phasediff.json: the IntendedFor value of the bidsmap entry was empty
2021-01-28 14:53:36 - bidscoin - ERROR Could not find expected magnitude1 image associated with: /Volumes/toshiba/DICOMS/BIDS/sub-3002/ses-1/fmap/sub-3002_ses-1_phasediff.json
2021-01-28 14:53:36 - bidscoin - ERROR Could not find expected magnitude2 image associated with: /Volumes/toshiba/DICOMS/BIDS/sub-3002/ses-1/fmap/sub-3002_ses-1_phasediff.json
2021-01-28 14:53:36 - bidscoin - ERROR Cannot find and add valid EchoTime1=None and EchoTime2=None data to: /Volumes/toshiba/DICOMS/BIDS/sub-3002/ses-1/fmap/sub-3002_ses-1_phasediff.json

Thanks!

The README.md is also used as documentation on https://pypi.org/project/bidscoin/. However, the images which appear properly on github are not shown on PyPi.

I had the same issue with EEGsynth on https://github.com/eegsynth/eegsynth and https://pypi.org/project/eegsynth/ but did manage to solve that.

This dataset should contain all different kinds of scan-protocols so that:

1. Python routines can be tested against this
2. New users can practice and see if they can get the reference result themselves using the normal workflow

This dataset could then perhaps be hosted on the DR data sharing collection and be downloaded using a (to-be-developed) bidscoin tool

Write a session.tsv file for multi-session studies. See:
https://bids-specification.readthedocs.io/en/stable/05-longitudinal-and-multi-site-studies.html#sessions-file

Now DICOM and PAR/REC attributes are used to identify runs, add the possibility to identify runs based on their filename (e.g. "StopTask.dcm"

I had a raw directory structure that looked like this

raw
|-- sub-100261
|   `-- ses-mri01
|       `-- 002-t1_mpr_ns_PH8
|-- sub-100444
|   `-- ses-mri01
|       `-- 20150601_SUB16
|-- sub-100594
|   `-- ses-mri01
|       |-- 001-localizer_PH8
|       `-- 002-t1_mpr_ns_PH8
|-- sub-100903
|   `-- ses-mri01
|       |-- 001-localizer_PH8
|       `-- 002-t1_mpr_ns_PH8
...

and in one of these was a directory Header with in that directory an Analyze img file. I guess I had already removed the hdr file in an attempt to clean up and forgot the img, but would expect that not to really make a difference.

This caused bidsmapper to crash hard. When I removed the Header directory and content, all was fine.

You can reproduce this if you make a copy of /home/common/anatomical_mri/61978 and pass this through dicomsort.

In the file browser tab and/or in the edit window, it would be nice to have the option to inspect the full DICOM header, as in bids.py:get_dicomfield:
dicomdict = pydicom.dcmread(dicomfile, force=True) # The DICM tag may be missing for anonymized DICOM files

[Feature Request]
We have a bit of a problem keeping large numbers of DICOMS in our file servers before conversion. Is it possible to add support for reading tarballs of DICOMS into bidsmapper.py?
Thanks!

Append a textbox in the Options tab of the main window where users can edit and test the list of plug-ins