Problem with importing MEG datasets ds003352 & ds003922 with File-IO about fieldtrip HOT 8 CLOSED

Sure, no problem! Could you kindly point me to a single dataset in more detail so that I can do a selective download, and check out what might be going on. I suspect some recent changes to the low-level reading code. Could you confirm that you are using a very recent version of Fieldtrip, and perhaps that if you downgrade to a version from prior to november 2023 the reading works?

from fieldtrip.

Thank you so much! We experienced the issue using v20230613 and v20240111 on ds003352 found here: https://nemar.org/dataexplorer/detail?dataset_id=ds003352

Please let me know if any additional details are need.

from fieldtrip.

yes, I would like to know whether this occurs for a particular subject from that dataset. I'd rather not download 200 GB of data...

from fieldtrip.

This error is given when importing Subject 1, Session 1, Task Color Spiraks, Run 00

Opening raw data file /expanse/projects/nemar/openneuro/ds003352/sub-1/ses-01/meg/sub-1_ses-01_task-ColorSpirals_run-00_meg.fif...

from fieldtrip.

hi @asalazar4, I downloaded the referenced data file. To me it looks as if the datafile is corrupt. Specifically, when the low level function throws this error:

Error in fiff_read_raw_segment (line 160)
one = double(reshape(tag.data,nchan,this.nsamp));

the number of elements in the tag.data field do not contain the expected number of data points to be the product of nchan*this.nsamp.

from fieldtrip.

Hi @asalazar4 as per my previous comment, it could be that some of the data uploaded into the nemar repository are corrupted. We could consider building in some extra checks in the code for this, but that would require a bit of overhead in the lower level reading functions, and it would require work. Do you think such checks are necessary, and if so, who is going to implement this? @robertoostenveld what do you think?

from fieldtrip.

Note that all the files seem to be there. Also, you need not download the entire archive. You can download this file only. Based on the message, it could be that the data is truncated. We will add a comment to that dataset. Anyway, thank you for your answer.

from fieldtrip.

Yes, indeed as I said the file is corrupted. The low-level reading function has read fewer bytes than the number of bytes suggested by the corresponding tag. In other words, there were fewer bytes available than numbers-of-channels times number-of-samples. My suggestion was related to the fact that we can build in a quick check (with additional overhead) that can throw a more informative error message than just a reshape error

from fieldtrip.