logos.zip

Hi all,

Could you please tell us when the MRIQC web API will be available again? Thanks.

After some deliberation we had to limit the metadata fields that will be sent to this API. This was motivated by the fact that we should avoid a situation when a misused metadata field would accidentally include personal health information (see nipreps/mriqc#464 (comment)).

The following are the only allowed metadata fields.

• ContrastBolusIngredient
• RepetitionTime
• TaskName
• Manufacturer
• ManufacturersModelName
• MagneticFieldStrength
• DeviceSerialNumber
• SoftwareVersions
• HardcopyDeviceSoftwareVersion
• ReceiveCoilName
• GradientSetType
• MRTransmitCoilSequence
• MatrixCoilMode
• CoilCombinationMethod
• PulseSequenceType
• PulseSequenceDetails
• NumberShots
• ParallelReductionFactorInPlane
• ParallelAcquisitionTechnique
• PartialFourier
• PartialFourierDirection
• PhaseEncodingDirection
• EffectiveEchoSpacing
• TotalReadoutTime
• EchoTime
• InversionTime
• SliceTiming
• SliceEncodingDirection
• NumberOfVolumesDiscardedByScanner
• NumberOfVolumesDiscardedByUser
• DelayTime
• FlipAngle
• MultibandAccelerationFactor
• Instructions
• TaskDescription
• CogAtlasID
• CogPOID
• InstitutionName
• InstitutionAddress
• ConversionSoftware
• ConversionSoftwareVersion
• md5sum
• modality
• mriqc_pred
• software
• subject_id
• version

Please adjust the https://github.com/poldracklab/mriqcwebapi/blob/master/dockereve-master/eve-app/settings.py file.

Before August 31.

@rwblair Is the mriqc server down?
We are getting complaints that data cannot be uploaded or retrieved.

PUT should not require any authentication. Here's an example payload:

{
  "cjv": 0.7248942255973816,
  "cnr": 11.335847854614258,
  "efc": 0.47647998100292316,
  "fber": 26.33627700805664,
  "fwhm_avg": 3.47466,
  "fwhm_x": 3.50783,
  "fwhm_y": 3.70268,
  "fwhm_z": 3.22984,
  "icvs_csf": 0.2430190456707652,
  "icvs_gm": 0.3959101040697155,
  "icvs_wm": 0.3610708502595193,
  "inu_med": 1.2151005268096924,
  "inu_range": 0.35875076055526733,
  "metadata": {
    "AccelNumReferenceLines": 32,
    "AccelerationFactorPE": 2,
    "AcquisitionMatrix": "0/256/256/0",
    "DeviceSerialNumber": "35343",
    "EchoTime": 0.00331,
    "EchoTrainLength": 1,
    "FlipAngle": 7,
    "ImageType": "ORIGINAL/PRIMARY/M/ND/NORM",
    "ImagingFrequency": 123249959,
    "InPlanePhaseEncodingDirection": "ROW",
    "InversionTime": 1.1,
    "MRAcquisitionType": "3D",
    "MagneticFieldStrength": 3,
    "ManufacturerModelName": "TrioTim",
    "NumberOfAverages": 1,
    "NumberOfPhaseEncodingSteps": 255,
    "PatientPosition": "HFS",
    "PercentPhaseFieldOfView": 100,
    "PercentSampling": 100,
    "PhaseEncodingDirection": "i-",
    "PixelBandwidth": 200,
    "ProtocolName": "MPRAGE - BWM",
    "ReceiveCoilName": "HeadMatrix",
    "RepetitionTime": 2.53,
    "ScanOptions": "IR",
    "ScanningSequence": "GR/IR",
    "SequenceName": "*tfl3d1_ns",
    "SequenceVariant": "SP/MP",
    "SoftwareVersions": "syngo MR B15",
    "TotalScanTimeSec": 363,
    "TransmitCoilName": "Body",
    "VariableFlipAngleFlag": "N",
    "md5sum": "75cab8005361c2504ba5a7f02ecbacd7",
    "modality": "T1w",
    "mriqc_pred": 1,
    "software": "mriqc",
    "subject_id": "10159",
    "version": "99.99.99"
  },
  "qi_1": 5.289201412128623e-06,
  "qi_2": 0.0042996791690604694,
  "rpve_csf": 83261.9375,
  "rpve_gm": 152862.53125,
  "rpve_wm": 82412.921875,
  "size_x": 176,
  "size_y": 256,
  "size_z": 256,
  "snr_csf": 3.691214433820612,
  "snr_gm": 8.794440881363833,
  "snr_total": 7.911386668025524,
  "snr_wm": 11.248504688892124,
  "snrd_csf": 26.613233468186966,
  "snrd_gm": 44.12607332564048,
  "snrd_total": 43.8475075764988,
  "snrd_wm": 60.80321593566894,
  "spacing_x": 1.0,
  "spacing_y": 1.0,
  "spacing_z": 1.0,
  "summary_bg_k": 177.84711440664685,
  "summary_bg_mean": 4.691673278808594,
  "summary_bg_p05": 0.0,
  "summary_bg_p95": 12.0,
  "summary_bg_stdv": 4.799960136413574,
  "summary_csf_k": 2.800222210830917,
  "summary_csf_mean": 126.74657440185547,
  "summary_csf_p05": 68.0,
  "summary_csf_p95": 170.0,
  "summary_csf_stdv": 32.83399963378906,
  "summary_gm_k": -0.6324611944326,
  "summary_gm_mean": 211.44973754882812,
  "summary_gm_p05": 167.0,
  "summary_gm_p95": 260.0,
  "summary_gm_stdv": 28.741302490234375,
  "summary_wm_k": -0.15745795522916461,
  "summary_wm_mean": 309.4393005371094,
  "summary_wm_p05": 255.0,
  "summary_wm_p95": 368.0,
  "summary_wm_stdv": 35.2022590637207,
  "tpm_overlap_csf": 0.21703721582889557,
  "tpm_overlap_gm": 0.479917049407959,
  "tpm_overlap_wm": 0.5890556573867798,
  "wm2max": 0.6761487964989059
}

Here are some example JSONs you can use for testing: https://drive.google.com/file/d/0B77zr9yIiKOTaUZQWWE4NjRqM0U/view?usp=drivesdk

I've seen the above message for a few different runs recently (v22.0.0rc1 ). I have no idea if this is a TACC issue or an mriqc issue but I thought I'd share it. Not a big deal for me because the runs still complete successfully. Here's the full error message:

QC metrics upload failed to create an ID for the record uplOADED. rEsponse from server follows:

<title>502 Proxy Error</title>

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request POST /api/v1/bold.

Reason: Error reading from remote server

220429-00:13:14,560 nipype.interface WARNING:
QC metrics failed to upload. Status 502:

<title>502 Proxy Error</title>

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request POST /api/v1/bold.

Reason: Error reading from remote server

https://github.com/poldracklab/mriqcwebapi/blob/Conrad/dockereve-master/eve-app/settings.py

Current tests check POST and GET for individual images. We should also test doing GET at the main endpoint that should return all images.

After some deliberation we had to limit the metadata fields that will be sent to this API. This was motivated by the fact that we should avoid a situation when a misused metadata field would accidentally include personal health information (see nipreps/mriqc#464 (comment)).

The following are the only allowed metadata fields.

• ContrastBolusIngredient
• RepetitionTime
• TaskName
• Manufacturer
• ManufacturersModelName
• MagneticFieldStrength
• DeviceSerialNumber
• SoftwareVersions
• HardcopyDeviceSoftwareVersion
• ReceiveCoilName
• GradientSetType
• MRTransmitCoilSequence
• MatrixCoilMode
• CoilCombinationMethod
• PulseSequenceType
• PulseSequenceDetails
• NumberShots
• ParallelReductionFactorInPlane
• ParallelAcquisitionTechnique
• PartialFourier
• PartialFourierDirection
• PhaseEncodingDirection
• EffectiveEchoSpacing
• TotalReadoutTime
• Timing Parameters
• EchoTime
• InversionTime
• SliceTiming
• SliceEncodingDirection
• NumberOfVolumesDiscardedByScanner
• NumberOfVolumesDiscardedByUser
• DelayTime
• FlipAngle
• MultibandAccelerationFactor
• Instructions
• TaskDescription
• CogAtlasID
• CogPOID
• InstitutionName
• InstitutionAddress
• ConversionSoftware
• ConversionSoftwareVersion
• md5sum
• modality
• mriqc_pred
• software
• subject_id
• version

Please adjust the https://github.com/poldracklab/mriqcwebapi/blob/master/dockereve-master/eve-app/settings.py file.

2017-12-12 19:21:47,966 interface:WARNING QC metrics failed to upload. Status 422: {"_error": {"code": 422, "message": "Insertion failure: 1 document(s) contain(s) error(s)"}, "_status": "ERR", "_issues": {"bids_meta": {"ImageType": "must be of string type"}}}
2017-12-12 19:23:00,331 interface:WARNING QC metrics failed to upload. Status 422: {"_error": {"code": 422, "message": "Insertion failure: 1 document(s) contain(s) error(s)"}, "_status": "ERR", "_issues": {"bids_meta": {"ImageType": "must be of string type"}}}
2017-12-12 19:24:10,697 interface:WARNING QC metrics failed to upload. Status 422: {"_error": {"code": 422, "message": "Insertion failure: 1 document(s) contain(s) error(s)"}, "_status": "ERR", "_issues": {"bids_meta": {"ImageType": "must be of string type"}}}

Recently dcm2niix started including ImageType fields in the JSON sidecars which are of the form:

    "ImageType": [
        "ORIGINAL",
        "PRIMARY",
        "M",
        "MB",
        "ND",
        "NORM",
        "MOSAIC"
    ]

Web API seems to not accept such data.

We should be encrypting all of the traffic for mriqc, but currently the proxy server points to http://34.224.227.208. We need to enable https connection on the mriqc end point server so that the proxy server can point to https://34.224.227.208.

Appears this field was removed from the schema awhile back:
6882f33

Wonder how the tests continued to pass for so long.

@frank-24 @394822740 Some questions:

• Is 5000 the default port for MRIQCWebAPI?
• How can I change the port where the endpoints are open?
• Is there a dynamic way to do it (meaning, without changing the docker-compose.yml file)

What would you like to see added in this software?

This is such a valuable resource, and I'm very grateful that it's still running (and so only getting better!). However, I think the REST interface can make it difficult to download the dataset. This is partly because of the limit on the number of records that can be downloaded in one request (meaning that someone wanting the full dataset needs to coordinate many requests). I was recently able to grab all of the bold data (over 1 million records!) and was struck that the resulting table was under 200 MB (that's an M!). A file that small seems like it could be relatively straightforward to distribute. So, this is a request for monthly (quarterly, semiannually) snapshots that are made available as a table.

Do you have any interest in helping implement the feature?

Yes

Additional information / screenshots

No response

Right now, we flatten the dictionary in the client side, making it impossible for the server to know if a field is an IQM or if it is under an special key ('metadata', 'settings' and 'warnings').

This is a limiting factor for both the client and the server, because it adds an unnecessary complexity into the client (e.g. this rather unreliable dictionary). But also limits the server: we designed the settings/software field so that, at some point, tools other than mriqc would upload their own IQMs. Right now, all the accepted keywords are hard-coded in both client and server: so only mriqc IQMs would be valid. Right now, it is a bit painful to add new IQMs: it requires a big deal of cohesion between mriqcwebapi and mriqc. And cohesion is generally a bad idea (I can see similar tensions and problems to those we have with niworkflows and nipype, trying to keep versions consistent).

The client should take care of not sending identifiable personal information. So it makes sense to hard-code a dictionary of BIDS metadata keys that can be copied to the JSON derivatives. That could be done with the list linked before, but restricted to only BIDS keys (so no md5sum in that list, for instance).

The server should take care of all the rest: split IQMs vs. metadata (and settings), validate the field, and double check no personal info is inserted. If the dictionary comes flattened, this is a lot harder.

Does it make sense?

Currently MRIQC Web API only supports structural (T1w) files. MRIQC also calculates quality of functional (BOLD) files. Here is an example payload of such file:

{
aor: 0.034428499999999994,
aqi: 0.0005224403999999999,
dvars_nstd: 10.191722418947368,
dvars_std: 1.0800233642105264,
dvars_vstd: 1.6403423489473683,
efc: 0.5638485423014671,
fber: 958.6884765625,
fd_mean: 0.16413075091049215,
fd_num: 4,
fd_perc: 20,
fwhm_avg: 40.5562,
fwhm_x: 37.569,
fwhm_y: 43.0101,
fwhm_z: 41.283,
gcor: 0.2162099838256836,
gsr_x: -0.004743137396872044,
gsr_y: 0.02154964953660965,
RepetitionTime: 2,
TaskName: "rhyme judgment",
md5sum: "bae64aa6144d8f5d19e56d4c28ff2bda",
modality: "bold",
software: "mriqc",
subject_id: "ds003s01",
task_id: "rhymejudgment",
version: "99.99.99",
size_t: 20,
size_x: 16,
size_y: 16,
size_z: 9,
snr: 1.1260140530360272,
spacing_tr: 2,
spacing_x: 12.5,
spacing_y: 12.5,
spacing_z: 16,
summary_bg_k: 10.199746320897564,
summary_bg_mean: 10.67599868774414,
summary_bg_p05: 3.4488043785095215,
summary_bg_p95: 21.972539520263666,
summary_bg_stdv: 5.769111156463623,
summary_fg_k: -1.6145134278870812,
summary_fg_mean: 290.6355895996094,
summary_fg_p05: 13.376275062561035,
summary_fg_p95: 631.4849853515625,
summary_fg_stdv: 238.1324462890625,
tsnr: 108.249267578125
}

As you can see it includes different, but overlapping set of measures. To support this data type the API schema needs to be adjusted.

I was hoping to pull data from the database with specific qualities (TR, MB, manufacturer etc.). I tried the following URL: https://mriqc.nimh.nih.gov/api/v1/bold?max_results=10&where=bids_meta.MultibandAccelerationFactor%3C4&RepetitionTime=2.0&Manufacturer=Siemens&MagneticFieldStrength=3

Although this runs, the results are not filtered by the BIDS meta data. Am I missing something about how to do this? Is this a planned feature? It would be very useful.

Thank you.

Hi, @oesteban @dbirman
After MRIQC (most recent docker version) finished succesfully on my small dataset, I would like to interprete the results based on the distribution of all the IQMs of my dataset and the shared dataset from mriqc web-api. When I am trying the code below, it is giving the below problems after a while. Can you tell me why this is happening? Thanks.

code:
`# T1
df_t1w = get_iqms('T1w', software=None)
df_t1w_unique = df_t1w.drop_duplicates(subset=['provenance.md5sum'])

BOLD

df_bold = get_iqms('bold')
df_bold_unique = df_bold.drop_duplicates(subset=['provenance.md5sum'])`

error message:
`HTTPError Traceback (most recent call last)
in
1 # T1
----> 2 df_t1w = get_iqms('T1w', software=None)
3 df_t1w_unique = df_t1w.drop_duplicates(subset=['provenance.md5sum'])
4
5 # BOLD

in get_iqms(modality, versions, software)
24 query='where={%s}&page=%d' % (','.join(query), page)
25 )
---> 26 with urllib.request.urlopen(page_url) as url:
27 data = json.loads(url.read().decode())
28 dfs.append(json_normalize(data['_items']))

/usr/local/miniconda/lib/python3.7/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
220 else:
221 opener = _opener
--> 222 return opener.open(url, data, timeout)
223
224 def install_opener(opener):

/usr/local/miniconda/lib/python3.7/urllib/request.py in open(self, fullurl, data, timeout)
529 for processor in self.process_response.get(protocol, []):
530 meth = getattr(processor, meth_name)
--> 531 response = meth(req, response)
532
533 return response

/usr/local/miniconda/lib/python3.7/urllib/request.py in http_response(self, request, response)
639 if not (200 <= code < 300):
640 response = self.parent.error(
--> 641 'http', request, response, code, msg, hdrs)
642
643 return response

/usr/local/miniconda/lib/python3.7/urllib/request.py in error(self, proto, *args)
567 if http_err:
568 args = (dict, 'default', 'http_error_default') + orig_args
--> 569 return self._call_chain(*args)
570
571 # XXX probably also want an abstract factory that knows when it makes

/usr/local/miniconda/lib/python3.7/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)
501 for handler in handlers:
502 func = getattr(handler, meth_name)
--> 503 result = func(*args)
504 if result is not None:
505 return result

/usr/local/miniconda/lib/python3.7/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)
647 class HTTPDefaultErrorHandler(BaseHandler):
648 def http_error_default(self, req, fp, code, msg, hdrs):
--> 649 raise HTTPError(req.full_url, code, msg, hdrs, fp)
650
651 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 502: Bad Gateway`

We are seeing this on OpenNeuro for some but not all jobs. Could be related to ECS setup or something with the mriqcwebapi

  2017-06-04 06:50:33,934 interface:WARNING QC metrics failed to upload. Status 1: QC metrics failed to upload due to connection error shown below:
  HTTPSConnectionPool(host='mriqc.nimh.nih.gov', port=443): Max retries exceeded with url: /api/v1/T1w (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f561dfa6278>: Failed to establish a new connection: [Errno 101] Network is unreachable',))
  2017-06-04 06:50:33,941 workflow:INFO [Job finished] jobname: UploadMetrics.a4 jobid: 20
  2017-06-04 06:51:10,542 interface:WARNING QC metrics failed to upload. Status 1: QC metrics failed to upload due to connection error shown below:
  HTTPSConnectionPool(host='mriqc.nimh.nih.gov', port=443): Max retries exceeded with url: /api/v1/T1w (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f55a9f1d0b8>: Failed to establish a new connection: [Errno 101] Network is unreachable',))
  2017-06-04 06:51:10,545 workflow:INFO [Job finished] jobname: UploadMetrics.a3 jobid: 46

@rwblair do you think this could be due to some limit of how many concurrent connections mriqcwebapi can handle right now?

http://python-eve.org/authentication.html#token-based-authentication

It seems that someone added .DS_store files to this repository by accident. You can learn how to get rid of them and avoid this happening again here: http://stackoverflow.com/questions/107701/how-can-i-remove-ds-store-files-from-a-git-repository