gbook / nidb Goto Github PK

Hi Greg... Two more simple ideas for you to consider whether they’d be useful additions to ADO2. (In other words, neither are critically needed, just stuff I noticed along the way). The first is whether or not an email notification system might be a good thing to add back into the mix, e.g., once a script successfully/unsuccessfully finishes, a list of email addresses could be contacted with a standard notification.

Second... A potentially useful flexibility on the Pipeline Specification for Primary Data might be to... instead of having a flag up above (Data download) that specifies “or” “and” or “none”... Perhaps instead have options down in the data specification section that says “Click if MUST” be present vs. another box that says “Click to download IF it’s there” (i.e., but don’t prevent the processing of other data just ‘cause it’s not). For instance, I can picture the convenience of having a single pipeline that processes the fixed battery of scans that comes from someone’s paradigm. But what if someone completes all but 1 of those scans? Right now, you’d have to set up a completely separate pipeline (e.g., HCP) to do the processing... or do it by hand. But, by increasing the flexibility of the data specificiation, one can not only have the full flexibility of “OR”... But one can specify situations where absolutely required scans (e.g., fieldmaps) MUST be present, but that any and all fMRI data in that study could be run. Again, not a huge tweak, but maybe worth thinking about?

Maybe only give permissions to the super-user, but a button to remove an MR series (including the QA, etc) would be useful when cleaning merged datasets

When copying pipelines, the script copies lines multiple times

Create an option for a read-only user account. Meaning they cannot upload, edit, change, or delete existing data.

1. Needs to change /var/lib/php/session permissions during install. This directory needs to be owned by the same user that apache is running under.

2. Perl modules installed under the root directory /root/perl5/lib/perl5 are not accessible. These modules should be copied to one of the other perl directories that non-root users can read from.

It could start as a reverse data download, where the process is initiated by the local server, and downloads the data from the public server to the local one

1. Zip the download at end, not during the data pulling process. Expanding the .zip file takes more time the larger it gets
2. Provide another way to download large files other than over httpd. Difficult if the only port open is 80, however

Pipeline was saved with no name and could not click on it to be able to delete it.

Add a check box next to each data item to indicate whether it is required or optional.

When adding subjects to groups, add the date. This will require adding a field to the groups table. When repopulating the list of group members from the text box, the original dates should be retained, unless they are deleted.

Display if pipeline is enabled or disabled on all pipeline specific pages, including analysis

Under the MR QC tab, make it more of a summary for the data. Only display details about the MRI data if there is a QC problem. Kind of like a table that lists the QC issues at the top, the subjects on the left, and if they meet the QC criteria or not. Clicking the QC item will give details about the error.

Really, for all modalities.

It's not working when sending to remote NiDB servers

Multi-part change request to ratings:

• rename 1-5 ratings scale to something more meaningful
• allow deleting of existing ratings
• display average rating on search page instead of number of ratings

• (complete) Associate assessments with projects
• Allow copying of assessments between projects. Should create new, unpublished, assessment that is a copy of the original one. Allow them to change the name, also record which assessment it is derived from
• Fix creation of assessments
• Add scoring. Start with simple scoring (coding/categorization of variables), then sections, etc.
  • Store computed value(s) to allow for searching of computed values. Also allow for option of re-scoring if scoring formula changes
• Fix display of data-entry forms, and display of completed forms
• Add searching of assessments, by question and by sub-score, complete score

Add the galvanic skin response modality

Sometimes data downloaded through Linux NFS has 0755 permissions instead of 0777.

During import from the uploader, and possibly the inter-instance transfer, empty transactions are created. Possibly in the createtransaction() function in api.php. This doesn't appear to be a problem, as it is not causing data to be lost or not uploaded, but it is disconcerting to see empty transactions.

Add an option to create groups of subjects or studies from a text file. On the groups.php page, we should add an option for the user to enter a list of subjects or studies into a large text box when creating a group. It should probably check for validity of the subject UIDs (S1234ABC) and study IDs (S1234ABC1) when creating it, and list the IDs that it could not find.

while trying to upload DICOM data via NiDB uploader, everything goes well, all files get received successfully but can't get archived. I can't find any problem with anything.

Add alternate ID to the Search-->Table results

Add a criteria for the number of volumes. For example, only perform the analysis if the data has exactly, or at least N volumes. Otherwise mark the analysis as ignored with a note as to why.

Modify the system.php page to allow editing of the nidb.cfg file

beh data added with . if there is no extension with filename

if beh data file already exists - it adds random numbers to the new files added with the same name

rating repeats on a study page in all the following sequences

Hi Greg,

I would like to submit the following enhancement requests to NiDB:

[1]

To avoid confusion in the future, is it possible to add the internal ado2 and project IDs (e.g. 4237) as additional fields that can be viewed and downloaded in the file name / as metadata? This would mean that when someone searches for or downloads data a subject's data based on the Olin NiBD external project ID, the metadata would also include fields that showed the corresponding internal MRI ID, as well as the ADAPTING ID used by JB and the folks collecting the clinical and cognitive data at HH. This is important for matching data across external data pulls, internal MRI data logs and the logs of out-of-scanner testing at HH. I'm not sure if all the other projects have all three ID designations, but if they could be added for the AA Connectivity project, that would be extremely helpful.

[2]

Similarly, it would be good to easily tell if subject data had already been downloaded. Having a field that showed the date of last download or that indicated the data had not been downloaded would greatly enhance the efficiency of the manual curation of these data pull downs. If such a tag was included at the subject data set and individual MRI series level, that would be most advantageous.

Thanks!
-Craig.

If you create a child pipeline, you must still specify data to be downloaded, even if you don't need any data.

This should be changed so that the data spec can be blank if there is a parent pipeline.

Display a summary of demographics for the completed analyses for a pipeline.

The pipeline dependency tree is not displaying correctly on pipelines.php though the <select> box is displaying correctly

Subjects may be enrolled in different projects, and with each of those projects comes another ID. For example, Subject S1234ABC is enrolled in project A with ID "341790" and in project B with ID "GP582". Add a field to the subject_altuid table with enrollment_id and associate that way. The primary ID should be shown on the project summary pages with their respective projects.

Sometimes the username is not included in the email. It may be in the cookie, or for some reason is not set in the global variables by the time the SQL error occurs.

When merging subjects, the alternate IDs are lost

Create an interface to allow users to add/remove these protocol names

other modalities do not show correct project report

Add a button or link to the MR studies pages to allow an admin to reset the QA without needing to go into the database to do it.

The parser is not handling certain bash commands correctly

• If a hash # exists in a command and is valid, the parser will assume it is a comment, and separate it
• Lines ending in double parentheses () cannot have an output redirect, and require a {NOLOG} variable
• If a hash is the first character of a line, any space before the hash may delete the comment line
• Comment lines should have all tabs converted to spaces

If a data spec item exists in the database but the files don't exist, throw an error

Permissions are on a project-level basis. Add permissions for read-only write-only per project

Create "rdoc_uploads" table with the following fields:
rdocupload_id - int, primary index
project_id - int
modality - varchar(20)
series_id - int
dateuploaded - datetime
label - varchar(255)
iscomplete - bool

Add a tab to the Projects webpage to display the contents of the table for that particular project. Fist display the list grouped by label. Clicking the label will display a list of the subjects and series that were part of that upload group.

The table will be populated when the data is downloaded via the Search page, and will contain labels like "July2017-MR"

The fields are missing from the Add Project page.

For:

1. Pipeline analysis completion (daily summary)
2. maybe per-project studies done (daily summary)

There are several orphaned import directories and no pending imports. Most likely the orphans were marked as imported, but an error had actually occured.

On an action=viewanalyses page, I'd like to be able to check one analysis, then hold shift when clicking another analysis and "fill in" all the boxes in-between those two boxes with clicks. This allows for quick selection of many checkboxes.

This is especially useful for situations in which approximately half of analyses must be selected (e.g., to make links to only successful analyses, to delete unsuccessful ones, etc). In that situation, the "select all" button does not help.

Here is a userscript that essentially does what I am describing. It can no doubt be improved.

age-in-months for the NDAR exports is not correct

Don't join in the blank modality table

This sort of works now, but sent data is not archived correctly at the remote site, and is not matched by alternate ID.

When sending data from NiDB-A to NiDB-B, the alternate IDs on A should be sent and used to determine an existing subject on B.

Sometimes sends 0 or fewer studies than were collected.

I'm using the setup-centos7.sh script and it fails to copy dcmrcv near the end. Here's a simple patch:

diff --git a/setup/setup-centos7.sh b/setup/setup-centos7.sh
old mode 100644
new mode 100755
index 2cfa0fc..4b4a8bc
--- a/setup/setup-centos7.sh
+++ b/setup/setup-centos7.sh
@@ -221,7 +221,7 @@ echo "Installing dcm4che receiver to listen on port 8104"
 #read -p "Press [enter] to continue"

 # add dcmrcv service at boot
-cp dcmrcv /etc/init.d
+cp ${NIDBROOT}/install/programs/dcmrcv /etc/init.d
 chmod 755 /etc/init.d/dcmrcv
 chkconfig --add dcmrcv

• Some thumbnails for DICOMs are not being generated
• create script to generate thumbnails for previously not existant thumbs (static images only, not animated gifs)
• generate thumbnails for PAR/REC files

The "MRAdvancedStatsPhantom" and "MRAdvancedStatsHuman" QC scripts occasionally crash and hang. Killing the spawned fmriqa_phantomqa.pl processes allow the qc.pl module to contine running. The bug appears to be within dicom2xcede.