[Loris-dev] Import mri - scripts

[Sotirios Nikoloutsopoulos]

The "Loris-MRI Data Directory" should be in the environment file?

Στις Πέμ, 25 Ιουλ 2019 στις 12:59 π.μ., ο/η Sotirios Nikoloutsopoulos <
sotirisnik at gmail.com> έγραψε:

> The scripts are located in  /data/loris/bin/mri/. Somewhere I might have
> messed something up but i remember following the instructions in the
> wiki.For example when i tried to execute the batch_uploads_imageuploader
> perl script the line
>
> my $command = "$bin_dir/uploadNeuroDB/imaging_upload_file.pl "
>
> was pointing to an invalid path, for example: it was repeating a pattern
> and had LORIS instead of loris ( i think  the mistake was that in the wiki
> it suggested to use $PROJECT  for the name of the folder and i choosed
> 'loris'  https://github.com/aces/Loris/wiki/Imaging-Database ). See also
> the in image for the modification i did. Also i didn't find the section to
> config the $data_dir variable.
>
> Thank you,
>
> Sotirios
>
>
>
> Στις Τετ, 24 Ιουλ 2019 στις 6:05 μ.μ., ο/η Cecile Madjar <
> cecile.madjar at mcin.ca> έγραψε:
>
>> Hi Sotirios,
>>
>> It looks like you have one Config setting not set (based on the first
>> error message you got when running it the first time). You need to set a
>> value for "Loris-MRI Data Directory" in the Config module under the Imaging
>> pipeline section (from what I can see, it would be /data/loris/data/ for
>> your setup).
>>
>> The reason you got that error message the second time is that the
>> pipeline checks if the DICOM study was already inserted in the tarchive
>> table. Since you ran the study once already, the study is inserted into the
>> tarchive, hence the message. However, the pipeline did not complete as the
>> MINC files were not created and inserted into the files/parameter_file
>> table. In order to continue the insertion, you will then need to run the
>> tarchiveLoader command suggested in the terminal during the second run:
>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile
>> prod
>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar
>> -verbose
>>
>> Finally, I am a bit confused by your setup. Where are the scripts
>> located? In /data/loris/bin/mri or /data/loris/data? It feels like the
>> Config setting "LORIS-MRI code" in the Paths section was set to
>> "/data/LORIS/data" but it should have been set to the directory where the
>> scripts are located (most probably /data/loris/bin/mri).
>>
>> Hope this helps,
>>
>> Cécile
>>
>>
>> On Wed, Jul 24, 2019 at 10:55 AM Sotirios Nikoloutsopoulos <
>> sotirisnik at gmail.com> wrote:
>>
>>> Ok first i execute this command:
>>>
>>> "lorisadmin at hbp:/data/loris/bin/mri$  ./batch_uploads_imageuploader
>>> -profile prod < ~/Desktop/input.txt > log.txt
>>> Use of uninitialized value $data_dir in concatenation (.) or string at
>>> ./batch_uploads_imageuploader line 143.
>>> Use of uninitialized value $data_dir in concatenation (.) or string at
>>> ./batch_uploads_imageuploader line 143.
>>> Use of uninitialized value $_ in pattern match (m//) at
>>> ./batch_uploads_imageuploader line 147.
>>> base is DCC0000_258024_V01
>>>  path is /data/incoming/
>>>  type is .tar.gz
>>>  fullpath is /data/incoming/DCC0000_258024_V01.tar.gz
>>>
>>> source /tmp/ImagingUpload-17-43-ZUEFjb
>>> targetlocation /data/loris/data/tarchive
>>>
>>> Use of uninitialized value $data_dir in concatenation (.) or string at
>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256.
>>> Use of uninitialized value $data_dir in concatenation (.) or string at
>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284.
>>> Use of uninitialized value $data_dir in concatenation (.) or string at
>>> /data/loris/bin/mri/uploadNeuroDB/tarchive_validation.pl line 219.
>>> Use of uninitialized value $data_dir in concatenation (.) or string at
>>> /data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl line 270.
>>> Use of uninitialized value $data_dir in concatenation (.) or string at
>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711.
>>> Can't exec "mail": No such file or directory at
>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721.
>>> print() on closed filehandle MAIL at
>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722.
>>> print() on closed filehandle MAIL at
>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723.
>>> Use of uninitialized value $data_dir in concatenation (.) or string at
>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724.
>>> print() on closed filehandle MAIL at
>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724.
>>> print() on closed filehandle MAIL at
>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725.
>>> Use of uninitialized value $data_dir in concatenation (.) or string at
>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735.
>>>
>>>  No Mincs inserted
>>>
>>>
>>> The tarchiveLoader insertion script has failed.
>>> Running now the following command: /data/loris/data/uploadNeuroDB/
>>> imaging_upload_file.pl -profile prod -upload_id 3
>>> /data/incoming/DCC0000_258024_V01.tar.gz -verbose"
>>>
>>> I inserted the VisitLabel, but still the Mincs are not inserted and i am
>>> getting this error
>>>
>>> "lorisadmin at hbp:/data/loris/bin/mri$ /data/loris/data/uploadNeuroDB/
>>> imaging_upload_file.pl -profile prod -upload_id 3
>>> /data/incoming/DCC0000_258024_V01.tar.gz -verbose
>>>
>>>  find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name '__MACOSX'
>>> -delete
>>> Spool message is:
>>> The Scan for the uploadID 3 has already been run with tarchiveID: 7.
>>> To continue with the rest of the insertion pipeline, please run
>>> tarchiveLoader from a terminal as follows:
>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod
>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar
>>> -verbose"
>>>
>>> Which files should i delete? I thought that deleting the files from the
>>> tarchive was enough. And i executed the last proposed command to continue
>>> to rest of the insertion pipeline ( check the image ).
>>>
>>> Thank you
>>>
>>> Στις Τετ, 24 Ιουλ 2019 στις 5:05 μ.μ., ο/η Cecile Madjar <
>>> cecile.madjar at mcin.ca> έγραψε:
>>>
>>>> Hi Sotirios,
>>>>
>>>> Sorry I was not clear before. Every visits of candidates are created in
>>>> the session table with one entry per CandID/VisitLabel. So if the imaging
>>>> pipeline created visits in the session table (that are not attached to
>>>> instruments) and you want to clean your database completely, you can delete
>>>> entries in that table as well.
>>>>
>>>> Regarding your point 2, the imaging pipeline can create the visit in
>>>> the session table as long as the visit label is present in the
>>>> Visit_Windows table (which means the visit label stored in the PatientName
>>>> is a valid visit label).
>>>> However, in your prod file, if the
>>>> variable $subjectID{'createVisitLabel'} is set to 1 for candidates, the
>>>> visit label should be created in the Visit_Windows table I believe. Here
>>>> is the line of the prod file you would need to change
>>>> <https://github.com/aces/Loris-MRI/blob/81bae73ea6e86c9498519dadf574468ee1d992ca/dicom-archive/profileTemplate.pl#L59>
>>>> .
>>>> In general, we prefer populating the Visit_Windows table and set the
>>>> $subjectID{'createVisitLabel'} to 0 for candidate data to avoid insertion
>>>> of badly labelled MRI data that are complex to relabel.
>>>>
>>>> Finally, regarding the candidate creation, if you execute the following
>>>> steps, the pipeline should be able to create candidates:
>>>> - set in the Imaging Pipeline section of the config module the config
>>>> setting "Upload creation of candidates" to Yes
>>>> - manually transfer the scans to the LORIS server instead of uploading
>>>> it via the imaging browser
>>>> - run batch_imaging_upload.pl as you did until now.
>>>>
>>>> Hope this helps!
>>>> Best,
>>>>
>>>> Cécile
>>>>
>>>> On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos <
>>>> sotirisnik at gmail.com> wrote:
>>>>
>>>>> About my second question i replaced the name of the patient in the
>>>>> .dcm files and now i am getting an error that the visit label does not
>>>>> exist.
>>>>> I thought it was supposed to be automatically created.
>>>>>
>>>>> "Done adding archive info into database
>>>>>
>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile
>>>>> prod
>>>>> /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar
>>>>> -verbose
>>>>>  md5sum
>>>>> /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar
>>>>> PSCID is: DCC0000
>>>>>  CandID id: 258024
>>>>>  visit_label is: V1
>>>>> PSCID is: DCC0000
>>>>>  CandID id: 258024
>>>>>  visit_label is: V1
>>>>> candidate id 258024
>>>>>
>>>>>
>>>>> => No Visit labelVisit label does not exist
>>>>> Set centerID = 1
>>>>> PSCID is: DCC0000
>>>>>  CandID id: 258024
>>>>>  visit_label is: V1
>>>>> PSCID is: DCC0000
>>>>>  CandID id: 258024
>>>>>  visit_label is: V1
>>>>>
>>>>> Number of MINC files that will be considered for inserting into the
>>>>> database: 1
>>>>>
>>>>> log dir is /logs and log file is /logs/TarLoad-15-42-L6pocU.log
>>>>> PSCID is: DCC0000
>>>>>  CandID id: 258024
>>>>>  visit_label is: V1
>>>>> PSCID is: DCC0000
>>>>>  CandID id: 258024
>>>>>  visit_label is: V1
>>>>> candidate id 258024
>>>>>
>>>>>
>>>>> => No Visit label
>>>>> Cleaning up temp files: rm -rf
>>>>> /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*"
>>>>>
>>>>> Στις Τετ, 24 Ιουλ 2019 στις 3:38 μ.μ., ο/η Sotirios Nikoloutsopoulos <
>>>>> sotirisnik at gmail.com> έγραψε:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> 1) What are the records in the session table?
>>>>>>
>>>>>> 2) I tried to upload a non-phantom file like this
>>>>>>
>>>>>> input_file.txt:
>>>>>> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1
>>>>>>
>>>>>> Terminal:
>>>>>>  ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt >
>>>>>> log.txt
>>>>>>
>>>>>> and i get the following error in the log.txt file
>>>>>>
>>>>>> "Running now the following command: /data/loris/data/uploadNeuroDB/
>>>>>> imaging_upload_file.pl -profile prod -upload_id 7
>>>>>> /data/incoming/DCC0000_258024_V1.tar.gz -verbose
>>>>>>
>>>>>>  find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name '__MACOSX'
>>>>>> -delete
>>>>>> Spool message is:
>>>>>> The PatientName read from the DICOM header does not start with
>>>>>> DCC0000_258024_V1 from the mri_upload table
>>>>>>  "
>>>>>>
>>>>>> and the spool message is repeated for each .dcm file. Should i
>>>>>> preprocess every .dcm file and change the PatientName to
>>>>>> "DCC0000_258024_V1"?
>>>>>>
>>>>>> 3) As far as it concerns the candidate profile i have to create a new
>>>>>> one each time before i try to upload a file? Is there already a way to
>>>>>> create a candidate profile and get the DCCID and PSCID without the user
>>>>>> interface?
>>>>>>
>>>>>> Thank you,
>>>>>>
>>>>>> Sotirios
>>>>>>
>>>>>> Στις Πέμ, 18 Ιουλ 2019 στις 5:00 μ.μ., ο/η Cecile Madjar <
>>>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>>>
>>>>>>> Hi Sotirios,
>>>>>>>
>>>>>>> See answers to your questions below.
>>>>>>>
>>>>>>> About the 1st case we had a dicom with a blank studyUID, maybe
>>>>>>>> someone accidentally removed it.
>>>>>>>>
>>>>>>> That is strange. Something must have been done to the DICOM files. I
>>>>>>> have never seen a study with no StudyUID out of the scanner. Usually, you
>>>>>>> need to manually erase it with some tool. However, DICOMs never cease to
>>>>>>> surprise me...
>>>>>>>
>>>>>>>
>>>>>>>> Some other questions:
>>>>>>>> 1) What are the steps to remove all of my patients and uploads?:
>>>>>>>> because i want to try to reupload all the testing dicoms we had with a
>>>>>>>> clean database.
>>>>>>>>
>>>>>>>
>>>>>>> Release 21.0 of LORIS-MRI will offer a script to remove all the
>>>>>>> entries and files specific to an upload. However, this release is not yet
>>>>>>> out. Hopefully in the next few weeks.
>>>>>>> In the meantime, since your case is particular and that you just
>>>>>>> want to start with a clean database, the following deletes should work
>>>>>>> (hopefully I won't forget any tables):
>>>>>>> delete from mri_violations_log;
>>>>>>> delete from mri_protocol_violated_scans;
>>>>>>> delete from MRICandidateErrors;
>>>>>>> delete from parameter_file;
>>>>>>> delete from files_qcstatus; # (if you played with the QC part of the
>>>>>>> imaging browser for testing)
>>>>>>> delete from feedback_mri_comments; # (if you played with the QC part
>>>>>>> of the imaging browser for testing)
>>>>>>> delete from tarchive_series;
>>>>>>> delete from tarchive_files;
>>>>>>> delete from mri_upload;
>>>>>>> delete from tarchive;
>>>>>>> delete from mri_scanner;
>>>>>>> delete from candidate where PSCID="scanner";
>>>>>>>
>>>>>>> I don't know if you want to also delete entries in the session and
>>>>>>> candidate table too?
>>>>>>>
>>>>>>> 2) We do not need to create the profile of a patient ( are the
>>>>>>>> patients stored in the candidate table? because i can't see the ghosts ),
>>>>>>>> because will it automatically be created when importing a dicom?
>>>>>>>>
>>>>>>>
>>>>>>> I am not sure I understand fully this question. All candidates are
>>>>>>> stored in the candidate table. The phantom scans however are attached to a
>>>>>>> scanner candidate depending on where the scan happened. It is a bit of a
>>>>>>> weird concept that we have to redesign eventually but never got a chance to
>>>>>>> get to it.
>>>>>>>
>>>>>>> Hopefully this answered your question, otherwise, don't hesitate to
>>>>>>> let me know.
>>>>>>>
>>>>>>> 3) What is the difference between uploading a dicom as phantom
>>>>>>>> instead of importing as PSCCID_DCCID_VisitLabel?
>>>>>>>>
>>>>>>>
>>>>>>> When you upload the scan as PSCID_DCCID_VisitLabel, there is a check
>>>>>>> that makes sure the candidate IDs and visit label are valid at the time of
>>>>>>> upload.
>>>>>>> When you upload a scan as phantom, it expects that the DICOM field
>>>>>>> PatientName and uploaded filename contains the string "phantom". We
>>>>>>> enforced this behaviour on the imaging uploader side in recent releases but
>>>>>>> I can't remember which one. Probably the upcoming 21.0 release.
>>>>>>>
>>>>>>> All those verifications might seem cumbersome but they are here to
>>>>>>> ensure that the files inserted are all valid and labelled properly as it is
>>>>>>> a bit messy to have to delete files that were wrongly labelled. At least,
>>>>>>> the delete script present in release 21.0 will make this process easier but
>>>>>>> it is still good practice to verify those things.
>>>>>>>
>>>>>>>
>>>>>>>> Thank you,
>>>>>>>>
>>>>>>>
>>>>>>> With pleasure. Hopefully the answers to your questions will be
>>>>>>> helpful.
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Cécile
>>>>>>>
>>>>>>> PS: loris-dev mailing list: you will find the initial answer I gave
>>>>>>> Sotirios below. I forgot to cc the loris-dev in my earlier reply...
>>>>>>>
>>>>>>> Στις Τρί, 16 Ιουλ 2019 στις 12:02 π.μ., ο/η Cecile Madjar <
>>>>>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>>>>>
>>>>>>>>> Dear Sotirios,
>>>>>>>>>
>>>>>>>>> Thank you for reaching out. Since your email was already organized
>>>>>>>>> in points, I will reply directly below your questions below.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>>
>>>>>>>>> Cécile
>>>>>>>>>
>>>>>>>>> 1) What do we do when the StudyUID is blank on the header?
>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md
>>>>>>>>>> There solution provided at the table 3 does not guide us how to find the
>>>>>>>>>> information ( is possible ).
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The StudyUID is the field used in our pipeline at the moment to
>>>>>>>>> check whether a DICOM study was already inserted into LORIS with that same
>>>>>>>>> StudyUID since it is supposed to be unique for every single study.
>>>>>>>>> Currently, we do not support insertion of DICOM studies if they do not have
>>>>>>>>> a StudyUID associated with them.
>>>>>>>>>
>>>>>>>>> Is there a specific reason why the StudyUID is blank in the DICOM
>>>>>>>>> headers in your DICOM files? Did they go through some processes before
>>>>>>>>> upload to LORIS?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2) When i am about to upload a non-phanton case i need a 'Visit
>>>>>>>>>> Label'. Is there already a script that can create a new visit label for a
>>>>>>>>>> specific Candidate, so i can provide it as input afterwards to the
>>>>>>>>>>
>>>>>>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md )
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> I am not sure if the front-end allows for it but if you use
>>>>>>>>> batch_uploads_imageuploader, the insertion scripts could create the visit
>>>>>>>>> label for you. However, it does need to be included in the patient name in
>>>>>>>>> order for the script to know which label it should use.
>>>>>>>>> For example: you upload V02 for MTL0123 (DCCID: 456789)  but V02
>>>>>>>>> was not yet created for MTL0123; the insertion scripts will create a V02
>>>>>>>>> visit for MTL0123 if the PatientName field for the dataset is
>>>>>>>>> MTL0123_456789_V02 with Stage marked as "Not Started".
>>>>>>>>>
>>>>>>>>> However, *note that for the script to create the visit for the
>>>>>>>>> candidate, you have to make sure that all the visit label of your projects
>>>>>>>>> were inserted in the Visit_Windows table* (otherwise, the visit
>>>>>>>>> will never be created as they were not specified as being part of the list
>>>>>>>>> of visit label to expect). So in the example above, you should have one row
>>>>>>>>> in Visit_Windows with Visit_label="V02" set.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> 3) Section 4.3
>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md
>>>>>>>>>>
>>>>>>>>>> What do we do when the folder contains multiple mnc files?
>>>>>>>>>>
>>>>>>>>>> This is a case by case. Most of the time, the MINC files that
>>>>>>>>> failed insertion into the imaging browser end up in the MRI violation
>>>>>>>>> module where you can see what went wrong with the acquisitions not
>>>>>>>>> inserted. If you notice that the MINC file should be inserted as a specific
>>>>>>>>> protocol, you can force the insertion as explained in section 4.3.
>>>>>>>>>
>>>>>>>>> In your case, I have a feeling that you want to be able to insert
>>>>>>>>> MINC files that did not go through the whole pipeline insertion (DICOM
>>>>>>>>> archival, then dcm2mnc, then protocol identification, then insertion...).
>>>>>>>>> In theory, you could call minc_insertion.pl on any MINC file,
>>>>>>>>> provided you have at least the following information:
>>>>>>>>> - path to the MINC  file
>>>>>>>>> - uploadID associated with the MINC files or DICOM archive path
>>>>>>>>> from which those MINC files were created from
>>>>>>>>>
>>>>>>>>> Then the minc_insertion.pl script will do the subject information
>>>>>>>>> and protocol validation etc...
>>>>>>>>>
>>>>>>>>> If all the MINC files come from the same uploadID or TarchiveID,
>>>>>>>>> then you could run a loop in bash calling the minc_insertion.pl
>>>>>>>>> script like this for files deriving from uploadID=1:
>>>>>>>>>
>>>>>>>>> ls /path/to/mnc/folder/* | while read f; do minc_insertion.pl
>>>>>>>>> -profile prod -uploadID 1 -mincPath $f; done
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Thank you,
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hope this helped!
>>>>>>>>>
>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190725/fce10f5e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: environment.png
Type: image/png
Size: 50080 bytes
Desc: not available
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190725/fce10f5e/attachment-0001.png>