[Loris-dev] Import mri - scripts

Wed Jul 24 11:05:08 EDT 2019

Hi Sotirios,

It looks like you have one Config setting not set (based on the first error
message you got when running it the first time). You need to set a value
for "Loris-MRI Data Directory" in the Config module under the Imaging
pipeline section (from what I can see, it would be /data/loris/data/ for
your setup).

The reason you got that error message the second time is that the pipeline
checks if the DICOM study was already inserted in the tarchive table. Since
you ran the study once already, the study is inserted into the tarchive,
hence the message. However, the pipeline did not complete as the MINC files
were not created and inserted into the files/parameter_file table. In order
to continue the insertion, you will then need to run the tarchiveLoader
command suggested in the terminal during the second run:
/data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod
/data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar
-verbose

Finally, I am a bit confused by your setup. Where are the scripts located?
In /data/loris/bin/mri or /data/loris/data? It feels like the Config
setting "LORIS-MRI code" in the Paths section was set to "/data/LORIS/data"
but it should have been set to the directory where the scripts are located
(most probably /data/loris/bin/mri).

Hope this helps,

Cécile

On Wed, Jul 24, 2019 at 10:55 AM Sotirios Nikoloutsopoulos <
sotirisnik at gmail.com> wrote:

> Ok first i execute this command:
>
> "lorisadmin at hbp:/data/loris/bin/mri$  ./batch_uploads_imageuploader
> -profile prod < ~/Desktop/input.txt > log.txt
> Use of uninitialized value $data_dir in concatenation (.) or string at
> ./batch_uploads_imageuploader line 143.
> Use of uninitialized value $data_dir in concatenation (.) or string at
> ./batch_uploads_imageuploader line 143.
> Use of uninitialized value $_ in pattern match (m//) at
> ./batch_uploads_imageuploader line 147.
> base is DCC0000_258024_V01
>  path is /data/incoming/
>  type is .tar.gz
>  fullpath is /data/incoming/DCC0000_258024_V01.tar.gz
>
> source /tmp/ImagingUpload-17-43-ZUEFjb
> targetlocation /data/loris/data/tarchive
>
> Use of uninitialized value $data_dir in concatenation (.) or string at
> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256.
> Use of uninitialized value $data_dir in concatenation (.) or string at
> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284.
> Use of uninitialized value $data_dir in concatenation (.) or string at
> /data/loris/bin/mri/uploadNeuroDB/tarchive_validation.pl line 219.
> Use of uninitialized value $data_dir in concatenation (.) or string at
> /data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl line 270.
> Use of uninitialized value $data_dir in concatenation (.) or string at
> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711.
> Can't exec "mail": No such file or directory at
> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721.
> print() on closed filehandle MAIL at
> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722.
> print() on closed filehandle MAIL at
> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723.
> Use of uninitialized value $data_dir in concatenation (.) or string at
> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724.
> print() on closed filehandle MAIL at
> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724.
> print() on closed filehandle MAIL at
> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725.
> Use of uninitialized value $data_dir in concatenation (.) or string at
> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735.
>
>  No Mincs inserted
>
>
> The tarchiveLoader insertion script has failed.
> Running now the following command: /data/loris/data/uploadNeuroDB/
> imaging_upload_file.pl -profile prod -upload_id 3
> /data/incoming/DCC0000_258024_V01.tar.gz -verbose"
>
> I inserted the VisitLabel, but still the Mincs are not inserted and i am
> getting this error
>
> "lorisadmin at hbp:/data/loris/bin/mri$ /data/loris/data/uploadNeuroDB/
> imaging_upload_file.pl -profile prod -upload_id 3
> /data/incoming/DCC0000_258024_V01.tar.gz -verbose
>
>  find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name '__MACOSX' -delete
> Spool message is:
> The Scan for the uploadID 3 has already been run with tarchiveID: 7.
> To continue with the rest of the insertion pipeline, please run
> tarchiveLoader from a terminal as follows:
> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod
> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar
> -verbose"
>
> Which files should i delete? I thought that deleting the files from the
> tarchive was enough. And i executed the last proposed command to continue
> to rest of the insertion pipeline ( check the image ).
>
> Thank you
>
> Στις Τετ, 24 Ιουλ 2019 στις 5:05 μ.μ., ο/η Cecile Madjar <
> cecile.madjar at mcin.ca> έγραψε:
>
>> Hi Sotirios,
>>
>> Sorry I was not clear before. Every visits of candidates are created in
>> the session table with one entry per CandID/VisitLabel. So if the imaging
>> pipeline created visits in the session table (that are not attached to
>> instruments) and you want to clean your database completely, you can delete
>> entries in that table as well.
>>
>> Regarding your point 2, the imaging pipeline can create the visit in the
>> session table as long as the visit label is present in the Visit_Windows
>> table (which means the visit label stored in the PatientName is a valid
>> visit label).
>> However, in your prod file, if the
>> variable $subjectID{'createVisitLabel'} is set to 1 for candidates, the
>> visit label should be created in the Visit_Windows table I believe. Here
>> is the line of the prod file you would need to change
>> <https://github.com/aces/Loris-MRI/blob/81bae73ea6e86c9498519dadf574468ee1d992ca/dicom-archive/profileTemplate.pl#L59>
>> .
>> In general, we prefer populating the Visit_Windows table and set the
>> $subjectID{'createVisitLabel'} to 0 for candidate data to avoid insertion
>> of badly labelled MRI data that are complex to relabel.
>>
>> Finally, regarding the candidate creation, if you execute the following
>> steps, the pipeline should be able to create candidates:
>> - set in the Imaging Pipeline section of the config module the config
>> setting "Upload creation of candidates" to Yes
>> - manually transfer the scans to the LORIS server instead of uploading it
>> via the imaging browser
>> - run batch_imaging_upload.pl as you did until now.
>>
>> Hope this helps!
>> Best,
>>
>> Cécile
>>
>> On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos <
>> sotirisnik at gmail.com> wrote:
>>
>>> About my second question i replaced the name of the patient in the .dcm
>>> files and now i am getting an error that the visit label does not exist.
>>> I thought it was supposed to be automatically created.
>>>
>>> "Done adding archive info into database
>>>
>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile
>>> prod
>>> /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar
>>> -verbose
>>>  md5sum
>>> /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar
>>> PSCID is: DCC0000
>>>  CandID id: 258024
>>>  visit_label is: V1
>>> PSCID is: DCC0000
>>>  CandID id: 258024
>>>  visit_label is: V1
>>> candidate id 258024
>>>
>>>
>>> => No Visit labelVisit label does not exist
>>> Set centerID = 1
>>> PSCID is: DCC0000
>>>  CandID id: 258024
>>>  visit_label is: V1
>>> PSCID is: DCC0000
>>>  CandID id: 258024
>>>  visit_label is: V1
>>>
>>> Number of MINC files that will be considered for inserting into the
>>> database: 1
>>>
>>> log dir is /logs and log file is /logs/TarLoad-15-42-L6pocU.log
>>> PSCID is: DCC0000
>>>  CandID id: 258024
>>>  visit_label is: V1
>>> PSCID is: DCC0000
>>>  CandID id: 258024
>>>  visit_label is: V1
>>> candidate id 258024
>>>
>>>
>>> => No Visit label
>>> Cleaning up temp files: rm -rf
>>> /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*"
>>>
>>> Στις Τετ, 24 Ιουλ 2019 στις 3:38 μ.μ., ο/η Sotirios Nikoloutsopoulos <
>>> sotirisnik at gmail.com> έγραψε:
>>>
>>>> Hi,
>>>>
>>>> 1) What are the records in the session table?
>>>>
>>>> 2) I tried to upload a non-phantom file like this
>>>>
>>>> input_file.txt:
>>>> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1
>>>>
>>>> Terminal:
>>>>  ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt >
>>>> log.txt
>>>>
>>>> and i get the following error in the log.txt file
>>>>
>>>> "Running now the following command: /data/loris/data/uploadNeuroDB/
>>>> imaging_upload_file.pl -profile prod -upload_id 7
>>>> /data/incoming/DCC0000_258024_V1.tar.gz -verbose
>>>>
>>>>  find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name '__MACOSX'
>>>> -delete
>>>> Spool message is:
>>>> The PatientName read from the DICOM header does not start with
>>>> DCC0000_258024_V1 from the mri_upload table
>>>>  "
>>>>
>>>> and the spool message is repeated for each .dcm file. Should i
>>>> preprocess every .dcm file and change the PatientName to
>>>> "DCC0000_258024_V1"?
>>>>
>>>> 3) As far as it concerns the candidate profile i have to create a new
>>>> one each time before i try to upload a file? Is there already a way to
>>>> create a candidate profile and get the DCCID and PSCID without the user
>>>> interface?
>>>>
>>>> Thank you,
>>>>
>>>> Sotirios
>>>>
>>>> Στις Πέμ, 18 Ιουλ 2019 στις 5:00 μ.μ., ο/η Cecile Madjar <
>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>
>>>>> Hi Sotirios,
>>>>>
>>>>> See answers to your questions below.
>>>>>
>>>>> About the 1st case we had a dicom with a blank studyUID, maybe someone
>>>>>> accidentally removed it.
>>>>>>
>>>>> That is strange. Something must have been done to the DICOM files. I
>>>>> have never seen a study with no StudyUID out of the scanner. Usually, you
>>>>> need to manually erase it with some tool. However, DICOMs never cease to
>>>>> surprise me...
>>>>>
>>>>>
>>>>>> Some other questions:
>>>>>> 1) What are the steps to remove all of my patients and uploads?:
>>>>>> because i want to try to reupload all the testing dicoms we had with a
>>>>>> clean database.
>>>>>>
>>>>>
>>>>> Release 21.0 of LORIS-MRI will offer a script to remove all the
>>>>> entries and files specific to an upload. However, this release is not yet
>>>>> out. Hopefully in the next few weeks.
>>>>> In the meantime, since your case is particular and that you just want
>>>>> to start with a clean database, the following deletes should work
>>>>> (hopefully I won't forget any tables):
>>>>> delete from mri_violations_log;
>>>>> delete from mri_protocol_violated_scans;
>>>>> delete from MRICandidateErrors;
>>>>> delete from parameter_file;
>>>>> delete from files_qcstatus; # (if you played with the QC part of the
>>>>> imaging browser for testing)
>>>>> delete from feedback_mri_comments; # (if you played with the QC part
>>>>> of the imaging browser for testing)
>>>>> delete from tarchive_series;
>>>>> delete from tarchive_files;
>>>>> delete from mri_upload;
>>>>> delete from tarchive;
>>>>> delete from mri_scanner;
>>>>> delete from candidate where PSCID="scanner";
>>>>>
>>>>> I don't know if you want to also delete entries in the session and
>>>>> candidate table too?
>>>>>
>>>>> 2) We do not need to create the profile of a patient ( are the
>>>>>> patients stored in the candidate table? because i can't see the ghosts ),
>>>>>> because will it automatically be created when importing a dicom?
>>>>>>
>>>>>
>>>>> I am not sure I understand fully this question. All candidates are
>>>>> stored in the candidate table. The phantom scans however are attached to a
>>>>> scanner candidate depending on where the scan happened. It is a bit of a
>>>>> weird concept that we have to redesign eventually but never got a chance to
>>>>> get to it.
>>>>>
>>>>> Hopefully this answered your question, otherwise, don't hesitate to
>>>>> let me know.
>>>>>
>>>>> 3) What is the difference between uploading a dicom as phantom instead
>>>>>> of importing as PSCCID_DCCID_VisitLabel?
>>>>>>
>>>>>
>>>>> When you upload the scan as PSCID_DCCID_VisitLabel, there is a check
>>>>> that makes sure the candidate IDs and visit label are valid at the time of
>>>>> upload.
>>>>> When you upload a scan as phantom, it expects that the DICOM field
>>>>> PatientName and uploaded filename contains the string "phantom". We
>>>>> enforced this behaviour on the imaging uploader side in recent releases but
>>>>> I can't remember which one. Probably the upcoming 21.0 release.
>>>>>
>>>>> All those verifications might seem cumbersome but they are here to
>>>>> ensure that the files inserted are all valid and labelled properly as it is
>>>>> a bit messy to have to delete files that were wrongly labelled. At least,
>>>>> the delete script present in release 21.0 will make this process easier but
>>>>> it is still good practice to verify those things.
>>>>>
>>>>>
>>>>>> Thank you,
>>>>>>
>>>>>
>>>>> With pleasure. Hopefully the answers to your questions will be helpful.
>>>>>
>>>>> Best,
>>>>>
>>>>> Cécile
>>>>>
>>>>> PS: loris-dev mailing list: you will find the initial answer I gave
>>>>> Sotirios below. I forgot to cc the loris-dev in my earlier reply...
>>>>>
>>>>> Στις Τρί, 16 Ιουλ 2019 στις 12:02 π.μ., ο/η Cecile Madjar <
>>>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>>>
>>>>>>> Dear Sotirios,
>>>>>>>
>>>>>>> Thank you for reaching out. Since your email was already organized
>>>>>>> in points, I will reply directly below your questions below.
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Cécile
>>>>>>>
>>>>>>> 1) What do we do when the StudyUID is blank on the header?
>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md
>>>>>>>> There solution provided at the table 3 does not guide us how to find the
>>>>>>>> information ( is possible ).
>>>>>>>>
>>>>>>>
>>>>>>> The StudyUID is the field used in our pipeline at the moment to
>>>>>>> check whether a DICOM study was already inserted into LORIS with that same
>>>>>>> StudyUID since it is supposed to be unique for every single study.
>>>>>>> Currently, we do not support insertion of DICOM studies if they do not have
>>>>>>> a StudyUID associated with them.
>>>>>>>
>>>>>>> Is there a specific reason why the StudyUID is blank in the DICOM
>>>>>>> headers in your DICOM files? Did they go through some processes before
>>>>>>> upload to LORIS?
>>>>>>>
>>>>>>>
>>>>>>> 2) When i am about to upload a non-phanton case i need a 'Visit
>>>>>>>> Label'. Is there already a script that can create a new visit label for a
>>>>>>>> specific Candidate, so i can provide it as input afterwards to the
>>>>>>>>
>>>>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md )
>>>>>>>>
>>>>>>>>
>>>>>>> I am not sure if the front-end allows for it but if you use
>>>>>>> batch_uploads_imageuploader, the insertion scripts could create the visit
>>>>>>> label for you. However, it does need to be included in the patient name in
>>>>>>> order for the script to know which label it should use.
>>>>>>> For example: you upload V02 for MTL0123 (DCCID: 456789)  but V02 was
>>>>>>> not yet created for MTL0123; the insertion scripts will create a V02 visit
>>>>>>> for MTL0123 if the PatientName field for the dataset is MTL0123_456789_V02
>>>>>>> with Stage marked as "Not Started".
>>>>>>>
>>>>>>> However, *note that for the script to create the visit for the
>>>>>>> candidate, you have to make sure that all the visit label of your projects
>>>>>>> were inserted in the Visit_Windows table* (otherwise, the visit
>>>>>>> will never be created as they were not specified as being part of the list
>>>>>>> of visit label to expect). So in the example above, you should have one row
>>>>>>> in Visit_Windows with Visit_label="V02" set.
>>>>>>>
>>>>>>>
>>>>>>>> 3) Section 4.3
>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md
>>>>>>>>
>>>>>>>> What do we do when the folder contains multiple mnc files?
>>>>>>>>
>>>>>>>> This is a case by case. Most of the time, the MINC files that
>>>>>>> failed insertion into the imaging browser end up in the MRI violation
>>>>>>> module where you can see what went wrong with the acquisitions not
>>>>>>> inserted. If you notice that the MINC file should be inserted as a specific
>>>>>>> protocol, you can force the insertion as explained in section 4.3.
>>>>>>>
>>>>>>> In your case, I have a feeling that you want to be able to insert
>>>>>>> MINC files that did not go through the whole pipeline insertion (DICOM
>>>>>>> archival, then dcm2mnc, then protocol identification, then insertion...).
>>>>>>> In theory, you could call minc_insertion.pl on any MINC file,
>>>>>>> provided you have at least the following information:
>>>>>>> - path to the MINC  file
>>>>>>> - uploadID associated with the MINC files or DICOM archive path from
>>>>>>> which those MINC files were created from
>>>>>>>
>>>>>>> Then the minc_insertion.pl script will do the subject information
>>>>>>> and protocol validation etc...
>>>>>>>
>>>>>>> If all the MINC files come from the same uploadID or TarchiveID,
>>>>>>> then you could run a loop in bash calling the minc_insertion.pl
>>>>>>> script like this for files deriving from uploadID=1:
>>>>>>>
>>>>>>> ls /path/to/mnc/folder/* | while read f; do minc_insertion.pl
>>>>>>> -profile prod -uploadID 1 -mincPath $f; done
>>>>>>>
>>>>>>>
>>>>>>>> Thank you,
>>>>>>>>
>>>>>>>
>>>>>>> Hope this helped!
>>>>>>>
>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190724/4e56a876/attachment-0001.html>