[Loris-dev] Import mri - scripts
Sotirios Nikoloutsopoulos
sotirisnik at gmail.com
Wed Jul 24 10:55:13 EDT 2019
Ok first i execute this command:
"lorisadmin at hbp:/data/loris/bin/mri$ ./batch_uploads_imageuploader
-profile prod < ~/Desktop/input.txt > log.txt
Use of uninitialized value $data_dir in concatenation (.) or string at
./batch_uploads_imageuploader line 143.
Use of uninitialized value $data_dir in concatenation (.) or string at
./batch_uploads_imageuploader line 143.
Use of uninitialized value $_ in pattern match (m//) at
./batch_uploads_imageuploader line 147.
base is DCC0000_258024_V01
path is /data/incoming/
type is .tar.gz
fullpath is /data/incoming/DCC0000_258024_V01.tar.gz
source /tmp/ImagingUpload-17-43-ZUEFjb
targetlocation /data/loris/data/tarchive
Use of uninitialized value $data_dir in concatenation (.) or string at
/data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256.
Use of uninitialized value $data_dir in concatenation (.) or string at
/data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284.
Use of uninitialized value $data_dir in concatenation (.) or string at
/data/loris/bin/mri/uploadNeuroDB/tarchive_validation.pl line 219.
Use of uninitialized value $data_dir in concatenation (.) or string at
/data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl line 270.
Use of uninitialized value $data_dir in concatenation (.) or string at
/data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711.
Can't exec "mail": No such file or directory at
/data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721.
print() on closed filehandle MAIL at
/data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722.
print() on closed filehandle MAIL at
/data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723.
Use of uninitialized value $data_dir in concatenation (.) or string at
/data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724.
print() on closed filehandle MAIL at
/data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724.
print() on closed filehandle MAIL at
/data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725.
Use of uninitialized value $data_dir in concatenation (.) or string at
/data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735.
No Mincs inserted
The tarchiveLoader insertion script has failed.
Running now the following command: /data/loris/data/uploadNeuroDB/
imaging_upload_file.pl -profile prod -upload_id 3
/data/incoming/DCC0000_258024_V01.tar.gz -verbose"
I inserted the VisitLabel, but still the Mincs are not inserted and i am
getting this error
"lorisadmin at hbp:/data/loris/bin/mri$ /data/loris/data/uploadNeuroDB/
imaging_upload_file.pl -profile prod -upload_id 3
/data/incoming/DCC0000_258024_V01.tar.gz -verbose
find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name '__MACOSX' -delete
Spool message is:
The Scan for the uploadID 3 has already been run with tarchiveID: 7.
To continue with the rest of the insertion pipeline, please run
tarchiveLoader from a terminal as follows:
/data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod
/data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar
-verbose"
Which files should i delete? I thought that deleting the files from the
tarchive was enough. And i executed the last proposed command to continue
to rest of the insertion pipeline ( check the image ).
Thank you
Στις Τετ, 24 Ιουλ 2019 στις 5:05 μ.μ., ο/η Cecile Madjar <
cecile.madjar at mcin.ca> έγραψε:
> Hi Sotirios,
>
> Sorry I was not clear before. Every visits of candidates are created in
> the session table with one entry per CandID/VisitLabel. So if the imaging
> pipeline created visits in the session table (that are not attached to
> instruments) and you want to clean your database completely, you can delete
> entries in that table as well.
>
> Regarding your point 2, the imaging pipeline can create the visit in the
> session table as long as the visit label is present in the Visit_Windows
> table (which means the visit label stored in the PatientName is a valid
> visit label).
> However, in your prod file, if the variable $subjectID{'createVisitLabel'}
> is set to 1 for candidates, the visit label should be created in the
> Visit_Windows table I believe. Here is the line of the prod file you
> would need to change
> <https://github.com/aces/Loris-MRI/blob/81bae73ea6e86c9498519dadf574468ee1d992ca/dicom-archive/profileTemplate.pl#L59>
> .
> In general, we prefer populating the Visit_Windows table and set the
> $subjectID{'createVisitLabel'} to 0 for candidate data to avoid insertion
> of badly labelled MRI data that are complex to relabel.
>
> Finally, regarding the candidate creation, if you execute the following
> steps, the pipeline should be able to create candidates:
> - set in the Imaging Pipeline section of the config module the config
> setting "Upload creation of candidates" to Yes
> - manually transfer the scans to the LORIS server instead of uploading it
> via the imaging browser
> - run batch_imaging_upload.pl as you did until now.
>
> Hope this helps!
> Best,
>
> Cécile
>
> On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos <
> sotirisnik at gmail.com> wrote:
>
>> About my second question i replaced the name of the patient in the .dcm
>> files and now i am getting an error that the visit label does not exist.
>> I thought it was supposed to be automatically created.
>>
>> "Done adding archive info into database
>>
>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile
>> prod
>> /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar
>> -verbose
>> md5sum
>> /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar
>> PSCID is: DCC0000
>> CandID id: 258024
>> visit_label is: V1
>> PSCID is: DCC0000
>> CandID id: 258024
>> visit_label is: V1
>> candidate id 258024
>>
>>
>> => No Visit labelVisit label does not exist
>> Set centerID = 1
>> PSCID is: DCC0000
>> CandID id: 258024
>> visit_label is: V1
>> PSCID is: DCC0000
>> CandID id: 258024
>> visit_label is: V1
>>
>> Number of MINC files that will be considered for inserting into the
>> database: 1
>>
>> log dir is /logs and log file is /logs/TarLoad-15-42-L6pocU.log
>> PSCID is: DCC0000
>> CandID id: 258024
>> visit_label is: V1
>> PSCID is: DCC0000
>> CandID id: 258024
>> visit_label is: V1
>> candidate id 258024
>>
>>
>> => No Visit label
>> Cleaning up temp files: rm -rf
>> /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*"
>>
>> Στις Τετ, 24 Ιουλ 2019 στις 3:38 μ.μ., ο/η Sotirios Nikoloutsopoulos <
>> sotirisnik at gmail.com> έγραψε:
>>
>>> Hi,
>>>
>>> 1) What are the records in the session table?
>>>
>>> 2) I tried to upload a non-phantom file like this
>>>
>>> input_file.txt:
>>> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1
>>>
>>> Terminal:
>>> ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt >
>>> log.txt
>>>
>>> and i get the following error in the log.txt file
>>>
>>> "Running now the following command: /data/loris/data/uploadNeuroDB/
>>> imaging_upload_file.pl -profile prod -upload_id 7
>>> /data/incoming/DCC0000_258024_V1.tar.gz -verbose
>>>
>>> find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name '__MACOSX'
>>> -delete
>>> Spool message is:
>>> The PatientName read from the DICOM header does not start with
>>> DCC0000_258024_V1 from the mri_upload table
>>> "
>>>
>>> and the spool message is repeated for each .dcm file. Should i
>>> preprocess every .dcm file and change the PatientName to
>>> "DCC0000_258024_V1"?
>>>
>>> 3) As far as it concerns the candidate profile i have to create a new
>>> one each time before i try to upload a file? Is there already a way to
>>> create a candidate profile and get the DCCID and PSCID without the user
>>> interface?
>>>
>>> Thank you,
>>>
>>> Sotirios
>>>
>>> Στις Πέμ, 18 Ιουλ 2019 στις 5:00 μ.μ., ο/η Cecile Madjar <
>>> cecile.madjar at mcin.ca> έγραψε:
>>>
>>>> Hi Sotirios,
>>>>
>>>> See answers to your questions below.
>>>>
>>>> About the 1st case we had a dicom with a blank studyUID, maybe someone
>>>>> accidentally removed it.
>>>>>
>>>> That is strange. Something must have been done to the DICOM files. I
>>>> have never seen a study with no StudyUID out of the scanner. Usually, you
>>>> need to manually erase it with some tool. However, DICOMs never cease to
>>>> surprise me...
>>>>
>>>>
>>>>> Some other questions:
>>>>> 1) What are the steps to remove all of my patients and uploads?:
>>>>> because i want to try to reupload all the testing dicoms we had with a
>>>>> clean database.
>>>>>
>>>>
>>>> Release 21.0 of LORIS-MRI will offer a script to remove all the entries
>>>> and files specific to an upload. However, this release is not yet out.
>>>> Hopefully in the next few weeks.
>>>> In the meantime, since your case is particular and that you just want
>>>> to start with a clean database, the following deletes should work
>>>> (hopefully I won't forget any tables):
>>>> delete from mri_violations_log;
>>>> delete from mri_protocol_violated_scans;
>>>> delete from MRICandidateErrors;
>>>> delete from parameter_file;
>>>> delete from files_qcstatus; # (if you played with the QC part of the
>>>> imaging browser for testing)
>>>> delete from feedback_mri_comments; # (if you played with the QC part of
>>>> the imaging browser for testing)
>>>> delete from tarchive_series;
>>>> delete from tarchive_files;
>>>> delete from mri_upload;
>>>> delete from tarchive;
>>>> delete from mri_scanner;
>>>> delete from candidate where PSCID="scanner";
>>>>
>>>> I don't know if you want to also delete entries in the session and
>>>> candidate table too?
>>>>
>>>> 2) We do not need to create the profile of a patient ( are the patients
>>>>> stored in the candidate table? because i can't see the ghosts ), because
>>>>> will it automatically be created when importing a dicom?
>>>>>
>>>>
>>>> I am not sure I understand fully this question. All candidates are
>>>> stored in the candidate table. The phantom scans however are attached to a
>>>> scanner candidate depending on where the scan happened. It is a bit of a
>>>> weird concept that we have to redesign eventually but never got a chance to
>>>> get to it.
>>>>
>>>> Hopefully this answered your question, otherwise, don't hesitate to let
>>>> me know.
>>>>
>>>> 3) What is the difference between uploading a dicom as phantom instead
>>>>> of importing as PSCCID_DCCID_VisitLabel?
>>>>>
>>>>
>>>> When you upload the scan as PSCID_DCCID_VisitLabel, there is a check
>>>> that makes sure the candidate IDs and visit label are valid at the time of
>>>> upload.
>>>> When you upload a scan as phantom, it expects that the DICOM field
>>>> PatientName and uploaded filename contains the string "phantom". We
>>>> enforced this behaviour on the imaging uploader side in recent releases but
>>>> I can't remember which one. Probably the upcoming 21.0 release.
>>>>
>>>> All those verifications might seem cumbersome but they are here to
>>>> ensure that the files inserted are all valid and labelled properly as it is
>>>> a bit messy to have to delete files that were wrongly labelled. At least,
>>>> the delete script present in release 21.0 will make this process easier but
>>>> it is still good practice to verify those things.
>>>>
>>>>
>>>>> Thank you,
>>>>>
>>>>
>>>> With pleasure. Hopefully the answers to your questions will be helpful.
>>>>
>>>> Best,
>>>>
>>>> Cécile
>>>>
>>>> PS: loris-dev mailing list: you will find the initial answer I gave
>>>> Sotirios below. I forgot to cc the loris-dev in my earlier reply...
>>>>
>>>> Στις Τρί, 16 Ιουλ 2019 στις 12:02 π.μ., ο/η Cecile Madjar <
>>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>>
>>>>>> Dear Sotirios,
>>>>>>
>>>>>> Thank you for reaching out. Since your email was already organized in
>>>>>> points, I will reply directly below your questions below.
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Cécile
>>>>>>
>>>>>> 1) What do we do when the StudyUID is blank on the header?
>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md
>>>>>>> There solution provided at the table 3 does not guide us how to find the
>>>>>>> information ( is possible ).
>>>>>>>
>>>>>>
>>>>>> The StudyUID is the field used in our pipeline at the moment to check
>>>>>> whether a DICOM study was already inserted into LORIS with that same
>>>>>> StudyUID since it is supposed to be unique for every single study.
>>>>>> Currently, we do not support insertion of DICOM studies if they do not have
>>>>>> a StudyUID associated with them.
>>>>>>
>>>>>> Is there a specific reason why the StudyUID is blank in the DICOM
>>>>>> headers in your DICOM files? Did they go through some processes before
>>>>>> upload to LORIS?
>>>>>>
>>>>>>
>>>>>> 2) When i am about to upload a non-phanton case i need a 'Visit
>>>>>>> Label'. Is there already a script that can create a new visit label for a
>>>>>>> specific Candidate, so i can provide it as input afterwards to the
>>>>>>>
>>>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md )
>>>>>>>
>>>>>>>
>>>>>> I am not sure if the front-end allows for it but if you use
>>>>>> batch_uploads_imageuploader, the insertion scripts could create the visit
>>>>>> label for you. However, it does need to be included in the patient name in
>>>>>> order for the script to know which label it should use.
>>>>>> For example: you upload V02 for MTL0123 (DCCID: 456789) but V02 was
>>>>>> not yet created for MTL0123; the insertion scripts will create a V02 visit
>>>>>> for MTL0123 if the PatientName field for the dataset is MTL0123_456789_V02
>>>>>> with Stage marked as "Not Started".
>>>>>>
>>>>>> However, *note that for the script to create the visit for the
>>>>>> candidate, you have to make sure that all the visit label of your projects
>>>>>> were inserted in the Visit_Windows table* (otherwise, the visit will
>>>>>> never be created as they were not specified as being part of the list of
>>>>>> visit label to expect). So in the example above, you should have one row in
>>>>>> Visit_Windows with Visit_label="V02" set.
>>>>>>
>>>>>>
>>>>>>> 3) Section 4.3
>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md
>>>>>>>
>>>>>>> What do we do when the folder contains multiple mnc files?
>>>>>>>
>>>>>>> This is a case by case. Most of the time, the MINC files that failed
>>>>>> insertion into the imaging browser end up in the MRI violation module where
>>>>>> you can see what went wrong with the acquisitions not inserted. If you
>>>>>> notice that the MINC file should be inserted as a specific protocol, you
>>>>>> can force the insertion as explained in section 4.3.
>>>>>>
>>>>>> In your case, I have a feeling that you want to be able to insert
>>>>>> MINC files that did not go through the whole pipeline insertion (DICOM
>>>>>> archival, then dcm2mnc, then protocol identification, then insertion...).
>>>>>> In theory, you could call minc_insertion.pl on any MINC file,
>>>>>> provided you have at least the following information:
>>>>>> - path to the MINC file
>>>>>> - uploadID associated with the MINC files or DICOM archive path from
>>>>>> which those MINC files were created from
>>>>>>
>>>>>> Then the minc_insertion.pl script will do the subject information
>>>>>> and protocol validation etc...
>>>>>>
>>>>>> If all the MINC files come from the same uploadID or TarchiveID, then
>>>>>> you could run a loop in bash calling the minc_insertion.pl script
>>>>>> like this for files deriving from uploadID=1:
>>>>>>
>>>>>> ls /path/to/mnc/folder/* | while read f; do minc_insertion.pl
>>>>>> -profile prod -uploadID 1 -mincPath $f; done
>>>>>>
>>>>>>
>>>>>>> Thank you,
>>>>>>>
>>>>>>
>>>>>> Hope this helped!
>>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190724/2f8a0bca/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: loris.png
Type: image/png
Size: 202034 bytes
Desc: not available
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190724/2f8a0bca/attachment-0001.png>
More information about the Loris-dev
mailing list