[Loris-dev] Import mri - scripts
Sotirios Nikoloutsopoulos
sotirisnik at gmail.com
Fri Jul 26 08:02:38 EDT 2019
Hi Cecile,
The Center error was fixed. Still the insertion of mincs fails. Please see
the output.txt file i attached. Also is there something else i need to
change at the paths configuration?
Thank you,
Sotirios
Στις Πέμ, 25 Ιουλ 2019 στις 6:55 μ.μ., ο/η Cecile Madjar <
cecile.madjar at mcin.ca> έγραψε:
> Hi Sotirios,
>
> We are making progress!
>
> I have a feeling this error comes from the fact that your psc table does
> not have the MRI_alias populated for the DCC site. Running the following
> query in the mysql database should fix this:
> UPDATE psc SET MRI_alias="DCC", Alias="DCC" WHERE Name="Data Coordinating
> Center";
> Repeat the same with all your sites (make sure to change the Aliases
> between sites, they should all be unique).
>
> Let me know how it goes!
> Best,
>
> Cécile
>
>
>
>
>
> On Thu, Jul 25, 2019 at 11:43 AM Sotirios Nikoloutsopoulos <
> sotirisnik at gmail.com> wrote:
>
>> Dear Cecile,
>> You were right about your guesses. I did the modifications you suggested,
>> i still get the "no mincs inserted" error, now i am getting the "No center
>> found this candidate" error ( how do i fix that? from what i see the .dcm
>> files contains a 'Institution Name', is that the center? ), so i guess
>> that's why the validation fails, but i changed the force variable of
>> tarchiveLoader to 1, shouldn't it have inserted the mincs then? Also it is
>> optional to have a mail server, right? ( from what i see the mail error
>> does not affect the $valid_study, so i assume we are okay with that )
>> Thank you a lot,
>> Sotirios
>>
>> Στις Πέμ, 25 Ιουλ 2019 στις 5:48 μ.μ., ο/η Cecile Madjar <
>> cecile.madjar at mcin.ca> έγραψε:
>>
>>> Hi Sotirios,
>>>
>>> From what I understand, I believe you just need to update a few config
>>> settings from the frontend in the Config module (under the Admin menu).
>>> [image: Screen Shot 2019-07-25 at 10.31.35 AM.png]
>>>
>>> Then go to the Paths section of the config module and update the
>>> "LORIS-MRI code" config setting to /data/loris/bin/mri (my guess is that
>>> currently, it is set to /data/LORIS/data) and click on the submit button.
>>> [image: Screen Shot 2019-07-25 at 10.32.50 AM.png]
>>>
>>> Then, go to Imaging Pipeline section of the Configuration module and
>>> update the "Loris-MRI Data Directory" to /data/loris/data/ (my guess is
>>> that it is empty at the moment in your database which is why $data_dir is
>>> not defined when you run the tarchiveLoader step of the insertion pipeline).
>>> [image: Screen Shot 2019-07-25 at 10.40.52 AM.png]
>>>
>>> Finally, I took a look at your environment file and it might be a good
>>> idea to update your PERL5LIB variable to the following (it has been added
>>> to the code a few releases ago as we realized this was needed for the
>>> libraries. Not sure which version of LORIS-MRI you are using but it does
>>> not hurt to update this variable to what I suggest below):
>>> export
>>> PERL5LIB=/data/loris/bin/mri/uploadNeuroDB:/data/loris/bin/mri/dicom-archive:$PERL5LIB
>>>
>>> Hope this helps. Sorry you are experiencing so much trouble setting
>>> things up. We probably need to update our WIKI.
>>>
>>> Best,
>>>
>>> Cécile
>>>
>>> On Thu, Jul 25, 2019 at 9:48 AM Sotirios Nikoloutsopoulos <
>>> sotirisnik at gmail.com> wrote:
>>>
>>>> The "Loris-MRI Data Directory" should be in the environment file?
>>>>
>>>> Στις Πέμ, 25 Ιουλ 2019 στις 12:59 π.μ., ο/η Sotirios Nikoloutsopoulos <
>>>> sotirisnik at gmail.com> έγραψε:
>>>>
>>>>> The scripts are located in /data/loris/bin/mri/. Somewhere I might
>>>>> have messed something up but i remember following the instructions in the
>>>>> wiki.For example when i tried to execute the batch_uploads_imageuploader
>>>>> perl script the line
>>>>>
>>>>> my $command = "$bin_dir/uploadNeuroDB/imaging_upload_file.pl "
>>>>>
>>>>> was pointing to an invalid path, for example: it was repeating a
>>>>> pattern and had LORIS instead of loris ( i think the mistake was that in
>>>>> the wiki it suggested to use $PROJECT for the name of the folder and i
>>>>> choosed 'loris' https://github.com/aces/Loris/wiki/Imaging-Database
>>>>> ). See also the in image for the modification i did. Also i didn't find the
>>>>> section to config the $data_dir variable.
>>>>>
>>>>> Thank you,
>>>>>
>>>>> Sotirios
>>>>>
>>>>>
>>>>>
>>>>> Στις Τετ, 24 Ιουλ 2019 στις 6:05 μ.μ., ο/η Cecile Madjar <
>>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>>
>>>>>> Hi Sotirios,
>>>>>>
>>>>>> It looks like you have one Config setting not set (based on the first
>>>>>> error message you got when running it the first time). You need to set a
>>>>>> value for "Loris-MRI Data Directory" in the Config module under the Imaging
>>>>>> pipeline section (from what I can see, it would be /data/loris/data/ for
>>>>>> your setup).
>>>>>>
>>>>>> The reason you got that error message the second time is that the
>>>>>> pipeline checks if the DICOM study was already inserted in the tarchive
>>>>>> table. Since you ran the study once already, the study is inserted into the
>>>>>> tarchive, hence the message. However, the pipeline did not complete as the
>>>>>> MINC files were not created and inserted into the files/parameter_file
>>>>>> table. In order to continue the insertion, you will then need to run the
>>>>>> tarchiveLoader command suggested in the terminal during the second run:
>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile
>>>>>> prod
>>>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar
>>>>>> -verbose
>>>>>>
>>>>>> Finally, I am a bit confused by your setup. Where are the scripts
>>>>>> located? In /data/loris/bin/mri or /data/loris/data? It feels like the
>>>>>> Config setting "LORIS-MRI code" in the Paths section was set to
>>>>>> "/data/LORIS/data" but it should have been set to the directory where the
>>>>>> scripts are located (most probably /data/loris/bin/mri).
>>>>>>
>>>>>> Hope this helps,
>>>>>>
>>>>>> Cécile
>>>>>>
>>>>>>
>>>>>> On Wed, Jul 24, 2019 at 10:55 AM Sotirios Nikoloutsopoulos <
>>>>>> sotirisnik at gmail.com> wrote:
>>>>>>
>>>>>>> Ok first i execute this command:
>>>>>>>
>>>>>>> "lorisadmin at hbp:/data/loris/bin/mri$ ./batch_uploads_imageuploader
>>>>>>> -profile prod < ~/Desktop/input.txt > log.txt
>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string
>>>>>>> at ./batch_uploads_imageuploader line 143.
>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string
>>>>>>> at ./batch_uploads_imageuploader line 143.
>>>>>>> Use of uninitialized value $_ in pattern match (m//) at
>>>>>>> ./batch_uploads_imageuploader line 147.
>>>>>>> base is DCC0000_258024_V01
>>>>>>> path is /data/incoming/
>>>>>>> type is .tar.gz
>>>>>>> fullpath is /data/incoming/DCC0000_258024_V01.tar.gz
>>>>>>>
>>>>>>> source /tmp/ImagingUpload-17-43-ZUEFjb
>>>>>>> targetlocation /data/loris/data/tarchive
>>>>>>>
>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string
>>>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256.
>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string
>>>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284.
>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string
>>>>>>> at /data/loris/bin/mri/uploadNeuroDB/tarchive_validation.pl line
>>>>>>> 219.
>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string
>>>>>>> at /data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl line 270.
>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string
>>>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711.
>>>>>>> Can't exec "mail": No such file or directory at
>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721.
>>>>>>> print() on closed filehandle MAIL at
>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722.
>>>>>>> print() on closed filehandle MAIL at
>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723.
>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string
>>>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724.
>>>>>>> print() on closed filehandle MAIL at
>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724.
>>>>>>> print() on closed filehandle MAIL at
>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725.
>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string
>>>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735.
>>>>>>>
>>>>>>> No Mincs inserted
>>>>>>>
>>>>>>>
>>>>>>> The tarchiveLoader insertion script has failed.
>>>>>>> Running now the following command: /data/loris/data/uploadNeuroDB/
>>>>>>> imaging_upload_file.pl -profile prod -upload_id 3
>>>>>>> /data/incoming/DCC0000_258024_V01.tar.gz -verbose"
>>>>>>>
>>>>>>> I inserted the VisitLabel, but still the Mincs are not inserted and
>>>>>>> i am getting this error
>>>>>>>
>>>>>>> "lorisadmin at hbp:/data/loris/bin/mri$ /data/loris/data/uploadNeuroDB/
>>>>>>> imaging_upload_file.pl -profile prod -upload_id 3
>>>>>>> /data/incoming/DCC0000_258024_V01.tar.gz -verbose
>>>>>>>
>>>>>>> find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name '__MACOSX'
>>>>>>> -delete
>>>>>>> Spool message is:
>>>>>>> The Scan for the uploadID 3 has already been run with tarchiveID: 7.
>>>>>>> To continue with the rest of the insertion pipeline, please run
>>>>>>> tarchiveLoader from a terminal as follows:
>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod
>>>>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar
>>>>>>> -verbose"
>>>>>>>
>>>>>>> Which files should i delete? I thought that deleting the files from
>>>>>>> the tarchive was enough. And i executed the last proposed command to
>>>>>>> continue to rest of the insertion pipeline ( check the image ).
>>>>>>>
>>>>>>> Thank you
>>>>>>>
>>>>>>> Στις Τετ, 24 Ιουλ 2019 στις 5:05 μ.μ., ο/η Cecile Madjar <
>>>>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>>>>
>>>>>>>> Hi Sotirios,
>>>>>>>>
>>>>>>>> Sorry I was not clear before. Every visits of candidates are
>>>>>>>> created in the session table with one entry per CandID/VisitLabel. So if
>>>>>>>> the imaging pipeline created visits in the session table (that are not
>>>>>>>> attached to instruments) and you want to clean your database completely,
>>>>>>>> you can delete entries in that table as well.
>>>>>>>>
>>>>>>>> Regarding your point 2, the imaging pipeline can create the visit
>>>>>>>> in the session table as long as the visit label is present in the
>>>>>>>> Visit_Windows table (which means the visit label stored in the PatientName
>>>>>>>> is a valid visit label).
>>>>>>>> However, in your prod file, if the
>>>>>>>> variable $subjectID{'createVisitLabel'} is set to 1 for candidates, the
>>>>>>>> visit label should be created in the Visit_Windows table I believe. Here
>>>>>>>> is the line of the prod file you would need to change
>>>>>>>> <https://github.com/aces/Loris-MRI/blob/81bae73ea6e86c9498519dadf574468ee1d992ca/dicom-archive/profileTemplate.pl#L59>
>>>>>>>> .
>>>>>>>> In general, we prefer populating the Visit_Windows table and set
>>>>>>>> the $subjectID{'createVisitLabel'} to 0 for candidate data to avoid
>>>>>>>> insertion of badly labelled MRI data that are complex to relabel.
>>>>>>>>
>>>>>>>> Finally, regarding the candidate creation, if you execute the
>>>>>>>> following steps, the pipeline should be able to create candidates:
>>>>>>>> - set in the Imaging Pipeline section of the config module the
>>>>>>>> config setting "Upload creation of candidates" to Yes
>>>>>>>> - manually transfer the scans to the LORIS server instead of
>>>>>>>> uploading it via the imaging browser
>>>>>>>> - run batch_imaging_upload.pl as you did until now.
>>>>>>>>
>>>>>>>> Hope this helps!
>>>>>>>> Best,
>>>>>>>>
>>>>>>>> Cécile
>>>>>>>>
>>>>>>>> On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos <
>>>>>>>> sotirisnik at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> About my second question i replaced the name of the patient in the
>>>>>>>>> .dcm files and now i am getting an error that the visit label does not
>>>>>>>>> exist.
>>>>>>>>> I thought it was supposed to be automatically created.
>>>>>>>>>
>>>>>>>>> "Done adding archive info into database
>>>>>>>>>
>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation
>>>>>>>>> -profile prod
>>>>>>>>> /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar
>>>>>>>>> -verbose
>>>>>>>>> md5sum
>>>>>>>>> /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar
>>>>>>>>> PSCID is: DCC0000
>>>>>>>>> CandID id: 258024
>>>>>>>>> visit_label is: V1
>>>>>>>>> PSCID is: DCC0000
>>>>>>>>> CandID id: 258024
>>>>>>>>> visit_label is: V1
>>>>>>>>> candidate id 258024
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> => No Visit labelVisit label does not exist
>>>>>>>>> Set centerID = 1
>>>>>>>>> PSCID is: DCC0000
>>>>>>>>> CandID id: 258024
>>>>>>>>> visit_label is: V1
>>>>>>>>> PSCID is: DCC0000
>>>>>>>>> CandID id: 258024
>>>>>>>>> visit_label is: V1
>>>>>>>>>
>>>>>>>>> Number of MINC files that will be considered for inserting into
>>>>>>>>> the database: 1
>>>>>>>>>
>>>>>>>>> log dir is /logs and log file is /logs/TarLoad-15-42-L6pocU.log
>>>>>>>>> PSCID is: DCC0000
>>>>>>>>> CandID id: 258024
>>>>>>>>> visit_label is: V1
>>>>>>>>> PSCID is: DCC0000
>>>>>>>>> CandID id: 258024
>>>>>>>>> visit_label is: V1
>>>>>>>>> candidate id 258024
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> => No Visit label
>>>>>>>>> Cleaning up temp files: rm -rf
>>>>>>>>> /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*"
>>>>>>>>>
>>>>>>>>> Στις Τετ, 24 Ιουλ 2019 στις 3:38 μ.μ., ο/η Sotirios
>>>>>>>>> Nikoloutsopoulos <sotirisnik at gmail.com> έγραψε:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> 1) What are the records in the session table?
>>>>>>>>>>
>>>>>>>>>> 2) I tried to upload a non-phantom file like this
>>>>>>>>>>
>>>>>>>>>> input_file.txt:
>>>>>>>>>> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1
>>>>>>>>>>
>>>>>>>>>> Terminal:
>>>>>>>>>> ./batch_uploads_imageuploader -profile prod <
>>>>>>>>>> ~/Desktop/input.txt > log.txt
>>>>>>>>>>
>>>>>>>>>> and i get the following error in the log.txt file
>>>>>>>>>>
>>>>>>>>>> "Running now the following command:
>>>>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile
>>>>>>>>>> prod -upload_id 7 /data/incoming/DCC0000_258024_V1.tar.gz -verbose
>>>>>>>>>>
>>>>>>>>>> find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name '__MACOSX'
>>>>>>>>>> -delete
>>>>>>>>>> Spool message is:
>>>>>>>>>> The PatientName read from the DICOM header does not start with
>>>>>>>>>> DCC0000_258024_V1 from the mri_upload table
>>>>>>>>>> "
>>>>>>>>>>
>>>>>>>>>> and the spool message is repeated for each .dcm file. Should i
>>>>>>>>>> preprocess every .dcm file and change the PatientName to
>>>>>>>>>> "DCC0000_258024_V1"?
>>>>>>>>>>
>>>>>>>>>> 3) As far as it concerns the candidate profile i have to create a
>>>>>>>>>> new one each time before i try to upload a file? Is there already a way to
>>>>>>>>>> create a candidate profile and get the DCCID and PSCID without the user
>>>>>>>>>> interface?
>>>>>>>>>>
>>>>>>>>>> Thank you,
>>>>>>>>>>
>>>>>>>>>> Sotirios
>>>>>>>>>>
>>>>>>>>>> Στις Πέμ, 18 Ιουλ 2019 στις 5:00 μ.μ., ο/η Cecile Madjar <
>>>>>>>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>>>>>>>
>>>>>>>>>>> Hi Sotirios,
>>>>>>>>>>>
>>>>>>>>>>> See answers to your questions below.
>>>>>>>>>>>
>>>>>>>>>>> About the 1st case we had a dicom with a blank studyUID, maybe
>>>>>>>>>>>> someone accidentally removed it.
>>>>>>>>>>>>
>>>>>>>>>>> That is strange. Something must have been done to the DICOM
>>>>>>>>>>> files. I have never seen a study with no StudyUID out of the scanner.
>>>>>>>>>>> Usually, you need to manually erase it with some tool. However, DICOMs
>>>>>>>>>>> never cease to surprise me...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Some other questions:
>>>>>>>>>>>> 1) What are the steps to remove all of my patients and
>>>>>>>>>>>> uploads?: because i want to try to reupload all the testing dicoms we had
>>>>>>>>>>>> with a clean database.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Release 21.0 of LORIS-MRI will offer a script to remove all the
>>>>>>>>>>> entries and files specific to an upload. However, this release is not yet
>>>>>>>>>>> out. Hopefully in the next few weeks.
>>>>>>>>>>> In the meantime, since your case is particular and that you just
>>>>>>>>>>> want to start with a clean database, the following deletes should work
>>>>>>>>>>> (hopefully I won't forget any tables):
>>>>>>>>>>> delete from mri_violations_log;
>>>>>>>>>>> delete from mri_protocol_violated_scans;
>>>>>>>>>>> delete from MRICandidateErrors;
>>>>>>>>>>> delete from parameter_file;
>>>>>>>>>>> delete from files_qcstatus; # (if you played with the QC part of
>>>>>>>>>>> the imaging browser for testing)
>>>>>>>>>>> delete from feedback_mri_comments; # (if you played with the QC
>>>>>>>>>>> part of the imaging browser for testing)
>>>>>>>>>>> delete from tarchive_series;
>>>>>>>>>>> delete from tarchive_files;
>>>>>>>>>>> delete from mri_upload;
>>>>>>>>>>> delete from tarchive;
>>>>>>>>>>> delete from mri_scanner;
>>>>>>>>>>> delete from candidate where PSCID="scanner";
>>>>>>>>>>>
>>>>>>>>>>> I don't know if you want to also delete entries in the session
>>>>>>>>>>> and candidate table too?
>>>>>>>>>>>
>>>>>>>>>>> 2) We do not need to create the profile of a patient ( are the
>>>>>>>>>>>> patients stored in the candidate table? because i can't see the ghosts ),
>>>>>>>>>>>> because will it automatically be created when importing a dicom?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I am not sure I understand fully this question. All candidates
>>>>>>>>>>> are stored in the candidate table. The phantom scans however are attached
>>>>>>>>>>> to a scanner candidate depending on where the scan happened. It is a bit of
>>>>>>>>>>> a weird concept that we have to redesign eventually but never got a chance
>>>>>>>>>>> to get to it.
>>>>>>>>>>>
>>>>>>>>>>> Hopefully this answered your question, otherwise, don't hesitate
>>>>>>>>>>> to let me know.
>>>>>>>>>>>
>>>>>>>>>>> 3) What is the difference between uploading a dicom as phantom
>>>>>>>>>>>> instead of importing as PSCCID_DCCID_VisitLabel?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> When you upload the scan as PSCID_DCCID_VisitLabel, there is a
>>>>>>>>>>> check that makes sure the candidate IDs and visit label are valid at the
>>>>>>>>>>> time of upload.
>>>>>>>>>>> When you upload a scan as phantom, it expects that the DICOM
>>>>>>>>>>> field PatientName and uploaded filename contains the string "phantom". We
>>>>>>>>>>> enforced this behaviour on the imaging uploader side in recent releases but
>>>>>>>>>>> I can't remember which one. Probably the upcoming 21.0 release.
>>>>>>>>>>>
>>>>>>>>>>> All those verifications might seem cumbersome but they are here
>>>>>>>>>>> to ensure that the files inserted are all valid and labelled properly as it
>>>>>>>>>>> is a bit messy to have to delete files that were wrongly labelled. At
>>>>>>>>>>> least, the delete script present in release 21.0 will make this process
>>>>>>>>>>> easier but it is still good practice to verify those things.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> With pleasure. Hopefully the answers to your questions will be
>>>>>>>>>>> helpful.
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>>
>>>>>>>>>>> Cécile
>>>>>>>>>>>
>>>>>>>>>>> PS: loris-dev mailing list: you will find the initial answer I
>>>>>>>>>>> gave Sotirios below. I forgot to cc the loris-dev in my earlier reply...
>>>>>>>>>>>
>>>>>>>>>>> Στις Τρί, 16 Ιουλ 2019 στις 12:02 π.μ., ο/η Cecile Madjar <
>>>>>>>>>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>>>>>>>>>
>>>>>>>>>>>>> Dear Sotirios,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you for reaching out. Since your email was already
>>>>>>>>>>>>> organized in points, I will reply directly below your questions below.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cécile
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1) What do we do when the StudyUID is blank on the header?
>>>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md
>>>>>>>>>>>>>> There solution provided at the table 3 does not guide us how to find the
>>>>>>>>>>>>>> information ( is possible ).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> The StudyUID is the field used in our pipeline at the moment
>>>>>>>>>>>>> to check whether a DICOM study was already inserted into LORIS with that
>>>>>>>>>>>>> same StudyUID since it is supposed to be unique for every single study.
>>>>>>>>>>>>> Currently, we do not support insertion of DICOM studies if they do not have
>>>>>>>>>>>>> a StudyUID associated with them.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is there a specific reason why the StudyUID is blank in the
>>>>>>>>>>>>> DICOM headers in your DICOM files? Did they go through some processes
>>>>>>>>>>>>> before upload to LORIS?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2) When i am about to upload a non-phanton case i need a
>>>>>>>>>>>>>> 'Visit Label'. Is there already a script that can create a new visit label
>>>>>>>>>>>>>> for a specific Candidate, so i can provide it as input afterwards to the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md )
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> I am not sure if the front-end allows for it but if you use
>>>>>>>>>>>>> batch_uploads_imageuploader, the insertion scripts could create the visit
>>>>>>>>>>>>> label for you. However, it does need to be included in the patient name in
>>>>>>>>>>>>> order for the script to know which label it should use.
>>>>>>>>>>>>> For example: you upload V02 for MTL0123 (DCCID: 456789) but
>>>>>>>>>>>>> V02 was not yet created for MTL0123; the insertion scripts will create a
>>>>>>>>>>>>> V02 visit for MTL0123 if the PatientName field for the dataset is
>>>>>>>>>>>>> MTL0123_456789_V02 with Stage marked as "Not Started".
>>>>>>>>>>>>>
>>>>>>>>>>>>> However, *note that for the script to create the visit for
>>>>>>>>>>>>> the candidate, you have to make sure that all the visit label of your
>>>>>>>>>>>>> projects were inserted in the Visit_Windows table*
>>>>>>>>>>>>> (otherwise, the visit will never be created as they were not specified as
>>>>>>>>>>>>> being part of the list of visit label to expect). So in the example above,
>>>>>>>>>>>>> you should have one row in Visit_Windows with Visit_label="V02" set.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 3) Section 4.3
>>>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What do we do when the folder contains multiple mnc files?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This is a case by case. Most of the time, the MINC files that
>>>>>>>>>>>>> failed insertion into the imaging browser end up in the MRI violation
>>>>>>>>>>>>> module where you can see what went wrong with the acquisitions not
>>>>>>>>>>>>> inserted. If you notice that the MINC file should be inserted as a specific
>>>>>>>>>>>>> protocol, you can force the insertion as explained in section 4.3.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In your case, I have a feeling that you want to be able to
>>>>>>>>>>>>> insert MINC files that did not go through the whole pipeline insertion
>>>>>>>>>>>>> (DICOM archival, then dcm2mnc, then protocol identification, then
>>>>>>>>>>>>> insertion...). In theory, you could call minc_insertion.pl on
>>>>>>>>>>>>> any MINC file, provided you have at least the following information:
>>>>>>>>>>>>> - path to the MINC file
>>>>>>>>>>>>> - uploadID associated with the MINC files or DICOM archive
>>>>>>>>>>>>> path from which those MINC files were created from
>>>>>>>>>>>>>
>>>>>>>>>>>>> Then the minc_insertion.pl script will do the subject
>>>>>>>>>>>>> information and protocol validation etc...
>>>>>>>>>>>>>
>>>>>>>>>>>>> If all the MINC files come from the same uploadID or
>>>>>>>>>>>>> TarchiveID, then you could run a loop in bash calling the
>>>>>>>>>>>>> minc_insertion.pl script like this for files deriving from
>>>>>>>>>>>>> uploadID=1:
>>>>>>>>>>>>>
>>>>>>>>>>>>> ls /path/to/mnc/folder/* | while read f; do minc_insertion.pl
>>>>>>>>>>>>> -profile prod -uploadID 1 -mincPath $f; done
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hope this helped!
>>>>>>>>>>>>>
>>>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190726/b63f3a39/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2019-07-25 at 10.31.35 AM.png
Type: image/png
Size: 45707 bytes
Desc: not available
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190726/b63f3a39/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2019-07-25 at 10.32.50 AM.png
Type: image/png
Size: 134887 bytes
Desc: not available
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190726/b63f3a39/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2019-07-25 at 10.40.52 AM.png
Type: image/png
Size: 95814 bytes
Desc: not available
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190726/b63f3a39/attachment-0006.png>
-------------- next part --------------
lorisadmin at hbp:/data/loris/bin/mri$ ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > log.txt
Use of uninitialized value $_ in pattern match (m//) at ./batch_uploads_imageuploader line 147.
base is DCC0000_258024_V01
path is /data/incoming/
type is .tar.gz
fullpath is /data/incoming/DCC0000_258024_V01.tar.gz
Can't exec "mail": No such file or directory at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 721.
print() on closed filehandle MAIL at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 722.
print() on closed filehandle MAIL at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 723.
print() on closed filehandle MAIL at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 724.
print() on closed filehandle MAIL at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 725.
No Mincs inserted
The tarchiveLoader insertion script has failed.
Running now the following command: /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile prod -upload_id 8 /data/incoming/DCC0000_258024_V01.tar.gz -verbose
so you came till herefileparse(): need a valid pathname at ./batch_uploads_imageuploader line 200.
lorisadmin at hbp:/data/loris/bin/mri$ /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile prod -upload_id 8 /data/incoming/DCC0000_258024_V01.tar.gz -verbose
find -path \/tmp\/ImagingUpload\-14\-58\-ikg2PS -name '__MACOSX' -delete
Spool message is:
The Scan for the uploadID 8 has already been run with tarchiveID: 6.
To continue with the rest of the insertion pipeline, please run tarchiveLoader from a terminal as follows: /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader -globLocation -profile prod /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-14-57-3YMeQO.tar -verbose
The candidate info validation has failed.
lorisadmin at hbp:/data/loris/bin/mri$ /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader -globLocation -profile prod /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-14-57-3YMeQO.tar -verbose
md5sum /data/loris/data/tarchive/2012/DCM_2012-12-05_ImagingUpload-14-57-3YMeQO.tar
PSCID is: DCC0000
CandID id: 258024
visit_label is: V01
PSCID is: DCC0000
CandID id: 258024
visit_label is: V01
candidate id 258024
=> No Visit labelVisit label does not exist
PSCID is: DCC0000
CandID id: 258024
visit_label is: V01
PSCID is: DCC0000
CandID id: 258024
visit_label is: V01
Number of MINC files that will be considered for inserting into the database: 1
log dir is /data/loris/data//logs and log file is /data/loris/data//logs/TarLoad-14-58-9Xog7H.log
PSCID is: DCC0000
CandID id: 258024
visit_label is: V01
PSCID is: DCC0000
CandID id: 258024
visit_label is: V01
candidate id 258024
=> No Visit label
Cleaning up temp files: rm -rf /tmp/TarLoad-14-58-5Z4hYX/ImagingUpload-14-57-3YMeQO*
Can't exec "mail": No such file or directory at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 721.
print() on closed filehandle MAIL at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 722.
print() on closed filehandle MAIL at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 723.
print() on closed filehandle MAIL at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 724.
print() on closed filehandle MAIL at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 725.
No Mincs inserted
lorisadmin at hbp:/data/loris/bin/mri$ cat log.txt
Running now the following command: /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile prod -upload_id 8 /data/incoming/DCC0000_258024_V01.tar.gz -verbose
find -path \/tmp\/ImagingUpload\-14\-57\-3YMeQO -name '__MACOSX' -delete
/data/loris/bin/mri/dicom-archive/dicomTar.pl /tmp/ImagingUpload-14-57-3YMeQO /data/loris/data/tarchive/ -clobber -database -profile prod -verbose
Source: /tmp/ImagingUpload-14-57-3YMeQO
Target: /data/loris/data/tarchive
Testing for database connectivity.
Database is available.
You will archive the dir : ImagingUpload-14-57-3YMeQO
You are creating a tar with the following command:
tar -cf /data/loris/data/tarchive/ImagingUpload-14-57-3YMeQO.tar ImagingUpload-14-57-3YMeQO
getting md5sums and gzipping!!
* Taken from dir : /tmp/ImagingUpload-14-57-3YMeQO
* Archive target location : /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-14-57-3YMeQO.tar
* Name of creating host : 127.0.1.1
* Name of host OS : Linux
* Created by user : lorisadmin
* Archived on : 2019-07-26 14:57:52
* dicomSummary version : 1
* dicomTar version : 1
* md5sum for DICOM tarball : 6d7ddce539a8e6557caed22994677be5 ImagingUpload-14-57-3YMeQO.tar
* md5sum for DICOM tarball gzipped : fa0106eff0f8ce8043ad2286d312b593 ImagingUpload-14-57-3YMeQO.tar.gz
* md5sum for complete archive : beb3b0aa9fa9fc7609d428733da3d7ff DCM_2012-12-05_ImagingUpload-14-57-3YMeQO.tar
Adding archive info into database
Removing temporary files from target location
Done adding archive info into database
/data/loris/bin/mri/uploadNeuroDB/tarchiveLoader -globLocation -profile prod /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-14-57-3YMeQO.tar -verbose
md5sum /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-14-57-3YMeQO.tar
PSCID is: DCC0000
CandID id: 258024
visit_label is: V01
PSCID is: DCC0000
CandID id: 258024
visit_label is: V01
candidate id 258024
=> No Visit labelVisit label does not exist
PSCID is: DCC0000
CandID id: 258024
visit_label is: V01
PSCID is: DCC0000
CandID id: 258024
visit_label is: V01
Number of MINC files that will be considered for inserting into the database: 1
log dir is /data/loris/data//logs and log file is /data/loris/data//logs/TarLoad-14-57-4g4PdT.log
PSCID is: DCC0000
CandID id: 258024
visit_label is: V01
PSCID is: DCC0000
CandID id: 258024
visit_label is: V01
candidate id 258024
=> No Visit label
Cleaning up temp files: rm -rf /tmp/TarLoad-14-57-8lyAAU/ImagingUpload-14-57-3YMeQO*
lorisadmin at hbp:/data/loris/bin/mri$
-------------- next part --------------
A non-text attachment was scrubbed...
Name: paths.png
Type: image/png
Size: 44398 bytes
Desc: not available
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190726/b63f3a39/attachment-0007.png>
More information about the Loris-dev
mailing list