[Loris-dev] Import mri - scripts
Sotirios Nikoloutsopoulos
sotirisnik at gmail.com
Tue Jul 30 10:46:31 EDT 2019
I hope this is sufficient as the error for the visit label disappeared.
[image: image.png]
Please see the attached file. Also something i found.
lorisadmin at hbp:/data/loris/bin/mri$ cat
/data/loris/data//logs/TarLoad-17-39-xubBdv.log
==> Loading file from disk
/tmp/TarLoad-17-39-UyQcG2/dcc0000_258024_v1_20121205_102108_501e1_mri.mnc
--> mapping DICOM parameter for
/tmp/TarLoad-17-39-UyQcG2/dcc0000_258024_v1_20121205_102108_501e1_mri.mnc
==> computing md5 hash for MINC body.
--> md5: 6b1202fd63e29de4c9be1cd9925888a5
==> verifying acquisition protocol
Acquisition protocol is unknown
--> The minc file cannot be registered since the AcquisitionProtocol is
unknown
Στις Τρί, 30 Ιουλ 2019 στις 4:58 μ.μ., ο/η Sotirios Nikoloutsopoulos <
sotirisnik at gmail.com> έγραψε:
> Ok I fixed the login problem by using a known hash and by deleting the
> rows from the login_history i was able to login.
>
> Στις Τρί, 30 Ιουλ 2019 στις 4:50 μ.μ., ο/η Sotirios Nikoloutsopoulos <
> sotirisnik at gmail.com> έγραψε:
>
>> Hi,
>>
>> there was no visit label 01 in the Visit_Windows table and i don't know
>> how to insert it because the record requires lots of attributes. Also the
>> system for a reason asked me to update my password and now i don't remember
>> it and my account got suspended. Is there a way to reset it or set a simple
>> password? I updated the password hash in the users table by using the hash
>> from another LORIS setup we had and i still couldn't login ( also the
>> attribute password is always NULL and the expiration date is 1999-01-01 for
>> some reason ).
>>
>> Thanks
>>
>>
>>
>> Στις Δευ, 29 Ιουλ 2019 στις 5:21 μ.μ., ο/η Cecile Madjar <
>> cecile.madjar at mcin.ca> έγραψε:
>>
>>> Hi Sotirios,
>>>
>>> Can you try adding the visit label v01 to the Visit_Windows table?
>>>
>>> However, I noticed that in a previous dataset, the label was V1. Does
>>> that refer to the same visit? If so, you might want to harmonize the visit
>>> labels so they are all the same across datasets for a given timepoint,
>>> meaning you will need to relabel scans when the visit label is different
>>> but referring to the same visit label (v01 = V1 etc...).
>>>
>>> Hope this helps,
>>>
>>> Cécile
>>>
>>> On Mon, Jul 29, 2019 at 10:15 AM Sotirios Nikoloutsopoulos <
>>> sotirisnik at gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Could you help me with the "Visit label does not exist"? I had changed
>>>> $subjectID{'createVisitLabel'} to 1 as you had suggested but it appears
>>>> that this problem hasn't been resolved yet.
>>>>
>>>> lorisadmin at hbp:/data/loris/bin/mri$ cat
>>>> /data/loris/data//logs/TarLoad-14-58-9Xog7H.log
>>>>
>>>> ==> Loading file from disk
>>>> /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc
>>>>
>>>> --> mapping DICOM parameter for
>>>> /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc
>>>>
>>>> --> using user-defined filterParameters for
>>>> /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc
>>>>
>>>> Candidate Mismatch Error is Visit label does not exist
>>>> -> WARNING: This candidate was invalid. Logging to
>>>> MRICandidateErrors table with reason Visit label does not
>>>> exist
>>>>
>>>> Thank you,
>>>>
>>>> Sotirios
>>>>
>>>> Στις Παρ, 26 Ιουλ 2019 στις 3:02 μ.μ., ο/η Sotirios Nikoloutsopoulos <
>>>> sotirisnik at gmail.com> έγραψε:
>>>>
>>>>> Hi Cecile,
>>>>>
>>>>> The Center error was fixed. Still the insertion of mincs fails. Please
>>>>> see the output.txt file i attached. Also is there something else i need to
>>>>> change at the paths configuration?
>>>>>
>>>>> Thank you,
>>>>>
>>>>> Sotirios
>>>>>
>>>>> Στις Πέμ, 25 Ιουλ 2019 στις 6:55 μ.μ., ο/η Cecile Madjar <
>>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>>
>>>>>> Hi Sotirios,
>>>>>>
>>>>>> We are making progress!
>>>>>>
>>>>>> I have a feeling this error comes from the fact that your psc table
>>>>>> does not have the MRI_alias populated for the DCC site. Running the
>>>>>> following query in the mysql database should fix this:
>>>>>> UPDATE psc SET MRI_alias="DCC", Alias="DCC" WHERE Name="Data
>>>>>> Coordinating Center";
>>>>>> Repeat the same with all your sites (make sure to change the Aliases
>>>>>> between sites, they should all be unique).
>>>>>>
>>>>>> Let me know how it goes!
>>>>>> Best,
>>>>>>
>>>>>> Cécile
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Jul 25, 2019 at 11:43 AM Sotirios Nikoloutsopoulos <
>>>>>> sotirisnik at gmail.com> wrote:
>>>>>>
>>>>>>> Dear Cecile,
>>>>>>> You were right about your guesses. I did the modifications you
>>>>>>> suggested, i still get the "no mincs inserted" error, now i am getting the
>>>>>>> "No center found this candidate" error ( how do i fix that? from what i see
>>>>>>> the .dcm files contains a 'Institution Name', is that the center? ), so i
>>>>>>> guess that's why the validation fails, but i changed the force variable of
>>>>>>> tarchiveLoader to 1, shouldn't it have inserted the mincs then? Also it is
>>>>>>> optional to have a mail server, right? ( from what i see the mail error
>>>>>>> does not affect the $valid_study, so i assume we are okay with that )
>>>>>>> Thank you a lot,
>>>>>>> Sotirios
>>>>>>>
>>>>>>> Στις Πέμ, 25 Ιουλ 2019 στις 5:48 μ.μ., ο/η Cecile Madjar <
>>>>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>>>>
>>>>>>>> Hi Sotirios,
>>>>>>>>
>>>>>>>> From what I understand, I believe you just need to update a few
>>>>>>>> config settings from the frontend in the Config module (under the Admin
>>>>>>>> menu).
>>>>>>>> [image: Screen Shot 2019-07-25 at 10.31.35 AM.png]
>>>>>>>>
>>>>>>>> Then go to the Paths section of the config module and update the
>>>>>>>> "LORIS-MRI code" config setting to /data/loris/bin/mri (my guess is that
>>>>>>>> currently, it is set to /data/LORIS/data) and click on the submit button.
>>>>>>>> [image: Screen Shot 2019-07-25 at 10.32.50 AM.png]
>>>>>>>>
>>>>>>>> Then, go to Imaging Pipeline section of the Configuration module
>>>>>>>> and update the "Loris-MRI Data Directory" to /data/loris/data/ (my guess is
>>>>>>>> that it is empty at the moment in your database which is why $data_dir is
>>>>>>>> not defined when you run the tarchiveLoader step of the insertion pipeline).
>>>>>>>> [image: Screen Shot 2019-07-25 at 10.40.52 AM.png]
>>>>>>>>
>>>>>>>> Finally, I took a look at your environment file and it might be a
>>>>>>>> good idea to update your PERL5LIB variable to the following (it has been
>>>>>>>> added to the code a few releases ago as we realized this was needed for the
>>>>>>>> libraries. Not sure which version of LORIS-MRI you are using but it does
>>>>>>>> not hurt to update this variable to what I suggest below):
>>>>>>>> export
>>>>>>>> PERL5LIB=/data/loris/bin/mri/uploadNeuroDB:/data/loris/bin/mri/dicom-archive:$PERL5LIB
>>>>>>>>
>>>>>>>> Hope this helps. Sorry you are experiencing so much trouble setting
>>>>>>>> things up. We probably need to update our WIKI.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>>
>>>>>>>> Cécile
>>>>>>>>
>>>>>>>> On Thu, Jul 25, 2019 at 9:48 AM Sotirios Nikoloutsopoulos <
>>>>>>>> sotirisnik at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> The "Loris-MRI Data Directory" should be in the environment file?
>>>>>>>>>
>>>>>>>>> Στις Πέμ, 25 Ιουλ 2019 στις 12:59 π.μ., ο/η Sotirios
>>>>>>>>> Nikoloutsopoulos <sotirisnik at gmail.com> έγραψε:
>>>>>>>>>
>>>>>>>>>> The scripts are located in /data/loris/bin/mri/. Somewhere I
>>>>>>>>>> might have messed something up but i remember following the instructions in
>>>>>>>>>> the wiki.For example when i tried to execute the
>>>>>>>>>> batch_uploads_imageuploader perl script the line
>>>>>>>>>>
>>>>>>>>>> my $command = "$bin_dir/uploadNeuroDB/imaging_upload_file.pl "
>>>>>>>>>>
>>>>>>>>>> was pointing to an invalid path, for example: it was repeating a
>>>>>>>>>> pattern and had LORIS instead of loris ( i think the mistake was that in
>>>>>>>>>> the wiki it suggested to use $PROJECT for the name of the folder and i
>>>>>>>>>> choosed 'loris'
>>>>>>>>>> https://github.com/aces/Loris/wiki/Imaging-Database ). See also
>>>>>>>>>> the in image for the modification i did. Also i didn't find the section to
>>>>>>>>>> config the $data_dir variable.
>>>>>>>>>>
>>>>>>>>>> Thank you,
>>>>>>>>>>
>>>>>>>>>> Sotirios
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Στις Τετ, 24 Ιουλ 2019 στις 6:05 μ.μ., ο/η Cecile Madjar <
>>>>>>>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>>>>>>>
>>>>>>>>>>> Hi Sotirios,
>>>>>>>>>>>
>>>>>>>>>>> It looks like you have one Config setting not set (based on the
>>>>>>>>>>> first error message you got when running it the first time). You need to
>>>>>>>>>>> set a value for "Loris-MRI Data Directory" in the Config module under the
>>>>>>>>>>> Imaging pipeline section (from what I can see, it would be
>>>>>>>>>>> /data/loris/data/ for your setup).
>>>>>>>>>>>
>>>>>>>>>>> The reason you got that error message the second time is that
>>>>>>>>>>> the pipeline checks if the DICOM study was already inserted in the tarchive
>>>>>>>>>>> table. Since you ran the study once already, the study is inserted into the
>>>>>>>>>>> tarchive, hence the message. However, the pipeline did not complete as the
>>>>>>>>>>> MINC files were not created and inserted into the files/parameter_file
>>>>>>>>>>> table. In order to continue the insertion, you will then need to run the
>>>>>>>>>>> tarchiveLoader command suggested in the terminal during the second run:
>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation
>>>>>>>>>>> -profile prod
>>>>>>>>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar
>>>>>>>>>>> -verbose
>>>>>>>>>>>
>>>>>>>>>>> Finally, I am a bit confused by your setup. Where are the
>>>>>>>>>>> scripts located? In /data/loris/bin/mri or /data/loris/data? It feels like
>>>>>>>>>>> the Config setting "LORIS-MRI code" in the Paths section was set to
>>>>>>>>>>> "/data/LORIS/data" but it should have been set to the directory where the
>>>>>>>>>>> scripts are located (most probably /data/loris/bin/mri).
>>>>>>>>>>>
>>>>>>>>>>> Hope this helps,
>>>>>>>>>>>
>>>>>>>>>>> Cécile
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jul 24, 2019 at 10:55 AM Sotirios Nikoloutsopoulos <
>>>>>>>>>>> sotirisnik at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Ok first i execute this command:
>>>>>>>>>>>>
>>>>>>>>>>>> "lorisadmin at hbp:/data/loris/bin/mri$
>>>>>>>>>>>> ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > log.txt
>>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or
>>>>>>>>>>>> string at ./batch_uploads_imageuploader line 143.
>>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or
>>>>>>>>>>>> string at ./batch_uploads_imageuploader line 143.
>>>>>>>>>>>> Use of uninitialized value $_ in pattern match (m//) at
>>>>>>>>>>>> ./batch_uploads_imageuploader line 147.
>>>>>>>>>>>> base is DCC0000_258024_V01
>>>>>>>>>>>> path is /data/incoming/
>>>>>>>>>>>> type is .tar.gz
>>>>>>>>>>>> fullpath is /data/incoming/DCC0000_258024_V01.tar.gz
>>>>>>>>>>>>
>>>>>>>>>>>> source /tmp/ImagingUpload-17-43-ZUEFjb
>>>>>>>>>>>> targetlocation /data/loris/data/tarchive
>>>>>>>>>>>>
>>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or
>>>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256.
>>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or
>>>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284.
>>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or
>>>>>>>>>>>> string at /data/loris/bin/mri/uploadNeuroDB/
>>>>>>>>>>>> tarchive_validation.pl line 219.
>>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or
>>>>>>>>>>>> string at /data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl
>>>>>>>>>>>> line 270.
>>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or
>>>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711.
>>>>>>>>>>>> Can't exec "mail": No such file or directory at
>>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721.
>>>>>>>>>>>> print() on closed filehandle MAIL at
>>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722.
>>>>>>>>>>>> print() on closed filehandle MAIL at
>>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723.
>>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or
>>>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724.
>>>>>>>>>>>> print() on closed filehandle MAIL at
>>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724.
>>>>>>>>>>>> print() on closed filehandle MAIL at
>>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725.
>>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or
>>>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735.
>>>>>>>>>>>>
>>>>>>>>>>>> No Mincs inserted
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> The tarchiveLoader insertion script has failed.
>>>>>>>>>>>> Running now the following command:
>>>>>>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile
>>>>>>>>>>>> prod -upload_id 3 /data/incoming/DCC0000_258024_V01.tar.gz -verbose"
>>>>>>>>>>>>
>>>>>>>>>>>> I inserted the VisitLabel, but still the Mincs are not inserted
>>>>>>>>>>>> and i am getting this error
>>>>>>>>>>>>
>>>>>>>>>>>> "lorisadmin at hbp:/data/loris/bin/mri$
>>>>>>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile
>>>>>>>>>>>> prod -upload_id 3 /data/incoming/DCC0000_258024_V01.tar.gz -verbose
>>>>>>>>>>>>
>>>>>>>>>>>> find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name
>>>>>>>>>>>> '__MACOSX' -delete
>>>>>>>>>>>> Spool message is:
>>>>>>>>>>>> The Scan for the uploadID 3 has already been run with
>>>>>>>>>>>> tarchiveID: 7.
>>>>>>>>>>>> To continue with the rest of the insertion pipeline, please run
>>>>>>>>>>>> tarchiveLoader from a terminal as follows:
>>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod
>>>>>>>>>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar
>>>>>>>>>>>> -verbose"
>>>>>>>>>>>>
>>>>>>>>>>>> Which files should i delete? I thought that deleting the files
>>>>>>>>>>>> from the tarchive was enough. And i executed the last proposed command to
>>>>>>>>>>>> continue to rest of the insertion pipeline ( check the image ).
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you
>>>>>>>>>>>>
>>>>>>>>>>>> Στις Τετ, 24 Ιουλ 2019 στις 5:05 μ.μ., ο/η Cecile Madjar <
>>>>>>>>>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Sotirios,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sorry I was not clear before. Every visits of candidates are
>>>>>>>>>>>>> created in the session table with one entry per CandID/VisitLabel. So if
>>>>>>>>>>>>> the imaging pipeline created visits in the session table (that are not
>>>>>>>>>>>>> attached to instruments) and you want to clean your database completely,
>>>>>>>>>>>>> you can delete entries in that table as well.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regarding your point 2, the imaging pipeline can create the
>>>>>>>>>>>>> visit in the session table as long as the visit label is present in the
>>>>>>>>>>>>> Visit_Windows table (which means the visit label stored in the PatientName
>>>>>>>>>>>>> is a valid visit label).
>>>>>>>>>>>>> However, in your prod file, if the
>>>>>>>>>>>>> variable $subjectID{'createVisitLabel'} is set to 1 for candidates, the
>>>>>>>>>>>>> visit label should be created in the Visit_Windows table I believe. Here
>>>>>>>>>>>>> is the line of the prod file you would need to change
>>>>>>>>>>>>> <https://github.com/aces/Loris-MRI/blob/81bae73ea6e86c9498519dadf574468ee1d992ca/dicom-archive/profileTemplate.pl#L59>
>>>>>>>>>>>>> .
>>>>>>>>>>>>> In general, we prefer populating the Visit_Windows table and
>>>>>>>>>>>>> set the $subjectID{'createVisitLabel'} to 0 for candidate data to avoid
>>>>>>>>>>>>> insertion of badly labelled MRI data that are complex to relabel.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Finally, regarding the candidate creation, if you execute the
>>>>>>>>>>>>> following steps, the pipeline should be able to create candidates:
>>>>>>>>>>>>> - set in the Imaging Pipeline section of the config module the
>>>>>>>>>>>>> config setting "Upload creation of candidates" to Yes
>>>>>>>>>>>>> - manually transfer the scans to the LORIS server instead of
>>>>>>>>>>>>> uploading it via the imaging browser
>>>>>>>>>>>>> - run batch_imaging_upload.pl as you did until now.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hope this helps!
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cécile
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos <
>>>>>>>>>>>>> sotirisnik at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> About my second question i replaced the name of the patient
>>>>>>>>>>>>>> in the .dcm files and now i am getting an error that the visit label does
>>>>>>>>>>>>>> not exist.
>>>>>>>>>>>>>> I thought it was supposed to be automatically created.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Done adding archive info into database
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation
>>>>>>>>>>>>>> -profile prod
>>>>>>>>>>>>>> /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar
>>>>>>>>>>>>>> -verbose
>>>>>>>>>>>>>> md5sum
>>>>>>>>>>>>>> /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar
>>>>>>>>>>>>>> PSCID is: DCC0000
>>>>>>>>>>>>>> CandID id: 258024
>>>>>>>>>>>>>> visit_label is: V1
>>>>>>>>>>>>>> PSCID is: DCC0000
>>>>>>>>>>>>>> CandID id: 258024
>>>>>>>>>>>>>> visit_label is: V1
>>>>>>>>>>>>>> candidate id 258024
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> => No Visit labelVisit label does not exist
>>>>>>>>>>>>>> Set centerID = 1
>>>>>>>>>>>>>> PSCID is: DCC0000
>>>>>>>>>>>>>> CandID id: 258024
>>>>>>>>>>>>>> visit_label is: V1
>>>>>>>>>>>>>> PSCID is: DCC0000
>>>>>>>>>>>>>> CandID id: 258024
>>>>>>>>>>>>>> visit_label is: V1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Number of MINC files that will be considered for inserting
>>>>>>>>>>>>>> into the database: 1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> log dir is /logs and log file is
>>>>>>>>>>>>>> /logs/TarLoad-15-42-L6pocU.log
>>>>>>>>>>>>>> PSCID is: DCC0000
>>>>>>>>>>>>>> CandID id: 258024
>>>>>>>>>>>>>> visit_label is: V1
>>>>>>>>>>>>>> PSCID is: DCC0000
>>>>>>>>>>>>>> CandID id: 258024
>>>>>>>>>>>>>> visit_label is: V1
>>>>>>>>>>>>>> candidate id 258024
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> => No Visit label
>>>>>>>>>>>>>> Cleaning up temp files: rm -rf
>>>>>>>>>>>>>> /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*"
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Στις Τετ, 24 Ιουλ 2019 στις 3:38 μ.μ., ο/η Sotirios
>>>>>>>>>>>>>> Nikoloutsopoulos <sotirisnik at gmail.com> έγραψε:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1) What are the records in the session table?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2) I tried to upload a non-phantom file like this
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> input_file.txt:
>>>>>>>>>>>>>>> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Terminal:
>>>>>>>>>>>>>>> ./batch_uploads_imageuploader -profile prod <
>>>>>>>>>>>>>>> ~/Desktop/input.txt > log.txt
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and i get the following error in the log.txt file
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "Running now the following command:
>>>>>>>>>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl
>>>>>>>>>>>>>>> -profile prod -upload_id 7 /data/incoming/DCC0000_258024_V1.tar.gz -verbose
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name
>>>>>>>>>>>>>>> '__MACOSX' -delete
>>>>>>>>>>>>>>> Spool message is:
>>>>>>>>>>>>>>> The PatientName read from the DICOM header does not start
>>>>>>>>>>>>>>> with DCC0000_258024_V1 from the mri_upload table
>>>>>>>>>>>>>>> "
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and the spool message is repeated for each .dcm file. Should
>>>>>>>>>>>>>>> i preprocess every .dcm file and change the PatientName to
>>>>>>>>>>>>>>> "DCC0000_258024_V1"?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 3) As far as it concerns the candidate profile i have to
>>>>>>>>>>>>>>> create a new one each time before i try to upload a file? Is there already
>>>>>>>>>>>>>>> a way to create a candidate profile and get the DCCID and PSCID without the
>>>>>>>>>>>>>>> user interface?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sotirios
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Στις Πέμ, 18 Ιουλ 2019 στις 5:00 μ.μ., ο/η Cecile Madjar <
>>>>>>>>>>>>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Sotirios,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> See answers to your questions below.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> About the 1st case we had a dicom with a blank studyUID,
>>>>>>>>>>>>>>>>> maybe someone accidentally removed it.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> That is strange. Something must have been done to the DICOM
>>>>>>>>>>>>>>>> files. I have never seen a study with no StudyUID out of the scanner.
>>>>>>>>>>>>>>>> Usually, you need to manually erase it with some tool. However, DICOMs
>>>>>>>>>>>>>>>> never cease to surprise me...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Some other questions:
>>>>>>>>>>>>>>>>> 1) What are the steps to remove all of my patients and
>>>>>>>>>>>>>>>>> uploads?: because i want to try to reupload all the testing dicoms we had
>>>>>>>>>>>>>>>>> with a clean database.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Release 21.0 of LORIS-MRI will offer a script to remove all
>>>>>>>>>>>>>>>> the entries and files specific to an upload. However, this release is not
>>>>>>>>>>>>>>>> yet out. Hopefully in the next few weeks.
>>>>>>>>>>>>>>>> In the meantime, since your case is particular and that you
>>>>>>>>>>>>>>>> just want to start with a clean database, the following deletes should work
>>>>>>>>>>>>>>>> (hopefully I won't forget any tables):
>>>>>>>>>>>>>>>> delete from mri_violations_log;
>>>>>>>>>>>>>>>> delete from mri_protocol_violated_scans;
>>>>>>>>>>>>>>>> delete from MRICandidateErrors;
>>>>>>>>>>>>>>>> delete from parameter_file;
>>>>>>>>>>>>>>>> delete from files_qcstatus; # (if you played with the QC
>>>>>>>>>>>>>>>> part of the imaging browser for testing)
>>>>>>>>>>>>>>>> delete from feedback_mri_comments; # (if you played with
>>>>>>>>>>>>>>>> the QC part of the imaging browser for testing)
>>>>>>>>>>>>>>>> delete from tarchive_series;
>>>>>>>>>>>>>>>> delete from tarchive_files;
>>>>>>>>>>>>>>>> delete from mri_upload;
>>>>>>>>>>>>>>>> delete from tarchive;
>>>>>>>>>>>>>>>> delete from mri_scanner;
>>>>>>>>>>>>>>>> delete from candidate where PSCID="scanner";
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I don't know if you want to also delete entries in the
>>>>>>>>>>>>>>>> session and candidate table too?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2) We do not need to create the profile of a patient ( are
>>>>>>>>>>>>>>>>> the patients stored in the candidate table? because i can't see the ghosts
>>>>>>>>>>>>>>>>> ), because will it automatically be created when importing a dicom?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I am not sure I understand fully this question. All
>>>>>>>>>>>>>>>> candidates are stored in the candidate table. The phantom scans however are
>>>>>>>>>>>>>>>> attached to a scanner candidate depending on where the scan happened. It is
>>>>>>>>>>>>>>>> a bit of a weird concept that we have to redesign eventually but never got
>>>>>>>>>>>>>>>> a chance to get to it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hopefully this answered your question, otherwise, don't
>>>>>>>>>>>>>>>> hesitate to let me know.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 3) What is the difference between uploading a dicom as
>>>>>>>>>>>>>>>>> phantom instead of importing as PSCCID_DCCID_VisitLabel?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When you upload the scan as PSCID_DCCID_VisitLabel, there
>>>>>>>>>>>>>>>> is a check that makes sure the candidate IDs and visit label are valid at
>>>>>>>>>>>>>>>> the time of upload.
>>>>>>>>>>>>>>>> When you upload a scan as phantom, it expects that the
>>>>>>>>>>>>>>>> DICOM field PatientName and uploaded filename contains the string
>>>>>>>>>>>>>>>> "phantom". We enforced this behaviour on the imaging uploader side in
>>>>>>>>>>>>>>>> recent releases but I can't remember which one. Probably the upcoming 21.0
>>>>>>>>>>>>>>>> release.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> All those verifications might seem cumbersome but they are
>>>>>>>>>>>>>>>> here to ensure that the files inserted are all valid and labelled properly
>>>>>>>>>>>>>>>> as it is a bit messy to have to delete files that were wrongly labelled. At
>>>>>>>>>>>>>>>> least, the delete script present in release 21.0 will make this process
>>>>>>>>>>>>>>>> easier but it is still good practice to verify those things.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> With pleasure. Hopefully the answers to your questions will
>>>>>>>>>>>>>>>> be helpful.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cécile
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> PS: loris-dev mailing list: you will find the initial
>>>>>>>>>>>>>>>> answer I gave Sotirios below. I forgot to cc the loris-dev in my earlier
>>>>>>>>>>>>>>>> reply...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Στις Τρί, 16 Ιουλ 2019 στις 12:02 π.μ., ο/η Cecile Madjar <
>>>>>>>>>>>>>>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Dear Sotirios,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thank you for reaching out. Since your email was already
>>>>>>>>>>>>>>>>>> organized in points, I will reply directly below your questions below.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cécile
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 1) What do we do when the StudyUID is blank on the
>>>>>>>>>>>>>>>>>>> header?
>>>>>>>>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md
>>>>>>>>>>>>>>>>>>> There solution provided at the table 3 does not guide us how to find the
>>>>>>>>>>>>>>>>>>> information ( is possible ).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The StudyUID is the field used in our pipeline at the
>>>>>>>>>>>>>>>>>> moment to check whether a DICOM study was already inserted into LORIS with
>>>>>>>>>>>>>>>>>> that same StudyUID since it is supposed to be unique for every single
>>>>>>>>>>>>>>>>>> study. Currently, we do not support insertion of DICOM studies if they do
>>>>>>>>>>>>>>>>>> not have a StudyUID associated with them.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Is there a specific reason why the StudyUID is blank in
>>>>>>>>>>>>>>>>>> the DICOM headers in your DICOM files? Did they go through some processes
>>>>>>>>>>>>>>>>>> before upload to LORIS?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2) When i am about to upload a non-phanton case i need a
>>>>>>>>>>>>>>>>>>> 'Visit Label'. Is there already a script that can create a new visit label
>>>>>>>>>>>>>>>>>>> for a specific Candidate, so i can provide it as input afterwards to the
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md )
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I am not sure if the front-end allows for it but if you
>>>>>>>>>>>>>>>>>> use batch_uploads_imageuploader, the insertion scripts could create the
>>>>>>>>>>>>>>>>>> visit label for you. However, it does need to be included in the patient
>>>>>>>>>>>>>>>>>> name in order for the script to know which label it should use.
>>>>>>>>>>>>>>>>>> For example: you upload V02 for MTL0123 (DCCID: 456789)
>>>>>>>>>>>>>>>>>> but V02 was not yet created for MTL0123; the insertion scripts will create
>>>>>>>>>>>>>>>>>> a V02 visit for MTL0123 if the PatientName field for the dataset is
>>>>>>>>>>>>>>>>>> MTL0123_456789_V02 with Stage marked as "Not Started".
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> However, *note that for the script to create the visit
>>>>>>>>>>>>>>>>>> for the candidate, you have to make sure that all the visit label of your
>>>>>>>>>>>>>>>>>> projects were inserted in the Visit_Windows table*
>>>>>>>>>>>>>>>>>> (otherwise, the visit will never be created as they were not specified as
>>>>>>>>>>>>>>>>>> being part of the list of visit label to expect). So in the example above,
>>>>>>>>>>>>>>>>>> you should have one row in Visit_Windows with Visit_label="V02" set.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 3) Section 4.3
>>>>>>>>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> What do we do when the folder contains multiple mnc files?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This is a case by case. Most of the time, the MINC files
>>>>>>>>>>>>>>>>>> that failed insertion into the imaging browser end up in the MRI violation
>>>>>>>>>>>>>>>>>> module where you can see what went wrong with the acquisitions not
>>>>>>>>>>>>>>>>>> inserted. If you notice that the MINC file should be inserted as a specific
>>>>>>>>>>>>>>>>>> protocol, you can force the insertion as explained in section 4.3.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> In your case, I have a feeling that you want to be able
>>>>>>>>>>>>>>>>>> to insert MINC files that did not go through the whole pipeline insertion
>>>>>>>>>>>>>>>>>> (DICOM archival, then dcm2mnc, then protocol identification, then
>>>>>>>>>>>>>>>>>> insertion...). In theory, you could call
>>>>>>>>>>>>>>>>>> minc_insertion.pl on any MINC file, provided you have at
>>>>>>>>>>>>>>>>>> least the following information:
>>>>>>>>>>>>>>>>>> - path to the MINC file
>>>>>>>>>>>>>>>>>> - uploadID associated with the MINC files or DICOM
>>>>>>>>>>>>>>>>>> archive path from which those MINC files were created from
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Then the minc_insertion.pl script will do the subject
>>>>>>>>>>>>>>>>>> information and protocol validation etc...
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> If all the MINC files come from the same uploadID or
>>>>>>>>>>>>>>>>>> TarchiveID, then you could run a loop in bash calling the
>>>>>>>>>>>>>>>>>> minc_insertion.pl script like this for files deriving
>>>>>>>>>>>>>>>>>> from uploadID=1:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ls /path/to/mnc/folder/* | while read f; do
>>>>>>>>>>>>>>>>>> minc_insertion.pl -profile prod -uploadID 1 -mincPath
>>>>>>>>>>>>>>>>>> $f; done
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hope this helped!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190730/c710f4b6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2019-07-25 at 10.31.35 AM.png
Type: image/png
Size: 45707 bytes
Desc: not available
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190730/c710f4b6/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2019-07-25 at 10.32.50 AM.png
Type: image/png
Size: 134887 bytes
Desc: not available
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190730/c710f4b6/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2019-07-25 at 10.40.52 AM.png
Type: image/png
Size: 95814 bytes
Desc: not available
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190730/c710f4b6/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 27691 bytes
Desc: not available
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190730/c710f4b6/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: output_loris
Type: application/octet-stream
Size: 5906 bytes
Desc: not available
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190730/c710f4b6/attachment-0001.obj>
More information about the Loris-dev
mailing list