[Loris-dev] Import mri - scripts
Sotirios Nikoloutsopoulos
sotirisnik at gmail.com
Tue Jul 30 09:58:17 EDT 2019
Ok I fixed the login problem by using a known hash and by deleting the rows
from the login_history i was able to login.
Στις Τρί, 30 Ιουλ 2019 στις 4:50 μ.μ., ο/η Sotirios Nikoloutsopoulos <
sotirisnik at gmail.com> έγραψε:
> Hi,
>
> there was no visit label 01 in the Visit_Windows table and i don't know
> how to insert it because the record requires lots of attributes. Also the
> system for a reason asked me to update my password and now i don't remember
> it and my account got suspended. Is there a way to reset it or set a simple
> password? I updated the password hash in the users table by using the hash
> from another LORIS setup we had and i still couldn't login ( also the
> attribute password is always NULL and the expiration date is 1999-01-01 for
> some reason ).
>
> Thanks
>
>
>
> Στις Δευ, 29 Ιουλ 2019 στις 5:21 μ.μ., ο/η Cecile Madjar <
> cecile.madjar at mcin.ca> έγραψε:
>
>> Hi Sotirios,
>>
>> Can you try adding the visit label v01 to the Visit_Windows table?
>>
>> However, I noticed that in a previous dataset, the label was V1. Does
>> that refer to the same visit? If so, you might want to harmonize the visit
>> labels so they are all the same across datasets for a given timepoint,
>> meaning you will need to relabel scans when the visit label is different
>> but referring to the same visit label (v01 = V1 etc...).
>>
>> Hope this helps,
>>
>> Cécile
>>
>> On Mon, Jul 29, 2019 at 10:15 AM Sotirios Nikoloutsopoulos <
>> sotirisnik at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Could you help me with the "Visit label does not exist"? I had changed
>>> $subjectID{'createVisitLabel'} to 1 as you had suggested but it appears
>>> that this problem hasn't been resolved yet.
>>>
>>> lorisadmin at hbp:/data/loris/bin/mri$ cat
>>> /data/loris/data//logs/TarLoad-14-58-9Xog7H.log
>>>
>>> ==> Loading file from disk
>>> /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc
>>>
>>> --> mapping DICOM parameter for
>>> /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc
>>>
>>> --> using user-defined filterParameters for
>>> /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc
>>>
>>> Candidate Mismatch Error is Visit label does not exist
>>> -> WARNING: This candidate was invalid. Logging to
>>> MRICandidateErrors table with reason Visit label does not
>>> exist
>>>
>>> Thank you,
>>>
>>> Sotirios
>>>
>>> Στις Παρ, 26 Ιουλ 2019 στις 3:02 μ.μ., ο/η Sotirios Nikoloutsopoulos <
>>> sotirisnik at gmail.com> έγραψε:
>>>
>>>> Hi Cecile,
>>>>
>>>> The Center error was fixed. Still the insertion of mincs fails. Please
>>>> see the output.txt file i attached. Also is there something else i need to
>>>> change at the paths configuration?
>>>>
>>>> Thank you,
>>>>
>>>> Sotirios
>>>>
>>>> Στις Πέμ, 25 Ιουλ 2019 στις 6:55 μ.μ., ο/η Cecile Madjar <
>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>
>>>>> Hi Sotirios,
>>>>>
>>>>> We are making progress!
>>>>>
>>>>> I have a feeling this error comes from the fact that your psc table
>>>>> does not have the MRI_alias populated for the DCC site. Running the
>>>>> following query in the mysql database should fix this:
>>>>> UPDATE psc SET MRI_alias="DCC", Alias="DCC" WHERE Name="Data
>>>>> Coordinating Center";
>>>>> Repeat the same with all your sites (make sure to change the Aliases
>>>>> between sites, they should all be unique).
>>>>>
>>>>> Let me know how it goes!
>>>>> Best,
>>>>>
>>>>> Cécile
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jul 25, 2019 at 11:43 AM Sotirios Nikoloutsopoulos <
>>>>> sotirisnik at gmail.com> wrote:
>>>>>
>>>>>> Dear Cecile,
>>>>>> You were right about your guesses. I did the modifications you
>>>>>> suggested, i still get the "no mincs inserted" error, now i am getting the
>>>>>> "No center found this candidate" error ( how do i fix that? from what i see
>>>>>> the .dcm files contains a 'Institution Name', is that the center? ), so i
>>>>>> guess that's why the validation fails, but i changed the force variable of
>>>>>> tarchiveLoader to 1, shouldn't it have inserted the mincs then? Also it is
>>>>>> optional to have a mail server, right? ( from what i see the mail error
>>>>>> does not affect the $valid_study, so i assume we are okay with that )
>>>>>> Thank you a lot,
>>>>>> Sotirios
>>>>>>
>>>>>> Στις Πέμ, 25 Ιουλ 2019 στις 5:48 μ.μ., ο/η Cecile Madjar <
>>>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>>>
>>>>>>> Hi Sotirios,
>>>>>>>
>>>>>>> From what I understand, I believe you just need to update a few
>>>>>>> config settings from the frontend in the Config module (under the Admin
>>>>>>> menu).
>>>>>>> [image: Screen Shot 2019-07-25 at 10.31.35 AM.png]
>>>>>>>
>>>>>>> Then go to the Paths section of the config module and update the
>>>>>>> "LORIS-MRI code" config setting to /data/loris/bin/mri (my guess is that
>>>>>>> currently, it is set to /data/LORIS/data) and click on the submit button.
>>>>>>> [image: Screen Shot 2019-07-25 at 10.32.50 AM.png]
>>>>>>>
>>>>>>> Then, go to Imaging Pipeline section of the Configuration module and
>>>>>>> update the "Loris-MRI Data Directory" to /data/loris/data/ (my guess is
>>>>>>> that it is empty at the moment in your database which is why $data_dir is
>>>>>>> not defined when you run the tarchiveLoader step of the insertion pipeline).
>>>>>>> [image: Screen Shot 2019-07-25 at 10.40.52 AM.png]
>>>>>>>
>>>>>>> Finally, I took a look at your environment file and it might be a
>>>>>>> good idea to update your PERL5LIB variable to the following (it has been
>>>>>>> added to the code a few releases ago as we realized this was needed for the
>>>>>>> libraries. Not sure which version of LORIS-MRI you are using but it does
>>>>>>> not hurt to update this variable to what I suggest below):
>>>>>>> export
>>>>>>> PERL5LIB=/data/loris/bin/mri/uploadNeuroDB:/data/loris/bin/mri/dicom-archive:$PERL5LIB
>>>>>>>
>>>>>>> Hope this helps. Sorry you are experiencing so much trouble setting
>>>>>>> things up. We probably need to update our WIKI.
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Cécile
>>>>>>>
>>>>>>> On Thu, Jul 25, 2019 at 9:48 AM Sotirios Nikoloutsopoulos <
>>>>>>> sotirisnik at gmail.com> wrote:
>>>>>>>
>>>>>>>> The "Loris-MRI Data Directory" should be in the environment file?
>>>>>>>>
>>>>>>>> Στις Πέμ, 25 Ιουλ 2019 στις 12:59 π.μ., ο/η Sotirios
>>>>>>>> Nikoloutsopoulos <sotirisnik at gmail.com> έγραψε:
>>>>>>>>
>>>>>>>>> The scripts are located in /data/loris/bin/mri/. Somewhere I
>>>>>>>>> might have messed something up but i remember following the instructions in
>>>>>>>>> the wiki.For example when i tried to execute the
>>>>>>>>> batch_uploads_imageuploader perl script the line
>>>>>>>>>
>>>>>>>>> my $command = "$bin_dir/uploadNeuroDB/imaging_upload_file.pl "
>>>>>>>>>
>>>>>>>>> was pointing to an invalid path, for example: it was repeating a
>>>>>>>>> pattern and had LORIS instead of loris ( i think the mistake was that in
>>>>>>>>> the wiki it suggested to use $PROJECT for the name of the folder and i
>>>>>>>>> choosed 'loris'
>>>>>>>>> https://github.com/aces/Loris/wiki/Imaging-Database ). See also
>>>>>>>>> the in image for the modification i did. Also i didn't find the section to
>>>>>>>>> config the $data_dir variable.
>>>>>>>>>
>>>>>>>>> Thank you,
>>>>>>>>>
>>>>>>>>> Sotirios
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Στις Τετ, 24 Ιουλ 2019 στις 6:05 μ.μ., ο/η Cecile Madjar <
>>>>>>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>>>>>>
>>>>>>>>>> Hi Sotirios,
>>>>>>>>>>
>>>>>>>>>> It looks like you have one Config setting not set (based on the
>>>>>>>>>> first error message you got when running it the first time). You need to
>>>>>>>>>> set a value for "Loris-MRI Data Directory" in the Config module under the
>>>>>>>>>> Imaging pipeline section (from what I can see, it would be
>>>>>>>>>> /data/loris/data/ for your setup).
>>>>>>>>>>
>>>>>>>>>> The reason you got that error message the second time is that the
>>>>>>>>>> pipeline checks if the DICOM study was already inserted in the tarchive
>>>>>>>>>> table. Since you ran the study once already, the study is inserted into the
>>>>>>>>>> tarchive, hence the message. However, the pipeline did not complete as the
>>>>>>>>>> MINC files were not created and inserted into the files/parameter_file
>>>>>>>>>> table. In order to continue the insertion, you will then need to run the
>>>>>>>>>> tarchiveLoader command suggested in the terminal during the second run:
>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation
>>>>>>>>>> -profile prod
>>>>>>>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar
>>>>>>>>>> -verbose
>>>>>>>>>>
>>>>>>>>>> Finally, I am a bit confused by your setup. Where are the scripts
>>>>>>>>>> located? In /data/loris/bin/mri or /data/loris/data? It feels like the
>>>>>>>>>> Config setting "LORIS-MRI code" in the Paths section was set to
>>>>>>>>>> "/data/LORIS/data" but it should have been set to the directory where the
>>>>>>>>>> scripts are located (most probably /data/loris/bin/mri).
>>>>>>>>>>
>>>>>>>>>> Hope this helps,
>>>>>>>>>>
>>>>>>>>>> Cécile
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Jul 24, 2019 at 10:55 AM Sotirios Nikoloutsopoulos <
>>>>>>>>>> sotirisnik at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Ok first i execute this command:
>>>>>>>>>>>
>>>>>>>>>>> "lorisadmin at hbp:/data/loris/bin/mri$
>>>>>>>>>>> ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > log.txt
>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or
>>>>>>>>>>> string at ./batch_uploads_imageuploader line 143.
>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or
>>>>>>>>>>> string at ./batch_uploads_imageuploader line 143.
>>>>>>>>>>> Use of uninitialized value $_ in pattern match (m//) at
>>>>>>>>>>> ./batch_uploads_imageuploader line 147.
>>>>>>>>>>> base is DCC0000_258024_V01
>>>>>>>>>>> path is /data/incoming/
>>>>>>>>>>> type is .tar.gz
>>>>>>>>>>> fullpath is /data/incoming/DCC0000_258024_V01.tar.gz
>>>>>>>>>>>
>>>>>>>>>>> source /tmp/ImagingUpload-17-43-ZUEFjb
>>>>>>>>>>> targetlocation /data/loris/data/tarchive
>>>>>>>>>>>
>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or
>>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256.
>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or
>>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284.
>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or
>>>>>>>>>>> string at /data/loris/bin/mri/uploadNeuroDB/
>>>>>>>>>>> tarchive_validation.pl line 219.
>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or
>>>>>>>>>>> string at /data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl
>>>>>>>>>>> line 270.
>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or
>>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711.
>>>>>>>>>>> Can't exec "mail": No such file or directory at
>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721.
>>>>>>>>>>> print() on closed filehandle MAIL at
>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722.
>>>>>>>>>>> print() on closed filehandle MAIL at
>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723.
>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or
>>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724.
>>>>>>>>>>> print() on closed filehandle MAIL at
>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724.
>>>>>>>>>>> print() on closed filehandle MAIL at
>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725.
>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or
>>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735.
>>>>>>>>>>>
>>>>>>>>>>> No Mincs inserted
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The tarchiveLoader insertion script has failed.
>>>>>>>>>>> Running now the following command:
>>>>>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile
>>>>>>>>>>> prod -upload_id 3 /data/incoming/DCC0000_258024_V01.tar.gz -verbose"
>>>>>>>>>>>
>>>>>>>>>>> I inserted the VisitLabel, but still the Mincs are not inserted
>>>>>>>>>>> and i am getting this error
>>>>>>>>>>>
>>>>>>>>>>> "lorisadmin at hbp:/data/loris/bin/mri$
>>>>>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile
>>>>>>>>>>> prod -upload_id 3 /data/incoming/DCC0000_258024_V01.tar.gz -verbose
>>>>>>>>>>>
>>>>>>>>>>> find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name
>>>>>>>>>>> '__MACOSX' -delete
>>>>>>>>>>> Spool message is:
>>>>>>>>>>> The Scan for the uploadID 3 has already been run with
>>>>>>>>>>> tarchiveID: 7.
>>>>>>>>>>> To continue with the rest of the insertion pipeline, please run
>>>>>>>>>>> tarchiveLoader from a terminal as follows:
>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod
>>>>>>>>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar
>>>>>>>>>>> -verbose"
>>>>>>>>>>>
>>>>>>>>>>> Which files should i delete? I thought that deleting the files
>>>>>>>>>>> from the tarchive was enough. And i executed the last proposed command to
>>>>>>>>>>> continue to rest of the insertion pipeline ( check the image ).
>>>>>>>>>>>
>>>>>>>>>>> Thank you
>>>>>>>>>>>
>>>>>>>>>>> Στις Τετ, 24 Ιουλ 2019 στις 5:05 μ.μ., ο/η Cecile Madjar <
>>>>>>>>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Sotirios,
>>>>>>>>>>>>
>>>>>>>>>>>> Sorry I was not clear before. Every visits of candidates are
>>>>>>>>>>>> created in the session table with one entry per CandID/VisitLabel. So if
>>>>>>>>>>>> the imaging pipeline created visits in the session table (that are not
>>>>>>>>>>>> attached to instruments) and you want to clean your database completely,
>>>>>>>>>>>> you can delete entries in that table as well.
>>>>>>>>>>>>
>>>>>>>>>>>> Regarding your point 2, the imaging pipeline can create the
>>>>>>>>>>>> visit in the session table as long as the visit label is present in the
>>>>>>>>>>>> Visit_Windows table (which means the visit label stored in the PatientName
>>>>>>>>>>>> is a valid visit label).
>>>>>>>>>>>> However, in your prod file, if the
>>>>>>>>>>>> variable $subjectID{'createVisitLabel'} is set to 1 for candidates, the
>>>>>>>>>>>> visit label should be created in the Visit_Windows table I believe. Here
>>>>>>>>>>>> is the line of the prod file you would need to change
>>>>>>>>>>>> <https://github.com/aces/Loris-MRI/blob/81bae73ea6e86c9498519dadf574468ee1d992ca/dicom-archive/profileTemplate.pl#L59>
>>>>>>>>>>>> .
>>>>>>>>>>>> In general, we prefer populating the Visit_Windows table and
>>>>>>>>>>>> set the $subjectID{'createVisitLabel'} to 0 for candidate data to avoid
>>>>>>>>>>>> insertion of badly labelled MRI data that are complex to relabel.
>>>>>>>>>>>>
>>>>>>>>>>>> Finally, regarding the candidate creation, if you execute the
>>>>>>>>>>>> following steps, the pipeline should be able to create candidates:
>>>>>>>>>>>> - set in the Imaging Pipeline section of the config module the
>>>>>>>>>>>> config setting "Upload creation of candidates" to Yes
>>>>>>>>>>>> - manually transfer the scans to the LORIS server instead of
>>>>>>>>>>>> uploading it via the imaging browser
>>>>>>>>>>>> - run batch_imaging_upload.pl as you did until now.
>>>>>>>>>>>>
>>>>>>>>>>>> Hope this helps!
>>>>>>>>>>>> Best,
>>>>>>>>>>>>
>>>>>>>>>>>> Cécile
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos <
>>>>>>>>>>>> sotirisnik at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> About my second question i replaced the name of the patient in
>>>>>>>>>>>>> the .dcm files and now i am getting an error that the visit label does not
>>>>>>>>>>>>> exist.
>>>>>>>>>>>>> I thought it was supposed to be automatically created.
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Done adding archive info into database
>>>>>>>>>>>>>
>>>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation
>>>>>>>>>>>>> -profile prod
>>>>>>>>>>>>> /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar
>>>>>>>>>>>>> -verbose
>>>>>>>>>>>>> md5sum
>>>>>>>>>>>>> /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar
>>>>>>>>>>>>> PSCID is: DCC0000
>>>>>>>>>>>>> CandID id: 258024
>>>>>>>>>>>>> visit_label is: V1
>>>>>>>>>>>>> PSCID is: DCC0000
>>>>>>>>>>>>> CandID id: 258024
>>>>>>>>>>>>> visit_label is: V1
>>>>>>>>>>>>> candidate id 258024
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> => No Visit labelVisit label does not exist
>>>>>>>>>>>>> Set centerID = 1
>>>>>>>>>>>>> PSCID is: DCC0000
>>>>>>>>>>>>> CandID id: 258024
>>>>>>>>>>>>> visit_label is: V1
>>>>>>>>>>>>> PSCID is: DCC0000
>>>>>>>>>>>>> CandID id: 258024
>>>>>>>>>>>>> visit_label is: V1
>>>>>>>>>>>>>
>>>>>>>>>>>>> Number of MINC files that will be considered for inserting
>>>>>>>>>>>>> into the database: 1
>>>>>>>>>>>>>
>>>>>>>>>>>>> log dir is /logs and log file is
>>>>>>>>>>>>> /logs/TarLoad-15-42-L6pocU.log
>>>>>>>>>>>>> PSCID is: DCC0000
>>>>>>>>>>>>> CandID id: 258024
>>>>>>>>>>>>> visit_label is: V1
>>>>>>>>>>>>> PSCID is: DCC0000
>>>>>>>>>>>>> CandID id: 258024
>>>>>>>>>>>>> visit_label is: V1
>>>>>>>>>>>>> candidate id 258024
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> => No Visit label
>>>>>>>>>>>>> Cleaning up temp files: rm -rf
>>>>>>>>>>>>> /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*"
>>>>>>>>>>>>>
>>>>>>>>>>>>> Στις Τετ, 24 Ιουλ 2019 στις 3:38 μ.μ., ο/η Sotirios
>>>>>>>>>>>>> Nikoloutsopoulos <sotirisnik at gmail.com> έγραψε:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1) What are the records in the session table?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2) I tried to upload a non-phantom file like this
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> input_file.txt:
>>>>>>>>>>>>>> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Terminal:
>>>>>>>>>>>>>> ./batch_uploads_imageuploader -profile prod <
>>>>>>>>>>>>>> ~/Desktop/input.txt > log.txt
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> and i get the following error in the log.txt file
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Running now the following command:
>>>>>>>>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl
>>>>>>>>>>>>>> -profile prod -upload_id 7 /data/incoming/DCC0000_258024_V1.tar.gz -verbose
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name
>>>>>>>>>>>>>> '__MACOSX' -delete
>>>>>>>>>>>>>> Spool message is:
>>>>>>>>>>>>>> The PatientName read from the DICOM header does not start
>>>>>>>>>>>>>> with DCC0000_258024_V1 from the mri_upload table
>>>>>>>>>>>>>> "
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> and the spool message is repeated for each .dcm file. Should
>>>>>>>>>>>>>> i preprocess every .dcm file and change the PatientName to
>>>>>>>>>>>>>> "DCC0000_258024_V1"?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 3) As far as it concerns the candidate profile i have to
>>>>>>>>>>>>>> create a new one each time before i try to upload a file? Is there already
>>>>>>>>>>>>>> a way to create a candidate profile and get the DCCID and PSCID without the
>>>>>>>>>>>>>> user interface?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sotirios
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Στις Πέμ, 18 Ιουλ 2019 στις 5:00 μ.μ., ο/η Cecile Madjar <
>>>>>>>>>>>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Sotirios,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> See answers to your questions below.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> About the 1st case we had a dicom with a blank studyUID,
>>>>>>>>>>>>>>>> maybe someone accidentally removed it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That is strange. Something must have been done to the DICOM
>>>>>>>>>>>>>>> files. I have never seen a study with no StudyUID out of the scanner.
>>>>>>>>>>>>>>> Usually, you need to manually erase it with some tool. However, DICOMs
>>>>>>>>>>>>>>> never cease to surprise me...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Some other questions:
>>>>>>>>>>>>>>>> 1) What are the steps to remove all of my patients and
>>>>>>>>>>>>>>>> uploads?: because i want to try to reupload all the testing dicoms we had
>>>>>>>>>>>>>>>> with a clean database.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Release 21.0 of LORIS-MRI will offer a script to remove all
>>>>>>>>>>>>>>> the entries and files specific to an upload. However, this release is not
>>>>>>>>>>>>>>> yet out. Hopefully in the next few weeks.
>>>>>>>>>>>>>>> In the meantime, since your case is particular and that you
>>>>>>>>>>>>>>> just want to start with a clean database, the following deletes should work
>>>>>>>>>>>>>>> (hopefully I won't forget any tables):
>>>>>>>>>>>>>>> delete from mri_violations_log;
>>>>>>>>>>>>>>> delete from mri_protocol_violated_scans;
>>>>>>>>>>>>>>> delete from MRICandidateErrors;
>>>>>>>>>>>>>>> delete from parameter_file;
>>>>>>>>>>>>>>> delete from files_qcstatus; # (if you played with the QC
>>>>>>>>>>>>>>> part of the imaging browser for testing)
>>>>>>>>>>>>>>> delete from feedback_mri_comments; # (if you played with the
>>>>>>>>>>>>>>> QC part of the imaging browser for testing)
>>>>>>>>>>>>>>> delete from tarchive_series;
>>>>>>>>>>>>>>> delete from tarchive_files;
>>>>>>>>>>>>>>> delete from mri_upload;
>>>>>>>>>>>>>>> delete from tarchive;
>>>>>>>>>>>>>>> delete from mri_scanner;
>>>>>>>>>>>>>>> delete from candidate where PSCID="scanner";
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I don't know if you want to also delete entries in the
>>>>>>>>>>>>>>> session and candidate table too?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2) We do not need to create the profile of a patient ( are
>>>>>>>>>>>>>>>> the patients stored in the candidate table? because i can't see the ghosts
>>>>>>>>>>>>>>>> ), because will it automatically be created when importing a dicom?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am not sure I understand fully this question. All
>>>>>>>>>>>>>>> candidates are stored in the candidate table. The phantom scans however are
>>>>>>>>>>>>>>> attached to a scanner candidate depending on where the scan happened. It is
>>>>>>>>>>>>>>> a bit of a weird concept that we have to redesign eventually but never got
>>>>>>>>>>>>>>> a chance to get to it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hopefully this answered your question, otherwise, don't
>>>>>>>>>>>>>>> hesitate to let me know.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 3) What is the difference between uploading a dicom as
>>>>>>>>>>>>>>>> phantom instead of importing as PSCCID_DCCID_VisitLabel?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When you upload the scan as PSCID_DCCID_VisitLabel, there is
>>>>>>>>>>>>>>> a check that makes sure the candidate IDs and visit label are valid at the
>>>>>>>>>>>>>>> time of upload.
>>>>>>>>>>>>>>> When you upload a scan as phantom, it expects that the DICOM
>>>>>>>>>>>>>>> field PatientName and uploaded filename contains the string "phantom". We
>>>>>>>>>>>>>>> enforced this behaviour on the imaging uploader side in recent releases but
>>>>>>>>>>>>>>> I can't remember which one. Probably the upcoming 21.0 release.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> All those verifications might seem cumbersome but they are
>>>>>>>>>>>>>>> here to ensure that the files inserted are all valid and labelled properly
>>>>>>>>>>>>>>> as it is a bit messy to have to delete files that were wrongly labelled. At
>>>>>>>>>>>>>>> least, the delete script present in release 21.0 will make this process
>>>>>>>>>>>>>>> easier but it is still good practice to verify those things.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> With pleasure. Hopefully the answers to your questions will
>>>>>>>>>>>>>>> be helpful.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cécile
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> PS: loris-dev mailing list: you will find the initial answer
>>>>>>>>>>>>>>> I gave Sotirios below. I forgot to cc the loris-dev in my earlier reply...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Στις Τρί, 16 Ιουλ 2019 στις 12:02 π.μ., ο/η Cecile Madjar <
>>>>>>>>>>>>>>>> cecile.madjar at mcin.ca> έγραψε:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Dear Sotirios,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thank you for reaching out. Since your email was already
>>>>>>>>>>>>>>>>> organized in points, I will reply directly below your questions below.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cécile
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 1) What do we do when the StudyUID is blank on the header?
>>>>>>>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md
>>>>>>>>>>>>>>>>>> There solution provided at the table 3 does not guide us how to find the
>>>>>>>>>>>>>>>>>> information ( is possible ).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The StudyUID is the field used in our pipeline at the
>>>>>>>>>>>>>>>>> moment to check whether a DICOM study was already inserted into LORIS with
>>>>>>>>>>>>>>>>> that same StudyUID since it is supposed to be unique for every single
>>>>>>>>>>>>>>>>> study. Currently, we do not support insertion of DICOM studies if they do
>>>>>>>>>>>>>>>>> not have a StudyUID associated with them.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is there a specific reason why the StudyUID is blank in
>>>>>>>>>>>>>>>>> the DICOM headers in your DICOM files? Did they go through some processes
>>>>>>>>>>>>>>>>> before upload to LORIS?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2) When i am about to upload a non-phanton case i need a
>>>>>>>>>>>>>>>>>> 'Visit Label'. Is there already a script that can create a new visit label
>>>>>>>>>>>>>>>>>> for a specific Candidate, so i can provide it as input afterwards to the
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md )
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I am not sure if the front-end allows for it but if you
>>>>>>>>>>>>>>>>> use batch_uploads_imageuploader, the insertion scripts could create the
>>>>>>>>>>>>>>>>> visit label for you. However, it does need to be included in the patient
>>>>>>>>>>>>>>>>> name in order for the script to know which label it should use.
>>>>>>>>>>>>>>>>> For example: you upload V02 for MTL0123 (DCCID: 456789)
>>>>>>>>>>>>>>>>> but V02 was not yet created for MTL0123; the insertion scripts will create
>>>>>>>>>>>>>>>>> a V02 visit for MTL0123 if the PatientName field for the dataset is
>>>>>>>>>>>>>>>>> MTL0123_456789_V02 with Stage marked as "Not Started".
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> However, *note that for the script to create the visit
>>>>>>>>>>>>>>>>> for the candidate, you have to make sure that all the visit label of your
>>>>>>>>>>>>>>>>> projects were inserted in the Visit_Windows table*
>>>>>>>>>>>>>>>>> (otherwise, the visit will never be created as they were not specified as
>>>>>>>>>>>>>>>>> being part of the list of visit label to expect). So in the example above,
>>>>>>>>>>>>>>>>> you should have one row in Visit_Windows with Visit_label="V02" set.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 3) Section 4.3
>>>>>>>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> What do we do when the folder contains multiple mnc files?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This is a case by case. Most of the time, the MINC files
>>>>>>>>>>>>>>>>> that failed insertion into the imaging browser end up in the MRI violation
>>>>>>>>>>>>>>>>> module where you can see what went wrong with the acquisitions not
>>>>>>>>>>>>>>>>> inserted. If you notice that the MINC file should be inserted as a specific
>>>>>>>>>>>>>>>>> protocol, you can force the insertion as explained in section 4.3.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In your case, I have a feeling that you want to be able to
>>>>>>>>>>>>>>>>> insert MINC files that did not go through the whole pipeline insertion
>>>>>>>>>>>>>>>>> (DICOM archival, then dcm2mnc, then protocol identification, then
>>>>>>>>>>>>>>>>> insertion...). In theory, you could call minc_insertion.pl
>>>>>>>>>>>>>>>>> on any MINC file, provided you have at least the following information:
>>>>>>>>>>>>>>>>> - path to the MINC file
>>>>>>>>>>>>>>>>> - uploadID associated with the MINC files or DICOM archive
>>>>>>>>>>>>>>>>> path from which those MINC files were created from
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Then the minc_insertion.pl script will do the subject
>>>>>>>>>>>>>>>>> information and protocol validation etc...
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If all the MINC files come from the same uploadID or
>>>>>>>>>>>>>>>>> TarchiveID, then you could run a loop in bash calling the
>>>>>>>>>>>>>>>>> minc_insertion.pl script like this for files deriving
>>>>>>>>>>>>>>>>> from uploadID=1:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ls /path/to/mnc/folder/* | while read f; do
>>>>>>>>>>>>>>>>> minc_insertion.pl -profile prod -uploadID 1 -mincPath $f;
>>>>>>>>>>>>>>>>> done
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hope this helped!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190730/0ad04ba2/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2019-07-25 at 10.31.35 AM.png
Type: image/png
Size: 45707 bytes
Desc: not available
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190730/0ad04ba2/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2019-07-25 at 10.32.50 AM.png
Type: image/png
Size: 134887 bytes
Desc: not available
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190730/0ad04ba2/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2019-07-25 at 10.40.52 AM.png
Type: image/png
Size: 95814 bytes
Desc: not available
URL: <http://mailman.bic.mni.mcgill.ca/pipermail/loris-dev/attachments/20190730/0ad04ba2/attachment-0005.png>
More information about the Loris-dev
mailing list