From sotirisnik at gmail.com Wed Jul 10 09:36:26 2019 From: sotirisnik at gmail.com (Sotirios Nikoloutsopoulos) Date: Wed, 10 Jul 2019 16:36:26 +0300 Subject: [Loris-dev] BrainBrowser Message-ID: Hello, Is the Longitudinal view under construction? Example https://demo.loris.ca/imaging_browser/viewSession/?sessionID=1053 it always saying 'Loading' Thanks, Sotiris -------------- next part -------------- An HTML attachment was scrubbed... URL: From cecile.madjar at mcin.ca Wed Jul 10 12:17:12 2019 From: cecile.madjar at mcin.ca (Cecile Madjar) Date: Wed, 10 Jul 2019 12:17:12 -0400 Subject: [Loris-dev] BrainBrowser In-Reply-To: References: Message-ID: Hello Sotirios, Thank you for letting us know. The demo VM was missing a config for the image viewer to work. It should be fixed and working now if you try again. Sorry for the inconvenience. Best, C?cile On Wed, Jul 10, 2019 at 11:24 AM Sotirios Nikoloutsopoulos < sotirisnik at gmail.com> wrote: > Hello, > > Is the Longitudinal view under construction? Example > https://demo.loris.ca/imaging_browser/viewSession/?sessionID=1053 it > always saying 'Loading' > > Thanks, > > Sotiris > _______________________________________________ > Loris-dev mailing list > Loris-dev at bic.mni.mcgill.ca > https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sotirisnik at gmail.com Sat Jul 13 11:39:19 2019 From: sotirisnik at gmail.com (Sotirios Nikoloutsopoulos) Date: Sat, 13 Jul 2019 18:39:19 +0300 Subject: [Loris-dev] Import mri - scripts Message-ID: Hello, I have a few questions about importing mri files. 1) What do we do when the StudyUID is blank on the header? https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md There solution provided at the table 3 does not guide us how to find the information ( is possible ). 2) When i am about to upload a non-phanton case i need a 'Visit Label'. Is there already a script that can create a new visit label for a specific Candidate, so i can provide it as input afterwards to the batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md ) 3) Section 4.3 https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md What do we do when the folder contains multiple mnc files? Thank you, Sotirios -------------- next part -------------- An HTML attachment was scrubbed... URL: From cecile.madjar at mcin.ca Thu Jul 18 10:00:25 2019 From: cecile.madjar at mcin.ca (Cecile Madjar) Date: Thu, 18 Jul 2019 10:00:25 -0400 Subject: [Loris-dev] Import mri - scripts In-Reply-To: References: Message-ID: Hi Sotirios, See answers to your questions below. About the 1st case we had a dicom with a blank studyUID, maybe someone > accidentally removed it. > That is strange. Something must have been done to the DICOM files. I have never seen a study with no StudyUID out of the scanner. Usually, you need to manually erase it with some tool. However, DICOMs never cease to surprise me... > Some other questions: > 1) What are the steps to remove all of my patients and uploads?: because i > want to try to reupload all the testing dicoms we had with a clean database. > Release 21.0 of LORIS-MRI will offer a script to remove all the entries and files specific to an upload. However, this release is not yet out. Hopefully in the next few weeks. In the meantime, since your case is particular and that you just want to start with a clean database, the following deletes should work (hopefully I won't forget any tables): delete from mri_violations_log; delete from mri_protocol_violated_scans; delete from MRICandidateErrors; delete from parameter_file; delete from files_qcstatus; # (if you played with the QC part of the imaging browser for testing) delete from feedback_mri_comments; # (if you played with the QC part of the imaging browser for testing) delete from tarchive_series; delete from tarchive_files; delete from mri_upload; delete from tarchive; delete from mri_scanner; delete from candidate where PSCID="scanner"; I don't know if you want to also delete entries in the session and candidate table too? 2) We do not need to create the profile of a patient ( are the patients > stored in the candidate table? because i can't see the ghosts ), because > will it automatically be created when importing a dicom? > I am not sure I understand fully this question. All candidates are stored in the candidate table. The phantom scans however are attached to a scanner candidate depending on where the scan happened. It is a bit of a weird concept that we have to redesign eventually but never got a chance to get to it. Hopefully this answered your question, otherwise, don't hesitate to let me know. 3) What is the difference between uploading a dicom as phantom instead of > importing as PSCCID_DCCID_VisitLabel? > When you upload the scan as PSCID_DCCID_VisitLabel, there is a check that makes sure the candidate IDs and visit label are valid at the time of upload. When you upload a scan as phantom, it expects that the DICOM field PatientName and uploaded filename contains the string "phantom". We enforced this behaviour on the imaging uploader side in recent releases but I can't remember which one. Probably the upcoming 21.0 release. All those verifications might seem cumbersome but they are here to ensure that the files inserted are all valid and labelled properly as it is a bit messy to have to delete files that were wrongly labelled. At least, the delete script present in release 21.0 will make this process easier but it is still good practice to verify those things. > Thank you, > With pleasure. Hopefully the answers to your questions will be helpful. Best, C?cile PS: loris-dev mailing list: you will find the initial answer I gave Sotirios below. I forgot to cc the loris-dev in my earlier reply... ???? ???, 16 ???? 2019 ???? 12:02 ?.?., ?/? Cecile Madjar < > cecile.madjar at mcin.ca> ??????: > >> Dear Sotirios, >> >> Thank you for reaching out. Since your email was already organized in >> points, I will reply directly below your questions below. >> >> Best, >> >> C?cile >> >> 1) What do we do when the StudyUID is blank on the header? >>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md >>> There solution provided at the table 3 does not guide us how to find the >>> information ( is possible ). >>> >> >> The StudyUID is the field used in our pipeline at the moment to check >> whether a DICOM study was already inserted into LORIS with that same >> StudyUID since it is supposed to be unique for every single study. >> Currently, we do not support insertion of DICOM studies if they do not have >> a StudyUID associated with them. >> >> Is there a specific reason why the StudyUID is blank in the DICOM headers >> in your DICOM files? Did they go through some processes before upload to >> LORIS? >> >> >> 2) When i am about to upload a non-phanton case i need a 'Visit Label'. >>> Is there already a script that can create a new visit label for a specific >>> Candidate, so i can provide it as input afterwards to the >>> >>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md ) >>> >>> >> I am not sure if the front-end allows for it but if you use >> batch_uploads_imageuploader, the insertion scripts could create the visit >> label for you. However, it does need to be included in the patient name in >> order for the script to know which label it should use. >> For example: you upload V02 for MTL0123 (DCCID: 456789) but V02 was not >> yet created for MTL0123; the insertion scripts will create a V02 visit for >> MTL0123 if the PatientName field for the dataset is MTL0123_456789_V02 with >> Stage marked as "Not Started". >> >> However, *note that for the script to create the visit for the >> candidate, you have to make sure that all the visit label of your projects >> were inserted in the Visit_Windows table* (otherwise, the visit will >> never be created as they were not specified as being part of the list of >> visit label to expect). So in the example above, you should have one row in >> Visit_Windows with Visit_label="V02" set. >> >> >>> 3) Section 4.3 >>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md >>> >>> What do we do when the folder contains multiple mnc files? >>> >>> This is a case by case. Most of the time, the MINC files that failed >> insertion into the imaging browser end up in the MRI violation module where >> you can see what went wrong with the acquisitions not inserted. If you >> notice that the MINC file should be inserted as a specific protocol, you >> can force the insertion as explained in section 4.3. >> >> In your case, I have a feeling that you want to be able to insert MINC >> files that did not go through the whole pipeline insertion (DICOM archival, >> then dcm2mnc, then protocol identification, then insertion...). In theory, >> you could call minc_insertion.pl on any MINC file, provided you have at >> least the following information: >> - path to the MINC file >> - uploadID associated with the MINC files or DICOM archive path from >> which those MINC files were created from >> >> Then the minc_insertion.pl script will do the subject information and >> protocol validation etc... >> >> If all the MINC files come from the same uploadID or TarchiveID, then you >> could run a loop in bash calling the minc_insertion.pl script like this >> for files deriving from uploadID=1: >> >> ls /path/to/mnc/folder/* | while read f; do minc_insertion.pl -profile >> prod -uploadID 1 -mincPath $f; done >> >> >>> Thank you, >>> >> >> Hope this helped! >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cecile.madjar at gmail.com Thu Jul 18 16:29:43 2019 From: cecile.madjar at gmail.com (Cecile Madjar) Date: Thu, 18 Jul 2019 16:29:43 -0400 Subject: [Loris-dev] another LORIS question In-Reply-To: References: Message-ID: Hi Louis, I am cc'ing the loris-dev mailing list in case someone wants to jump in and either rectify or add information to my reply. :) I don't see why what you propose would not work. In our MCIN setup, the VM hosting LORIS and the MRI files are on two different filesystems: - one filesystem that is hosting LORIS - one filesystem that is hosting the MRI data and (other large datasets). This one is then mounted on the VM that hosts LORIS so that LORIS can have access to the files and display it in the imaging browser. Separating the two also helps for the backup of the VM (way shorter without terabytes of data). Of course, back of the MRI data is done separately as well since we don't want to lose those precious datasets! All that will be needed once this is done would be to change a few config path in LORIS (in the Config module) to point to the mounted directory and voil?! Hopefully that answers your question. Best, C?cile On Thu, Jul 18, 2019 at 1:18 PM Louis Collins, Dr. wrote: > Cecile, > > I have another question about LORIS. It is taking us a huge amount of > time to upload clinical data into the LORIS DB. I think the slowdown may > be caused by NFS delays because the DB is hosted on network drives in the > computer science building on lower campus. > > Do you think it is possible to store a local copy of the DB on a machine > in the lab in the MNI, where the DB would point to MRI minc volumes that > are on the network drives? This way, the DB would be local, and I assume > that access would be (almost) instantaneous. In addition, because the MRI > data is not local, we would not need 13-15 TB of space locally? > > What do you think? > > -Louis > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sotirisnik at gmail.com Wed Jul 24 08:38:45 2019 From: sotirisnik at gmail.com (Sotirios Nikoloutsopoulos) Date: Wed, 24 Jul 2019 15:38:45 +0300 Subject: [Loris-dev] Import mri - scripts In-Reply-To: References: Message-ID: Hi, 1) What are the records in the session table? 2) I tried to upload a non-phantom file like this input_file.txt: /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1 Terminal: ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > log.txt and i get the following error in the log.txt file "Running now the following command: /data/loris/data/uploadNeuroDB/ imaging_upload_file.pl -profile prod -upload_id 7 /data/incoming/DCC0000_258024_V1.tar.gz -verbose find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name '__MACOSX' -delete Spool message is: The PatientName read from the DICOM header does not start with DCC0000_258024_V1 from the mri_upload table " and the spool message is repeated for each .dcm file. Should i preprocess every .dcm file and change the PatientName to "DCC0000_258024_V1"? 3) As far as it concerns the candidate profile i have to create a new one each time before i try to upload a file? Is there already a way to create a candidate profile and get the DCCID and PSCID without the user interface? Thank you, Sotirios ???? ???, 18 ???? 2019 ???? 5:00 ?.?., ?/? Cecile Madjar < cecile.madjar at mcin.ca> ??????: > Hi Sotirios, > > See answers to your questions below. > > About the 1st case we had a dicom with a blank studyUID, maybe someone >> accidentally removed it. >> > That is strange. Something must have been done to the DICOM files. I have > never seen a study with no StudyUID out of the scanner. Usually, you need > to manually erase it with some tool. However, DICOMs never cease to > surprise me... > > >> Some other questions: >> 1) What are the steps to remove all of my patients and uploads?: because >> i want to try to reupload all the testing dicoms we had with a clean >> database. >> > > Release 21.0 of LORIS-MRI will offer a script to remove all the entries > and files specific to an upload. However, this release is not yet out. > Hopefully in the next few weeks. > In the meantime, since your case is particular and that you just want to > start with a clean database, the following deletes should work (hopefully I > won't forget any tables): > delete from mri_violations_log; > delete from mri_protocol_violated_scans; > delete from MRICandidateErrors; > delete from parameter_file; > delete from files_qcstatus; # (if you played with the QC part of the > imaging browser for testing) > delete from feedback_mri_comments; # (if you played with the QC part of > the imaging browser for testing) > delete from tarchive_series; > delete from tarchive_files; > delete from mri_upload; > delete from tarchive; > delete from mri_scanner; > delete from candidate where PSCID="scanner"; > > I don't know if you want to also delete entries in the session and > candidate table too? > > 2) We do not need to create the profile of a patient ( are the patients >> stored in the candidate table? because i can't see the ghosts ), because >> will it automatically be created when importing a dicom? >> > > I am not sure I understand fully this question. All candidates are stored > in the candidate table. The phantom scans however are attached to a scanner > candidate depending on where the scan happened. It is a bit of a weird > concept that we have to redesign eventually but never got a chance to get > to it. > > Hopefully this answered your question, otherwise, don't hesitate to let me > know. > > 3) What is the difference between uploading a dicom as phantom instead of >> importing as PSCCID_DCCID_VisitLabel? >> > > When you upload the scan as PSCID_DCCID_VisitLabel, there is a check that > makes sure the candidate IDs and visit label are valid at the time of > upload. > When you upload a scan as phantom, it expects that the DICOM field > PatientName and uploaded filename contains the string "phantom". We > enforced this behaviour on the imaging uploader side in recent releases but > I can't remember which one. Probably the upcoming 21.0 release. > > All those verifications might seem cumbersome but they are here to ensure > that the files inserted are all valid and labelled properly as it is a bit > messy to have to delete files that were wrongly labelled. At least, the > delete script present in release 21.0 will make this process easier but it > is still good practice to verify those things. > > >> Thank you, >> > > With pleasure. Hopefully the answers to your questions will be helpful. > > Best, > > C?cile > > PS: loris-dev mailing list: you will find the initial answer I gave > Sotirios below. I forgot to cc the loris-dev in my earlier reply... > > ???? ???, 16 ???? 2019 ???? 12:02 ?.?., ?/? Cecile Madjar < >> cecile.madjar at mcin.ca> ??????: >> >>> Dear Sotirios, >>> >>> Thank you for reaching out. Since your email was already organized in >>> points, I will reply directly below your questions below. >>> >>> Best, >>> >>> C?cile >>> >>> 1) What do we do when the StudyUID is blank on the header? >>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md >>>> There solution provided at the table 3 does not guide us how to find the >>>> information ( is possible ). >>>> >>> >>> The StudyUID is the field used in our pipeline at the moment to check >>> whether a DICOM study was already inserted into LORIS with that same >>> StudyUID since it is supposed to be unique for every single study. >>> Currently, we do not support insertion of DICOM studies if they do not have >>> a StudyUID associated with them. >>> >>> Is there a specific reason why the StudyUID is blank in the DICOM >>> headers in your DICOM files? Did they go through some processes before >>> upload to LORIS? >>> >>> >>> 2) When i am about to upload a non-phanton case i need a 'Visit Label'. >>>> Is there already a script that can create a new visit label for a specific >>>> Candidate, so i can provide it as input afterwards to the >>>> >>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md ) >>>> >>>> >>> I am not sure if the front-end allows for it but if you use >>> batch_uploads_imageuploader, the insertion scripts could create the visit >>> label for you. However, it does need to be included in the patient name in >>> order for the script to know which label it should use. >>> For example: you upload V02 for MTL0123 (DCCID: 456789) but V02 was not >>> yet created for MTL0123; the insertion scripts will create a V02 visit for >>> MTL0123 if the PatientName field for the dataset is MTL0123_456789_V02 with >>> Stage marked as "Not Started". >>> >>> However, *note that for the script to create the visit for the >>> candidate, you have to make sure that all the visit label of your projects >>> were inserted in the Visit_Windows table* (otherwise, the visit will >>> never be created as they were not specified as being part of the list of >>> visit label to expect). So in the example above, you should have one row in >>> Visit_Windows with Visit_label="V02" set. >>> >>> >>>> 3) Section 4.3 >>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md >>>> >>>> What do we do when the folder contains multiple mnc files? >>>> >>>> This is a case by case. Most of the time, the MINC files that failed >>> insertion into the imaging browser end up in the MRI violation module where >>> you can see what went wrong with the acquisitions not inserted. If you >>> notice that the MINC file should be inserted as a specific protocol, you >>> can force the insertion as explained in section 4.3. >>> >>> In your case, I have a feeling that you want to be able to insert MINC >>> files that did not go through the whole pipeline insertion (DICOM archival, >>> then dcm2mnc, then protocol identification, then insertion...). In theory, >>> you could call minc_insertion.pl on any MINC file, provided you have at >>> least the following information: >>> - path to the MINC file >>> - uploadID associated with the MINC files or DICOM archive path from >>> which those MINC files were created from >>> >>> Then the minc_insertion.pl script will do the subject information and >>> protocol validation etc... >>> >>> If all the MINC files come from the same uploadID or TarchiveID, then >>> you could run a loop in bash calling the minc_insertion.pl script like >>> this for files deriving from uploadID=1: >>> >>> ls /path/to/mnc/folder/* | while read f; do minc_insertion.pl -profile >>> prod -uploadID 1 -mincPath $f; done >>> >>> >>>> Thank you, >>>> >>> >>> Hope this helped! >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From sotirisnik at gmail.com Wed Jul 24 08:47:02 2019 From: sotirisnik at gmail.com (Sotirios Nikoloutsopoulos) Date: Wed, 24 Jul 2019 15:47:02 +0300 Subject: [Loris-dev] Import mri - scripts In-Reply-To: References: Message-ID: About my second question i replaced the name of the patient in the .dcm files and now i am getting an error that the visit label does not exist. I thought it was supposed to be automatically created. "Done adding archive info into database /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar -verbose md5sum /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar PSCID is: DCC0000 CandID id: 258024 visit_label is: V1 PSCID is: DCC0000 CandID id: 258024 visit_label is: V1 candidate id 258024 => No Visit labelVisit label does not exist Set centerID = 1 PSCID is: DCC0000 CandID id: 258024 visit_label is: V1 PSCID is: DCC0000 CandID id: 258024 visit_label is: V1 Number of MINC files that will be considered for inserting into the database: 1 log dir is /logs and log file is /logs/TarLoad-15-42-L6pocU.log PSCID is: DCC0000 CandID id: 258024 visit_label is: V1 PSCID is: DCC0000 CandID id: 258024 visit_label is: V1 candidate id 258024 => No Visit label Cleaning up temp files: rm -rf /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*" ???? ???, 24 ???? 2019 ???? 3:38 ?.?., ?/? Sotirios Nikoloutsopoulos < sotirisnik at gmail.com> ??????: > Hi, > > 1) What are the records in the session table? > > 2) I tried to upload a non-phantom file like this > > input_file.txt: > /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1 > > Terminal: > ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > > log.txt > > and i get the following error in the log.txt file > > "Running now the following command: /data/loris/data/uploadNeuroDB/ > imaging_upload_file.pl -profile prod -upload_id 7 > /data/incoming/DCC0000_258024_V1.tar.gz -verbose > > find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name '__MACOSX' -delete > Spool message is: > The PatientName read from the DICOM header does not start with > DCC0000_258024_V1 from the mri_upload table > " > > and the spool message is repeated for each .dcm file. Should i preprocess > every .dcm file and change the PatientName to "DCC0000_258024_V1"? > > 3) As far as it concerns the candidate profile i have to create a new one > each time before i try to upload a file? Is there already a way to create a > candidate profile and get the DCCID and PSCID without the user interface? > > Thank you, > > Sotirios > > ???? ???, 18 ???? 2019 ???? 5:00 ?.?., ?/? Cecile Madjar < > cecile.madjar at mcin.ca> ??????: > >> Hi Sotirios, >> >> See answers to your questions below. >> >> About the 1st case we had a dicom with a blank studyUID, maybe someone >>> accidentally removed it. >>> >> That is strange. Something must have been done to the DICOM files. I have >> never seen a study with no StudyUID out of the scanner. Usually, you need >> to manually erase it with some tool. However, DICOMs never cease to >> surprise me... >> >> >>> Some other questions: >>> 1) What are the steps to remove all of my patients and uploads?: because >>> i want to try to reupload all the testing dicoms we had with a clean >>> database. >>> >> >> Release 21.0 of LORIS-MRI will offer a script to remove all the entries >> and files specific to an upload. However, this release is not yet out. >> Hopefully in the next few weeks. >> In the meantime, since your case is particular and that you just want to >> start with a clean database, the following deletes should work (hopefully I >> won't forget any tables): >> delete from mri_violations_log; >> delete from mri_protocol_violated_scans; >> delete from MRICandidateErrors; >> delete from parameter_file; >> delete from files_qcstatus; # (if you played with the QC part of the >> imaging browser for testing) >> delete from feedback_mri_comments; # (if you played with the QC part of >> the imaging browser for testing) >> delete from tarchive_series; >> delete from tarchive_files; >> delete from mri_upload; >> delete from tarchive; >> delete from mri_scanner; >> delete from candidate where PSCID="scanner"; >> >> I don't know if you want to also delete entries in the session and >> candidate table too? >> >> 2) We do not need to create the profile of a patient ( are the patients >>> stored in the candidate table? because i can't see the ghosts ), because >>> will it automatically be created when importing a dicom? >>> >> >> I am not sure I understand fully this question. All candidates are stored >> in the candidate table. The phantom scans however are attached to a scanner >> candidate depending on where the scan happened. It is a bit of a weird >> concept that we have to redesign eventually but never got a chance to get >> to it. >> >> Hopefully this answered your question, otherwise, don't hesitate to let >> me know. >> >> 3) What is the difference between uploading a dicom as phantom instead of >>> importing as PSCCID_DCCID_VisitLabel? >>> >> >> When you upload the scan as PSCID_DCCID_VisitLabel, there is a check that >> makes sure the candidate IDs and visit label are valid at the time of >> upload. >> When you upload a scan as phantom, it expects that the DICOM field >> PatientName and uploaded filename contains the string "phantom". We >> enforced this behaviour on the imaging uploader side in recent releases but >> I can't remember which one. Probably the upcoming 21.0 release. >> >> All those verifications might seem cumbersome but they are here to ensure >> that the files inserted are all valid and labelled properly as it is a bit >> messy to have to delete files that were wrongly labelled. At least, the >> delete script present in release 21.0 will make this process easier but it >> is still good practice to verify those things. >> >> >>> Thank you, >>> >> >> With pleasure. Hopefully the answers to your questions will be helpful. >> >> Best, >> >> C?cile >> >> PS: loris-dev mailing list: you will find the initial answer I gave >> Sotirios below. I forgot to cc the loris-dev in my earlier reply... >> >> ???? ???, 16 ???? 2019 ???? 12:02 ?.?., ?/? Cecile Madjar < >>> cecile.madjar at mcin.ca> ??????: >>> >>>> Dear Sotirios, >>>> >>>> Thank you for reaching out. Since your email was already organized in >>>> points, I will reply directly below your questions below. >>>> >>>> Best, >>>> >>>> C?cile >>>> >>>> 1) What do we do when the StudyUID is blank on the header? >>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md >>>>> There solution provided at the table 3 does not guide us how to find the >>>>> information ( is possible ). >>>>> >>>> >>>> The StudyUID is the field used in our pipeline at the moment to check >>>> whether a DICOM study was already inserted into LORIS with that same >>>> StudyUID since it is supposed to be unique for every single study. >>>> Currently, we do not support insertion of DICOM studies if they do not have >>>> a StudyUID associated with them. >>>> >>>> Is there a specific reason why the StudyUID is blank in the DICOM >>>> headers in your DICOM files? Did they go through some processes before >>>> upload to LORIS? >>>> >>>> >>>> 2) When i am about to upload a non-phanton case i need a 'Visit Label'. >>>>> Is there already a script that can create a new visit label for a specific >>>>> Candidate, so i can provide it as input afterwards to the >>>>> >>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md ) >>>>> >>>>> >>>> I am not sure if the front-end allows for it but if you use >>>> batch_uploads_imageuploader, the insertion scripts could create the visit >>>> label for you. However, it does need to be included in the patient name in >>>> order for the script to know which label it should use. >>>> For example: you upload V02 for MTL0123 (DCCID: 456789) but V02 was >>>> not yet created for MTL0123; the insertion scripts will create a V02 visit >>>> for MTL0123 if the PatientName field for the dataset is MTL0123_456789_V02 >>>> with Stage marked as "Not Started". >>>> >>>> However, *note that for the script to create the visit for the >>>> candidate, you have to make sure that all the visit label of your projects >>>> were inserted in the Visit_Windows table* (otherwise, the visit will >>>> never be created as they were not specified as being part of the list of >>>> visit label to expect). So in the example above, you should have one row in >>>> Visit_Windows with Visit_label="V02" set. >>>> >>>> >>>>> 3) Section 4.3 >>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md >>>>> >>>>> What do we do when the folder contains multiple mnc files? >>>>> >>>>> This is a case by case. Most of the time, the MINC files that failed >>>> insertion into the imaging browser end up in the MRI violation module where >>>> you can see what went wrong with the acquisitions not inserted. If you >>>> notice that the MINC file should be inserted as a specific protocol, you >>>> can force the insertion as explained in section 4.3. >>>> >>>> In your case, I have a feeling that you want to be able to insert MINC >>>> files that did not go through the whole pipeline insertion (DICOM archival, >>>> then dcm2mnc, then protocol identification, then insertion...). In theory, >>>> you could call minc_insertion.pl on any MINC file, provided you have >>>> at least the following information: >>>> - path to the MINC file >>>> - uploadID associated with the MINC files or DICOM archive path from >>>> which those MINC files were created from >>>> >>>> Then the minc_insertion.pl script will do the subject information and >>>> protocol validation etc... >>>> >>>> If all the MINC files come from the same uploadID or TarchiveID, then >>>> you could run a loop in bash calling the minc_insertion.pl script like >>>> this for files deriving from uploadID=1: >>>> >>>> ls /path/to/mnc/folder/* | while read f; do minc_insertion.pl -profile >>>> prod -uploadID 1 -mincPath $f; done >>>> >>>> >>>>> Thank you, >>>>> >>>> >>>> Hope this helped! >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From cecile.madjar at mcin.ca Wed Jul 24 10:05:44 2019 From: cecile.madjar at mcin.ca (Cecile Madjar) Date: Wed, 24 Jul 2019 10:05:44 -0400 Subject: [Loris-dev] Import mri - scripts In-Reply-To: References: Message-ID: Hi Sotirios, Sorry I was not clear before. Every visits of candidates are created in the session table with one entry per CandID/VisitLabel. So if the imaging pipeline created visits in the session table (that are not attached to instruments) and you want to clean your database completely, you can delete entries in that table as well. Regarding your point 2, the imaging pipeline can create the visit in the session table as long as the visit label is present in the Visit_Windows table (which means the visit label stored in the PatientName is a valid visit label). However, in your prod file, if the variable $subjectID{'createVisitLabel'} is set to 1 for candidates, the visit label should be created in the Visit_Windows table I believe. Here is the line of the prod file you would need to change . In general, we prefer populating the Visit_Windows table and set the $subjectID{'createVisitLabel'} to 0 for candidate data to avoid insertion of badly labelled MRI data that are complex to relabel. Finally, regarding the candidate creation, if you execute the following steps, the pipeline should be able to create candidates: - set in the Imaging Pipeline section of the config module the config setting "Upload creation of candidates" to Yes - manually transfer the scans to the LORIS server instead of uploading it via the imaging browser - run batch_imaging_upload.pl as you did until now. Hope this helps! Best, C?cile On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos < sotirisnik at gmail.com> wrote: > About my second question i replaced the name of the patient in the .dcm > files and now i am getting an error that the visit label does not exist. > I thought it was supposed to be automatically created. > > "Done adding archive info into database > > /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod > /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar > -verbose > md5sum > /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar > PSCID is: DCC0000 > CandID id: 258024 > visit_label is: V1 > PSCID is: DCC0000 > CandID id: 258024 > visit_label is: V1 > candidate id 258024 > > > => No Visit labelVisit label does not exist > Set centerID = 1 > PSCID is: DCC0000 > CandID id: 258024 > visit_label is: V1 > PSCID is: DCC0000 > CandID id: 258024 > visit_label is: V1 > > Number of MINC files that will be considered for inserting into the > database: 1 > > log dir is /logs and log file is /logs/TarLoad-15-42-L6pocU.log > PSCID is: DCC0000 > CandID id: 258024 > visit_label is: V1 > PSCID is: DCC0000 > CandID id: 258024 > visit_label is: V1 > candidate id 258024 > > > => No Visit label > Cleaning up temp files: rm -rf > /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*" > > ???? ???, 24 ???? 2019 ???? 3:38 ?.?., ?/? Sotirios Nikoloutsopoulos < > sotirisnik at gmail.com> ??????: > >> Hi, >> >> 1) What are the records in the session table? >> >> 2) I tried to upload a non-phantom file like this >> >> input_file.txt: >> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1 >> >> Terminal: >> ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > >> log.txt >> >> and i get the following error in the log.txt file >> >> "Running now the following command: /data/loris/data/uploadNeuroDB/ >> imaging_upload_file.pl -profile prod -upload_id 7 >> /data/incoming/DCC0000_258024_V1.tar.gz -verbose >> >> find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name '__MACOSX' -delete >> >> Spool message is: >> The PatientName read from the DICOM header does not start with >> DCC0000_258024_V1 from the mri_upload table >> " >> >> and the spool message is repeated for each .dcm file. Should i preprocess >> every .dcm file and change the PatientName to "DCC0000_258024_V1"? >> >> 3) As far as it concerns the candidate profile i have to create a new one >> each time before i try to upload a file? Is there already a way to create a >> candidate profile and get the DCCID and PSCID without the user interface? >> >> Thank you, >> >> Sotirios >> >> ???? ???, 18 ???? 2019 ???? 5:00 ?.?., ?/? Cecile Madjar < >> cecile.madjar at mcin.ca> ??????: >> >>> Hi Sotirios, >>> >>> See answers to your questions below. >>> >>> About the 1st case we had a dicom with a blank studyUID, maybe someone >>>> accidentally removed it. >>>> >>> That is strange. Something must have been done to the DICOM files. I >>> have never seen a study with no StudyUID out of the scanner. Usually, you >>> need to manually erase it with some tool. However, DICOMs never cease to >>> surprise me... >>> >>> >>>> Some other questions: >>>> 1) What are the steps to remove all of my patients and uploads?: >>>> because i want to try to reupload all the testing dicoms we had with a >>>> clean database. >>>> >>> >>> Release 21.0 of LORIS-MRI will offer a script to remove all the entries >>> and files specific to an upload. However, this release is not yet out. >>> Hopefully in the next few weeks. >>> In the meantime, since your case is particular and that you just want to >>> start with a clean database, the following deletes should work (hopefully I >>> won't forget any tables): >>> delete from mri_violations_log; >>> delete from mri_protocol_violated_scans; >>> delete from MRICandidateErrors; >>> delete from parameter_file; >>> delete from files_qcstatus; # (if you played with the QC part of the >>> imaging browser for testing) >>> delete from feedback_mri_comments; # (if you played with the QC part of >>> the imaging browser for testing) >>> delete from tarchive_series; >>> delete from tarchive_files; >>> delete from mri_upload; >>> delete from tarchive; >>> delete from mri_scanner; >>> delete from candidate where PSCID="scanner"; >>> >>> I don't know if you want to also delete entries in the session and >>> candidate table too? >>> >>> 2) We do not need to create the profile of a patient ( are the patients >>>> stored in the candidate table? because i can't see the ghosts ), because >>>> will it automatically be created when importing a dicom? >>>> >>> >>> I am not sure I understand fully this question. All candidates are >>> stored in the candidate table. The phantom scans however are attached to a >>> scanner candidate depending on where the scan happened. It is a bit of a >>> weird concept that we have to redesign eventually but never got a chance to >>> get to it. >>> >>> Hopefully this answered your question, otherwise, don't hesitate to let >>> me know. >>> >>> 3) What is the difference between uploading a dicom as phantom instead >>>> of importing as PSCCID_DCCID_VisitLabel? >>>> >>> >>> When you upload the scan as PSCID_DCCID_VisitLabel, there is a check >>> that makes sure the candidate IDs and visit label are valid at the time of >>> upload. >>> When you upload a scan as phantom, it expects that the DICOM field >>> PatientName and uploaded filename contains the string "phantom". We >>> enforced this behaviour on the imaging uploader side in recent releases but >>> I can't remember which one. Probably the upcoming 21.0 release. >>> >>> All those verifications might seem cumbersome but they are here to >>> ensure that the files inserted are all valid and labelled properly as it is >>> a bit messy to have to delete files that were wrongly labelled. At least, >>> the delete script present in release 21.0 will make this process easier but >>> it is still good practice to verify those things. >>> >>> >>>> Thank you, >>>> >>> >>> With pleasure. Hopefully the answers to your questions will be helpful. >>> >>> Best, >>> >>> C?cile >>> >>> PS: loris-dev mailing list: you will find the initial answer I gave >>> Sotirios below. I forgot to cc the loris-dev in my earlier reply... >>> >>> ???? ???, 16 ???? 2019 ???? 12:02 ?.?., ?/? Cecile Madjar < >>>> cecile.madjar at mcin.ca> ??????: >>>> >>>>> Dear Sotirios, >>>>> >>>>> Thank you for reaching out. Since your email was already organized in >>>>> points, I will reply directly below your questions below. >>>>> >>>>> Best, >>>>> >>>>> C?cile >>>>> >>>>> 1) What do we do when the StudyUID is blank on the header? >>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md >>>>>> There solution provided at the table 3 does not guide us how to find the >>>>>> information ( is possible ). >>>>>> >>>>> >>>>> The StudyUID is the field used in our pipeline at the moment to check >>>>> whether a DICOM study was already inserted into LORIS with that same >>>>> StudyUID since it is supposed to be unique for every single study. >>>>> Currently, we do not support insertion of DICOM studies if they do not have >>>>> a StudyUID associated with them. >>>>> >>>>> Is there a specific reason why the StudyUID is blank in the DICOM >>>>> headers in your DICOM files? Did they go through some processes before >>>>> upload to LORIS? >>>>> >>>>> >>>>> 2) When i am about to upload a non-phanton case i need a 'Visit >>>>>> Label'. Is there already a script that can create a new visit label for a >>>>>> specific Candidate, so i can provide it as input afterwards to the >>>>>> >>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md ) >>>>>> >>>>>> >>>>> I am not sure if the front-end allows for it but if you use >>>>> batch_uploads_imageuploader, the insertion scripts could create the visit >>>>> label for you. However, it does need to be included in the patient name in >>>>> order for the script to know which label it should use. >>>>> For example: you upload V02 for MTL0123 (DCCID: 456789) but V02 was >>>>> not yet created for MTL0123; the insertion scripts will create a V02 visit >>>>> for MTL0123 if the PatientName field for the dataset is MTL0123_456789_V02 >>>>> with Stage marked as "Not Started". >>>>> >>>>> However, *note that for the script to create the visit for the >>>>> candidate, you have to make sure that all the visit label of your projects >>>>> were inserted in the Visit_Windows table* (otherwise, the visit will >>>>> never be created as they were not specified as being part of the list of >>>>> visit label to expect). So in the example above, you should have one row in >>>>> Visit_Windows with Visit_label="V02" set. >>>>> >>>>> >>>>>> 3) Section 4.3 >>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md >>>>>> >>>>>> What do we do when the folder contains multiple mnc files? >>>>>> >>>>>> This is a case by case. Most of the time, the MINC files that failed >>>>> insertion into the imaging browser end up in the MRI violation module where >>>>> you can see what went wrong with the acquisitions not inserted. If you >>>>> notice that the MINC file should be inserted as a specific protocol, you >>>>> can force the insertion as explained in section 4.3. >>>>> >>>>> In your case, I have a feeling that you want to be able to insert MINC >>>>> files that did not go through the whole pipeline insertion (DICOM archival, >>>>> then dcm2mnc, then protocol identification, then insertion...). In theory, >>>>> you could call minc_insertion.pl on any MINC file, provided you have >>>>> at least the following information: >>>>> - path to the MINC file >>>>> - uploadID associated with the MINC files or DICOM archive path from >>>>> which those MINC files were created from >>>>> >>>>> Then the minc_insertion.pl script will do the subject information and >>>>> protocol validation etc... >>>>> >>>>> If all the MINC files come from the same uploadID or TarchiveID, then >>>>> you could run a loop in bash calling the minc_insertion.pl script >>>>> like this for files deriving from uploadID=1: >>>>> >>>>> ls /path/to/mnc/folder/* | while read f; do minc_insertion.pl >>>>> -profile prod -uploadID 1 -mincPath $f; done >>>>> >>>>> >>>>>> Thank you, >>>>>> >>>>> >>>>> Hope this helped! >>>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From sotirisnik at gmail.com Wed Jul 24 10:55:13 2019 From: sotirisnik at gmail.com (Sotirios Nikoloutsopoulos) Date: Wed, 24 Jul 2019 17:55:13 +0300 Subject: [Loris-dev] Import mri - scripts In-Reply-To: References: Message-ID: Ok first i execute this command: "lorisadmin at hbp:/data/loris/bin/mri$ ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > log.txt Use of uninitialized value $data_dir in concatenation (.) or string at ./batch_uploads_imageuploader line 143. Use of uninitialized value $data_dir in concatenation (.) or string at ./batch_uploads_imageuploader line 143. Use of uninitialized value $_ in pattern match (m//) at ./batch_uploads_imageuploader line 147. base is DCC0000_258024_V01 path is /data/incoming/ type is .tar.gz fullpath is /data/incoming/DCC0000_258024_V01.tar.gz source /tmp/ImagingUpload-17-43-ZUEFjb targetlocation /data/loris/data/tarchive Use of uninitialized value $data_dir in concatenation (.) or string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256. Use of uninitialized value $data_dir in concatenation (.) or string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284. Use of uninitialized value $data_dir in concatenation (.) or string at /data/loris/bin/mri/uploadNeuroDB/tarchive_validation.pl line 219. Use of uninitialized value $data_dir in concatenation (.) or string at /data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl line 270. Use of uninitialized value $data_dir in concatenation (.) or string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711. Can't exec "mail": No such file or directory at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721. print() on closed filehandle MAIL at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722. print() on closed filehandle MAIL at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723. Use of uninitialized value $data_dir in concatenation (.) or string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. print() on closed filehandle MAIL at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. print() on closed filehandle MAIL at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725. Use of uninitialized value $data_dir in concatenation (.) or string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735. No Mincs inserted The tarchiveLoader insertion script has failed. Running now the following command: /data/loris/data/uploadNeuroDB/ imaging_upload_file.pl -profile prod -upload_id 3 /data/incoming/DCC0000_258024_V01.tar.gz -verbose" I inserted the VisitLabel, but still the Mincs are not inserted and i am getting this error "lorisadmin at hbp:/data/loris/bin/mri$ /data/loris/data/uploadNeuroDB/ imaging_upload_file.pl -profile prod -upload_id 3 /data/incoming/DCC0000_258024_V01.tar.gz -verbose find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name '__MACOSX' -delete Spool message is: The Scan for the uploadID 3 has already been run with tarchiveID: 7. To continue with the rest of the insertion pipeline, please run tarchiveLoader from a terminal as follows: /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar -verbose" Which files should i delete? I thought that deleting the files from the tarchive was enough. And i executed the last proposed command to continue to rest of the insertion pipeline ( check the image ). Thank you ???? ???, 24 ???? 2019 ???? 5:05 ?.?., ?/? Cecile Madjar < cecile.madjar at mcin.ca> ??????: > Hi Sotirios, > > Sorry I was not clear before. Every visits of candidates are created in > the session table with one entry per CandID/VisitLabel. So if the imaging > pipeline created visits in the session table (that are not attached to > instruments) and you want to clean your database completely, you can delete > entries in that table as well. > > Regarding your point 2, the imaging pipeline can create the visit in the > session table as long as the visit label is present in the Visit_Windows > table (which means the visit label stored in the PatientName is a valid > visit label). > However, in your prod file, if the variable $subjectID{'createVisitLabel'} > is set to 1 for candidates, the visit label should be created in the > Visit_Windows table I believe. Here is the line of the prod file you > would need to change > > . > In general, we prefer populating the Visit_Windows table and set the > $subjectID{'createVisitLabel'} to 0 for candidate data to avoid insertion > of badly labelled MRI data that are complex to relabel. > > Finally, regarding the candidate creation, if you execute the following > steps, the pipeline should be able to create candidates: > - set in the Imaging Pipeline section of the config module the config > setting "Upload creation of candidates" to Yes > - manually transfer the scans to the LORIS server instead of uploading it > via the imaging browser > - run batch_imaging_upload.pl as you did until now. > > Hope this helps! > Best, > > C?cile > > On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos < > sotirisnik at gmail.com> wrote: > >> About my second question i replaced the name of the patient in the .dcm >> files and now i am getting an error that the visit label does not exist. >> I thought it was supposed to be automatically created. >> >> "Done adding archive info into database >> >> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile >> prod >> /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >> -verbose >> md5sum >> /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >> PSCID is: DCC0000 >> CandID id: 258024 >> visit_label is: V1 >> PSCID is: DCC0000 >> CandID id: 258024 >> visit_label is: V1 >> candidate id 258024 >> >> >> => No Visit labelVisit label does not exist >> Set centerID = 1 >> PSCID is: DCC0000 >> CandID id: 258024 >> visit_label is: V1 >> PSCID is: DCC0000 >> CandID id: 258024 >> visit_label is: V1 >> >> Number of MINC files that will be considered for inserting into the >> database: 1 >> >> log dir is /logs and log file is /logs/TarLoad-15-42-L6pocU.log >> PSCID is: DCC0000 >> CandID id: 258024 >> visit_label is: V1 >> PSCID is: DCC0000 >> CandID id: 258024 >> visit_label is: V1 >> candidate id 258024 >> >> >> => No Visit label >> Cleaning up temp files: rm -rf >> /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*" >> >> ???? ???, 24 ???? 2019 ???? 3:38 ?.?., ?/? Sotirios Nikoloutsopoulos < >> sotirisnik at gmail.com> ??????: >> >>> Hi, >>> >>> 1) What are the records in the session table? >>> >>> 2) I tried to upload a non-phantom file like this >>> >>> input_file.txt: >>> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1 >>> >>> Terminal: >>> ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > >>> log.txt >>> >>> and i get the following error in the log.txt file >>> >>> "Running now the following command: /data/loris/data/uploadNeuroDB/ >>> imaging_upload_file.pl -profile prod -upload_id 7 >>> /data/incoming/DCC0000_258024_V1.tar.gz -verbose >>> >>> find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name '__MACOSX' >>> -delete >>> Spool message is: >>> The PatientName read from the DICOM header does not start with >>> DCC0000_258024_V1 from the mri_upload table >>> " >>> >>> and the spool message is repeated for each .dcm file. Should i >>> preprocess every .dcm file and change the PatientName to >>> "DCC0000_258024_V1"? >>> >>> 3) As far as it concerns the candidate profile i have to create a new >>> one each time before i try to upload a file? Is there already a way to >>> create a candidate profile and get the DCCID and PSCID without the user >>> interface? >>> >>> Thank you, >>> >>> Sotirios >>> >>> ???? ???, 18 ???? 2019 ???? 5:00 ?.?., ?/? Cecile Madjar < >>> cecile.madjar at mcin.ca> ??????: >>> >>>> Hi Sotirios, >>>> >>>> See answers to your questions below. >>>> >>>> About the 1st case we had a dicom with a blank studyUID, maybe someone >>>>> accidentally removed it. >>>>> >>>> That is strange. Something must have been done to the DICOM files. I >>>> have never seen a study with no StudyUID out of the scanner. Usually, you >>>> need to manually erase it with some tool. However, DICOMs never cease to >>>> surprise me... >>>> >>>> >>>>> Some other questions: >>>>> 1) What are the steps to remove all of my patients and uploads?: >>>>> because i want to try to reupload all the testing dicoms we had with a >>>>> clean database. >>>>> >>>> >>>> Release 21.0 of LORIS-MRI will offer a script to remove all the entries >>>> and files specific to an upload. However, this release is not yet out. >>>> Hopefully in the next few weeks. >>>> In the meantime, since your case is particular and that you just want >>>> to start with a clean database, the following deletes should work >>>> (hopefully I won't forget any tables): >>>> delete from mri_violations_log; >>>> delete from mri_protocol_violated_scans; >>>> delete from MRICandidateErrors; >>>> delete from parameter_file; >>>> delete from files_qcstatus; # (if you played with the QC part of the >>>> imaging browser for testing) >>>> delete from feedback_mri_comments; # (if you played with the QC part of >>>> the imaging browser for testing) >>>> delete from tarchive_series; >>>> delete from tarchive_files; >>>> delete from mri_upload; >>>> delete from tarchive; >>>> delete from mri_scanner; >>>> delete from candidate where PSCID="scanner"; >>>> >>>> I don't know if you want to also delete entries in the session and >>>> candidate table too? >>>> >>>> 2) We do not need to create the profile of a patient ( are the patients >>>>> stored in the candidate table? because i can't see the ghosts ), because >>>>> will it automatically be created when importing a dicom? >>>>> >>>> >>>> I am not sure I understand fully this question. All candidates are >>>> stored in the candidate table. The phantom scans however are attached to a >>>> scanner candidate depending on where the scan happened. It is a bit of a >>>> weird concept that we have to redesign eventually but never got a chance to >>>> get to it. >>>> >>>> Hopefully this answered your question, otherwise, don't hesitate to let >>>> me know. >>>> >>>> 3) What is the difference between uploading a dicom as phantom instead >>>>> of importing as PSCCID_DCCID_VisitLabel? >>>>> >>>> >>>> When you upload the scan as PSCID_DCCID_VisitLabel, there is a check >>>> that makes sure the candidate IDs and visit label are valid at the time of >>>> upload. >>>> When you upload a scan as phantom, it expects that the DICOM field >>>> PatientName and uploaded filename contains the string "phantom". We >>>> enforced this behaviour on the imaging uploader side in recent releases but >>>> I can't remember which one. Probably the upcoming 21.0 release. >>>> >>>> All those verifications might seem cumbersome but they are here to >>>> ensure that the files inserted are all valid and labelled properly as it is >>>> a bit messy to have to delete files that were wrongly labelled. At least, >>>> the delete script present in release 21.0 will make this process easier but >>>> it is still good practice to verify those things. >>>> >>>> >>>>> Thank you, >>>>> >>>> >>>> With pleasure. Hopefully the answers to your questions will be helpful. >>>> >>>> Best, >>>> >>>> C?cile >>>> >>>> PS: loris-dev mailing list: you will find the initial answer I gave >>>> Sotirios below. I forgot to cc the loris-dev in my earlier reply... >>>> >>>> ???? ???, 16 ???? 2019 ???? 12:02 ?.?., ?/? Cecile Madjar < >>>>> cecile.madjar at mcin.ca> ??????: >>>>> >>>>>> Dear Sotirios, >>>>>> >>>>>> Thank you for reaching out. Since your email was already organized in >>>>>> points, I will reply directly below your questions below. >>>>>> >>>>>> Best, >>>>>> >>>>>> C?cile >>>>>> >>>>>> 1) What do we do when the StudyUID is blank on the header? >>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md >>>>>>> There solution provided at the table 3 does not guide us how to find the >>>>>>> information ( is possible ). >>>>>>> >>>>>> >>>>>> The StudyUID is the field used in our pipeline at the moment to check >>>>>> whether a DICOM study was already inserted into LORIS with that same >>>>>> StudyUID since it is supposed to be unique for every single study. >>>>>> Currently, we do not support insertion of DICOM studies if they do not have >>>>>> a StudyUID associated with them. >>>>>> >>>>>> Is there a specific reason why the StudyUID is blank in the DICOM >>>>>> headers in your DICOM files? Did they go through some processes before >>>>>> upload to LORIS? >>>>>> >>>>>> >>>>>> 2) When i am about to upload a non-phanton case i need a 'Visit >>>>>>> Label'. Is there already a script that can create a new visit label for a >>>>>>> specific Candidate, so i can provide it as input afterwards to the >>>>>>> >>>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md ) >>>>>>> >>>>>>> >>>>>> I am not sure if the front-end allows for it but if you use >>>>>> batch_uploads_imageuploader, the insertion scripts could create the visit >>>>>> label for you. However, it does need to be included in the patient name in >>>>>> order for the script to know which label it should use. >>>>>> For example: you upload V02 for MTL0123 (DCCID: 456789) but V02 was >>>>>> not yet created for MTL0123; the insertion scripts will create a V02 visit >>>>>> for MTL0123 if the PatientName field for the dataset is MTL0123_456789_V02 >>>>>> with Stage marked as "Not Started". >>>>>> >>>>>> However, *note that for the script to create the visit for the >>>>>> candidate, you have to make sure that all the visit label of your projects >>>>>> were inserted in the Visit_Windows table* (otherwise, the visit will >>>>>> never be created as they were not specified as being part of the list of >>>>>> visit label to expect). So in the example above, you should have one row in >>>>>> Visit_Windows with Visit_label="V02" set. >>>>>> >>>>>> >>>>>>> 3) Section 4.3 >>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md >>>>>>> >>>>>>> What do we do when the folder contains multiple mnc files? >>>>>>> >>>>>>> This is a case by case. Most of the time, the MINC files that failed >>>>>> insertion into the imaging browser end up in the MRI violation module where >>>>>> you can see what went wrong with the acquisitions not inserted. If you >>>>>> notice that the MINC file should be inserted as a specific protocol, you >>>>>> can force the insertion as explained in section 4.3. >>>>>> >>>>>> In your case, I have a feeling that you want to be able to insert >>>>>> MINC files that did not go through the whole pipeline insertion (DICOM >>>>>> archival, then dcm2mnc, then protocol identification, then insertion...). >>>>>> In theory, you could call minc_insertion.pl on any MINC file, >>>>>> provided you have at least the following information: >>>>>> - path to the MINC file >>>>>> - uploadID associated with the MINC files or DICOM archive path from >>>>>> which those MINC files were created from >>>>>> >>>>>> Then the minc_insertion.pl script will do the subject information >>>>>> and protocol validation etc... >>>>>> >>>>>> If all the MINC files come from the same uploadID or TarchiveID, then >>>>>> you could run a loop in bash calling the minc_insertion.pl script >>>>>> like this for files deriving from uploadID=1: >>>>>> >>>>>> ls /path/to/mnc/folder/* | while read f; do minc_insertion.pl >>>>>> -profile prod -uploadID 1 -mincPath $f; done >>>>>> >>>>>> >>>>>>> Thank you, >>>>>>> >>>>>> >>>>>> Hope this helped! >>>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: loris.png Type: image/png Size: 202034 bytes Desc: not available URL: From cecile.madjar at mcin.ca Wed Jul 24 11:05:08 2019 From: cecile.madjar at mcin.ca (Cecile Madjar) Date: Wed, 24 Jul 2019 11:05:08 -0400 Subject: [Loris-dev] Import mri - scripts In-Reply-To: References: Message-ID: Hi Sotirios, It looks like you have one Config setting not set (based on the first error message you got when running it the first time). You need to set a value for "Loris-MRI Data Directory" in the Config module under the Imaging pipeline section (from what I can see, it would be /data/loris/data/ for your setup). The reason you got that error message the second time is that the pipeline checks if the DICOM study was already inserted in the tarchive table. Since you ran the study once already, the study is inserted into the tarchive, hence the message. However, the pipeline did not complete as the MINC files were not created and inserted into the files/parameter_file table. In order to continue the insertion, you will then need to run the tarchiveLoader command suggested in the terminal during the second run: /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar -verbose Finally, I am a bit confused by your setup. Where are the scripts located? In /data/loris/bin/mri or /data/loris/data? It feels like the Config setting "LORIS-MRI code" in the Paths section was set to "/data/LORIS/data" but it should have been set to the directory where the scripts are located (most probably /data/loris/bin/mri). Hope this helps, C?cile On Wed, Jul 24, 2019 at 10:55 AM Sotirios Nikoloutsopoulos < sotirisnik at gmail.com> wrote: > Ok first i execute this command: > > "lorisadmin at hbp:/data/loris/bin/mri$ ./batch_uploads_imageuploader > -profile prod < ~/Desktop/input.txt > log.txt > Use of uninitialized value $data_dir in concatenation (.) or string at > ./batch_uploads_imageuploader line 143. > Use of uninitialized value $data_dir in concatenation (.) or string at > ./batch_uploads_imageuploader line 143. > Use of uninitialized value $_ in pattern match (m//) at > ./batch_uploads_imageuploader line 147. > base is DCC0000_258024_V01 > path is /data/incoming/ > type is .tar.gz > fullpath is /data/incoming/DCC0000_258024_V01.tar.gz > > source /tmp/ImagingUpload-17-43-ZUEFjb > targetlocation /data/loris/data/tarchive > > Use of uninitialized value $data_dir in concatenation (.) or string at > /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256. > Use of uninitialized value $data_dir in concatenation (.) or string at > /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284. > Use of uninitialized value $data_dir in concatenation (.) or string at > /data/loris/bin/mri/uploadNeuroDB/tarchive_validation.pl line 219. > Use of uninitialized value $data_dir in concatenation (.) or string at > /data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl line 270. > Use of uninitialized value $data_dir in concatenation (.) or string at > /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711. > Can't exec "mail": No such file or directory at > /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721. > print() on closed filehandle MAIL at > /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722. > print() on closed filehandle MAIL at > /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723. > Use of uninitialized value $data_dir in concatenation (.) or string at > /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. > print() on closed filehandle MAIL at > /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. > print() on closed filehandle MAIL at > /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725. > Use of uninitialized value $data_dir in concatenation (.) or string at > /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735. > > No Mincs inserted > > > The tarchiveLoader insertion script has failed. > Running now the following command: /data/loris/data/uploadNeuroDB/ > imaging_upload_file.pl -profile prod -upload_id 3 > /data/incoming/DCC0000_258024_V01.tar.gz -verbose" > > I inserted the VisitLabel, but still the Mincs are not inserted and i am > getting this error > > "lorisadmin at hbp:/data/loris/bin/mri$ /data/loris/data/uploadNeuroDB/ > imaging_upload_file.pl -profile prod -upload_id 3 > /data/incoming/DCC0000_258024_V01.tar.gz -verbose > > find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name '__MACOSX' -delete > Spool message is: > The Scan for the uploadID 3 has already been run with tarchiveID: 7. > To continue with the rest of the insertion pipeline, please run > tarchiveLoader from a terminal as follows: > /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod > /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar > -verbose" > > Which files should i delete? I thought that deleting the files from the > tarchive was enough. And i executed the last proposed command to continue > to rest of the insertion pipeline ( check the image ). > > Thank you > > ???? ???, 24 ???? 2019 ???? 5:05 ?.?., ?/? Cecile Madjar < > cecile.madjar at mcin.ca> ??????: > >> Hi Sotirios, >> >> Sorry I was not clear before. Every visits of candidates are created in >> the session table with one entry per CandID/VisitLabel. So if the imaging >> pipeline created visits in the session table (that are not attached to >> instruments) and you want to clean your database completely, you can delete >> entries in that table as well. >> >> Regarding your point 2, the imaging pipeline can create the visit in the >> session table as long as the visit label is present in the Visit_Windows >> table (which means the visit label stored in the PatientName is a valid >> visit label). >> However, in your prod file, if the >> variable $subjectID{'createVisitLabel'} is set to 1 for candidates, the >> visit label should be created in the Visit_Windows table I believe. Here >> is the line of the prod file you would need to change >> >> . >> In general, we prefer populating the Visit_Windows table and set the >> $subjectID{'createVisitLabel'} to 0 for candidate data to avoid insertion >> of badly labelled MRI data that are complex to relabel. >> >> Finally, regarding the candidate creation, if you execute the following >> steps, the pipeline should be able to create candidates: >> - set in the Imaging Pipeline section of the config module the config >> setting "Upload creation of candidates" to Yes >> - manually transfer the scans to the LORIS server instead of uploading it >> via the imaging browser >> - run batch_imaging_upload.pl as you did until now. >> >> Hope this helps! >> Best, >> >> C?cile >> >> On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos < >> sotirisnik at gmail.com> wrote: >> >>> About my second question i replaced the name of the patient in the .dcm >>> files and now i am getting an error that the visit label does not exist. >>> I thought it was supposed to be automatically created. >>> >>> "Done adding archive info into database >>> >>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile >>> prod >>> /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>> -verbose >>> md5sum >>> /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>> PSCID is: DCC0000 >>> CandID id: 258024 >>> visit_label is: V1 >>> PSCID is: DCC0000 >>> CandID id: 258024 >>> visit_label is: V1 >>> candidate id 258024 >>> >>> >>> => No Visit labelVisit label does not exist >>> Set centerID = 1 >>> PSCID is: DCC0000 >>> CandID id: 258024 >>> visit_label is: V1 >>> PSCID is: DCC0000 >>> CandID id: 258024 >>> visit_label is: V1 >>> >>> Number of MINC files that will be considered for inserting into the >>> database: 1 >>> >>> log dir is /logs and log file is /logs/TarLoad-15-42-L6pocU.log >>> PSCID is: DCC0000 >>> CandID id: 258024 >>> visit_label is: V1 >>> PSCID is: DCC0000 >>> CandID id: 258024 >>> visit_label is: V1 >>> candidate id 258024 >>> >>> >>> => No Visit label >>> Cleaning up temp files: rm -rf >>> /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*" >>> >>> ???? ???, 24 ???? 2019 ???? 3:38 ?.?., ?/? Sotirios Nikoloutsopoulos < >>> sotirisnik at gmail.com> ??????: >>> >>>> Hi, >>>> >>>> 1) What are the records in the session table? >>>> >>>> 2) I tried to upload a non-phantom file like this >>>> >>>> input_file.txt: >>>> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1 >>>> >>>> Terminal: >>>> ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > >>>> log.txt >>>> >>>> and i get the following error in the log.txt file >>>> >>>> "Running now the following command: /data/loris/data/uploadNeuroDB/ >>>> imaging_upload_file.pl -profile prod -upload_id 7 >>>> /data/incoming/DCC0000_258024_V1.tar.gz -verbose >>>> >>>> find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name '__MACOSX' >>>> -delete >>>> Spool message is: >>>> The PatientName read from the DICOM header does not start with >>>> DCC0000_258024_V1 from the mri_upload table >>>> " >>>> >>>> and the spool message is repeated for each .dcm file. Should i >>>> preprocess every .dcm file and change the PatientName to >>>> "DCC0000_258024_V1"? >>>> >>>> 3) As far as it concerns the candidate profile i have to create a new >>>> one each time before i try to upload a file? Is there already a way to >>>> create a candidate profile and get the DCCID and PSCID without the user >>>> interface? >>>> >>>> Thank you, >>>> >>>> Sotirios >>>> >>>> ???? ???, 18 ???? 2019 ???? 5:00 ?.?., ?/? Cecile Madjar < >>>> cecile.madjar at mcin.ca> ??????: >>>> >>>>> Hi Sotirios, >>>>> >>>>> See answers to your questions below. >>>>> >>>>> About the 1st case we had a dicom with a blank studyUID, maybe someone >>>>>> accidentally removed it. >>>>>> >>>>> That is strange. Something must have been done to the DICOM files. I >>>>> have never seen a study with no StudyUID out of the scanner. Usually, you >>>>> need to manually erase it with some tool. However, DICOMs never cease to >>>>> surprise me... >>>>> >>>>> >>>>>> Some other questions: >>>>>> 1) What are the steps to remove all of my patients and uploads?: >>>>>> because i want to try to reupload all the testing dicoms we had with a >>>>>> clean database. >>>>>> >>>>> >>>>> Release 21.0 of LORIS-MRI will offer a script to remove all the >>>>> entries and files specific to an upload. However, this release is not yet >>>>> out. Hopefully in the next few weeks. >>>>> In the meantime, since your case is particular and that you just want >>>>> to start with a clean database, the following deletes should work >>>>> (hopefully I won't forget any tables): >>>>> delete from mri_violations_log; >>>>> delete from mri_protocol_violated_scans; >>>>> delete from MRICandidateErrors; >>>>> delete from parameter_file; >>>>> delete from files_qcstatus; # (if you played with the QC part of the >>>>> imaging browser for testing) >>>>> delete from feedback_mri_comments; # (if you played with the QC part >>>>> of the imaging browser for testing) >>>>> delete from tarchive_series; >>>>> delete from tarchive_files; >>>>> delete from mri_upload; >>>>> delete from tarchive; >>>>> delete from mri_scanner; >>>>> delete from candidate where PSCID="scanner"; >>>>> >>>>> I don't know if you want to also delete entries in the session and >>>>> candidate table too? >>>>> >>>>> 2) We do not need to create the profile of a patient ( are the >>>>>> patients stored in the candidate table? because i can't see the ghosts ), >>>>>> because will it automatically be created when importing a dicom? >>>>>> >>>>> >>>>> I am not sure I understand fully this question. All candidates are >>>>> stored in the candidate table. The phantom scans however are attached to a >>>>> scanner candidate depending on where the scan happened. It is a bit of a >>>>> weird concept that we have to redesign eventually but never got a chance to >>>>> get to it. >>>>> >>>>> Hopefully this answered your question, otherwise, don't hesitate to >>>>> let me know. >>>>> >>>>> 3) What is the difference between uploading a dicom as phantom instead >>>>>> of importing as PSCCID_DCCID_VisitLabel? >>>>>> >>>>> >>>>> When you upload the scan as PSCID_DCCID_VisitLabel, there is a check >>>>> that makes sure the candidate IDs and visit label are valid at the time of >>>>> upload. >>>>> When you upload a scan as phantom, it expects that the DICOM field >>>>> PatientName and uploaded filename contains the string "phantom". We >>>>> enforced this behaviour on the imaging uploader side in recent releases but >>>>> I can't remember which one. Probably the upcoming 21.0 release. >>>>> >>>>> All those verifications might seem cumbersome but they are here to >>>>> ensure that the files inserted are all valid and labelled properly as it is >>>>> a bit messy to have to delete files that were wrongly labelled. At least, >>>>> the delete script present in release 21.0 will make this process easier but >>>>> it is still good practice to verify those things. >>>>> >>>>> >>>>>> Thank you, >>>>>> >>>>> >>>>> With pleasure. Hopefully the answers to your questions will be helpful. >>>>> >>>>> Best, >>>>> >>>>> C?cile >>>>> >>>>> PS: loris-dev mailing list: you will find the initial answer I gave >>>>> Sotirios below. I forgot to cc the loris-dev in my earlier reply... >>>>> >>>>> ???? ???, 16 ???? 2019 ???? 12:02 ?.?., ?/? Cecile Madjar < >>>>>> cecile.madjar at mcin.ca> ??????: >>>>>> >>>>>>> Dear Sotirios, >>>>>>> >>>>>>> Thank you for reaching out. Since your email was already organized >>>>>>> in points, I will reply directly below your questions below. >>>>>>> >>>>>>> Best, >>>>>>> >>>>>>> C?cile >>>>>>> >>>>>>> 1) What do we do when the StudyUID is blank on the header? >>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md >>>>>>>> There solution provided at the table 3 does not guide us how to find the >>>>>>>> information ( is possible ). >>>>>>>> >>>>>>> >>>>>>> The StudyUID is the field used in our pipeline at the moment to >>>>>>> check whether a DICOM study was already inserted into LORIS with that same >>>>>>> StudyUID since it is supposed to be unique for every single study. >>>>>>> Currently, we do not support insertion of DICOM studies if they do not have >>>>>>> a StudyUID associated with them. >>>>>>> >>>>>>> Is there a specific reason why the StudyUID is blank in the DICOM >>>>>>> headers in your DICOM files? Did they go through some processes before >>>>>>> upload to LORIS? >>>>>>> >>>>>>> >>>>>>> 2) When i am about to upload a non-phanton case i need a 'Visit >>>>>>>> Label'. Is there already a script that can create a new visit label for a >>>>>>>> specific Candidate, so i can provide it as input afterwards to the >>>>>>>> >>>>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md ) >>>>>>>> >>>>>>>> >>>>>>> I am not sure if the front-end allows for it but if you use >>>>>>> batch_uploads_imageuploader, the insertion scripts could create the visit >>>>>>> label for you. However, it does need to be included in the patient name in >>>>>>> order for the script to know which label it should use. >>>>>>> For example: you upload V02 for MTL0123 (DCCID: 456789) but V02 was >>>>>>> not yet created for MTL0123; the insertion scripts will create a V02 visit >>>>>>> for MTL0123 if the PatientName field for the dataset is MTL0123_456789_V02 >>>>>>> with Stage marked as "Not Started". >>>>>>> >>>>>>> However, *note that for the script to create the visit for the >>>>>>> candidate, you have to make sure that all the visit label of your projects >>>>>>> were inserted in the Visit_Windows table* (otherwise, the visit >>>>>>> will never be created as they were not specified as being part of the list >>>>>>> of visit label to expect). So in the example above, you should have one row >>>>>>> in Visit_Windows with Visit_label="V02" set. >>>>>>> >>>>>>> >>>>>>>> 3) Section 4.3 >>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md >>>>>>>> >>>>>>>> What do we do when the folder contains multiple mnc files? >>>>>>>> >>>>>>>> This is a case by case. Most of the time, the MINC files that >>>>>>> failed insertion into the imaging browser end up in the MRI violation >>>>>>> module where you can see what went wrong with the acquisitions not >>>>>>> inserted. If you notice that the MINC file should be inserted as a specific >>>>>>> protocol, you can force the insertion as explained in section 4.3. >>>>>>> >>>>>>> In your case, I have a feeling that you want to be able to insert >>>>>>> MINC files that did not go through the whole pipeline insertion (DICOM >>>>>>> archival, then dcm2mnc, then protocol identification, then insertion...). >>>>>>> In theory, you could call minc_insertion.pl on any MINC file, >>>>>>> provided you have at least the following information: >>>>>>> - path to the MINC file >>>>>>> - uploadID associated with the MINC files or DICOM archive path from >>>>>>> which those MINC files were created from >>>>>>> >>>>>>> Then the minc_insertion.pl script will do the subject information >>>>>>> and protocol validation etc... >>>>>>> >>>>>>> If all the MINC files come from the same uploadID or TarchiveID, >>>>>>> then you could run a loop in bash calling the minc_insertion.pl >>>>>>> script like this for files deriving from uploadID=1: >>>>>>> >>>>>>> ls /path/to/mnc/folder/* | while read f; do minc_insertion.pl >>>>>>> -profile prod -uploadID 1 -mincPath $f; done >>>>>>> >>>>>>> >>>>>>>> Thank you, >>>>>>>> >>>>>>> >>>>>>> Hope this helped! >>>>>>> >>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From sotirisnik at gmail.com Wed Jul 24 17:59:59 2019 From: sotirisnik at gmail.com (Sotirios Nikoloutsopoulos) Date: Thu, 25 Jul 2019 00:59:59 +0300 Subject: [Loris-dev] Import mri - scripts In-Reply-To: References: Message-ID: The scripts are located in /data/loris/bin/mri/. Somewhere I might have messed something up but i remember following the instructions in the wiki.For example when i tried to execute the batch_uploads_imageuploader perl script the line my $command = "$bin_dir/uploadNeuroDB/imaging_upload_file.pl " was pointing to an invalid path, for example: it was repeating a pattern and had LORIS instead of loris ( i think the mistake was that in the wiki it suggested to use $PROJECT for the name of the folder and i choosed 'loris' https://github.com/aces/Loris/wiki/Imaging-Database ). See also the in image for the modification i did. Also i didn't find the section to config the $data_dir variable. Thank you, Sotirios ???? ???, 24 ???? 2019 ???? 6:05 ?.?., ?/? Cecile Madjar < cecile.madjar at mcin.ca> ??????: > Hi Sotirios, > > It looks like you have one Config setting not set (based on the first > error message you got when running it the first time). You need to set a > value for "Loris-MRI Data Directory" in the Config module under the Imaging > pipeline section (from what I can see, it would be /data/loris/data/ for > your setup). > > The reason you got that error message the second time is that the pipeline > checks if the DICOM study was already inserted in the tarchive table. Since > you ran the study once already, the study is inserted into the tarchive, > hence the message. However, the pipeline did not complete as the MINC files > were not created and inserted into the files/parameter_file table. In order > to continue the insertion, you will then need to run the tarchiveLoader > command suggested in the terminal during the second run: > /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod > /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar > -verbose > > Finally, I am a bit confused by your setup. Where are the scripts located? > In /data/loris/bin/mri or /data/loris/data? It feels like the Config > setting "LORIS-MRI code" in the Paths section was set to "/data/LORIS/data" > but it should have been set to the directory where the scripts are located > (most probably /data/loris/bin/mri). > > Hope this helps, > > C?cile > > > On Wed, Jul 24, 2019 at 10:55 AM Sotirios Nikoloutsopoulos < > sotirisnik at gmail.com> wrote: > >> Ok first i execute this command: >> >> "lorisadmin at hbp:/data/loris/bin/mri$ ./batch_uploads_imageuploader >> -profile prod < ~/Desktop/input.txt > log.txt >> Use of uninitialized value $data_dir in concatenation (.) or string at >> ./batch_uploads_imageuploader line 143. >> Use of uninitialized value $data_dir in concatenation (.) or string at >> ./batch_uploads_imageuploader line 143. >> Use of uninitialized value $_ in pattern match (m//) at >> ./batch_uploads_imageuploader line 147. >> base is DCC0000_258024_V01 >> path is /data/incoming/ >> type is .tar.gz >> fullpath is /data/incoming/DCC0000_258024_V01.tar.gz >> >> source /tmp/ImagingUpload-17-43-ZUEFjb >> targetlocation /data/loris/data/tarchive >> >> Use of uninitialized value $data_dir in concatenation (.) or string at >> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256. >> Use of uninitialized value $data_dir in concatenation (.) or string at >> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284. >> Use of uninitialized value $data_dir in concatenation (.) or string at >> /data/loris/bin/mri/uploadNeuroDB/tarchive_validation.pl line 219. >> Use of uninitialized value $data_dir in concatenation (.) or string at >> /data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl line 270. >> Use of uninitialized value $data_dir in concatenation (.) or string at >> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711. >> Can't exec "mail": No such file or directory at >> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721. >> print() on closed filehandle MAIL at >> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722. >> print() on closed filehandle MAIL at >> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723. >> Use of uninitialized value $data_dir in concatenation (.) or string at >> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >> print() on closed filehandle MAIL at >> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >> print() on closed filehandle MAIL at >> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725. >> Use of uninitialized value $data_dir in concatenation (.) or string at >> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735. >> >> No Mincs inserted >> >> >> The tarchiveLoader insertion script has failed. >> Running now the following command: /data/loris/data/uploadNeuroDB/ >> imaging_upload_file.pl -profile prod -upload_id 3 >> /data/incoming/DCC0000_258024_V01.tar.gz -verbose" >> >> I inserted the VisitLabel, but still the Mincs are not inserted and i am >> getting this error >> >> "lorisadmin at hbp:/data/loris/bin/mri$ /data/loris/data/uploadNeuroDB/ >> imaging_upload_file.pl -profile prod -upload_id 3 >> /data/incoming/DCC0000_258024_V01.tar.gz -verbose >> >> find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name '__MACOSX' -delete >> >> Spool message is: >> The Scan for the uploadID 3 has already been run with tarchiveID: 7. >> To continue with the rest of the insertion pipeline, please run >> tarchiveLoader from a terminal as follows: >> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod >> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >> -verbose" >> >> Which files should i delete? I thought that deleting the files from the >> tarchive was enough. And i executed the last proposed command to continue >> to rest of the insertion pipeline ( check the image ). >> >> Thank you >> >> ???? ???, 24 ???? 2019 ???? 5:05 ?.?., ?/? Cecile Madjar < >> cecile.madjar at mcin.ca> ??????: >> >>> Hi Sotirios, >>> >>> Sorry I was not clear before. Every visits of candidates are created in >>> the session table with one entry per CandID/VisitLabel. So if the imaging >>> pipeline created visits in the session table (that are not attached to >>> instruments) and you want to clean your database completely, you can delete >>> entries in that table as well. >>> >>> Regarding your point 2, the imaging pipeline can create the visit in the >>> session table as long as the visit label is present in the Visit_Windows >>> table (which means the visit label stored in the PatientName is a valid >>> visit label). >>> However, in your prod file, if the >>> variable $subjectID{'createVisitLabel'} is set to 1 for candidates, the >>> visit label should be created in the Visit_Windows table I believe. Here >>> is the line of the prod file you would need to change >>> >>> . >>> In general, we prefer populating the Visit_Windows table and set the >>> $subjectID{'createVisitLabel'} to 0 for candidate data to avoid insertion >>> of badly labelled MRI data that are complex to relabel. >>> >>> Finally, regarding the candidate creation, if you execute the following >>> steps, the pipeline should be able to create candidates: >>> - set in the Imaging Pipeline section of the config module the config >>> setting "Upload creation of candidates" to Yes >>> - manually transfer the scans to the LORIS server instead of uploading >>> it via the imaging browser >>> - run batch_imaging_upload.pl as you did until now. >>> >>> Hope this helps! >>> Best, >>> >>> C?cile >>> >>> On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos < >>> sotirisnik at gmail.com> wrote: >>> >>>> About my second question i replaced the name of the patient in the .dcm >>>> files and now i am getting an error that the visit label does not exist. >>>> I thought it was supposed to be automatically created. >>>> >>>> "Done adding archive info into database >>>> >>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile >>>> prod >>>> /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>> -verbose >>>> md5sum >>>> /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>> PSCID is: DCC0000 >>>> CandID id: 258024 >>>> visit_label is: V1 >>>> PSCID is: DCC0000 >>>> CandID id: 258024 >>>> visit_label is: V1 >>>> candidate id 258024 >>>> >>>> >>>> => No Visit labelVisit label does not exist >>>> Set centerID = 1 >>>> PSCID is: DCC0000 >>>> CandID id: 258024 >>>> visit_label is: V1 >>>> PSCID is: DCC0000 >>>> CandID id: 258024 >>>> visit_label is: V1 >>>> >>>> Number of MINC files that will be considered for inserting into the >>>> database: 1 >>>> >>>> log dir is /logs and log file is /logs/TarLoad-15-42-L6pocU.log >>>> PSCID is: DCC0000 >>>> CandID id: 258024 >>>> visit_label is: V1 >>>> PSCID is: DCC0000 >>>> CandID id: 258024 >>>> visit_label is: V1 >>>> candidate id 258024 >>>> >>>> >>>> => No Visit label >>>> Cleaning up temp files: rm -rf >>>> /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*" >>>> >>>> ???? ???, 24 ???? 2019 ???? 3:38 ?.?., ?/? Sotirios Nikoloutsopoulos < >>>> sotirisnik at gmail.com> ??????: >>>> >>>>> Hi, >>>>> >>>>> 1) What are the records in the session table? >>>>> >>>>> 2) I tried to upload a non-phantom file like this >>>>> >>>>> input_file.txt: >>>>> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1 >>>>> >>>>> Terminal: >>>>> ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > >>>>> log.txt >>>>> >>>>> and i get the following error in the log.txt file >>>>> >>>>> "Running now the following command: /data/loris/data/uploadNeuroDB/ >>>>> imaging_upload_file.pl -profile prod -upload_id 7 >>>>> /data/incoming/DCC0000_258024_V1.tar.gz -verbose >>>>> >>>>> find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name '__MACOSX' >>>>> -delete >>>>> Spool message is: >>>>> The PatientName read from the DICOM header does not start with >>>>> DCC0000_258024_V1 from the mri_upload table >>>>> " >>>>> >>>>> and the spool message is repeated for each .dcm file. Should i >>>>> preprocess every .dcm file and change the PatientName to >>>>> "DCC0000_258024_V1"? >>>>> >>>>> 3) As far as it concerns the candidate profile i have to create a new >>>>> one each time before i try to upload a file? Is there already a way to >>>>> create a candidate profile and get the DCCID and PSCID without the user >>>>> interface? >>>>> >>>>> Thank you, >>>>> >>>>> Sotirios >>>>> >>>>> ???? ???, 18 ???? 2019 ???? 5:00 ?.?., ?/? Cecile Madjar < >>>>> cecile.madjar at mcin.ca> ??????: >>>>> >>>>>> Hi Sotirios, >>>>>> >>>>>> See answers to your questions below. >>>>>> >>>>>> About the 1st case we had a dicom with a blank studyUID, maybe >>>>>>> someone accidentally removed it. >>>>>>> >>>>>> That is strange. Something must have been done to the DICOM files. I >>>>>> have never seen a study with no StudyUID out of the scanner. Usually, you >>>>>> need to manually erase it with some tool. However, DICOMs never cease to >>>>>> surprise me... >>>>>> >>>>>> >>>>>>> Some other questions: >>>>>>> 1) What are the steps to remove all of my patients and uploads?: >>>>>>> because i want to try to reupload all the testing dicoms we had with a >>>>>>> clean database. >>>>>>> >>>>>> >>>>>> Release 21.0 of LORIS-MRI will offer a script to remove all the >>>>>> entries and files specific to an upload. However, this release is not yet >>>>>> out. Hopefully in the next few weeks. >>>>>> In the meantime, since your case is particular and that you just want >>>>>> to start with a clean database, the following deletes should work >>>>>> (hopefully I won't forget any tables): >>>>>> delete from mri_violations_log; >>>>>> delete from mri_protocol_violated_scans; >>>>>> delete from MRICandidateErrors; >>>>>> delete from parameter_file; >>>>>> delete from files_qcstatus; # (if you played with the QC part of the >>>>>> imaging browser for testing) >>>>>> delete from feedback_mri_comments; # (if you played with the QC part >>>>>> of the imaging browser for testing) >>>>>> delete from tarchive_series; >>>>>> delete from tarchive_files; >>>>>> delete from mri_upload; >>>>>> delete from tarchive; >>>>>> delete from mri_scanner; >>>>>> delete from candidate where PSCID="scanner"; >>>>>> >>>>>> I don't know if you want to also delete entries in the session and >>>>>> candidate table too? >>>>>> >>>>>> 2) We do not need to create the profile of a patient ( are the >>>>>>> patients stored in the candidate table? because i can't see the ghosts ), >>>>>>> because will it automatically be created when importing a dicom? >>>>>>> >>>>>> >>>>>> I am not sure I understand fully this question. All candidates are >>>>>> stored in the candidate table. The phantom scans however are attached to a >>>>>> scanner candidate depending on where the scan happened. It is a bit of a >>>>>> weird concept that we have to redesign eventually but never got a chance to >>>>>> get to it. >>>>>> >>>>>> Hopefully this answered your question, otherwise, don't hesitate to >>>>>> let me know. >>>>>> >>>>>> 3) What is the difference between uploading a dicom as phantom >>>>>>> instead of importing as PSCCID_DCCID_VisitLabel? >>>>>>> >>>>>> >>>>>> When you upload the scan as PSCID_DCCID_VisitLabel, there is a check >>>>>> that makes sure the candidate IDs and visit label are valid at the time of >>>>>> upload. >>>>>> When you upload a scan as phantom, it expects that the DICOM field >>>>>> PatientName and uploaded filename contains the string "phantom". We >>>>>> enforced this behaviour on the imaging uploader side in recent releases but >>>>>> I can't remember which one. Probably the upcoming 21.0 release. >>>>>> >>>>>> All those verifications might seem cumbersome but they are here to >>>>>> ensure that the files inserted are all valid and labelled properly as it is >>>>>> a bit messy to have to delete files that were wrongly labelled. At least, >>>>>> the delete script present in release 21.0 will make this process easier but >>>>>> it is still good practice to verify those things. >>>>>> >>>>>> >>>>>>> Thank you, >>>>>>> >>>>>> >>>>>> With pleasure. Hopefully the answers to your questions will be >>>>>> helpful. >>>>>> >>>>>> Best, >>>>>> >>>>>> C?cile >>>>>> >>>>>> PS: loris-dev mailing list: you will find the initial answer I gave >>>>>> Sotirios below. I forgot to cc the loris-dev in my earlier reply... >>>>>> >>>>>> ???? ???, 16 ???? 2019 ???? 12:02 ?.?., ?/? Cecile Madjar < >>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>> >>>>>>>> Dear Sotirios, >>>>>>>> >>>>>>>> Thank you for reaching out. Since your email was already organized >>>>>>>> in points, I will reply directly below your questions below. >>>>>>>> >>>>>>>> Best, >>>>>>>> >>>>>>>> C?cile >>>>>>>> >>>>>>>> 1) What do we do when the StudyUID is blank on the header? >>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md >>>>>>>>> There solution provided at the table 3 does not guide us how to find the >>>>>>>>> information ( is possible ). >>>>>>>>> >>>>>>>> >>>>>>>> The StudyUID is the field used in our pipeline at the moment to >>>>>>>> check whether a DICOM study was already inserted into LORIS with that same >>>>>>>> StudyUID since it is supposed to be unique for every single study. >>>>>>>> Currently, we do not support insertion of DICOM studies if they do not have >>>>>>>> a StudyUID associated with them. >>>>>>>> >>>>>>>> Is there a specific reason why the StudyUID is blank in the DICOM >>>>>>>> headers in your DICOM files? Did they go through some processes before >>>>>>>> upload to LORIS? >>>>>>>> >>>>>>>> >>>>>>>> 2) When i am about to upload a non-phanton case i need a 'Visit >>>>>>>>> Label'. Is there already a script that can create a new visit label for a >>>>>>>>> specific Candidate, so i can provide it as input afterwards to the >>>>>>>>> >>>>>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md ) >>>>>>>>> >>>>>>>>> >>>>>>>> I am not sure if the front-end allows for it but if you use >>>>>>>> batch_uploads_imageuploader, the insertion scripts could create the visit >>>>>>>> label for you. However, it does need to be included in the patient name in >>>>>>>> order for the script to know which label it should use. >>>>>>>> For example: you upload V02 for MTL0123 (DCCID: 456789) but V02 >>>>>>>> was not yet created for MTL0123; the insertion scripts will create a V02 >>>>>>>> visit for MTL0123 if the PatientName field for the dataset is >>>>>>>> MTL0123_456789_V02 with Stage marked as "Not Started". >>>>>>>> >>>>>>>> However, *note that for the script to create the visit for the >>>>>>>> candidate, you have to make sure that all the visit label of your projects >>>>>>>> were inserted in the Visit_Windows table* (otherwise, the visit >>>>>>>> will never be created as they were not specified as being part of the list >>>>>>>> of visit label to expect). So in the example above, you should have one row >>>>>>>> in Visit_Windows with Visit_label="V02" set. >>>>>>>> >>>>>>>> >>>>>>>>> 3) Section 4.3 >>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md >>>>>>>>> >>>>>>>>> What do we do when the folder contains multiple mnc files? >>>>>>>>> >>>>>>>>> This is a case by case. Most of the time, the MINC files that >>>>>>>> failed insertion into the imaging browser end up in the MRI violation >>>>>>>> module where you can see what went wrong with the acquisitions not >>>>>>>> inserted. If you notice that the MINC file should be inserted as a specific >>>>>>>> protocol, you can force the insertion as explained in section 4.3. >>>>>>>> >>>>>>>> In your case, I have a feeling that you want to be able to insert >>>>>>>> MINC files that did not go through the whole pipeline insertion (DICOM >>>>>>>> archival, then dcm2mnc, then protocol identification, then insertion...). >>>>>>>> In theory, you could call minc_insertion.pl on any MINC file, >>>>>>>> provided you have at least the following information: >>>>>>>> - path to the MINC file >>>>>>>> - uploadID associated with the MINC files or DICOM archive path >>>>>>>> from which those MINC files were created from >>>>>>>> >>>>>>>> Then the minc_insertion.pl script will do the subject information >>>>>>>> and protocol validation etc... >>>>>>>> >>>>>>>> If all the MINC files come from the same uploadID or TarchiveID, >>>>>>>> then you could run a loop in bash calling the minc_insertion.pl >>>>>>>> script like this for files deriving from uploadID=1: >>>>>>>> >>>>>>>> ls /path/to/mnc/folder/* | while read f; do minc_insertion.pl >>>>>>>> -profile prod -uploadID 1 -mincPath $f; done >>>>>>>> >>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> >>>>>>>> >>>>>>>> Hope this helped! >>>>>>>> >>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: modification.png Type: image/png Size: 99870 bytes Desc: not available URL: From sotirisnik at gmail.com Thu Jul 25 09:48:39 2019 From: sotirisnik at gmail.com (Sotirios Nikoloutsopoulos) Date: Thu, 25 Jul 2019 16:48:39 +0300 Subject: [Loris-dev] Import mri - scripts In-Reply-To: References: Message-ID: The "Loris-MRI Data Directory" should be in the environment file? ???? ???, 25 ???? 2019 ???? 12:59 ?.?., ?/? Sotirios Nikoloutsopoulos < sotirisnik at gmail.com> ??????: > The scripts are located in /data/loris/bin/mri/. Somewhere I might have > messed something up but i remember following the instructions in the > wiki.For example when i tried to execute the batch_uploads_imageuploader > perl script the line > > my $command = "$bin_dir/uploadNeuroDB/imaging_upload_file.pl " > > was pointing to an invalid path, for example: it was repeating a pattern > and had LORIS instead of loris ( i think the mistake was that in the wiki > it suggested to use $PROJECT for the name of the folder and i choosed > 'loris' https://github.com/aces/Loris/wiki/Imaging-Database ). See also > the in image for the modification i did. Also i didn't find the section to > config the $data_dir variable. > > Thank you, > > Sotirios > > > > ???? ???, 24 ???? 2019 ???? 6:05 ?.?., ?/? Cecile Madjar < > cecile.madjar at mcin.ca> ??????: > >> Hi Sotirios, >> >> It looks like you have one Config setting not set (based on the first >> error message you got when running it the first time). You need to set a >> value for "Loris-MRI Data Directory" in the Config module under the Imaging >> pipeline section (from what I can see, it would be /data/loris/data/ for >> your setup). >> >> The reason you got that error message the second time is that the >> pipeline checks if the DICOM study was already inserted in the tarchive >> table. Since you ran the study once already, the study is inserted into the >> tarchive, hence the message. However, the pipeline did not complete as the >> MINC files were not created and inserted into the files/parameter_file >> table. In order to continue the insertion, you will then need to run the >> tarchiveLoader command suggested in the terminal during the second run: >> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile >> prod >> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >> -verbose >> >> Finally, I am a bit confused by your setup. Where are the scripts >> located? In /data/loris/bin/mri or /data/loris/data? It feels like the >> Config setting "LORIS-MRI code" in the Paths section was set to >> "/data/LORIS/data" but it should have been set to the directory where the >> scripts are located (most probably /data/loris/bin/mri). >> >> Hope this helps, >> >> C?cile >> >> >> On Wed, Jul 24, 2019 at 10:55 AM Sotirios Nikoloutsopoulos < >> sotirisnik at gmail.com> wrote: >> >>> Ok first i execute this command: >>> >>> "lorisadmin at hbp:/data/loris/bin/mri$ ./batch_uploads_imageuploader >>> -profile prod < ~/Desktop/input.txt > log.txt >>> Use of uninitialized value $data_dir in concatenation (.) or string at >>> ./batch_uploads_imageuploader line 143. >>> Use of uninitialized value $data_dir in concatenation (.) or string at >>> ./batch_uploads_imageuploader line 143. >>> Use of uninitialized value $_ in pattern match (m//) at >>> ./batch_uploads_imageuploader line 147. >>> base is DCC0000_258024_V01 >>> path is /data/incoming/ >>> type is .tar.gz >>> fullpath is /data/incoming/DCC0000_258024_V01.tar.gz >>> >>> source /tmp/ImagingUpload-17-43-ZUEFjb >>> targetlocation /data/loris/data/tarchive >>> >>> Use of uninitialized value $data_dir in concatenation (.) or string at >>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256. >>> Use of uninitialized value $data_dir in concatenation (.) or string at >>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284. >>> Use of uninitialized value $data_dir in concatenation (.) or string at >>> /data/loris/bin/mri/uploadNeuroDB/tarchive_validation.pl line 219. >>> Use of uninitialized value $data_dir in concatenation (.) or string at >>> /data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl line 270. >>> Use of uninitialized value $data_dir in concatenation (.) or string at >>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711. >>> Can't exec "mail": No such file or directory at >>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721. >>> print() on closed filehandle MAIL at >>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722. >>> print() on closed filehandle MAIL at >>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723. >>> Use of uninitialized value $data_dir in concatenation (.) or string at >>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>> print() on closed filehandle MAIL at >>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>> print() on closed filehandle MAIL at >>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725. >>> Use of uninitialized value $data_dir in concatenation (.) or string at >>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735. >>> >>> No Mincs inserted >>> >>> >>> The tarchiveLoader insertion script has failed. >>> Running now the following command: /data/loris/data/uploadNeuroDB/ >>> imaging_upload_file.pl -profile prod -upload_id 3 >>> /data/incoming/DCC0000_258024_V01.tar.gz -verbose" >>> >>> I inserted the VisitLabel, but still the Mincs are not inserted and i am >>> getting this error >>> >>> "lorisadmin at hbp:/data/loris/bin/mri$ /data/loris/data/uploadNeuroDB/ >>> imaging_upload_file.pl -profile prod -upload_id 3 >>> /data/incoming/DCC0000_258024_V01.tar.gz -verbose >>> >>> find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name '__MACOSX' >>> -delete >>> Spool message is: >>> The Scan for the uploadID 3 has already been run with tarchiveID: 7. >>> To continue with the rest of the insertion pipeline, please run >>> tarchiveLoader from a terminal as follows: >>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod >>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >>> -verbose" >>> >>> Which files should i delete? I thought that deleting the files from the >>> tarchive was enough. And i executed the last proposed command to continue >>> to rest of the insertion pipeline ( check the image ). >>> >>> Thank you >>> >>> ???? ???, 24 ???? 2019 ???? 5:05 ?.?., ?/? Cecile Madjar < >>> cecile.madjar at mcin.ca> ??????: >>> >>>> Hi Sotirios, >>>> >>>> Sorry I was not clear before. Every visits of candidates are created in >>>> the session table with one entry per CandID/VisitLabel. So if the imaging >>>> pipeline created visits in the session table (that are not attached to >>>> instruments) and you want to clean your database completely, you can delete >>>> entries in that table as well. >>>> >>>> Regarding your point 2, the imaging pipeline can create the visit in >>>> the session table as long as the visit label is present in the >>>> Visit_Windows table (which means the visit label stored in the PatientName >>>> is a valid visit label). >>>> However, in your prod file, if the >>>> variable $subjectID{'createVisitLabel'} is set to 1 for candidates, the >>>> visit label should be created in the Visit_Windows table I believe. Here >>>> is the line of the prod file you would need to change >>>> >>>> . >>>> In general, we prefer populating the Visit_Windows table and set the >>>> $subjectID{'createVisitLabel'} to 0 for candidate data to avoid insertion >>>> of badly labelled MRI data that are complex to relabel. >>>> >>>> Finally, regarding the candidate creation, if you execute the following >>>> steps, the pipeline should be able to create candidates: >>>> - set in the Imaging Pipeline section of the config module the config >>>> setting "Upload creation of candidates" to Yes >>>> - manually transfer the scans to the LORIS server instead of uploading >>>> it via the imaging browser >>>> - run batch_imaging_upload.pl as you did until now. >>>> >>>> Hope this helps! >>>> Best, >>>> >>>> C?cile >>>> >>>> On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos < >>>> sotirisnik at gmail.com> wrote: >>>> >>>>> About my second question i replaced the name of the patient in the >>>>> .dcm files and now i am getting an error that the visit label does not >>>>> exist. >>>>> I thought it was supposed to be automatically created. >>>>> >>>>> "Done adding archive info into database >>>>> >>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile >>>>> prod >>>>> /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>> -verbose >>>>> md5sum >>>>> /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>> PSCID is: DCC0000 >>>>> CandID id: 258024 >>>>> visit_label is: V1 >>>>> PSCID is: DCC0000 >>>>> CandID id: 258024 >>>>> visit_label is: V1 >>>>> candidate id 258024 >>>>> >>>>> >>>>> => No Visit labelVisit label does not exist >>>>> Set centerID = 1 >>>>> PSCID is: DCC0000 >>>>> CandID id: 258024 >>>>> visit_label is: V1 >>>>> PSCID is: DCC0000 >>>>> CandID id: 258024 >>>>> visit_label is: V1 >>>>> >>>>> Number of MINC files that will be considered for inserting into the >>>>> database: 1 >>>>> >>>>> log dir is /logs and log file is /logs/TarLoad-15-42-L6pocU.log >>>>> PSCID is: DCC0000 >>>>> CandID id: 258024 >>>>> visit_label is: V1 >>>>> PSCID is: DCC0000 >>>>> CandID id: 258024 >>>>> visit_label is: V1 >>>>> candidate id 258024 >>>>> >>>>> >>>>> => No Visit label >>>>> Cleaning up temp files: rm -rf >>>>> /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*" >>>>> >>>>> ???? ???, 24 ???? 2019 ???? 3:38 ?.?., ?/? Sotirios Nikoloutsopoulos < >>>>> sotirisnik at gmail.com> ??????: >>>>> >>>>>> Hi, >>>>>> >>>>>> 1) What are the records in the session table? >>>>>> >>>>>> 2) I tried to upload a non-phantom file like this >>>>>> >>>>>> input_file.txt: >>>>>> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1 >>>>>> >>>>>> Terminal: >>>>>> ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > >>>>>> log.txt >>>>>> >>>>>> and i get the following error in the log.txt file >>>>>> >>>>>> "Running now the following command: /data/loris/data/uploadNeuroDB/ >>>>>> imaging_upload_file.pl -profile prod -upload_id 7 >>>>>> /data/incoming/DCC0000_258024_V1.tar.gz -verbose >>>>>> >>>>>> find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name '__MACOSX' >>>>>> -delete >>>>>> Spool message is: >>>>>> The PatientName read from the DICOM header does not start with >>>>>> DCC0000_258024_V1 from the mri_upload table >>>>>> " >>>>>> >>>>>> and the spool message is repeated for each .dcm file. Should i >>>>>> preprocess every .dcm file and change the PatientName to >>>>>> "DCC0000_258024_V1"? >>>>>> >>>>>> 3) As far as it concerns the candidate profile i have to create a new >>>>>> one each time before i try to upload a file? Is there already a way to >>>>>> create a candidate profile and get the DCCID and PSCID without the user >>>>>> interface? >>>>>> >>>>>> Thank you, >>>>>> >>>>>> Sotirios >>>>>> >>>>>> ???? ???, 18 ???? 2019 ???? 5:00 ?.?., ?/? Cecile Madjar < >>>>>> cecile.madjar at mcin.ca> ??????: >>>>>> >>>>>>> Hi Sotirios, >>>>>>> >>>>>>> See answers to your questions below. >>>>>>> >>>>>>> About the 1st case we had a dicom with a blank studyUID, maybe >>>>>>>> someone accidentally removed it. >>>>>>>> >>>>>>> That is strange. Something must have been done to the DICOM files. I >>>>>>> have never seen a study with no StudyUID out of the scanner. Usually, you >>>>>>> need to manually erase it with some tool. However, DICOMs never cease to >>>>>>> surprise me... >>>>>>> >>>>>>> >>>>>>>> Some other questions: >>>>>>>> 1) What are the steps to remove all of my patients and uploads?: >>>>>>>> because i want to try to reupload all the testing dicoms we had with a >>>>>>>> clean database. >>>>>>>> >>>>>>> >>>>>>> Release 21.0 of LORIS-MRI will offer a script to remove all the >>>>>>> entries and files specific to an upload. However, this release is not yet >>>>>>> out. Hopefully in the next few weeks. >>>>>>> In the meantime, since your case is particular and that you just >>>>>>> want to start with a clean database, the following deletes should work >>>>>>> (hopefully I won't forget any tables): >>>>>>> delete from mri_violations_log; >>>>>>> delete from mri_protocol_violated_scans; >>>>>>> delete from MRICandidateErrors; >>>>>>> delete from parameter_file; >>>>>>> delete from files_qcstatus; # (if you played with the QC part of the >>>>>>> imaging browser for testing) >>>>>>> delete from feedback_mri_comments; # (if you played with the QC part >>>>>>> of the imaging browser for testing) >>>>>>> delete from tarchive_series; >>>>>>> delete from tarchive_files; >>>>>>> delete from mri_upload; >>>>>>> delete from tarchive; >>>>>>> delete from mri_scanner; >>>>>>> delete from candidate where PSCID="scanner"; >>>>>>> >>>>>>> I don't know if you want to also delete entries in the session and >>>>>>> candidate table too? >>>>>>> >>>>>>> 2) We do not need to create the profile of a patient ( are the >>>>>>>> patients stored in the candidate table? because i can't see the ghosts ), >>>>>>>> because will it automatically be created when importing a dicom? >>>>>>>> >>>>>>> >>>>>>> I am not sure I understand fully this question. All candidates are >>>>>>> stored in the candidate table. The phantom scans however are attached to a >>>>>>> scanner candidate depending on where the scan happened. It is a bit of a >>>>>>> weird concept that we have to redesign eventually but never got a chance to >>>>>>> get to it. >>>>>>> >>>>>>> Hopefully this answered your question, otherwise, don't hesitate to >>>>>>> let me know. >>>>>>> >>>>>>> 3) What is the difference between uploading a dicom as phantom >>>>>>>> instead of importing as PSCCID_DCCID_VisitLabel? >>>>>>>> >>>>>>> >>>>>>> When you upload the scan as PSCID_DCCID_VisitLabel, there is a check >>>>>>> that makes sure the candidate IDs and visit label are valid at the time of >>>>>>> upload. >>>>>>> When you upload a scan as phantom, it expects that the DICOM field >>>>>>> PatientName and uploaded filename contains the string "phantom". We >>>>>>> enforced this behaviour on the imaging uploader side in recent releases but >>>>>>> I can't remember which one. Probably the upcoming 21.0 release. >>>>>>> >>>>>>> All those verifications might seem cumbersome but they are here to >>>>>>> ensure that the files inserted are all valid and labelled properly as it is >>>>>>> a bit messy to have to delete files that were wrongly labelled. At least, >>>>>>> the delete script present in release 21.0 will make this process easier but >>>>>>> it is still good practice to verify those things. >>>>>>> >>>>>>> >>>>>>>> Thank you, >>>>>>>> >>>>>>> >>>>>>> With pleasure. Hopefully the answers to your questions will be >>>>>>> helpful. >>>>>>> >>>>>>> Best, >>>>>>> >>>>>>> C?cile >>>>>>> >>>>>>> PS: loris-dev mailing list: you will find the initial answer I gave >>>>>>> Sotirios below. I forgot to cc the loris-dev in my earlier reply... >>>>>>> >>>>>>> ???? ???, 16 ???? 2019 ???? 12:02 ?.?., ?/? Cecile Madjar < >>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>> >>>>>>>>> Dear Sotirios, >>>>>>>>> >>>>>>>>> Thank you for reaching out. Since your email was already organized >>>>>>>>> in points, I will reply directly below your questions below. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> >>>>>>>>> C?cile >>>>>>>>> >>>>>>>>> 1) What do we do when the StudyUID is blank on the header? >>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md >>>>>>>>>> There solution provided at the table 3 does not guide us how to find the >>>>>>>>>> information ( is possible ). >>>>>>>>>> >>>>>>>>> >>>>>>>>> The StudyUID is the field used in our pipeline at the moment to >>>>>>>>> check whether a DICOM study was already inserted into LORIS with that same >>>>>>>>> StudyUID since it is supposed to be unique for every single study. >>>>>>>>> Currently, we do not support insertion of DICOM studies if they do not have >>>>>>>>> a StudyUID associated with them. >>>>>>>>> >>>>>>>>> Is there a specific reason why the StudyUID is blank in the DICOM >>>>>>>>> headers in your DICOM files? Did they go through some processes before >>>>>>>>> upload to LORIS? >>>>>>>>> >>>>>>>>> >>>>>>>>> 2) When i am about to upload a non-phanton case i need a 'Visit >>>>>>>>>> Label'. Is there already a script that can create a new visit label for a >>>>>>>>>> specific Candidate, so i can provide it as input afterwards to the >>>>>>>>>> >>>>>>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md ) >>>>>>>>>> >>>>>>>>>> >>>>>>>>> I am not sure if the front-end allows for it but if you use >>>>>>>>> batch_uploads_imageuploader, the insertion scripts could create the visit >>>>>>>>> label for you. However, it does need to be included in the patient name in >>>>>>>>> order for the script to know which label it should use. >>>>>>>>> For example: you upload V02 for MTL0123 (DCCID: 456789) but V02 >>>>>>>>> was not yet created for MTL0123; the insertion scripts will create a V02 >>>>>>>>> visit for MTL0123 if the PatientName field for the dataset is >>>>>>>>> MTL0123_456789_V02 with Stage marked as "Not Started". >>>>>>>>> >>>>>>>>> However, *note that for the script to create the visit for the >>>>>>>>> candidate, you have to make sure that all the visit label of your projects >>>>>>>>> were inserted in the Visit_Windows table* (otherwise, the visit >>>>>>>>> will never be created as they were not specified as being part of the list >>>>>>>>> of visit label to expect). So in the example above, you should have one row >>>>>>>>> in Visit_Windows with Visit_label="V02" set. >>>>>>>>> >>>>>>>>> >>>>>>>>>> 3) Section 4.3 >>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md >>>>>>>>>> >>>>>>>>>> What do we do when the folder contains multiple mnc files? >>>>>>>>>> >>>>>>>>>> This is a case by case. Most of the time, the MINC files that >>>>>>>>> failed insertion into the imaging browser end up in the MRI violation >>>>>>>>> module where you can see what went wrong with the acquisitions not >>>>>>>>> inserted. If you notice that the MINC file should be inserted as a specific >>>>>>>>> protocol, you can force the insertion as explained in section 4.3. >>>>>>>>> >>>>>>>>> In your case, I have a feeling that you want to be able to insert >>>>>>>>> MINC files that did not go through the whole pipeline insertion (DICOM >>>>>>>>> archival, then dcm2mnc, then protocol identification, then insertion...). >>>>>>>>> In theory, you could call minc_insertion.pl on any MINC file, >>>>>>>>> provided you have at least the following information: >>>>>>>>> - path to the MINC file >>>>>>>>> - uploadID associated with the MINC files or DICOM archive path >>>>>>>>> from which those MINC files were created from >>>>>>>>> >>>>>>>>> Then the minc_insertion.pl script will do the subject information >>>>>>>>> and protocol validation etc... >>>>>>>>> >>>>>>>>> If all the MINC files come from the same uploadID or TarchiveID, >>>>>>>>> then you could run a loop in bash calling the minc_insertion.pl >>>>>>>>> script like this for files deriving from uploadID=1: >>>>>>>>> >>>>>>>>> ls /path/to/mnc/folder/* | while read f; do minc_insertion.pl >>>>>>>>> -profile prod -uploadID 1 -mincPath $f; done >>>>>>>>> >>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> >>>>>>>>> >>>>>>>>> Hope this helped! >>>>>>>>> >>>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: environment.png Type: image/png Size: 50080 bytes Desc: not available URL: From cecile.madjar at mcin.ca Thu Jul 25 10:47:52 2019 From: cecile.madjar at mcin.ca (Cecile Madjar) Date: Thu, 25 Jul 2019 10:47:52 -0400 Subject: [Loris-dev] Import mri - scripts In-Reply-To: References: Message-ID: Hi Sotirios, >From what I understand, I believe you just need to update a few config settings from the frontend in the Config module (under the Admin menu). [image: Screen Shot 2019-07-25 at 10.31.35 AM.png] Then go to the Paths section of the config module and update the "LORIS-MRI code" config setting to /data/loris/bin/mri (my guess is that currently, it is set to /data/LORIS/data) and click on the submit button. [image: Screen Shot 2019-07-25 at 10.32.50 AM.png] Then, go to Imaging Pipeline section of the Configuration module and update the "Loris-MRI Data Directory" to /data/loris/data/ (my guess is that it is empty at the moment in your database which is why $data_dir is not defined when you run the tarchiveLoader step of the insertion pipeline). [image: Screen Shot 2019-07-25 at 10.40.52 AM.png] Finally, I took a look at your environment file and it might be a good idea to update your PERL5LIB variable to the following (it has been added to the code a few releases ago as we realized this was needed for the libraries. Not sure which version of LORIS-MRI you are using but it does not hurt to update this variable to what I suggest below): export PERL5LIB=/data/loris/bin/mri/uploadNeuroDB:/data/loris/bin/mri/dicom-archive:$PERL5LIB Hope this helps. Sorry you are experiencing so much trouble setting things up. We probably need to update our WIKI. Best, C?cile On Thu, Jul 25, 2019 at 9:48 AM Sotirios Nikoloutsopoulos < sotirisnik at gmail.com> wrote: > The "Loris-MRI Data Directory" should be in the environment file? > > ???? ???, 25 ???? 2019 ???? 12:59 ?.?., ?/? Sotirios Nikoloutsopoulos < > sotirisnik at gmail.com> ??????: > >> The scripts are located in /data/loris/bin/mri/. Somewhere I might have >> messed something up but i remember following the instructions in the >> wiki.For example when i tried to execute the batch_uploads_imageuploader >> perl script the line >> >> my $command = "$bin_dir/uploadNeuroDB/imaging_upload_file.pl " >> >> was pointing to an invalid path, for example: it was repeating a pattern >> and had LORIS instead of loris ( i think the mistake was that in the wiki >> it suggested to use $PROJECT for the name of the folder and i choosed >> 'loris' https://github.com/aces/Loris/wiki/Imaging-Database ). See also >> the in image for the modification i did. Also i didn't find the section to >> config the $data_dir variable. >> >> Thank you, >> >> Sotirios >> >> >> >> ???? ???, 24 ???? 2019 ???? 6:05 ?.?., ?/? Cecile Madjar < >> cecile.madjar at mcin.ca> ??????: >> >>> Hi Sotirios, >>> >>> It looks like you have one Config setting not set (based on the first >>> error message you got when running it the first time). You need to set a >>> value for "Loris-MRI Data Directory" in the Config module under the Imaging >>> pipeline section (from what I can see, it would be /data/loris/data/ for >>> your setup). >>> >>> The reason you got that error message the second time is that the >>> pipeline checks if the DICOM study was already inserted in the tarchive >>> table. Since you ran the study once already, the study is inserted into the >>> tarchive, hence the message. However, the pipeline did not complete as the >>> MINC files were not created and inserted into the files/parameter_file >>> table. In order to continue the insertion, you will then need to run the >>> tarchiveLoader command suggested in the terminal during the second run: >>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile >>> prod >>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >>> -verbose >>> >>> Finally, I am a bit confused by your setup. Where are the scripts >>> located? In /data/loris/bin/mri or /data/loris/data? It feels like the >>> Config setting "LORIS-MRI code" in the Paths section was set to >>> "/data/LORIS/data" but it should have been set to the directory where the >>> scripts are located (most probably /data/loris/bin/mri). >>> >>> Hope this helps, >>> >>> C?cile >>> >>> >>> On Wed, Jul 24, 2019 at 10:55 AM Sotirios Nikoloutsopoulos < >>> sotirisnik at gmail.com> wrote: >>> >>>> Ok first i execute this command: >>>> >>>> "lorisadmin at hbp:/data/loris/bin/mri$ ./batch_uploads_imageuploader >>>> -profile prod < ~/Desktop/input.txt > log.txt >>>> Use of uninitialized value $data_dir in concatenation (.) or string at >>>> ./batch_uploads_imageuploader line 143. >>>> Use of uninitialized value $data_dir in concatenation (.) or string at >>>> ./batch_uploads_imageuploader line 143. >>>> Use of uninitialized value $_ in pattern match (m//) at >>>> ./batch_uploads_imageuploader line 147. >>>> base is DCC0000_258024_V01 >>>> path is /data/incoming/ >>>> type is .tar.gz >>>> fullpath is /data/incoming/DCC0000_258024_V01.tar.gz >>>> >>>> source /tmp/ImagingUpload-17-43-ZUEFjb >>>> targetlocation /data/loris/data/tarchive >>>> >>>> Use of uninitialized value $data_dir in concatenation (.) or string at >>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256. >>>> Use of uninitialized value $data_dir in concatenation (.) or string at >>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284. >>>> Use of uninitialized value $data_dir in concatenation (.) or string at >>>> /data/loris/bin/mri/uploadNeuroDB/tarchive_validation.pl line 219. >>>> Use of uninitialized value $data_dir in concatenation (.) or string at >>>> /data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl line 270. >>>> Use of uninitialized value $data_dir in concatenation (.) or string at >>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711. >>>> Can't exec "mail": No such file or directory at >>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721. >>>> print() on closed filehandle MAIL at >>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722. >>>> print() on closed filehandle MAIL at >>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723. >>>> Use of uninitialized value $data_dir in concatenation (.) or string at >>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>>> print() on closed filehandle MAIL at >>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>>> print() on closed filehandle MAIL at >>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725. >>>> Use of uninitialized value $data_dir in concatenation (.) or string at >>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735. >>>> >>>> No Mincs inserted >>>> >>>> >>>> The tarchiveLoader insertion script has failed. >>>> Running now the following command: /data/loris/data/uploadNeuroDB/ >>>> imaging_upload_file.pl -profile prod -upload_id 3 >>>> /data/incoming/DCC0000_258024_V01.tar.gz -verbose" >>>> >>>> I inserted the VisitLabel, but still the Mincs are not inserted and i >>>> am getting this error >>>> >>>> "lorisadmin at hbp:/data/loris/bin/mri$ /data/loris/data/uploadNeuroDB/ >>>> imaging_upload_file.pl -profile prod -upload_id 3 >>>> /data/incoming/DCC0000_258024_V01.tar.gz -verbose >>>> >>>> find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name '__MACOSX' >>>> -delete >>>> Spool message is: >>>> The Scan for the uploadID 3 has already been run with tarchiveID: 7. >>>> To continue with the rest of the insertion pipeline, please run >>>> tarchiveLoader from a terminal as follows: >>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod >>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >>>> -verbose" >>>> >>>> Which files should i delete? I thought that deleting the files from the >>>> tarchive was enough. And i executed the last proposed command to continue >>>> to rest of the insertion pipeline ( check the image ). >>>> >>>> Thank you >>>> >>>> ???? ???, 24 ???? 2019 ???? 5:05 ?.?., ?/? Cecile Madjar < >>>> cecile.madjar at mcin.ca> ??????: >>>> >>>>> Hi Sotirios, >>>>> >>>>> Sorry I was not clear before. Every visits of candidates are created >>>>> in the session table with one entry per CandID/VisitLabel. So if the >>>>> imaging pipeline created visits in the session table (that are not attached >>>>> to instruments) and you want to clean your database completely, you can >>>>> delete entries in that table as well. >>>>> >>>>> Regarding your point 2, the imaging pipeline can create the visit in >>>>> the session table as long as the visit label is present in the >>>>> Visit_Windows table (which means the visit label stored in the PatientName >>>>> is a valid visit label). >>>>> However, in your prod file, if the >>>>> variable $subjectID{'createVisitLabel'} is set to 1 for candidates, the >>>>> visit label should be created in the Visit_Windows table I believe. Here >>>>> is the line of the prod file you would need to change >>>>> >>>>> . >>>>> In general, we prefer populating the Visit_Windows table and set the >>>>> $subjectID{'createVisitLabel'} to 0 for candidate data to avoid insertion >>>>> of badly labelled MRI data that are complex to relabel. >>>>> >>>>> Finally, regarding the candidate creation, if you execute the >>>>> following steps, the pipeline should be able to create candidates: >>>>> - set in the Imaging Pipeline section of the config module the config >>>>> setting "Upload creation of candidates" to Yes >>>>> - manually transfer the scans to the LORIS server instead of uploading >>>>> it via the imaging browser >>>>> - run batch_imaging_upload.pl as you did until now. >>>>> >>>>> Hope this helps! >>>>> Best, >>>>> >>>>> C?cile >>>>> >>>>> On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos < >>>>> sotirisnik at gmail.com> wrote: >>>>> >>>>>> About my second question i replaced the name of the patient in the >>>>>> .dcm files and now i am getting an error that the visit label does not >>>>>> exist. >>>>>> I thought it was supposed to be automatically created. >>>>>> >>>>>> "Done adding archive info into database >>>>>> >>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile >>>>>> prod >>>>>> /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>>> -verbose >>>>>> md5sum >>>>>> /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>>> PSCID is: DCC0000 >>>>>> CandID id: 258024 >>>>>> visit_label is: V1 >>>>>> PSCID is: DCC0000 >>>>>> CandID id: 258024 >>>>>> visit_label is: V1 >>>>>> candidate id 258024 >>>>>> >>>>>> >>>>>> => No Visit labelVisit label does not exist >>>>>> Set centerID = 1 >>>>>> PSCID is: DCC0000 >>>>>> CandID id: 258024 >>>>>> visit_label is: V1 >>>>>> PSCID is: DCC0000 >>>>>> CandID id: 258024 >>>>>> visit_label is: V1 >>>>>> >>>>>> Number of MINC files that will be considered for inserting into the >>>>>> database: 1 >>>>>> >>>>>> log dir is /logs and log file is /logs/TarLoad-15-42-L6pocU.log >>>>>> PSCID is: DCC0000 >>>>>> CandID id: 258024 >>>>>> visit_label is: V1 >>>>>> PSCID is: DCC0000 >>>>>> CandID id: 258024 >>>>>> visit_label is: V1 >>>>>> candidate id 258024 >>>>>> >>>>>> >>>>>> => No Visit label >>>>>> Cleaning up temp files: rm -rf >>>>>> /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*" >>>>>> >>>>>> ???? ???, 24 ???? 2019 ???? 3:38 ?.?., ?/? Sotirios Nikoloutsopoulos < >>>>>> sotirisnik at gmail.com> ??????: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> 1) What are the records in the session table? >>>>>>> >>>>>>> 2) I tried to upload a non-phantom file like this >>>>>>> >>>>>>> input_file.txt: >>>>>>> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1 >>>>>>> >>>>>>> Terminal: >>>>>>> ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > >>>>>>> log.txt >>>>>>> >>>>>>> and i get the following error in the log.txt file >>>>>>> >>>>>>> "Running now the following command: /data/loris/data/uploadNeuroDB/ >>>>>>> imaging_upload_file.pl -profile prod -upload_id 7 >>>>>>> /data/incoming/DCC0000_258024_V1.tar.gz -verbose >>>>>>> >>>>>>> find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name '__MACOSX' >>>>>>> -delete >>>>>>> Spool message is: >>>>>>> The PatientName read from the DICOM header does not start with >>>>>>> DCC0000_258024_V1 from the mri_upload table >>>>>>> " >>>>>>> >>>>>>> and the spool message is repeated for each .dcm file. Should i >>>>>>> preprocess every .dcm file and change the PatientName to >>>>>>> "DCC0000_258024_V1"? >>>>>>> >>>>>>> 3) As far as it concerns the candidate profile i have to create a >>>>>>> new one each time before i try to upload a file? Is there already a way to >>>>>>> create a candidate profile and get the DCCID and PSCID without the user >>>>>>> interface? >>>>>>> >>>>>>> Thank you, >>>>>>> >>>>>>> Sotirios >>>>>>> >>>>>>> ???? ???, 18 ???? 2019 ???? 5:00 ?.?., ?/? Cecile Madjar < >>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>> >>>>>>>> Hi Sotirios, >>>>>>>> >>>>>>>> See answers to your questions below. >>>>>>>> >>>>>>>> About the 1st case we had a dicom with a blank studyUID, maybe >>>>>>>>> someone accidentally removed it. >>>>>>>>> >>>>>>>> That is strange. Something must have been done to the DICOM files. >>>>>>>> I have never seen a study with no StudyUID out of the scanner. Usually, you >>>>>>>> need to manually erase it with some tool. However, DICOMs never cease to >>>>>>>> surprise me... >>>>>>>> >>>>>>>> >>>>>>>>> Some other questions: >>>>>>>>> 1) What are the steps to remove all of my patients and uploads?: >>>>>>>>> because i want to try to reupload all the testing dicoms we had with a >>>>>>>>> clean database. >>>>>>>>> >>>>>>>> >>>>>>>> Release 21.0 of LORIS-MRI will offer a script to remove all the >>>>>>>> entries and files specific to an upload. However, this release is not yet >>>>>>>> out. Hopefully in the next few weeks. >>>>>>>> In the meantime, since your case is particular and that you just >>>>>>>> want to start with a clean database, the following deletes should work >>>>>>>> (hopefully I won't forget any tables): >>>>>>>> delete from mri_violations_log; >>>>>>>> delete from mri_protocol_violated_scans; >>>>>>>> delete from MRICandidateErrors; >>>>>>>> delete from parameter_file; >>>>>>>> delete from files_qcstatus; # (if you played with the QC part of >>>>>>>> the imaging browser for testing) >>>>>>>> delete from feedback_mri_comments; # (if you played with the QC >>>>>>>> part of the imaging browser for testing) >>>>>>>> delete from tarchive_series; >>>>>>>> delete from tarchive_files; >>>>>>>> delete from mri_upload; >>>>>>>> delete from tarchive; >>>>>>>> delete from mri_scanner; >>>>>>>> delete from candidate where PSCID="scanner"; >>>>>>>> >>>>>>>> I don't know if you want to also delete entries in the session and >>>>>>>> candidate table too? >>>>>>>> >>>>>>>> 2) We do not need to create the profile of a patient ( are the >>>>>>>>> patients stored in the candidate table? because i can't see the ghosts ), >>>>>>>>> because will it automatically be created when importing a dicom? >>>>>>>>> >>>>>>>> >>>>>>>> I am not sure I understand fully this question. All candidates are >>>>>>>> stored in the candidate table. The phantom scans however are attached to a >>>>>>>> scanner candidate depending on where the scan happened. It is a bit of a >>>>>>>> weird concept that we have to redesign eventually but never got a chance to >>>>>>>> get to it. >>>>>>>> >>>>>>>> Hopefully this answered your question, otherwise, don't hesitate to >>>>>>>> let me know. >>>>>>>> >>>>>>>> 3) What is the difference between uploading a dicom as phantom >>>>>>>>> instead of importing as PSCCID_DCCID_VisitLabel? >>>>>>>>> >>>>>>>> >>>>>>>> When you upload the scan as PSCID_DCCID_VisitLabel, there is a >>>>>>>> check that makes sure the candidate IDs and visit label are valid at the >>>>>>>> time of upload. >>>>>>>> When you upload a scan as phantom, it expects that the DICOM field >>>>>>>> PatientName and uploaded filename contains the string "phantom". We >>>>>>>> enforced this behaviour on the imaging uploader side in recent releases but >>>>>>>> I can't remember which one. Probably the upcoming 21.0 release. >>>>>>>> >>>>>>>> All those verifications might seem cumbersome but they are here to >>>>>>>> ensure that the files inserted are all valid and labelled properly as it is >>>>>>>> a bit messy to have to delete files that were wrongly labelled. At least, >>>>>>>> the delete script present in release 21.0 will make this process easier but >>>>>>>> it is still good practice to verify those things. >>>>>>>> >>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> >>>>>>>> >>>>>>>> With pleasure. Hopefully the answers to your questions will be >>>>>>>> helpful. >>>>>>>> >>>>>>>> Best, >>>>>>>> >>>>>>>> C?cile >>>>>>>> >>>>>>>> PS: loris-dev mailing list: you will find the initial answer I gave >>>>>>>> Sotirios below. I forgot to cc the loris-dev in my earlier reply... >>>>>>>> >>>>>>>> ???? ???, 16 ???? 2019 ???? 12:02 ?.?., ?/? Cecile Madjar < >>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>> >>>>>>>>>> Dear Sotirios, >>>>>>>>>> >>>>>>>>>> Thank you for reaching out. Since your email was already >>>>>>>>>> organized in points, I will reply directly below your questions below. >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> >>>>>>>>>> C?cile >>>>>>>>>> >>>>>>>>>> 1) What do we do when the StudyUID is blank on the header? >>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md >>>>>>>>>>> There solution provided at the table 3 does not guide us how to find the >>>>>>>>>>> information ( is possible ). >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The StudyUID is the field used in our pipeline at the moment to >>>>>>>>>> check whether a DICOM study was already inserted into LORIS with that same >>>>>>>>>> StudyUID since it is supposed to be unique for every single study. >>>>>>>>>> Currently, we do not support insertion of DICOM studies if they do not have >>>>>>>>>> a StudyUID associated with them. >>>>>>>>>> >>>>>>>>>> Is there a specific reason why the StudyUID is blank in the DICOM >>>>>>>>>> headers in your DICOM files? Did they go through some processes before >>>>>>>>>> upload to LORIS? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2) When i am about to upload a non-phanton case i need a 'Visit >>>>>>>>>>> Label'. Is there already a script that can create a new visit label for a >>>>>>>>>>> specific Candidate, so i can provide it as input afterwards to the >>>>>>>>>>> >>>>>>>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md ) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> I am not sure if the front-end allows for it but if you use >>>>>>>>>> batch_uploads_imageuploader, the insertion scripts could create the visit >>>>>>>>>> label for you. However, it does need to be included in the patient name in >>>>>>>>>> order for the script to know which label it should use. >>>>>>>>>> For example: you upload V02 for MTL0123 (DCCID: 456789) but V02 >>>>>>>>>> was not yet created for MTL0123; the insertion scripts will create a V02 >>>>>>>>>> visit for MTL0123 if the PatientName field for the dataset is >>>>>>>>>> MTL0123_456789_V02 with Stage marked as "Not Started". >>>>>>>>>> >>>>>>>>>> However, *note that for the script to create the visit for the >>>>>>>>>> candidate, you have to make sure that all the visit label of your projects >>>>>>>>>> were inserted in the Visit_Windows table* (otherwise, the visit >>>>>>>>>> will never be created as they were not specified as being part of the list >>>>>>>>>> of visit label to expect). So in the example above, you should have one row >>>>>>>>>> in Visit_Windows with Visit_label="V02" set. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> 3) Section 4.3 >>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md >>>>>>>>>>> >>>>>>>>>>> What do we do when the folder contains multiple mnc files? >>>>>>>>>>> >>>>>>>>>>> This is a case by case. Most of the time, the MINC files that >>>>>>>>>> failed insertion into the imaging browser end up in the MRI violation >>>>>>>>>> module where you can see what went wrong with the acquisitions not >>>>>>>>>> inserted. If you notice that the MINC file should be inserted as a specific >>>>>>>>>> protocol, you can force the insertion as explained in section 4.3. >>>>>>>>>> >>>>>>>>>> In your case, I have a feeling that you want to be able to insert >>>>>>>>>> MINC files that did not go through the whole pipeline insertion (DICOM >>>>>>>>>> archival, then dcm2mnc, then protocol identification, then insertion...). >>>>>>>>>> In theory, you could call minc_insertion.pl on any MINC file, >>>>>>>>>> provided you have at least the following information: >>>>>>>>>> - path to the MINC file >>>>>>>>>> - uploadID associated with the MINC files or DICOM archive path >>>>>>>>>> from which those MINC files were created from >>>>>>>>>> >>>>>>>>>> Then the minc_insertion.pl script will do the subject >>>>>>>>>> information and protocol validation etc... >>>>>>>>>> >>>>>>>>>> If all the MINC files come from the same uploadID or TarchiveID, >>>>>>>>>> then you could run a loop in bash calling the minc_insertion.pl >>>>>>>>>> script like this for files deriving from uploadID=1: >>>>>>>>>> >>>>>>>>>> ls /path/to/mnc/folder/* | while read f; do minc_insertion.pl >>>>>>>>>> -profile prod -uploadID 1 -mincPath $f; done >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Thank you, >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hope this helped! >>>>>>>>>> >>>>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.31.35 AM.png Type: image/png Size: 45707 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.32.50 AM.png Type: image/png Size: 134887 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.40.52 AM.png Type: image/png Size: 95814 bytes Desc: not available URL: From sotirisnik at gmail.com Thu Jul 25 11:42:58 2019 From: sotirisnik at gmail.com (Sotirios Nikoloutsopoulos) Date: Thu, 25 Jul 2019 18:42:58 +0300 Subject: [Loris-dev] Import mri - scripts In-Reply-To: References: Message-ID: Dear Cecile, You were right about your guesses. I did the modifications you suggested, i still get the "no mincs inserted" error, now i am getting the "No center found this candidate" error ( how do i fix that? from what i see the .dcm files contains a 'Institution Name', is that the center? ), so i guess that's why the validation fails, but i changed the force variable of tarchiveLoader to 1, shouldn't it have inserted the mincs then? Also it is optional to have a mail server, right? ( from what i see the mail error does not affect the $valid_study, so i assume we are okay with that ) Thank you a lot, Sotirios ???? ???, 25 ???? 2019 ???? 5:48 ?.?., ?/? Cecile Madjar < cecile.madjar at mcin.ca> ??????: > Hi Sotirios, > > From what I understand, I believe you just need to update a few config > settings from the frontend in the Config module (under the Admin menu). > [image: Screen Shot 2019-07-25 at 10.31.35 AM.png] > > Then go to the Paths section of the config module and update the > "LORIS-MRI code" config setting to /data/loris/bin/mri (my guess is that > currently, it is set to /data/LORIS/data) and click on the submit button. > [image: Screen Shot 2019-07-25 at 10.32.50 AM.png] > > Then, go to Imaging Pipeline section of the Configuration module and > update the "Loris-MRI Data Directory" to /data/loris/data/ (my guess is > that it is empty at the moment in your database which is why $data_dir is > not defined when you run the tarchiveLoader step of the insertion pipeline). > [image: Screen Shot 2019-07-25 at 10.40.52 AM.png] > > Finally, I took a look at your environment file and it might be a good > idea to update your PERL5LIB variable to the following (it has been added > to the code a few releases ago as we realized this was needed for the > libraries. Not sure which version of LORIS-MRI you are using but it does > not hurt to update this variable to what I suggest below): > export > PERL5LIB=/data/loris/bin/mri/uploadNeuroDB:/data/loris/bin/mri/dicom-archive:$PERL5LIB > > Hope this helps. Sorry you are experiencing so much trouble setting things > up. We probably need to update our WIKI. > > Best, > > C?cile > > On Thu, Jul 25, 2019 at 9:48 AM Sotirios Nikoloutsopoulos < > sotirisnik at gmail.com> wrote: > >> The "Loris-MRI Data Directory" should be in the environment file? >> >> ???? ???, 25 ???? 2019 ???? 12:59 ?.?., ?/? Sotirios Nikoloutsopoulos < >> sotirisnik at gmail.com> ??????: >> >>> The scripts are located in /data/loris/bin/mri/. Somewhere I might have >>> messed something up but i remember following the instructions in the >>> wiki.For example when i tried to execute the batch_uploads_imageuploader >>> perl script the line >>> >>> my $command = "$bin_dir/uploadNeuroDB/imaging_upload_file.pl " >>> >>> was pointing to an invalid path, for example: it was repeating a pattern >>> and had LORIS instead of loris ( i think the mistake was that in the wiki >>> it suggested to use $PROJECT for the name of the folder and i choosed >>> 'loris' https://github.com/aces/Loris/wiki/Imaging-Database ). See >>> also the in image for the modification i did. Also i didn't find the >>> section to config the $data_dir variable. >>> >>> Thank you, >>> >>> Sotirios >>> >>> >>> >>> ???? ???, 24 ???? 2019 ???? 6:05 ?.?., ?/? Cecile Madjar < >>> cecile.madjar at mcin.ca> ??????: >>> >>>> Hi Sotirios, >>>> >>>> It looks like you have one Config setting not set (based on the first >>>> error message you got when running it the first time). You need to set a >>>> value for "Loris-MRI Data Directory" in the Config module under the Imaging >>>> pipeline section (from what I can see, it would be /data/loris/data/ for >>>> your setup). >>>> >>>> The reason you got that error message the second time is that the >>>> pipeline checks if the DICOM study was already inserted in the tarchive >>>> table. Since you ran the study once already, the study is inserted into the >>>> tarchive, hence the message. However, the pipeline did not complete as the >>>> MINC files were not created and inserted into the files/parameter_file >>>> table. In order to continue the insertion, you will then need to run the >>>> tarchiveLoader command suggested in the terminal during the second run: >>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile >>>> prod >>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >>>> -verbose >>>> >>>> Finally, I am a bit confused by your setup. Where are the scripts >>>> located? In /data/loris/bin/mri or /data/loris/data? It feels like the >>>> Config setting "LORIS-MRI code" in the Paths section was set to >>>> "/data/LORIS/data" but it should have been set to the directory where the >>>> scripts are located (most probably /data/loris/bin/mri). >>>> >>>> Hope this helps, >>>> >>>> C?cile >>>> >>>> >>>> On Wed, Jul 24, 2019 at 10:55 AM Sotirios Nikoloutsopoulos < >>>> sotirisnik at gmail.com> wrote: >>>> >>>>> Ok first i execute this command: >>>>> >>>>> "lorisadmin at hbp:/data/loris/bin/mri$ ./batch_uploads_imageuploader >>>>> -profile prod < ~/Desktop/input.txt > log.txt >>>>> Use of uninitialized value $data_dir in concatenation (.) or string at >>>>> ./batch_uploads_imageuploader line 143. >>>>> Use of uninitialized value $data_dir in concatenation (.) or string at >>>>> ./batch_uploads_imageuploader line 143. >>>>> Use of uninitialized value $_ in pattern match (m//) at >>>>> ./batch_uploads_imageuploader line 147. >>>>> base is DCC0000_258024_V01 >>>>> path is /data/incoming/ >>>>> type is .tar.gz >>>>> fullpath is /data/incoming/DCC0000_258024_V01.tar.gz >>>>> >>>>> source /tmp/ImagingUpload-17-43-ZUEFjb >>>>> targetlocation /data/loris/data/tarchive >>>>> >>>>> Use of uninitialized value $data_dir in concatenation (.) or string at >>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256. >>>>> Use of uninitialized value $data_dir in concatenation (.) or string at >>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284. >>>>> Use of uninitialized value $data_dir in concatenation (.) or string at >>>>> /data/loris/bin/mri/uploadNeuroDB/tarchive_validation.pl line 219. >>>>> Use of uninitialized value $data_dir in concatenation (.) or string at >>>>> /data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl line 270. >>>>> Use of uninitialized value $data_dir in concatenation (.) or string at >>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711. >>>>> Can't exec "mail": No such file or directory at >>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721. >>>>> print() on closed filehandle MAIL at >>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722. >>>>> print() on closed filehandle MAIL at >>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723. >>>>> Use of uninitialized value $data_dir in concatenation (.) or string at >>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>>>> print() on closed filehandle MAIL at >>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>>>> print() on closed filehandle MAIL at >>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725. >>>>> Use of uninitialized value $data_dir in concatenation (.) or string at >>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735. >>>>> >>>>> No Mincs inserted >>>>> >>>>> >>>>> The tarchiveLoader insertion script has failed. >>>>> Running now the following command: /data/loris/data/uploadNeuroDB/ >>>>> imaging_upload_file.pl -profile prod -upload_id 3 >>>>> /data/incoming/DCC0000_258024_V01.tar.gz -verbose" >>>>> >>>>> I inserted the VisitLabel, but still the Mincs are not inserted and i >>>>> am getting this error >>>>> >>>>> "lorisadmin at hbp:/data/loris/bin/mri$ /data/loris/data/uploadNeuroDB/ >>>>> imaging_upload_file.pl -profile prod -upload_id 3 >>>>> /data/incoming/DCC0000_258024_V01.tar.gz -verbose >>>>> >>>>> find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name '__MACOSX' >>>>> -delete >>>>> Spool message is: >>>>> The Scan for the uploadID 3 has already been run with tarchiveID: 7. >>>>> To continue with the rest of the insertion pipeline, please run >>>>> tarchiveLoader from a terminal as follows: >>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod >>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >>>>> -verbose" >>>>> >>>>> Which files should i delete? I thought that deleting the files from >>>>> the tarchive was enough. And i executed the last proposed command to >>>>> continue to rest of the insertion pipeline ( check the image ). >>>>> >>>>> Thank you >>>>> >>>>> ???? ???, 24 ???? 2019 ???? 5:05 ?.?., ?/? Cecile Madjar < >>>>> cecile.madjar at mcin.ca> ??????: >>>>> >>>>>> Hi Sotirios, >>>>>> >>>>>> Sorry I was not clear before. Every visits of candidates are created >>>>>> in the session table with one entry per CandID/VisitLabel. So if the >>>>>> imaging pipeline created visits in the session table (that are not attached >>>>>> to instruments) and you want to clean your database completely, you can >>>>>> delete entries in that table as well. >>>>>> >>>>>> Regarding your point 2, the imaging pipeline can create the visit in >>>>>> the session table as long as the visit label is present in the >>>>>> Visit_Windows table (which means the visit label stored in the PatientName >>>>>> is a valid visit label). >>>>>> However, in your prod file, if the >>>>>> variable $subjectID{'createVisitLabel'} is set to 1 for candidates, the >>>>>> visit label should be created in the Visit_Windows table I believe. Here >>>>>> is the line of the prod file you would need to change >>>>>> >>>>>> . >>>>>> In general, we prefer populating the Visit_Windows table and set the >>>>>> $subjectID{'createVisitLabel'} to 0 for candidate data to avoid insertion >>>>>> of badly labelled MRI data that are complex to relabel. >>>>>> >>>>>> Finally, regarding the candidate creation, if you execute the >>>>>> following steps, the pipeline should be able to create candidates: >>>>>> - set in the Imaging Pipeline section of the config module the config >>>>>> setting "Upload creation of candidates" to Yes >>>>>> - manually transfer the scans to the LORIS server instead of >>>>>> uploading it via the imaging browser >>>>>> - run batch_imaging_upload.pl as you did until now. >>>>>> >>>>>> Hope this helps! >>>>>> Best, >>>>>> >>>>>> C?cile >>>>>> >>>>>> On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos < >>>>>> sotirisnik at gmail.com> wrote: >>>>>> >>>>>>> About my second question i replaced the name of the patient in the >>>>>>> .dcm files and now i am getting an error that the visit label does not >>>>>>> exist. >>>>>>> I thought it was supposed to be automatically created. >>>>>>> >>>>>>> "Done adding archive info into database >>>>>>> >>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation >>>>>>> -profile prod >>>>>>> /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>>>> -verbose >>>>>>> md5sum >>>>>>> /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>>>> PSCID is: DCC0000 >>>>>>> CandID id: 258024 >>>>>>> visit_label is: V1 >>>>>>> PSCID is: DCC0000 >>>>>>> CandID id: 258024 >>>>>>> visit_label is: V1 >>>>>>> candidate id 258024 >>>>>>> >>>>>>> >>>>>>> => No Visit labelVisit label does not exist >>>>>>> Set centerID = 1 >>>>>>> PSCID is: DCC0000 >>>>>>> CandID id: 258024 >>>>>>> visit_label is: V1 >>>>>>> PSCID is: DCC0000 >>>>>>> CandID id: 258024 >>>>>>> visit_label is: V1 >>>>>>> >>>>>>> Number of MINC files that will be considered for inserting into the >>>>>>> database: 1 >>>>>>> >>>>>>> log dir is /logs and log file is /logs/TarLoad-15-42-L6pocU.log >>>>>>> PSCID is: DCC0000 >>>>>>> CandID id: 258024 >>>>>>> visit_label is: V1 >>>>>>> PSCID is: DCC0000 >>>>>>> CandID id: 258024 >>>>>>> visit_label is: V1 >>>>>>> candidate id 258024 >>>>>>> >>>>>>> >>>>>>> => No Visit label >>>>>>> Cleaning up temp files: rm -rf >>>>>>> /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*" >>>>>>> >>>>>>> ???? ???, 24 ???? 2019 ???? 3:38 ?.?., ?/? Sotirios Nikoloutsopoulos >>>>>>> ??????: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> 1) What are the records in the session table? >>>>>>>> >>>>>>>> 2) I tried to upload a non-phantom file like this >>>>>>>> >>>>>>>> input_file.txt: >>>>>>>> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1 >>>>>>>> >>>>>>>> Terminal: >>>>>>>> ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt >>>>>>>> > log.txt >>>>>>>> >>>>>>>> and i get the following error in the log.txt file >>>>>>>> >>>>>>>> "Running now the following command: /data/loris/data/uploadNeuroDB/ >>>>>>>> imaging_upload_file.pl -profile prod -upload_id 7 >>>>>>>> /data/incoming/DCC0000_258024_V1.tar.gz -verbose >>>>>>>> >>>>>>>> find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name '__MACOSX' >>>>>>>> -delete >>>>>>>> Spool message is: >>>>>>>> The PatientName read from the DICOM header does not start with >>>>>>>> DCC0000_258024_V1 from the mri_upload table >>>>>>>> " >>>>>>>> >>>>>>>> and the spool message is repeated for each .dcm file. Should i >>>>>>>> preprocess every .dcm file and change the PatientName to >>>>>>>> "DCC0000_258024_V1"? >>>>>>>> >>>>>>>> 3) As far as it concerns the candidate profile i have to create a >>>>>>>> new one each time before i try to upload a file? Is there already a way to >>>>>>>> create a candidate profile and get the DCCID and PSCID without the user >>>>>>>> interface? >>>>>>>> >>>>>>>> Thank you, >>>>>>>> >>>>>>>> Sotirios >>>>>>>> >>>>>>>> ???? ???, 18 ???? 2019 ???? 5:00 ?.?., ?/? Cecile Madjar < >>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>> >>>>>>>>> Hi Sotirios, >>>>>>>>> >>>>>>>>> See answers to your questions below. >>>>>>>>> >>>>>>>>> About the 1st case we had a dicom with a blank studyUID, maybe >>>>>>>>>> someone accidentally removed it. >>>>>>>>>> >>>>>>>>> That is strange. Something must have been done to the DICOM files. >>>>>>>>> I have never seen a study with no StudyUID out of the scanner. Usually, you >>>>>>>>> need to manually erase it with some tool. However, DICOMs never cease to >>>>>>>>> surprise me... >>>>>>>>> >>>>>>>>> >>>>>>>>>> Some other questions: >>>>>>>>>> 1) What are the steps to remove all of my patients and uploads?: >>>>>>>>>> because i want to try to reupload all the testing dicoms we had with a >>>>>>>>>> clean database. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Release 21.0 of LORIS-MRI will offer a script to remove all the >>>>>>>>> entries and files specific to an upload. However, this release is not yet >>>>>>>>> out. Hopefully in the next few weeks. >>>>>>>>> In the meantime, since your case is particular and that you just >>>>>>>>> want to start with a clean database, the following deletes should work >>>>>>>>> (hopefully I won't forget any tables): >>>>>>>>> delete from mri_violations_log; >>>>>>>>> delete from mri_protocol_violated_scans; >>>>>>>>> delete from MRICandidateErrors; >>>>>>>>> delete from parameter_file; >>>>>>>>> delete from files_qcstatus; # (if you played with the QC part of >>>>>>>>> the imaging browser for testing) >>>>>>>>> delete from feedback_mri_comments; # (if you played with the QC >>>>>>>>> part of the imaging browser for testing) >>>>>>>>> delete from tarchive_series; >>>>>>>>> delete from tarchive_files; >>>>>>>>> delete from mri_upload; >>>>>>>>> delete from tarchive; >>>>>>>>> delete from mri_scanner; >>>>>>>>> delete from candidate where PSCID="scanner"; >>>>>>>>> >>>>>>>>> I don't know if you want to also delete entries in the session and >>>>>>>>> candidate table too? >>>>>>>>> >>>>>>>>> 2) We do not need to create the profile of a patient ( are the >>>>>>>>>> patients stored in the candidate table? because i can't see the ghosts ), >>>>>>>>>> because will it automatically be created when importing a dicom? >>>>>>>>>> >>>>>>>>> >>>>>>>>> I am not sure I understand fully this question. All candidates are >>>>>>>>> stored in the candidate table. The phantom scans however are attached to a >>>>>>>>> scanner candidate depending on where the scan happened. It is a bit of a >>>>>>>>> weird concept that we have to redesign eventually but never got a chance to >>>>>>>>> get to it. >>>>>>>>> >>>>>>>>> Hopefully this answered your question, otherwise, don't hesitate >>>>>>>>> to let me know. >>>>>>>>> >>>>>>>>> 3) What is the difference between uploading a dicom as phantom >>>>>>>>>> instead of importing as PSCCID_DCCID_VisitLabel? >>>>>>>>>> >>>>>>>>> >>>>>>>>> When you upload the scan as PSCID_DCCID_VisitLabel, there is a >>>>>>>>> check that makes sure the candidate IDs and visit label are valid at the >>>>>>>>> time of upload. >>>>>>>>> When you upload a scan as phantom, it expects that the DICOM field >>>>>>>>> PatientName and uploaded filename contains the string "phantom". We >>>>>>>>> enforced this behaviour on the imaging uploader side in recent releases but >>>>>>>>> I can't remember which one. Probably the upcoming 21.0 release. >>>>>>>>> >>>>>>>>> All those verifications might seem cumbersome but they are here to >>>>>>>>> ensure that the files inserted are all valid and labelled properly as it is >>>>>>>>> a bit messy to have to delete files that were wrongly labelled. At least, >>>>>>>>> the delete script present in release 21.0 will make this process easier but >>>>>>>>> it is still good practice to verify those things. >>>>>>>>> >>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> >>>>>>>>> >>>>>>>>> With pleasure. Hopefully the answers to your questions will be >>>>>>>>> helpful. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> >>>>>>>>> C?cile >>>>>>>>> >>>>>>>>> PS: loris-dev mailing list: you will find the initial answer I >>>>>>>>> gave Sotirios below. I forgot to cc the loris-dev in my earlier reply... >>>>>>>>> >>>>>>>>> ???? ???, 16 ???? 2019 ???? 12:02 ?.?., ?/? Cecile Madjar < >>>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>>> >>>>>>>>>>> Dear Sotirios, >>>>>>>>>>> >>>>>>>>>>> Thank you for reaching out. Since your email was already >>>>>>>>>>> organized in points, I will reply directly below your questions below. >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> >>>>>>>>>>> C?cile >>>>>>>>>>> >>>>>>>>>>> 1) What do we do when the StudyUID is blank on the header? >>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md >>>>>>>>>>>> There solution provided at the table 3 does not guide us how to find the >>>>>>>>>>>> information ( is possible ). >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The StudyUID is the field used in our pipeline at the moment to >>>>>>>>>>> check whether a DICOM study was already inserted into LORIS with that same >>>>>>>>>>> StudyUID since it is supposed to be unique for every single study. >>>>>>>>>>> Currently, we do not support insertion of DICOM studies if they do not have >>>>>>>>>>> a StudyUID associated with them. >>>>>>>>>>> >>>>>>>>>>> Is there a specific reason why the StudyUID is blank in the >>>>>>>>>>> DICOM headers in your DICOM files? Did they go through some processes >>>>>>>>>>> before upload to LORIS? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2) When i am about to upload a non-phanton case i need a 'Visit >>>>>>>>>>>> Label'. Is there already a script that can create a new visit label for a >>>>>>>>>>>> specific Candidate, so i can provide it as input afterwards to the >>>>>>>>>>>> >>>>>>>>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md ) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> I am not sure if the front-end allows for it but if you use >>>>>>>>>>> batch_uploads_imageuploader, the insertion scripts could create the visit >>>>>>>>>>> label for you. However, it does need to be included in the patient name in >>>>>>>>>>> order for the script to know which label it should use. >>>>>>>>>>> For example: you upload V02 for MTL0123 (DCCID: 456789) but V02 >>>>>>>>>>> was not yet created for MTL0123; the insertion scripts will create a V02 >>>>>>>>>>> visit for MTL0123 if the PatientName field for the dataset is >>>>>>>>>>> MTL0123_456789_V02 with Stage marked as "Not Started". >>>>>>>>>>> >>>>>>>>>>> However, *note that for the script to create the visit for the >>>>>>>>>>> candidate, you have to make sure that all the visit label of your projects >>>>>>>>>>> were inserted in the Visit_Windows table* (otherwise, the visit >>>>>>>>>>> will never be created as they were not specified as being part of the list >>>>>>>>>>> of visit label to expect). So in the example above, you should have one row >>>>>>>>>>> in Visit_Windows with Visit_label="V02" set. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> 3) Section 4.3 >>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md >>>>>>>>>>>> >>>>>>>>>>>> What do we do when the folder contains multiple mnc files? >>>>>>>>>>>> >>>>>>>>>>>> This is a case by case. Most of the time, the MINC files that >>>>>>>>>>> failed insertion into the imaging browser end up in the MRI violation >>>>>>>>>>> module where you can see what went wrong with the acquisitions not >>>>>>>>>>> inserted. If you notice that the MINC file should be inserted as a specific >>>>>>>>>>> protocol, you can force the insertion as explained in section 4.3. >>>>>>>>>>> >>>>>>>>>>> In your case, I have a feeling that you want to be able to >>>>>>>>>>> insert MINC files that did not go through the whole pipeline insertion >>>>>>>>>>> (DICOM archival, then dcm2mnc, then protocol identification, then >>>>>>>>>>> insertion...). In theory, you could call minc_insertion.pl on >>>>>>>>>>> any MINC file, provided you have at least the following information: >>>>>>>>>>> - path to the MINC file >>>>>>>>>>> - uploadID associated with the MINC files or DICOM archive path >>>>>>>>>>> from which those MINC files were created from >>>>>>>>>>> >>>>>>>>>>> Then the minc_insertion.pl script will do the subject >>>>>>>>>>> information and protocol validation etc... >>>>>>>>>>> >>>>>>>>>>> If all the MINC files come from the same uploadID or TarchiveID, >>>>>>>>>>> then you could run a loop in bash calling the minc_insertion.pl >>>>>>>>>>> script like this for files deriving from uploadID=1: >>>>>>>>>>> >>>>>>>>>>> ls /path/to/mnc/folder/* | while read f; do minc_insertion.pl >>>>>>>>>>> -profile prod -uploadID 1 -mincPath $f; done >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Thank you, >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Hope this helped! >>>>>>>>>>> >>>>>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.31.35 AM.png Type: image/png Size: 45707 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.32.50 AM.png Type: image/png Size: 134887 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.40.52 AM.png Type: image/png Size: 95814 bytes Desc: not available URL: -------------- next part -------------- Running now the following command: /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile prod -upload_id 4 /data/incoming/DCC0000_258024_V01.tar.gz -verbose find -path \/tmp\/ImagingUpload\-18\-24\-Tn9fQY -name '__MACOSX' -delete /data/loris/bin/mri/dicom-archive/dicomTar.pl /tmp/ImagingUpload-18-24-Tn9fQY /data/loris/data/tarchive/ -clobber -database -profile prod -verbose Source: /tmp/ImagingUpload-18-24-Tn9fQY Target: /data/loris/data/tarchive Testing for database connectivity. Database is available. You will archive the dir : ImagingUpload-18-24-Tn9fQY You are creating a tar with the following command: tar -cf /data/loris/data/tarchive/ImagingUpload-18-24-Tn9fQY.tar ImagingUpload-18-24-Tn9fQY getting md5sums and gzipping!! * Taken from dir : /tmp/ImagingUpload-18-24-Tn9fQY * Archive target location : /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-18-24-Tn9fQY.tar * Name of creating host : 127.0.1.1 * Name of host OS : Linux * Created by user : lorisadmin * Archived on : 2019-07-25 18:24:43 * dicomSummary version : 1 * dicomTar version : 1 * md5sum for DICOM tarball : d5d7701c95f54efbdb745b358ce7c133 ImagingUpload-18-24-Tn9fQY.tar * md5sum for DICOM tarball gzipped : 757a3d2ba29f885ea5dc0902e2a5cc62 ImagingUpload-18-24-Tn9fQY.tar.gz * md5sum for complete archive : f7940af43992811324440d8153942654 DCM_2012-12-05_ImagingUpload-18-24-Tn9fQY.tar Adding archive info into database Removing temporary files from target location Done adding archive info into database /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader -globLocation -profile prod /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-18-24-Tn9fQY.tar -verbose md5sum /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-18-24-Tn9fQY.tar PSCID is: DCC0000 CandID id: 258024 visit_label is: V01 Spool message is: ERROR: No center found for this candidate PSCID is: DCC0000 CandID id: 258024 visit_label is: V01 PSCID is: DCC0000 CandID id: 258024 visit_label is: V01 Number of MINC files that will be considered for inserting into the database: 1 log dir is /data/loris/data//logs and log file is /data/loris/data//logs/TarLoad-18-24-9_GgGT.log PSCID is: DCC0000 CandID id: 258024 visit_label is: V01 PSCID is: DCC0000 CandID id: 258024 visit_label is: V01 candidate id 258024 => No Visit label Cleaning up temp files: rm -rf /tmp/TarLoad-18-24-hzcram/ImagingUpload-18-24-Tn9fQY* -------------- next part -------------- A non-text attachment was scrubbed... Name: 1.png Type: image/png Size: 90809 bytes Desc: not available URL: From cecile.madjar at mcin.ca Thu Jul 25 11:55:24 2019 From: cecile.madjar at mcin.ca (Cecile Madjar) Date: Thu, 25 Jul 2019 11:55:24 -0400 Subject: [Loris-dev] Import mri - scripts In-Reply-To: References: Message-ID: Hi Sotirios, We are making progress! I have a feeling this error comes from the fact that your psc table does not have the MRI_alias populated for the DCC site. Running the following query in the mysql database should fix this: UPDATE psc SET MRI_alias="DCC", Alias="DCC" WHERE Name="Data Coordinating Center"; Repeat the same with all your sites (make sure to change the Aliases between sites, they should all be unique). Let me know how it goes! Best, C?cile On Thu, Jul 25, 2019 at 11:43 AM Sotirios Nikoloutsopoulos < sotirisnik at gmail.com> wrote: > Dear Cecile, > You were right about your guesses. I did the modifications you suggested, > i still get the "no mincs inserted" error, now i am getting the "No center > found this candidate" error ( how do i fix that? from what i see the .dcm > files contains a 'Institution Name', is that the center? ), so i guess > that's why the validation fails, but i changed the force variable of > tarchiveLoader to 1, shouldn't it have inserted the mincs then? Also it is > optional to have a mail server, right? ( from what i see the mail error > does not affect the $valid_study, so i assume we are okay with that ) > Thank you a lot, > Sotirios > > ???? ???, 25 ???? 2019 ???? 5:48 ?.?., ?/? Cecile Madjar < > cecile.madjar at mcin.ca> ??????: > >> Hi Sotirios, >> >> From what I understand, I believe you just need to update a few config >> settings from the frontend in the Config module (under the Admin menu). >> [image: Screen Shot 2019-07-25 at 10.31.35 AM.png] >> >> Then go to the Paths section of the config module and update the >> "LORIS-MRI code" config setting to /data/loris/bin/mri (my guess is that >> currently, it is set to /data/LORIS/data) and click on the submit button. >> [image: Screen Shot 2019-07-25 at 10.32.50 AM.png] >> >> Then, go to Imaging Pipeline section of the Configuration module and >> update the "Loris-MRI Data Directory" to /data/loris/data/ (my guess is >> that it is empty at the moment in your database which is why $data_dir is >> not defined when you run the tarchiveLoader step of the insertion pipeline). >> [image: Screen Shot 2019-07-25 at 10.40.52 AM.png] >> >> Finally, I took a look at your environment file and it might be a good >> idea to update your PERL5LIB variable to the following (it has been added >> to the code a few releases ago as we realized this was needed for the >> libraries. Not sure which version of LORIS-MRI you are using but it does >> not hurt to update this variable to what I suggest below): >> export >> PERL5LIB=/data/loris/bin/mri/uploadNeuroDB:/data/loris/bin/mri/dicom-archive:$PERL5LIB >> >> Hope this helps. Sorry you are experiencing so much trouble setting >> things up. We probably need to update our WIKI. >> >> Best, >> >> C?cile >> >> On Thu, Jul 25, 2019 at 9:48 AM Sotirios Nikoloutsopoulos < >> sotirisnik at gmail.com> wrote: >> >>> The "Loris-MRI Data Directory" should be in the environment file? >>> >>> ???? ???, 25 ???? 2019 ???? 12:59 ?.?., ?/? Sotirios Nikoloutsopoulos < >>> sotirisnik at gmail.com> ??????: >>> >>>> The scripts are located in /data/loris/bin/mri/. Somewhere I might >>>> have messed something up but i remember following the instructions in the >>>> wiki.For example when i tried to execute the batch_uploads_imageuploader >>>> perl script the line >>>> >>>> my $command = "$bin_dir/uploadNeuroDB/imaging_upload_file.pl " >>>> >>>> was pointing to an invalid path, for example: it was repeating a >>>> pattern and had LORIS instead of loris ( i think the mistake was that in >>>> the wiki it suggested to use $PROJECT for the name of the folder and i >>>> choosed 'loris' https://github.com/aces/Loris/wiki/Imaging-Database >>>> ). See also the in image for the modification i did. Also i didn't find the >>>> section to config the $data_dir variable. >>>> >>>> Thank you, >>>> >>>> Sotirios >>>> >>>> >>>> >>>> ???? ???, 24 ???? 2019 ???? 6:05 ?.?., ?/? Cecile Madjar < >>>> cecile.madjar at mcin.ca> ??????: >>>> >>>>> Hi Sotirios, >>>>> >>>>> It looks like you have one Config setting not set (based on the first >>>>> error message you got when running it the first time). You need to set a >>>>> value for "Loris-MRI Data Directory" in the Config module under the Imaging >>>>> pipeline section (from what I can see, it would be /data/loris/data/ for >>>>> your setup). >>>>> >>>>> The reason you got that error message the second time is that the >>>>> pipeline checks if the DICOM study was already inserted in the tarchive >>>>> table. Since you ran the study once already, the study is inserted into the >>>>> tarchive, hence the message. However, the pipeline did not complete as the >>>>> MINC files were not created and inserted into the files/parameter_file >>>>> table. In order to continue the insertion, you will then need to run the >>>>> tarchiveLoader command suggested in the terminal during the second run: >>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile >>>>> prod >>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >>>>> -verbose >>>>> >>>>> Finally, I am a bit confused by your setup. Where are the scripts >>>>> located? In /data/loris/bin/mri or /data/loris/data? It feels like the >>>>> Config setting "LORIS-MRI code" in the Paths section was set to >>>>> "/data/LORIS/data" but it should have been set to the directory where the >>>>> scripts are located (most probably /data/loris/bin/mri). >>>>> >>>>> Hope this helps, >>>>> >>>>> C?cile >>>>> >>>>> >>>>> On Wed, Jul 24, 2019 at 10:55 AM Sotirios Nikoloutsopoulos < >>>>> sotirisnik at gmail.com> wrote: >>>>> >>>>>> Ok first i execute this command: >>>>>> >>>>>> "lorisadmin at hbp:/data/loris/bin/mri$ ./batch_uploads_imageuploader >>>>>> -profile prod < ~/Desktop/input.txt > log.txt >>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>> at ./batch_uploads_imageuploader line 143. >>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>> at ./batch_uploads_imageuploader line 143. >>>>>> Use of uninitialized value $_ in pattern match (m//) at >>>>>> ./batch_uploads_imageuploader line 147. >>>>>> base is DCC0000_258024_V01 >>>>>> path is /data/incoming/ >>>>>> type is .tar.gz >>>>>> fullpath is /data/incoming/DCC0000_258024_V01.tar.gz >>>>>> >>>>>> source /tmp/ImagingUpload-17-43-ZUEFjb >>>>>> targetlocation /data/loris/data/tarchive >>>>>> >>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256. >>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284. >>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>> at /data/loris/bin/mri/uploadNeuroDB/tarchive_validation.pl line 219. >>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>> at /data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl line 270. >>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711. >>>>>> Can't exec "mail": No such file or directory at >>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721. >>>>>> print() on closed filehandle MAIL at >>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722. >>>>>> print() on closed filehandle MAIL at >>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723. >>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>>>>> print() on closed filehandle MAIL at >>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>>>>> print() on closed filehandle MAIL at >>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725. >>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735. >>>>>> >>>>>> No Mincs inserted >>>>>> >>>>>> >>>>>> The tarchiveLoader insertion script has failed. >>>>>> Running now the following command: /data/loris/data/uploadNeuroDB/ >>>>>> imaging_upload_file.pl -profile prod -upload_id 3 >>>>>> /data/incoming/DCC0000_258024_V01.tar.gz -verbose" >>>>>> >>>>>> I inserted the VisitLabel, but still the Mincs are not inserted and i >>>>>> am getting this error >>>>>> >>>>>> "lorisadmin at hbp:/data/loris/bin/mri$ /data/loris/data/uploadNeuroDB/ >>>>>> imaging_upload_file.pl -profile prod -upload_id 3 >>>>>> /data/incoming/DCC0000_258024_V01.tar.gz -verbose >>>>>> >>>>>> find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name '__MACOSX' >>>>>> -delete >>>>>> Spool message is: >>>>>> The Scan for the uploadID 3 has already been run with tarchiveID: 7. >>>>>> To continue with the rest of the insertion pipeline, please run >>>>>> tarchiveLoader from a terminal as follows: >>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod >>>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >>>>>> -verbose" >>>>>> >>>>>> Which files should i delete? I thought that deleting the files from >>>>>> the tarchive was enough. And i executed the last proposed command to >>>>>> continue to rest of the insertion pipeline ( check the image ). >>>>>> >>>>>> Thank you >>>>>> >>>>>> ???? ???, 24 ???? 2019 ???? 5:05 ?.?., ?/? Cecile Madjar < >>>>>> cecile.madjar at mcin.ca> ??????: >>>>>> >>>>>>> Hi Sotirios, >>>>>>> >>>>>>> Sorry I was not clear before. Every visits of candidates are created >>>>>>> in the session table with one entry per CandID/VisitLabel. So if the >>>>>>> imaging pipeline created visits in the session table (that are not attached >>>>>>> to instruments) and you want to clean your database completely, you can >>>>>>> delete entries in that table as well. >>>>>>> >>>>>>> Regarding your point 2, the imaging pipeline can create the visit in >>>>>>> the session table as long as the visit label is present in the >>>>>>> Visit_Windows table (which means the visit label stored in the PatientName >>>>>>> is a valid visit label). >>>>>>> However, in your prod file, if the >>>>>>> variable $subjectID{'createVisitLabel'} is set to 1 for candidates, the >>>>>>> visit label should be created in the Visit_Windows table I believe. Here >>>>>>> is the line of the prod file you would need to change >>>>>>> >>>>>>> . >>>>>>> In general, we prefer populating the Visit_Windows table and set the >>>>>>> $subjectID{'createVisitLabel'} to 0 for candidate data to avoid insertion >>>>>>> of badly labelled MRI data that are complex to relabel. >>>>>>> >>>>>>> Finally, regarding the candidate creation, if you execute the >>>>>>> following steps, the pipeline should be able to create candidates: >>>>>>> - set in the Imaging Pipeline section of the config module the >>>>>>> config setting "Upload creation of candidates" to Yes >>>>>>> - manually transfer the scans to the LORIS server instead of >>>>>>> uploading it via the imaging browser >>>>>>> - run batch_imaging_upload.pl as you did until now. >>>>>>> >>>>>>> Hope this helps! >>>>>>> Best, >>>>>>> >>>>>>> C?cile >>>>>>> >>>>>>> On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos < >>>>>>> sotirisnik at gmail.com> wrote: >>>>>>> >>>>>>>> About my second question i replaced the name of the patient in the >>>>>>>> .dcm files and now i am getting an error that the visit label does not >>>>>>>> exist. >>>>>>>> I thought it was supposed to be automatically created. >>>>>>>> >>>>>>>> "Done adding archive info into database >>>>>>>> >>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation >>>>>>>> -profile prod >>>>>>>> /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>>>>> -verbose >>>>>>>> md5sum >>>>>>>> /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>>>>> PSCID is: DCC0000 >>>>>>>> CandID id: 258024 >>>>>>>> visit_label is: V1 >>>>>>>> PSCID is: DCC0000 >>>>>>>> CandID id: 258024 >>>>>>>> visit_label is: V1 >>>>>>>> candidate id 258024 >>>>>>>> >>>>>>>> >>>>>>>> => No Visit labelVisit label does not exist >>>>>>>> Set centerID = 1 >>>>>>>> PSCID is: DCC0000 >>>>>>>> CandID id: 258024 >>>>>>>> visit_label is: V1 >>>>>>>> PSCID is: DCC0000 >>>>>>>> CandID id: 258024 >>>>>>>> visit_label is: V1 >>>>>>>> >>>>>>>> Number of MINC files that will be considered for inserting into the >>>>>>>> database: 1 >>>>>>>> >>>>>>>> log dir is /logs and log file is /logs/TarLoad-15-42-L6pocU.log >>>>>>>> PSCID is: DCC0000 >>>>>>>> CandID id: 258024 >>>>>>>> visit_label is: V1 >>>>>>>> PSCID is: DCC0000 >>>>>>>> CandID id: 258024 >>>>>>>> visit_label is: V1 >>>>>>>> candidate id 258024 >>>>>>>> >>>>>>>> >>>>>>>> => No Visit label >>>>>>>> Cleaning up temp files: rm -rf >>>>>>>> /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*" >>>>>>>> >>>>>>>> ???? ???, 24 ???? 2019 ???? 3:38 ?.?., ?/? Sotirios >>>>>>>> Nikoloutsopoulos ??????: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> 1) What are the records in the session table? >>>>>>>>> >>>>>>>>> 2) I tried to upload a non-phantom file like this >>>>>>>>> >>>>>>>>> input_file.txt: >>>>>>>>> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1 >>>>>>>>> >>>>>>>>> Terminal: >>>>>>>>> ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt >>>>>>>>> > log.txt >>>>>>>>> >>>>>>>>> and i get the following error in the log.txt file >>>>>>>>> >>>>>>>>> "Running now the following command: /data/loris/data/uploadNeuroDB/ >>>>>>>>> imaging_upload_file.pl -profile prod -upload_id 7 >>>>>>>>> /data/incoming/DCC0000_258024_V1.tar.gz -verbose >>>>>>>>> >>>>>>>>> find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name '__MACOSX' >>>>>>>>> -delete >>>>>>>>> Spool message is: >>>>>>>>> The PatientName read from the DICOM header does not start with >>>>>>>>> DCC0000_258024_V1 from the mri_upload table >>>>>>>>> " >>>>>>>>> >>>>>>>>> and the spool message is repeated for each .dcm file. Should i >>>>>>>>> preprocess every .dcm file and change the PatientName to >>>>>>>>> "DCC0000_258024_V1"? >>>>>>>>> >>>>>>>>> 3) As far as it concerns the candidate profile i have to create a >>>>>>>>> new one each time before i try to upload a file? Is there already a way to >>>>>>>>> create a candidate profile and get the DCCID and PSCID without the user >>>>>>>>> interface? >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> >>>>>>>>> Sotirios >>>>>>>>> >>>>>>>>> ???? ???, 18 ???? 2019 ???? 5:00 ?.?., ?/? Cecile Madjar < >>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>> >>>>>>>>>> Hi Sotirios, >>>>>>>>>> >>>>>>>>>> See answers to your questions below. >>>>>>>>>> >>>>>>>>>> About the 1st case we had a dicom with a blank studyUID, maybe >>>>>>>>>>> someone accidentally removed it. >>>>>>>>>>> >>>>>>>>>> That is strange. Something must have been done to the DICOM >>>>>>>>>> files. I have never seen a study with no StudyUID out of the scanner. >>>>>>>>>> Usually, you need to manually erase it with some tool. However, DICOMs >>>>>>>>>> never cease to surprise me... >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Some other questions: >>>>>>>>>>> 1) What are the steps to remove all of my patients and uploads?: >>>>>>>>>>> because i want to try to reupload all the testing dicoms we had with a >>>>>>>>>>> clean database. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Release 21.0 of LORIS-MRI will offer a script to remove all the >>>>>>>>>> entries and files specific to an upload. However, this release is not yet >>>>>>>>>> out. Hopefully in the next few weeks. >>>>>>>>>> In the meantime, since your case is particular and that you just >>>>>>>>>> want to start with a clean database, the following deletes should work >>>>>>>>>> (hopefully I won't forget any tables): >>>>>>>>>> delete from mri_violations_log; >>>>>>>>>> delete from mri_protocol_violated_scans; >>>>>>>>>> delete from MRICandidateErrors; >>>>>>>>>> delete from parameter_file; >>>>>>>>>> delete from files_qcstatus; # (if you played with the QC part of >>>>>>>>>> the imaging browser for testing) >>>>>>>>>> delete from feedback_mri_comments; # (if you played with the QC >>>>>>>>>> part of the imaging browser for testing) >>>>>>>>>> delete from tarchive_series; >>>>>>>>>> delete from tarchive_files; >>>>>>>>>> delete from mri_upload; >>>>>>>>>> delete from tarchive; >>>>>>>>>> delete from mri_scanner; >>>>>>>>>> delete from candidate where PSCID="scanner"; >>>>>>>>>> >>>>>>>>>> I don't know if you want to also delete entries in the session >>>>>>>>>> and candidate table too? >>>>>>>>>> >>>>>>>>>> 2) We do not need to create the profile of a patient ( are the >>>>>>>>>>> patients stored in the candidate table? because i can't see the ghosts ), >>>>>>>>>>> because will it automatically be created when importing a dicom? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I am not sure I understand fully this question. All candidates >>>>>>>>>> are stored in the candidate table. The phantom scans however are attached >>>>>>>>>> to a scanner candidate depending on where the scan happened. It is a bit of >>>>>>>>>> a weird concept that we have to redesign eventually but never got a chance >>>>>>>>>> to get to it. >>>>>>>>>> >>>>>>>>>> Hopefully this answered your question, otherwise, don't hesitate >>>>>>>>>> to let me know. >>>>>>>>>> >>>>>>>>>> 3) What is the difference between uploading a dicom as phantom >>>>>>>>>>> instead of importing as PSCCID_DCCID_VisitLabel? >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> When you upload the scan as PSCID_DCCID_VisitLabel, there is a >>>>>>>>>> check that makes sure the candidate IDs and visit label are valid at the >>>>>>>>>> time of upload. >>>>>>>>>> When you upload a scan as phantom, it expects that the DICOM >>>>>>>>>> field PatientName and uploaded filename contains the string "phantom". We >>>>>>>>>> enforced this behaviour on the imaging uploader side in recent releases but >>>>>>>>>> I can't remember which one. Probably the upcoming 21.0 release. >>>>>>>>>> >>>>>>>>>> All those verifications might seem cumbersome but they are here >>>>>>>>>> to ensure that the files inserted are all valid and labelled properly as it >>>>>>>>>> is a bit messy to have to delete files that were wrongly labelled. At >>>>>>>>>> least, the delete script present in release 21.0 will make this process >>>>>>>>>> easier but it is still good practice to verify those things. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Thank you, >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> With pleasure. Hopefully the answers to your questions will be >>>>>>>>>> helpful. >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> >>>>>>>>>> C?cile >>>>>>>>>> >>>>>>>>>> PS: loris-dev mailing list: you will find the initial answer I >>>>>>>>>> gave Sotirios below. I forgot to cc the loris-dev in my earlier reply... >>>>>>>>>> >>>>>>>>>> ???? ???, 16 ???? 2019 ???? 12:02 ?.?., ?/? Cecile Madjar < >>>>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>>>> >>>>>>>>>>>> Dear Sotirios, >>>>>>>>>>>> >>>>>>>>>>>> Thank you for reaching out. Since your email was already >>>>>>>>>>>> organized in points, I will reply directly below your questions below. >>>>>>>>>>>> >>>>>>>>>>>> Best, >>>>>>>>>>>> >>>>>>>>>>>> C?cile >>>>>>>>>>>> >>>>>>>>>>>> 1) What do we do when the StudyUID is blank on the header? >>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md >>>>>>>>>>>>> There solution provided at the table 3 does not guide us how to find the >>>>>>>>>>>>> information ( is possible ). >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> The StudyUID is the field used in our pipeline at the moment to >>>>>>>>>>>> check whether a DICOM study was already inserted into LORIS with that same >>>>>>>>>>>> StudyUID since it is supposed to be unique for every single study. >>>>>>>>>>>> Currently, we do not support insertion of DICOM studies if they do not have >>>>>>>>>>>> a StudyUID associated with them. >>>>>>>>>>>> >>>>>>>>>>>> Is there a specific reason why the StudyUID is blank in the >>>>>>>>>>>> DICOM headers in your DICOM files? Did they go through some processes >>>>>>>>>>>> before upload to LORIS? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 2) When i am about to upload a non-phanton case i need a 'Visit >>>>>>>>>>>>> Label'. Is there already a script that can create a new visit label for a >>>>>>>>>>>>> specific Candidate, so i can provide it as input afterwards to the >>>>>>>>>>>>> >>>>>>>>>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md ) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> I am not sure if the front-end allows for it but if you use >>>>>>>>>>>> batch_uploads_imageuploader, the insertion scripts could create the visit >>>>>>>>>>>> label for you. However, it does need to be included in the patient name in >>>>>>>>>>>> order for the script to know which label it should use. >>>>>>>>>>>> For example: you upload V02 for MTL0123 (DCCID: 456789) but >>>>>>>>>>>> V02 was not yet created for MTL0123; the insertion scripts will create a >>>>>>>>>>>> V02 visit for MTL0123 if the PatientName field for the dataset is >>>>>>>>>>>> MTL0123_456789_V02 with Stage marked as "Not Started". >>>>>>>>>>>> >>>>>>>>>>>> However, *note that for the script to create the visit for the >>>>>>>>>>>> candidate, you have to make sure that all the visit label of your projects >>>>>>>>>>>> were inserted in the Visit_Windows table* (otherwise, the >>>>>>>>>>>> visit will never be created as they were not specified as being part of the >>>>>>>>>>>> list of visit label to expect). So in the example above, you should have >>>>>>>>>>>> one row in Visit_Windows with Visit_label="V02" set. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> 3) Section 4.3 >>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md >>>>>>>>>>>>> >>>>>>>>>>>>> What do we do when the folder contains multiple mnc files? >>>>>>>>>>>>> >>>>>>>>>>>>> This is a case by case. Most of the time, the MINC files that >>>>>>>>>>>> failed insertion into the imaging browser end up in the MRI violation >>>>>>>>>>>> module where you can see what went wrong with the acquisitions not >>>>>>>>>>>> inserted. If you notice that the MINC file should be inserted as a specific >>>>>>>>>>>> protocol, you can force the insertion as explained in section 4.3. >>>>>>>>>>>> >>>>>>>>>>>> In your case, I have a feeling that you want to be able to >>>>>>>>>>>> insert MINC files that did not go through the whole pipeline insertion >>>>>>>>>>>> (DICOM archival, then dcm2mnc, then protocol identification, then >>>>>>>>>>>> insertion...). In theory, you could call minc_insertion.pl on >>>>>>>>>>>> any MINC file, provided you have at least the following information: >>>>>>>>>>>> - path to the MINC file >>>>>>>>>>>> - uploadID associated with the MINC files or DICOM archive path >>>>>>>>>>>> from which those MINC files were created from >>>>>>>>>>>> >>>>>>>>>>>> Then the minc_insertion.pl script will do the subject >>>>>>>>>>>> information and protocol validation etc... >>>>>>>>>>>> >>>>>>>>>>>> If all the MINC files come from the same uploadID or >>>>>>>>>>>> TarchiveID, then you could run a loop in bash calling the >>>>>>>>>>>> minc_insertion.pl script like this for files deriving from >>>>>>>>>>>> uploadID=1: >>>>>>>>>>>> >>>>>>>>>>>> ls /path/to/mnc/folder/* | while read f; do minc_insertion.pl >>>>>>>>>>>> -profile prod -uploadID 1 -mincPath $f; done >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Thank you, >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hope this helped! >>>>>>>>>>>> >>>>>>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.31.35 AM.png Type: image/png Size: 45707 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.32.50 AM.png Type: image/png Size: 134887 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.40.52 AM.png Type: image/png Size: 95814 bytes Desc: not available URL: From sotirisnik at gmail.com Fri Jul 26 08:02:38 2019 From: sotirisnik at gmail.com (Sotirios Nikoloutsopoulos) Date: Fri, 26 Jul 2019 15:02:38 +0300 Subject: [Loris-dev] Import mri - scripts In-Reply-To: References: Message-ID: Hi Cecile, The Center error was fixed. Still the insertion of mincs fails. Please see the output.txt file i attached. Also is there something else i need to change at the paths configuration? Thank you, Sotirios ???? ???, 25 ???? 2019 ???? 6:55 ?.?., ?/? Cecile Madjar < cecile.madjar at mcin.ca> ??????: > Hi Sotirios, > > We are making progress! > > I have a feeling this error comes from the fact that your psc table does > not have the MRI_alias populated for the DCC site. Running the following > query in the mysql database should fix this: > UPDATE psc SET MRI_alias="DCC", Alias="DCC" WHERE Name="Data Coordinating > Center"; > Repeat the same with all your sites (make sure to change the Aliases > between sites, they should all be unique). > > Let me know how it goes! > Best, > > C?cile > > > > > > On Thu, Jul 25, 2019 at 11:43 AM Sotirios Nikoloutsopoulos < > sotirisnik at gmail.com> wrote: > >> Dear Cecile, >> You were right about your guesses. I did the modifications you suggested, >> i still get the "no mincs inserted" error, now i am getting the "No center >> found this candidate" error ( how do i fix that? from what i see the .dcm >> files contains a 'Institution Name', is that the center? ), so i guess >> that's why the validation fails, but i changed the force variable of >> tarchiveLoader to 1, shouldn't it have inserted the mincs then? Also it is >> optional to have a mail server, right? ( from what i see the mail error >> does not affect the $valid_study, so i assume we are okay with that ) >> Thank you a lot, >> Sotirios >> >> ???? ???, 25 ???? 2019 ???? 5:48 ?.?., ?/? Cecile Madjar < >> cecile.madjar at mcin.ca> ??????: >> >>> Hi Sotirios, >>> >>> From what I understand, I believe you just need to update a few config >>> settings from the frontend in the Config module (under the Admin menu). >>> [image: Screen Shot 2019-07-25 at 10.31.35 AM.png] >>> >>> Then go to the Paths section of the config module and update the >>> "LORIS-MRI code" config setting to /data/loris/bin/mri (my guess is that >>> currently, it is set to /data/LORIS/data) and click on the submit button. >>> [image: Screen Shot 2019-07-25 at 10.32.50 AM.png] >>> >>> Then, go to Imaging Pipeline section of the Configuration module and >>> update the "Loris-MRI Data Directory" to /data/loris/data/ (my guess is >>> that it is empty at the moment in your database which is why $data_dir is >>> not defined when you run the tarchiveLoader step of the insertion pipeline). >>> [image: Screen Shot 2019-07-25 at 10.40.52 AM.png] >>> >>> Finally, I took a look at your environment file and it might be a good >>> idea to update your PERL5LIB variable to the following (it has been added >>> to the code a few releases ago as we realized this was needed for the >>> libraries. Not sure which version of LORIS-MRI you are using but it does >>> not hurt to update this variable to what I suggest below): >>> export >>> PERL5LIB=/data/loris/bin/mri/uploadNeuroDB:/data/loris/bin/mri/dicom-archive:$PERL5LIB >>> >>> Hope this helps. Sorry you are experiencing so much trouble setting >>> things up. We probably need to update our WIKI. >>> >>> Best, >>> >>> C?cile >>> >>> On Thu, Jul 25, 2019 at 9:48 AM Sotirios Nikoloutsopoulos < >>> sotirisnik at gmail.com> wrote: >>> >>>> The "Loris-MRI Data Directory" should be in the environment file? >>>> >>>> ???? ???, 25 ???? 2019 ???? 12:59 ?.?., ?/? Sotirios Nikoloutsopoulos < >>>> sotirisnik at gmail.com> ??????: >>>> >>>>> The scripts are located in /data/loris/bin/mri/. Somewhere I might >>>>> have messed something up but i remember following the instructions in the >>>>> wiki.For example when i tried to execute the batch_uploads_imageuploader >>>>> perl script the line >>>>> >>>>> my $command = "$bin_dir/uploadNeuroDB/imaging_upload_file.pl " >>>>> >>>>> was pointing to an invalid path, for example: it was repeating a >>>>> pattern and had LORIS instead of loris ( i think the mistake was that in >>>>> the wiki it suggested to use $PROJECT for the name of the folder and i >>>>> choosed 'loris' https://github.com/aces/Loris/wiki/Imaging-Database >>>>> ). See also the in image for the modification i did. Also i didn't find the >>>>> section to config the $data_dir variable. >>>>> >>>>> Thank you, >>>>> >>>>> Sotirios >>>>> >>>>> >>>>> >>>>> ???? ???, 24 ???? 2019 ???? 6:05 ?.?., ?/? Cecile Madjar < >>>>> cecile.madjar at mcin.ca> ??????: >>>>> >>>>>> Hi Sotirios, >>>>>> >>>>>> It looks like you have one Config setting not set (based on the first >>>>>> error message you got when running it the first time). You need to set a >>>>>> value for "Loris-MRI Data Directory" in the Config module under the Imaging >>>>>> pipeline section (from what I can see, it would be /data/loris/data/ for >>>>>> your setup). >>>>>> >>>>>> The reason you got that error message the second time is that the >>>>>> pipeline checks if the DICOM study was already inserted in the tarchive >>>>>> table. Since you ran the study once already, the study is inserted into the >>>>>> tarchive, hence the message. However, the pipeline did not complete as the >>>>>> MINC files were not created and inserted into the files/parameter_file >>>>>> table. In order to continue the insertion, you will then need to run the >>>>>> tarchiveLoader command suggested in the terminal during the second run: >>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile >>>>>> prod >>>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >>>>>> -verbose >>>>>> >>>>>> Finally, I am a bit confused by your setup. Where are the scripts >>>>>> located? In /data/loris/bin/mri or /data/loris/data? It feels like the >>>>>> Config setting "LORIS-MRI code" in the Paths section was set to >>>>>> "/data/LORIS/data" but it should have been set to the directory where the >>>>>> scripts are located (most probably /data/loris/bin/mri). >>>>>> >>>>>> Hope this helps, >>>>>> >>>>>> C?cile >>>>>> >>>>>> >>>>>> On Wed, Jul 24, 2019 at 10:55 AM Sotirios Nikoloutsopoulos < >>>>>> sotirisnik at gmail.com> wrote: >>>>>> >>>>>>> Ok first i execute this command: >>>>>>> >>>>>>> "lorisadmin at hbp:/data/loris/bin/mri$ ./batch_uploads_imageuploader >>>>>>> -profile prod < ~/Desktop/input.txt > log.txt >>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>>> at ./batch_uploads_imageuploader line 143. >>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>>> at ./batch_uploads_imageuploader line 143. >>>>>>> Use of uninitialized value $_ in pattern match (m//) at >>>>>>> ./batch_uploads_imageuploader line 147. >>>>>>> base is DCC0000_258024_V01 >>>>>>> path is /data/incoming/ >>>>>>> type is .tar.gz >>>>>>> fullpath is /data/incoming/DCC0000_258024_V01.tar.gz >>>>>>> >>>>>>> source /tmp/ImagingUpload-17-43-ZUEFjb >>>>>>> targetlocation /data/loris/data/tarchive >>>>>>> >>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256. >>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284. >>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>>> at /data/loris/bin/mri/uploadNeuroDB/tarchive_validation.pl line >>>>>>> 219. >>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>>> at /data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl line 270. >>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711. >>>>>>> Can't exec "mail": No such file or directory at >>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721. >>>>>>> print() on closed filehandle MAIL at >>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722. >>>>>>> print() on closed filehandle MAIL at >>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723. >>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>>>>>> print() on closed filehandle MAIL at >>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>>>>>> print() on closed filehandle MAIL at >>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725. >>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735. >>>>>>> >>>>>>> No Mincs inserted >>>>>>> >>>>>>> >>>>>>> The tarchiveLoader insertion script has failed. >>>>>>> Running now the following command: /data/loris/data/uploadNeuroDB/ >>>>>>> imaging_upload_file.pl -profile prod -upload_id 3 >>>>>>> /data/incoming/DCC0000_258024_V01.tar.gz -verbose" >>>>>>> >>>>>>> I inserted the VisitLabel, but still the Mincs are not inserted and >>>>>>> i am getting this error >>>>>>> >>>>>>> "lorisadmin at hbp:/data/loris/bin/mri$ /data/loris/data/uploadNeuroDB/ >>>>>>> imaging_upload_file.pl -profile prod -upload_id 3 >>>>>>> /data/incoming/DCC0000_258024_V01.tar.gz -verbose >>>>>>> >>>>>>> find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name '__MACOSX' >>>>>>> -delete >>>>>>> Spool message is: >>>>>>> The Scan for the uploadID 3 has already been run with tarchiveID: 7. >>>>>>> To continue with the rest of the insertion pipeline, please run >>>>>>> tarchiveLoader from a terminal as follows: >>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod >>>>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >>>>>>> -verbose" >>>>>>> >>>>>>> Which files should i delete? I thought that deleting the files from >>>>>>> the tarchive was enough. And i executed the last proposed command to >>>>>>> continue to rest of the insertion pipeline ( check the image ). >>>>>>> >>>>>>> Thank you >>>>>>> >>>>>>> ???? ???, 24 ???? 2019 ???? 5:05 ?.?., ?/? Cecile Madjar < >>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>> >>>>>>>> Hi Sotirios, >>>>>>>> >>>>>>>> Sorry I was not clear before. Every visits of candidates are >>>>>>>> created in the session table with one entry per CandID/VisitLabel. So if >>>>>>>> the imaging pipeline created visits in the session table (that are not >>>>>>>> attached to instruments) and you want to clean your database completely, >>>>>>>> you can delete entries in that table as well. >>>>>>>> >>>>>>>> Regarding your point 2, the imaging pipeline can create the visit >>>>>>>> in the session table as long as the visit label is present in the >>>>>>>> Visit_Windows table (which means the visit label stored in the PatientName >>>>>>>> is a valid visit label). >>>>>>>> However, in your prod file, if the >>>>>>>> variable $subjectID{'createVisitLabel'} is set to 1 for candidates, the >>>>>>>> visit label should be created in the Visit_Windows table I believe. Here >>>>>>>> is the line of the prod file you would need to change >>>>>>>> >>>>>>>> . >>>>>>>> In general, we prefer populating the Visit_Windows table and set >>>>>>>> the $subjectID{'createVisitLabel'} to 0 for candidate data to avoid >>>>>>>> insertion of badly labelled MRI data that are complex to relabel. >>>>>>>> >>>>>>>> Finally, regarding the candidate creation, if you execute the >>>>>>>> following steps, the pipeline should be able to create candidates: >>>>>>>> - set in the Imaging Pipeline section of the config module the >>>>>>>> config setting "Upload creation of candidates" to Yes >>>>>>>> - manually transfer the scans to the LORIS server instead of >>>>>>>> uploading it via the imaging browser >>>>>>>> - run batch_imaging_upload.pl as you did until now. >>>>>>>> >>>>>>>> Hope this helps! >>>>>>>> Best, >>>>>>>> >>>>>>>> C?cile >>>>>>>> >>>>>>>> On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos < >>>>>>>> sotirisnik at gmail.com> wrote: >>>>>>>> >>>>>>>>> About my second question i replaced the name of the patient in the >>>>>>>>> .dcm files and now i am getting an error that the visit label does not >>>>>>>>> exist. >>>>>>>>> I thought it was supposed to be automatically created. >>>>>>>>> >>>>>>>>> "Done adding archive info into database >>>>>>>>> >>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation >>>>>>>>> -profile prod >>>>>>>>> /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>>>>>> -verbose >>>>>>>>> md5sum >>>>>>>>> /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>>>>>> PSCID is: DCC0000 >>>>>>>>> CandID id: 258024 >>>>>>>>> visit_label is: V1 >>>>>>>>> PSCID is: DCC0000 >>>>>>>>> CandID id: 258024 >>>>>>>>> visit_label is: V1 >>>>>>>>> candidate id 258024 >>>>>>>>> >>>>>>>>> >>>>>>>>> => No Visit labelVisit label does not exist >>>>>>>>> Set centerID = 1 >>>>>>>>> PSCID is: DCC0000 >>>>>>>>> CandID id: 258024 >>>>>>>>> visit_label is: V1 >>>>>>>>> PSCID is: DCC0000 >>>>>>>>> CandID id: 258024 >>>>>>>>> visit_label is: V1 >>>>>>>>> >>>>>>>>> Number of MINC files that will be considered for inserting into >>>>>>>>> the database: 1 >>>>>>>>> >>>>>>>>> log dir is /logs and log file is /logs/TarLoad-15-42-L6pocU.log >>>>>>>>> PSCID is: DCC0000 >>>>>>>>> CandID id: 258024 >>>>>>>>> visit_label is: V1 >>>>>>>>> PSCID is: DCC0000 >>>>>>>>> CandID id: 258024 >>>>>>>>> visit_label is: V1 >>>>>>>>> candidate id 258024 >>>>>>>>> >>>>>>>>> >>>>>>>>> => No Visit label >>>>>>>>> Cleaning up temp files: rm -rf >>>>>>>>> /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*" >>>>>>>>> >>>>>>>>> ???? ???, 24 ???? 2019 ???? 3:38 ?.?., ?/? Sotirios >>>>>>>>> Nikoloutsopoulos ??????: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> 1) What are the records in the session table? >>>>>>>>>> >>>>>>>>>> 2) I tried to upload a non-phantom file like this >>>>>>>>>> >>>>>>>>>> input_file.txt: >>>>>>>>>> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1 >>>>>>>>>> >>>>>>>>>> Terminal: >>>>>>>>>> ./batch_uploads_imageuploader -profile prod < >>>>>>>>>> ~/Desktop/input.txt > log.txt >>>>>>>>>> >>>>>>>>>> and i get the following error in the log.txt file >>>>>>>>>> >>>>>>>>>> "Running now the following command: >>>>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile >>>>>>>>>> prod -upload_id 7 /data/incoming/DCC0000_258024_V1.tar.gz -verbose >>>>>>>>>> >>>>>>>>>> find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name '__MACOSX' >>>>>>>>>> -delete >>>>>>>>>> Spool message is: >>>>>>>>>> The PatientName read from the DICOM header does not start with >>>>>>>>>> DCC0000_258024_V1 from the mri_upload table >>>>>>>>>> " >>>>>>>>>> >>>>>>>>>> and the spool message is repeated for each .dcm file. Should i >>>>>>>>>> preprocess every .dcm file and change the PatientName to >>>>>>>>>> "DCC0000_258024_V1"? >>>>>>>>>> >>>>>>>>>> 3) As far as it concerns the candidate profile i have to create a >>>>>>>>>> new one each time before i try to upload a file? Is there already a way to >>>>>>>>>> create a candidate profile and get the DCCID and PSCID without the user >>>>>>>>>> interface? >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> >>>>>>>>>> Sotirios >>>>>>>>>> >>>>>>>>>> ???? ???, 18 ???? 2019 ???? 5:00 ?.?., ?/? Cecile Madjar < >>>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>>> >>>>>>>>>>> Hi Sotirios, >>>>>>>>>>> >>>>>>>>>>> See answers to your questions below. >>>>>>>>>>> >>>>>>>>>>> About the 1st case we had a dicom with a blank studyUID, maybe >>>>>>>>>>>> someone accidentally removed it. >>>>>>>>>>>> >>>>>>>>>>> That is strange. Something must have been done to the DICOM >>>>>>>>>>> files. I have never seen a study with no StudyUID out of the scanner. >>>>>>>>>>> Usually, you need to manually erase it with some tool. However, DICOMs >>>>>>>>>>> never cease to surprise me... >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Some other questions: >>>>>>>>>>>> 1) What are the steps to remove all of my patients and >>>>>>>>>>>> uploads?: because i want to try to reupload all the testing dicoms we had >>>>>>>>>>>> with a clean database. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Release 21.0 of LORIS-MRI will offer a script to remove all the >>>>>>>>>>> entries and files specific to an upload. However, this release is not yet >>>>>>>>>>> out. Hopefully in the next few weeks. >>>>>>>>>>> In the meantime, since your case is particular and that you just >>>>>>>>>>> want to start with a clean database, the following deletes should work >>>>>>>>>>> (hopefully I won't forget any tables): >>>>>>>>>>> delete from mri_violations_log; >>>>>>>>>>> delete from mri_protocol_violated_scans; >>>>>>>>>>> delete from MRICandidateErrors; >>>>>>>>>>> delete from parameter_file; >>>>>>>>>>> delete from files_qcstatus; # (if you played with the QC part of >>>>>>>>>>> the imaging browser for testing) >>>>>>>>>>> delete from feedback_mri_comments; # (if you played with the QC >>>>>>>>>>> part of the imaging browser for testing) >>>>>>>>>>> delete from tarchive_series; >>>>>>>>>>> delete from tarchive_files; >>>>>>>>>>> delete from mri_upload; >>>>>>>>>>> delete from tarchive; >>>>>>>>>>> delete from mri_scanner; >>>>>>>>>>> delete from candidate where PSCID="scanner"; >>>>>>>>>>> >>>>>>>>>>> I don't know if you want to also delete entries in the session >>>>>>>>>>> and candidate table too? >>>>>>>>>>> >>>>>>>>>>> 2) We do not need to create the profile of a patient ( are the >>>>>>>>>>>> patients stored in the candidate table? because i can't see the ghosts ), >>>>>>>>>>>> because will it automatically be created when importing a dicom? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I am not sure I understand fully this question. All candidates >>>>>>>>>>> are stored in the candidate table. The phantom scans however are attached >>>>>>>>>>> to a scanner candidate depending on where the scan happened. It is a bit of >>>>>>>>>>> a weird concept that we have to redesign eventually but never got a chance >>>>>>>>>>> to get to it. >>>>>>>>>>> >>>>>>>>>>> Hopefully this answered your question, otherwise, don't hesitate >>>>>>>>>>> to let me know. >>>>>>>>>>> >>>>>>>>>>> 3) What is the difference between uploading a dicom as phantom >>>>>>>>>>>> instead of importing as PSCCID_DCCID_VisitLabel? >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> When you upload the scan as PSCID_DCCID_VisitLabel, there is a >>>>>>>>>>> check that makes sure the candidate IDs and visit label are valid at the >>>>>>>>>>> time of upload. >>>>>>>>>>> When you upload a scan as phantom, it expects that the DICOM >>>>>>>>>>> field PatientName and uploaded filename contains the string "phantom". We >>>>>>>>>>> enforced this behaviour on the imaging uploader side in recent releases but >>>>>>>>>>> I can't remember which one. Probably the upcoming 21.0 release. >>>>>>>>>>> >>>>>>>>>>> All those verifications might seem cumbersome but they are here >>>>>>>>>>> to ensure that the files inserted are all valid and labelled properly as it >>>>>>>>>>> is a bit messy to have to delete files that were wrongly labelled. At >>>>>>>>>>> least, the delete script present in release 21.0 will make this process >>>>>>>>>>> easier but it is still good practice to verify those things. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Thank you, >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> With pleasure. Hopefully the answers to your questions will be >>>>>>>>>>> helpful. >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> >>>>>>>>>>> C?cile >>>>>>>>>>> >>>>>>>>>>> PS: loris-dev mailing list: you will find the initial answer I >>>>>>>>>>> gave Sotirios below. I forgot to cc the loris-dev in my earlier reply... >>>>>>>>>>> >>>>>>>>>>> ???? ???, 16 ???? 2019 ???? 12:02 ?.?., ?/? Cecile Madjar < >>>>>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>>>>> >>>>>>>>>>>>> Dear Sotirios, >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you for reaching out. Since your email was already >>>>>>>>>>>>> organized in points, I will reply directly below your questions below. >>>>>>>>>>>>> >>>>>>>>>>>>> Best, >>>>>>>>>>>>> >>>>>>>>>>>>> C?cile >>>>>>>>>>>>> >>>>>>>>>>>>> 1) What do we do when the StudyUID is blank on the header? >>>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md >>>>>>>>>>>>>> There solution provided at the table 3 does not guide us how to find the >>>>>>>>>>>>>> information ( is possible ). >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> The StudyUID is the field used in our pipeline at the moment >>>>>>>>>>>>> to check whether a DICOM study was already inserted into LORIS with that >>>>>>>>>>>>> same StudyUID since it is supposed to be unique for every single study. >>>>>>>>>>>>> Currently, we do not support insertion of DICOM studies if they do not have >>>>>>>>>>>>> a StudyUID associated with them. >>>>>>>>>>>>> >>>>>>>>>>>>> Is there a specific reason why the StudyUID is blank in the >>>>>>>>>>>>> DICOM headers in your DICOM files? Did they go through some processes >>>>>>>>>>>>> before upload to LORIS? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 2) When i am about to upload a non-phanton case i need a >>>>>>>>>>>>>> 'Visit Label'. Is there already a script that can create a new visit label >>>>>>>>>>>>>> for a specific Candidate, so i can provide it as input afterwards to the >>>>>>>>>>>>>> >>>>>>>>>>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md ) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> I am not sure if the front-end allows for it but if you use >>>>>>>>>>>>> batch_uploads_imageuploader, the insertion scripts could create the visit >>>>>>>>>>>>> label for you. However, it does need to be included in the patient name in >>>>>>>>>>>>> order for the script to know which label it should use. >>>>>>>>>>>>> For example: you upload V02 for MTL0123 (DCCID: 456789) but >>>>>>>>>>>>> V02 was not yet created for MTL0123; the insertion scripts will create a >>>>>>>>>>>>> V02 visit for MTL0123 if the PatientName field for the dataset is >>>>>>>>>>>>> MTL0123_456789_V02 with Stage marked as "Not Started". >>>>>>>>>>>>> >>>>>>>>>>>>> However, *note that for the script to create the visit for >>>>>>>>>>>>> the candidate, you have to make sure that all the visit label of your >>>>>>>>>>>>> projects were inserted in the Visit_Windows table* >>>>>>>>>>>>> (otherwise, the visit will never be created as they were not specified as >>>>>>>>>>>>> being part of the list of visit label to expect). So in the example above, >>>>>>>>>>>>> you should have one row in Visit_Windows with Visit_label="V02" set. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> 3) Section 4.3 >>>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md >>>>>>>>>>>>>> >>>>>>>>>>>>>> What do we do when the folder contains multiple mnc files? >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is a case by case. Most of the time, the MINC files that >>>>>>>>>>>>> failed insertion into the imaging browser end up in the MRI violation >>>>>>>>>>>>> module where you can see what went wrong with the acquisitions not >>>>>>>>>>>>> inserted. If you notice that the MINC file should be inserted as a specific >>>>>>>>>>>>> protocol, you can force the insertion as explained in section 4.3. >>>>>>>>>>>>> >>>>>>>>>>>>> In your case, I have a feeling that you want to be able to >>>>>>>>>>>>> insert MINC files that did not go through the whole pipeline insertion >>>>>>>>>>>>> (DICOM archival, then dcm2mnc, then protocol identification, then >>>>>>>>>>>>> insertion...). In theory, you could call minc_insertion.pl on >>>>>>>>>>>>> any MINC file, provided you have at least the following information: >>>>>>>>>>>>> - path to the MINC file >>>>>>>>>>>>> - uploadID associated with the MINC files or DICOM archive >>>>>>>>>>>>> path from which those MINC files were created from >>>>>>>>>>>>> >>>>>>>>>>>>> Then the minc_insertion.pl script will do the subject >>>>>>>>>>>>> information and protocol validation etc... >>>>>>>>>>>>> >>>>>>>>>>>>> If all the MINC files come from the same uploadID or >>>>>>>>>>>>> TarchiveID, then you could run a loop in bash calling the >>>>>>>>>>>>> minc_insertion.pl script like this for files deriving from >>>>>>>>>>>>> uploadID=1: >>>>>>>>>>>>> >>>>>>>>>>>>> ls /path/to/mnc/folder/* | while read f; do minc_insertion.pl >>>>>>>>>>>>> -profile prod -uploadID 1 -mincPath $f; done >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Hope this helped! >>>>>>>>>>>>> >>>>>>>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.31.35 AM.png Type: image/png Size: 45707 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.32.50 AM.png Type: image/png Size: 134887 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.40.52 AM.png Type: image/png Size: 95814 bytes Desc: not available URL: -------------- next part -------------- lorisadmin at hbp:/data/loris/bin/mri$ ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > log.txt Use of uninitialized value $_ in pattern match (m//) at ./batch_uploads_imageuploader line 147. base is DCC0000_258024_V01 path is /data/incoming/ type is .tar.gz fullpath is /data/incoming/DCC0000_258024_V01.tar.gz Can't exec "mail": No such file or directory at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 721. print() on closed filehandle MAIL at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 722. print() on closed filehandle MAIL at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 723. print() on closed filehandle MAIL at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 724. print() on closed filehandle MAIL at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 725. No Mincs inserted The tarchiveLoader insertion script has failed. Running now the following command: /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile prod -upload_id 8 /data/incoming/DCC0000_258024_V01.tar.gz -verbose so you came till herefileparse(): need a valid pathname at ./batch_uploads_imageuploader line 200. lorisadmin at hbp:/data/loris/bin/mri$ /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile prod -upload_id 8 /data/incoming/DCC0000_258024_V01.tar.gz -verbose find -path \/tmp\/ImagingUpload\-14\-58\-ikg2PS -name '__MACOSX' -delete Spool message is: The Scan for the uploadID 8 has already been run with tarchiveID: 6. To continue with the rest of the insertion pipeline, please run tarchiveLoader from a terminal as follows: /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader -globLocation -profile prod /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-14-57-3YMeQO.tar -verbose The candidate info validation has failed. lorisadmin at hbp:/data/loris/bin/mri$ /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader -globLocation -profile prod /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-14-57-3YMeQO.tar -verbose md5sum /data/loris/data/tarchive/2012/DCM_2012-12-05_ImagingUpload-14-57-3YMeQO.tar PSCID is: DCC0000 CandID id: 258024 visit_label is: V01 PSCID is: DCC0000 CandID id: 258024 visit_label is: V01 candidate id 258024 => No Visit labelVisit label does not exist PSCID is: DCC0000 CandID id: 258024 visit_label is: V01 PSCID is: DCC0000 CandID id: 258024 visit_label is: V01 Number of MINC files that will be considered for inserting into the database: 1 log dir is /data/loris/data//logs and log file is /data/loris/data//logs/TarLoad-14-58-9Xog7H.log PSCID is: DCC0000 CandID id: 258024 visit_label is: V01 PSCID is: DCC0000 CandID id: 258024 visit_label is: V01 candidate id 258024 => No Visit label Cleaning up temp files: rm -rf /tmp/TarLoad-14-58-5Z4hYX/ImagingUpload-14-57-3YMeQO* Can't exec "mail": No such file or directory at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 721. print() on closed filehandle MAIL at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 722. print() on closed filehandle MAIL at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 723. print() on closed filehandle MAIL at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 724. print() on closed filehandle MAIL at /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader line 725. No Mincs inserted lorisadmin at hbp:/data/loris/bin/mri$ cat log.txt Running now the following command: /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile prod -upload_id 8 /data/incoming/DCC0000_258024_V01.tar.gz -verbose find -path \/tmp\/ImagingUpload\-14\-57\-3YMeQO -name '__MACOSX' -delete /data/loris/bin/mri/dicom-archive/dicomTar.pl /tmp/ImagingUpload-14-57-3YMeQO /data/loris/data/tarchive/ -clobber -database -profile prod -verbose Source: /tmp/ImagingUpload-14-57-3YMeQO Target: /data/loris/data/tarchive Testing for database connectivity. Database is available. You will archive the dir : ImagingUpload-14-57-3YMeQO You are creating a tar with the following command: tar -cf /data/loris/data/tarchive/ImagingUpload-14-57-3YMeQO.tar ImagingUpload-14-57-3YMeQO getting md5sums and gzipping!! * Taken from dir : /tmp/ImagingUpload-14-57-3YMeQO * Archive target location : /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-14-57-3YMeQO.tar * Name of creating host : 127.0.1.1 * Name of host OS : Linux * Created by user : lorisadmin * Archived on : 2019-07-26 14:57:52 * dicomSummary version : 1 * dicomTar version : 1 * md5sum for DICOM tarball : 6d7ddce539a8e6557caed22994677be5 ImagingUpload-14-57-3YMeQO.tar * md5sum for DICOM tarball gzipped : fa0106eff0f8ce8043ad2286d312b593 ImagingUpload-14-57-3YMeQO.tar.gz * md5sum for complete archive : beb3b0aa9fa9fc7609d428733da3d7ff DCM_2012-12-05_ImagingUpload-14-57-3YMeQO.tar Adding archive info into database Removing temporary files from target location Done adding archive info into database /data/loris/bin/mri/uploadNeuroDB/tarchiveLoader -globLocation -profile prod /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-14-57-3YMeQO.tar -verbose md5sum /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-14-57-3YMeQO.tar PSCID is: DCC0000 CandID id: 258024 visit_label is: V01 PSCID is: DCC0000 CandID id: 258024 visit_label is: V01 candidate id 258024 => No Visit labelVisit label does not exist PSCID is: DCC0000 CandID id: 258024 visit_label is: V01 PSCID is: DCC0000 CandID id: 258024 visit_label is: V01 Number of MINC files that will be considered for inserting into the database: 1 log dir is /data/loris/data//logs and log file is /data/loris/data//logs/TarLoad-14-57-4g4PdT.log PSCID is: DCC0000 CandID id: 258024 visit_label is: V01 PSCID is: DCC0000 CandID id: 258024 visit_label is: V01 candidate id 258024 => No Visit label Cleaning up temp files: rm -rf /tmp/TarLoad-14-57-8lyAAU/ImagingUpload-14-57-3YMeQO* lorisadmin at hbp:/data/loris/bin/mri$ -------------- next part -------------- A non-text attachment was scrubbed... Name: paths.png Type: image/png Size: 44398 bytes Desc: not available URL: From sotirisnik at gmail.com Mon Jul 29 10:15:00 2019 From: sotirisnik at gmail.com (Sotirios Nikoloutsopoulos) Date: Mon, 29 Jul 2019 17:15:00 +0300 Subject: [Loris-dev] Import mri - scripts In-Reply-To: References: Message-ID: Hi, Could you help me with the "Visit label does not exist"? I had changed $subjectID{'createVisitLabel'} to 1 as you had suggested but it appears that this problem hasn't been resolved yet. lorisadmin at hbp:/data/loris/bin/mri$ cat /data/loris/data//logs/TarLoad-14-58-9Xog7H.log ==> Loading file from disk /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc --> mapping DICOM parameter for /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc --> using user-defined filterParameters for /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc Candidate Mismatch Error is Visit label does not exist -> WARNING: This candidate was invalid. Logging to MRICandidateErrors table with reason Visit label does not exist Thank you, Sotirios ???? ???, 26 ???? 2019 ???? 3:02 ?.?., ?/? Sotirios Nikoloutsopoulos < sotirisnik at gmail.com> ??????: > Hi Cecile, > > The Center error was fixed. Still the insertion of mincs fails. Please see > the output.txt file i attached. Also is there something else i need to > change at the paths configuration? > > Thank you, > > Sotirios > > ???? ???, 25 ???? 2019 ???? 6:55 ?.?., ?/? Cecile Madjar < > cecile.madjar at mcin.ca> ??????: > >> Hi Sotirios, >> >> We are making progress! >> >> I have a feeling this error comes from the fact that your psc table does >> not have the MRI_alias populated for the DCC site. Running the following >> query in the mysql database should fix this: >> UPDATE psc SET MRI_alias="DCC", Alias="DCC" WHERE Name="Data Coordinating >> Center"; >> Repeat the same with all your sites (make sure to change the Aliases >> between sites, they should all be unique). >> >> Let me know how it goes! >> Best, >> >> C?cile >> >> >> >> >> >> On Thu, Jul 25, 2019 at 11:43 AM Sotirios Nikoloutsopoulos < >> sotirisnik at gmail.com> wrote: >> >>> Dear Cecile, >>> You were right about your guesses. I did the modifications you >>> suggested, i still get the "no mincs inserted" error, now i am getting the >>> "No center found this candidate" error ( how do i fix that? from what i see >>> the .dcm files contains a 'Institution Name', is that the center? ), so i >>> guess that's why the validation fails, but i changed the force variable of >>> tarchiveLoader to 1, shouldn't it have inserted the mincs then? Also it is >>> optional to have a mail server, right? ( from what i see the mail error >>> does not affect the $valid_study, so i assume we are okay with that ) >>> Thank you a lot, >>> Sotirios >>> >>> ???? ???, 25 ???? 2019 ???? 5:48 ?.?., ?/? Cecile Madjar < >>> cecile.madjar at mcin.ca> ??????: >>> >>>> Hi Sotirios, >>>> >>>> From what I understand, I believe you just need to update a few config >>>> settings from the frontend in the Config module (under the Admin menu). >>>> [image: Screen Shot 2019-07-25 at 10.31.35 AM.png] >>>> >>>> Then go to the Paths section of the config module and update the >>>> "LORIS-MRI code" config setting to /data/loris/bin/mri (my guess is that >>>> currently, it is set to /data/LORIS/data) and click on the submit button. >>>> [image: Screen Shot 2019-07-25 at 10.32.50 AM.png] >>>> >>>> Then, go to Imaging Pipeline section of the Configuration module and >>>> update the "Loris-MRI Data Directory" to /data/loris/data/ (my guess is >>>> that it is empty at the moment in your database which is why $data_dir is >>>> not defined when you run the tarchiveLoader step of the insertion pipeline). >>>> [image: Screen Shot 2019-07-25 at 10.40.52 AM.png] >>>> >>>> Finally, I took a look at your environment file and it might be a good >>>> idea to update your PERL5LIB variable to the following (it has been added >>>> to the code a few releases ago as we realized this was needed for the >>>> libraries. Not sure which version of LORIS-MRI you are using but it does >>>> not hurt to update this variable to what I suggest below): >>>> export >>>> PERL5LIB=/data/loris/bin/mri/uploadNeuroDB:/data/loris/bin/mri/dicom-archive:$PERL5LIB >>>> >>>> Hope this helps. Sorry you are experiencing so much trouble setting >>>> things up. We probably need to update our WIKI. >>>> >>>> Best, >>>> >>>> C?cile >>>> >>>> On Thu, Jul 25, 2019 at 9:48 AM Sotirios Nikoloutsopoulos < >>>> sotirisnik at gmail.com> wrote: >>>> >>>>> The "Loris-MRI Data Directory" should be in the environment file? >>>>> >>>>> ???? ???, 25 ???? 2019 ???? 12:59 ?.?., ?/? Sotirios Nikoloutsopoulos < >>>>> sotirisnik at gmail.com> ??????: >>>>> >>>>>> The scripts are located in /data/loris/bin/mri/. Somewhere I might >>>>>> have messed something up but i remember following the instructions in the >>>>>> wiki.For example when i tried to execute the batch_uploads_imageuploader >>>>>> perl script the line >>>>>> >>>>>> my $command = "$bin_dir/uploadNeuroDB/imaging_upload_file.pl " >>>>>> >>>>>> was pointing to an invalid path, for example: it was repeating a >>>>>> pattern and had LORIS instead of loris ( i think the mistake was that in >>>>>> the wiki it suggested to use $PROJECT for the name of the folder and i >>>>>> choosed 'loris' https://github.com/aces/Loris/wiki/Imaging-Database >>>>>> ). See also the in image for the modification i did. Also i didn't find the >>>>>> section to config the $data_dir variable. >>>>>> >>>>>> Thank you, >>>>>> >>>>>> Sotirios >>>>>> >>>>>> >>>>>> >>>>>> ???? ???, 24 ???? 2019 ???? 6:05 ?.?., ?/? Cecile Madjar < >>>>>> cecile.madjar at mcin.ca> ??????: >>>>>> >>>>>>> Hi Sotirios, >>>>>>> >>>>>>> It looks like you have one Config setting not set (based on the >>>>>>> first error message you got when running it the first time). You need to >>>>>>> set a value for "Loris-MRI Data Directory" in the Config module under the >>>>>>> Imaging pipeline section (from what I can see, it would be >>>>>>> /data/loris/data/ for your setup). >>>>>>> >>>>>>> The reason you got that error message the second time is that the >>>>>>> pipeline checks if the DICOM study was already inserted in the tarchive >>>>>>> table. Since you ran the study once already, the study is inserted into the >>>>>>> tarchive, hence the message. However, the pipeline did not complete as the >>>>>>> MINC files were not created and inserted into the files/parameter_file >>>>>>> table. In order to continue the insertion, you will then need to run the >>>>>>> tarchiveLoader command suggested in the terminal during the second run: >>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation >>>>>>> -profile prod >>>>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >>>>>>> -verbose >>>>>>> >>>>>>> Finally, I am a bit confused by your setup. Where are the scripts >>>>>>> located? In /data/loris/bin/mri or /data/loris/data? It feels like the >>>>>>> Config setting "LORIS-MRI code" in the Paths section was set to >>>>>>> "/data/LORIS/data" but it should have been set to the directory where the >>>>>>> scripts are located (most probably /data/loris/bin/mri). >>>>>>> >>>>>>> Hope this helps, >>>>>>> >>>>>>> C?cile >>>>>>> >>>>>>> >>>>>>> On Wed, Jul 24, 2019 at 10:55 AM Sotirios Nikoloutsopoulos < >>>>>>> sotirisnik at gmail.com> wrote: >>>>>>> >>>>>>>> Ok first i execute this command: >>>>>>>> >>>>>>>> "lorisadmin at hbp:/data/loris/bin/mri$ >>>>>>>> ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > log.txt >>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>>>> at ./batch_uploads_imageuploader line 143. >>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>>>> at ./batch_uploads_imageuploader line 143. >>>>>>>> Use of uninitialized value $_ in pattern match (m//) at >>>>>>>> ./batch_uploads_imageuploader line 147. >>>>>>>> base is DCC0000_258024_V01 >>>>>>>> path is /data/incoming/ >>>>>>>> type is .tar.gz >>>>>>>> fullpath is /data/incoming/DCC0000_258024_V01.tar.gz >>>>>>>> >>>>>>>> source /tmp/ImagingUpload-17-43-ZUEFjb >>>>>>>> targetlocation /data/loris/data/tarchive >>>>>>>> >>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256. >>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284. >>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>>>> at /data/loris/bin/mri/uploadNeuroDB/tarchive_validation.pl line >>>>>>>> 219. >>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>>>> at /data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl line 270. >>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711. >>>>>>>> Can't exec "mail": No such file or directory at >>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721. >>>>>>>> print() on closed filehandle MAIL at >>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722. >>>>>>>> print() on closed filehandle MAIL at >>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723. >>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>>>>>>> print() on closed filehandle MAIL at >>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>>>>>>> print() on closed filehandle MAIL at >>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725. >>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or string >>>>>>>> at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735. >>>>>>>> >>>>>>>> No Mincs inserted >>>>>>>> >>>>>>>> >>>>>>>> The tarchiveLoader insertion script has failed. >>>>>>>> Running now the following command: /data/loris/data/uploadNeuroDB/ >>>>>>>> imaging_upload_file.pl -profile prod -upload_id 3 >>>>>>>> /data/incoming/DCC0000_258024_V01.tar.gz -verbose" >>>>>>>> >>>>>>>> I inserted the VisitLabel, but still the Mincs are not inserted and >>>>>>>> i am getting this error >>>>>>>> >>>>>>>> "lorisadmin at hbp:/data/loris/bin/mri$ >>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile >>>>>>>> prod -upload_id 3 /data/incoming/DCC0000_258024_V01.tar.gz -verbose >>>>>>>> >>>>>>>> find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name '__MACOSX' >>>>>>>> -delete >>>>>>>> Spool message is: >>>>>>>> The Scan for the uploadID 3 has already been run with tarchiveID: >>>>>>>> 7. >>>>>>>> To continue with the rest of the insertion pipeline, please run >>>>>>>> tarchiveLoader from a terminal as follows: >>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod >>>>>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >>>>>>>> -verbose" >>>>>>>> >>>>>>>> Which files should i delete? I thought that deleting the files from >>>>>>>> the tarchive was enough. And i executed the last proposed command to >>>>>>>> continue to rest of the insertion pipeline ( check the image ). >>>>>>>> >>>>>>>> Thank you >>>>>>>> >>>>>>>> ???? ???, 24 ???? 2019 ???? 5:05 ?.?., ?/? Cecile Madjar < >>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>> >>>>>>>>> Hi Sotirios, >>>>>>>>> >>>>>>>>> Sorry I was not clear before. Every visits of candidates are >>>>>>>>> created in the session table with one entry per CandID/VisitLabel. So if >>>>>>>>> the imaging pipeline created visits in the session table (that are not >>>>>>>>> attached to instruments) and you want to clean your database completely, >>>>>>>>> you can delete entries in that table as well. >>>>>>>>> >>>>>>>>> Regarding your point 2, the imaging pipeline can create the visit >>>>>>>>> in the session table as long as the visit label is present in the >>>>>>>>> Visit_Windows table (which means the visit label stored in the PatientName >>>>>>>>> is a valid visit label). >>>>>>>>> However, in your prod file, if the >>>>>>>>> variable $subjectID{'createVisitLabel'} is set to 1 for candidates, the >>>>>>>>> visit label should be created in the Visit_Windows table I believe. Here >>>>>>>>> is the line of the prod file you would need to change >>>>>>>>> >>>>>>>>> . >>>>>>>>> In general, we prefer populating the Visit_Windows table and set >>>>>>>>> the $subjectID{'createVisitLabel'} to 0 for candidate data to avoid >>>>>>>>> insertion of badly labelled MRI data that are complex to relabel. >>>>>>>>> >>>>>>>>> Finally, regarding the candidate creation, if you execute the >>>>>>>>> following steps, the pipeline should be able to create candidates: >>>>>>>>> - set in the Imaging Pipeline section of the config module the >>>>>>>>> config setting "Upload creation of candidates" to Yes >>>>>>>>> - manually transfer the scans to the LORIS server instead of >>>>>>>>> uploading it via the imaging browser >>>>>>>>> - run batch_imaging_upload.pl as you did until now. >>>>>>>>> >>>>>>>>> Hope this helps! >>>>>>>>> Best, >>>>>>>>> >>>>>>>>> C?cile >>>>>>>>> >>>>>>>>> On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos < >>>>>>>>> sotirisnik at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> About my second question i replaced the name of the patient in >>>>>>>>>> the .dcm files and now i am getting an error that the visit label does not >>>>>>>>>> exist. >>>>>>>>>> I thought it was supposed to be automatically created. >>>>>>>>>> >>>>>>>>>> "Done adding archive info into database >>>>>>>>>> >>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation >>>>>>>>>> -profile prod >>>>>>>>>> /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>>>>>>> -verbose >>>>>>>>>> md5sum >>>>>>>>>> /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>> CandID id: 258024 >>>>>>>>>> visit_label is: V1 >>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>> CandID id: 258024 >>>>>>>>>> visit_label is: V1 >>>>>>>>>> candidate id 258024 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> => No Visit labelVisit label does not exist >>>>>>>>>> Set centerID = 1 >>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>> CandID id: 258024 >>>>>>>>>> visit_label is: V1 >>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>> CandID id: 258024 >>>>>>>>>> visit_label is: V1 >>>>>>>>>> >>>>>>>>>> Number of MINC files that will be considered for inserting into >>>>>>>>>> the database: 1 >>>>>>>>>> >>>>>>>>>> log dir is /logs and log file is /logs/TarLoad-15-42-L6pocU.log >>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>> CandID id: 258024 >>>>>>>>>> visit_label is: V1 >>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>> CandID id: 258024 >>>>>>>>>> visit_label is: V1 >>>>>>>>>> candidate id 258024 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> => No Visit label >>>>>>>>>> Cleaning up temp files: rm -rf >>>>>>>>>> /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*" >>>>>>>>>> >>>>>>>>>> ???? ???, 24 ???? 2019 ???? 3:38 ?.?., ?/? Sotirios >>>>>>>>>> Nikoloutsopoulos ??????: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> 1) What are the records in the session table? >>>>>>>>>>> >>>>>>>>>>> 2) I tried to upload a non-phantom file like this >>>>>>>>>>> >>>>>>>>>>> input_file.txt: >>>>>>>>>>> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1 >>>>>>>>>>> >>>>>>>>>>> Terminal: >>>>>>>>>>> ./batch_uploads_imageuploader -profile prod < >>>>>>>>>>> ~/Desktop/input.txt > log.txt >>>>>>>>>>> >>>>>>>>>>> and i get the following error in the log.txt file >>>>>>>>>>> >>>>>>>>>>> "Running now the following command: >>>>>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile >>>>>>>>>>> prod -upload_id 7 /data/incoming/DCC0000_258024_V1.tar.gz -verbose >>>>>>>>>>> >>>>>>>>>>> find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name >>>>>>>>>>> '__MACOSX' -delete >>>>>>>>>>> Spool message is: >>>>>>>>>>> The PatientName read from the DICOM header does not start with >>>>>>>>>>> DCC0000_258024_V1 from the mri_upload table >>>>>>>>>>> " >>>>>>>>>>> >>>>>>>>>>> and the spool message is repeated for each .dcm file. Should i >>>>>>>>>>> preprocess every .dcm file and change the PatientName to >>>>>>>>>>> "DCC0000_258024_V1"? >>>>>>>>>>> >>>>>>>>>>> 3) As far as it concerns the candidate profile i have to create >>>>>>>>>>> a new one each time before i try to upload a file? Is there already a way >>>>>>>>>>> to create a candidate profile and get the DCCID and PSCID without the user >>>>>>>>>>> interface? >>>>>>>>>>> >>>>>>>>>>> Thank you, >>>>>>>>>>> >>>>>>>>>>> Sotirios >>>>>>>>>>> >>>>>>>>>>> ???? ???, 18 ???? 2019 ???? 5:00 ?.?., ?/? Cecile Madjar < >>>>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>>>> >>>>>>>>>>>> Hi Sotirios, >>>>>>>>>>>> >>>>>>>>>>>> See answers to your questions below. >>>>>>>>>>>> >>>>>>>>>>>> About the 1st case we had a dicom with a blank studyUID, maybe >>>>>>>>>>>>> someone accidentally removed it. >>>>>>>>>>>>> >>>>>>>>>>>> That is strange. Something must have been done to the DICOM >>>>>>>>>>>> files. I have never seen a study with no StudyUID out of the scanner. >>>>>>>>>>>> Usually, you need to manually erase it with some tool. However, DICOMs >>>>>>>>>>>> never cease to surprise me... >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Some other questions: >>>>>>>>>>>>> 1) What are the steps to remove all of my patients and >>>>>>>>>>>>> uploads?: because i want to try to reupload all the testing dicoms we had >>>>>>>>>>>>> with a clean database. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Release 21.0 of LORIS-MRI will offer a script to remove all the >>>>>>>>>>>> entries and files specific to an upload. However, this release is not yet >>>>>>>>>>>> out. Hopefully in the next few weeks. >>>>>>>>>>>> In the meantime, since your case is particular and that you >>>>>>>>>>>> just want to start with a clean database, the following deletes should work >>>>>>>>>>>> (hopefully I won't forget any tables): >>>>>>>>>>>> delete from mri_violations_log; >>>>>>>>>>>> delete from mri_protocol_violated_scans; >>>>>>>>>>>> delete from MRICandidateErrors; >>>>>>>>>>>> delete from parameter_file; >>>>>>>>>>>> delete from files_qcstatus; # (if you played with the QC part >>>>>>>>>>>> of the imaging browser for testing) >>>>>>>>>>>> delete from feedback_mri_comments; # (if you played with the QC >>>>>>>>>>>> part of the imaging browser for testing) >>>>>>>>>>>> delete from tarchive_series; >>>>>>>>>>>> delete from tarchive_files; >>>>>>>>>>>> delete from mri_upload; >>>>>>>>>>>> delete from tarchive; >>>>>>>>>>>> delete from mri_scanner; >>>>>>>>>>>> delete from candidate where PSCID="scanner"; >>>>>>>>>>>> >>>>>>>>>>>> I don't know if you want to also delete entries in the session >>>>>>>>>>>> and candidate table too? >>>>>>>>>>>> >>>>>>>>>>>> 2) We do not need to create the profile of a patient ( are the >>>>>>>>>>>>> patients stored in the candidate table? because i can't see the ghosts ), >>>>>>>>>>>>> because will it automatically be created when importing a dicom? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I am not sure I understand fully this question. All candidates >>>>>>>>>>>> are stored in the candidate table. The phantom scans however are attached >>>>>>>>>>>> to a scanner candidate depending on where the scan happened. It is a bit of >>>>>>>>>>>> a weird concept that we have to redesign eventually but never got a chance >>>>>>>>>>>> to get to it. >>>>>>>>>>>> >>>>>>>>>>>> Hopefully this answered your question, otherwise, don't >>>>>>>>>>>> hesitate to let me know. >>>>>>>>>>>> >>>>>>>>>>>> 3) What is the difference between uploading a dicom as phantom >>>>>>>>>>>>> instead of importing as PSCCID_DCCID_VisitLabel? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> When you upload the scan as PSCID_DCCID_VisitLabel, there is a >>>>>>>>>>>> check that makes sure the candidate IDs and visit label are valid at the >>>>>>>>>>>> time of upload. >>>>>>>>>>>> When you upload a scan as phantom, it expects that the DICOM >>>>>>>>>>>> field PatientName and uploaded filename contains the string "phantom". We >>>>>>>>>>>> enforced this behaviour on the imaging uploader side in recent releases but >>>>>>>>>>>> I can't remember which one. Probably the upcoming 21.0 release. >>>>>>>>>>>> >>>>>>>>>>>> All those verifications might seem cumbersome but they are here >>>>>>>>>>>> to ensure that the files inserted are all valid and labelled properly as it >>>>>>>>>>>> is a bit messy to have to delete files that were wrongly labelled. At >>>>>>>>>>>> least, the delete script present in release 21.0 will make this process >>>>>>>>>>>> easier but it is still good practice to verify those things. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Thank you, >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> With pleasure. Hopefully the answers to your questions will be >>>>>>>>>>>> helpful. >>>>>>>>>>>> >>>>>>>>>>>> Best, >>>>>>>>>>>> >>>>>>>>>>>> C?cile >>>>>>>>>>>> >>>>>>>>>>>> PS: loris-dev mailing list: you will find the initial answer I >>>>>>>>>>>> gave Sotirios below. I forgot to cc the loris-dev in my earlier reply... >>>>>>>>>>>> >>>>>>>>>>>> ???? ???, 16 ???? 2019 ???? 12:02 ?.?., ?/? Cecile Madjar < >>>>>>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>>>>>> >>>>>>>>>>>>>> Dear Sotirios, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you for reaching out. Since your email was already >>>>>>>>>>>>>> organized in points, I will reply directly below your questions below. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best, >>>>>>>>>>>>>> >>>>>>>>>>>>>> C?cile >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1) What do we do when the StudyUID is blank on the header? >>>>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md >>>>>>>>>>>>>>> There solution provided at the table 3 does not guide us how to find the >>>>>>>>>>>>>>> information ( is possible ). >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> The StudyUID is the field used in our pipeline at the moment >>>>>>>>>>>>>> to check whether a DICOM study was already inserted into LORIS with that >>>>>>>>>>>>>> same StudyUID since it is supposed to be unique for every single study. >>>>>>>>>>>>>> Currently, we do not support insertion of DICOM studies if they do not have >>>>>>>>>>>>>> a StudyUID associated with them. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Is there a specific reason why the StudyUID is blank in the >>>>>>>>>>>>>> DICOM headers in your DICOM files? Did they go through some processes >>>>>>>>>>>>>> before upload to LORIS? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2) When i am about to upload a non-phanton case i need a >>>>>>>>>>>>>>> 'Visit Label'. Is there already a script that can create a new visit label >>>>>>>>>>>>>>> for a specific Candidate, so i can provide it as input afterwards to the >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md ) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> I am not sure if the front-end allows for it but if you use >>>>>>>>>>>>>> batch_uploads_imageuploader, the insertion scripts could create the visit >>>>>>>>>>>>>> label for you. However, it does need to be included in the patient name in >>>>>>>>>>>>>> order for the script to know which label it should use. >>>>>>>>>>>>>> For example: you upload V02 for MTL0123 (DCCID: 456789) but >>>>>>>>>>>>>> V02 was not yet created for MTL0123; the insertion scripts will create a >>>>>>>>>>>>>> V02 visit for MTL0123 if the PatientName field for the dataset is >>>>>>>>>>>>>> MTL0123_456789_V02 with Stage marked as "Not Started". >>>>>>>>>>>>>> >>>>>>>>>>>>>> However, *note that for the script to create the visit for >>>>>>>>>>>>>> the candidate, you have to make sure that all the visit label of your >>>>>>>>>>>>>> projects were inserted in the Visit_Windows table* >>>>>>>>>>>>>> (otherwise, the visit will never be created as they were not specified as >>>>>>>>>>>>>> being part of the list of visit label to expect). So in the example above, >>>>>>>>>>>>>> you should have one row in Visit_Windows with Visit_label="V02" set. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> 3) Section 4.3 >>>>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> What do we do when the folder contains multiple mnc files? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This is a case by case. Most of the time, the MINC files >>>>>>>>>>>>>> that failed insertion into the imaging browser end up in the MRI violation >>>>>>>>>>>>>> module where you can see what went wrong with the acquisitions not >>>>>>>>>>>>>> inserted. If you notice that the MINC file should be inserted as a specific >>>>>>>>>>>>>> protocol, you can force the insertion as explained in section 4.3. >>>>>>>>>>>>>> >>>>>>>>>>>>>> In your case, I have a feeling that you want to be able to >>>>>>>>>>>>>> insert MINC files that did not go through the whole pipeline insertion >>>>>>>>>>>>>> (DICOM archival, then dcm2mnc, then protocol identification, then >>>>>>>>>>>>>> insertion...). In theory, you could call minc_insertion.pl >>>>>>>>>>>>>> on any MINC file, provided you have at least the following information: >>>>>>>>>>>>>> - path to the MINC file >>>>>>>>>>>>>> - uploadID associated with the MINC files or DICOM archive >>>>>>>>>>>>>> path from which those MINC files were created from >>>>>>>>>>>>>> >>>>>>>>>>>>>> Then the minc_insertion.pl script will do the subject >>>>>>>>>>>>>> information and protocol validation etc... >>>>>>>>>>>>>> >>>>>>>>>>>>>> If all the MINC files come from the same uploadID or >>>>>>>>>>>>>> TarchiveID, then you could run a loop in bash calling the >>>>>>>>>>>>>> minc_insertion.pl script like this for files deriving from >>>>>>>>>>>>>> uploadID=1: >>>>>>>>>>>>>> >>>>>>>>>>>>>> ls /path/to/mnc/folder/* | while read f; do minc_insertion.pl >>>>>>>>>>>>>> -profile prod -uploadID 1 -mincPath $f; done >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hope this helped! >>>>>>>>>>>>>> >>>>>>>>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.31.35 AM.png Type: image/png Size: 45707 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.32.50 AM.png Type: image/png Size: 134887 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.40.52 AM.png Type: image/png Size: 95814 bytes Desc: not available URL: From cecile.madjar at mcin.ca Mon Jul 29 10:21:34 2019 From: cecile.madjar at mcin.ca (Cecile Madjar) Date: Mon, 29 Jul 2019 10:21:34 -0400 Subject: [Loris-dev] Import mri - scripts In-Reply-To: References: Message-ID: Hi Sotirios, Can you try adding the visit label v01 to the Visit_Windows table? However, I noticed that in a previous dataset, the label was V1. Does that refer to the same visit? If so, you might want to harmonize the visit labels so they are all the same across datasets for a given timepoint, meaning you will need to relabel scans when the visit label is different but referring to the same visit label (v01 = V1 etc...). Hope this helps, C?cile On Mon, Jul 29, 2019 at 10:15 AM Sotirios Nikoloutsopoulos < sotirisnik at gmail.com> wrote: > Hi, > > Could you help me with the "Visit label does not exist"? I had changed > $subjectID{'createVisitLabel'} to 1 as you had suggested but it appears > that this problem hasn't been resolved yet. > > lorisadmin at hbp:/data/loris/bin/mri$ cat > /data/loris/data//logs/TarLoad-14-58-9Xog7H.log > > ==> Loading file from disk > /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc > > --> mapping DICOM parameter for > /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc > > --> using user-defined filterParameters for > /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc > > Candidate Mismatch Error is Visit label does not exist > -> WARNING: This candidate was invalid. Logging to > MRICandidateErrors table with reason Visit label does not > exist > > Thank you, > > Sotirios > > ???? ???, 26 ???? 2019 ???? 3:02 ?.?., ?/? Sotirios Nikoloutsopoulos < > sotirisnik at gmail.com> ??????: > >> Hi Cecile, >> >> The Center error was fixed. Still the insertion of mincs fails. Please >> see the output.txt file i attached. Also is there something else i need to >> change at the paths configuration? >> >> Thank you, >> >> Sotirios >> >> ???? ???, 25 ???? 2019 ???? 6:55 ?.?., ?/? Cecile Madjar < >> cecile.madjar at mcin.ca> ??????: >> >>> Hi Sotirios, >>> >>> We are making progress! >>> >>> I have a feeling this error comes from the fact that your psc table does >>> not have the MRI_alias populated for the DCC site. Running the following >>> query in the mysql database should fix this: >>> UPDATE psc SET MRI_alias="DCC", Alias="DCC" WHERE Name="Data >>> Coordinating Center"; >>> Repeat the same with all your sites (make sure to change the Aliases >>> between sites, they should all be unique). >>> >>> Let me know how it goes! >>> Best, >>> >>> C?cile >>> >>> >>> >>> >>> >>> On Thu, Jul 25, 2019 at 11:43 AM Sotirios Nikoloutsopoulos < >>> sotirisnik at gmail.com> wrote: >>> >>>> Dear Cecile, >>>> You were right about your guesses. I did the modifications you >>>> suggested, i still get the "no mincs inserted" error, now i am getting the >>>> "No center found this candidate" error ( how do i fix that? from what i see >>>> the .dcm files contains a 'Institution Name', is that the center? ), so i >>>> guess that's why the validation fails, but i changed the force variable of >>>> tarchiveLoader to 1, shouldn't it have inserted the mincs then? Also it is >>>> optional to have a mail server, right? ( from what i see the mail error >>>> does not affect the $valid_study, so i assume we are okay with that ) >>>> Thank you a lot, >>>> Sotirios >>>> >>>> ???? ???, 25 ???? 2019 ???? 5:48 ?.?., ?/? Cecile Madjar < >>>> cecile.madjar at mcin.ca> ??????: >>>> >>>>> Hi Sotirios, >>>>> >>>>> From what I understand, I believe you just need to update a few config >>>>> settings from the frontend in the Config module (under the Admin menu). >>>>> [image: Screen Shot 2019-07-25 at 10.31.35 AM.png] >>>>> >>>>> Then go to the Paths section of the config module and update the >>>>> "LORIS-MRI code" config setting to /data/loris/bin/mri (my guess is that >>>>> currently, it is set to /data/LORIS/data) and click on the submit button. >>>>> [image: Screen Shot 2019-07-25 at 10.32.50 AM.png] >>>>> >>>>> Then, go to Imaging Pipeline section of the Configuration module and >>>>> update the "Loris-MRI Data Directory" to /data/loris/data/ (my guess is >>>>> that it is empty at the moment in your database which is why $data_dir is >>>>> not defined when you run the tarchiveLoader step of the insertion pipeline). >>>>> [image: Screen Shot 2019-07-25 at 10.40.52 AM.png] >>>>> >>>>> Finally, I took a look at your environment file and it might be a good >>>>> idea to update your PERL5LIB variable to the following (it has been added >>>>> to the code a few releases ago as we realized this was needed for the >>>>> libraries. Not sure which version of LORIS-MRI you are using but it does >>>>> not hurt to update this variable to what I suggest below): >>>>> export >>>>> PERL5LIB=/data/loris/bin/mri/uploadNeuroDB:/data/loris/bin/mri/dicom-archive:$PERL5LIB >>>>> >>>>> Hope this helps. Sorry you are experiencing so much trouble setting >>>>> things up. We probably need to update our WIKI. >>>>> >>>>> Best, >>>>> >>>>> C?cile >>>>> >>>>> On Thu, Jul 25, 2019 at 9:48 AM Sotirios Nikoloutsopoulos < >>>>> sotirisnik at gmail.com> wrote: >>>>> >>>>>> The "Loris-MRI Data Directory" should be in the environment file? >>>>>> >>>>>> ???? ???, 25 ???? 2019 ???? 12:59 ?.?., ?/? Sotirios Nikoloutsopoulos >>>>>> ??????: >>>>>> >>>>>>> The scripts are located in /data/loris/bin/mri/. Somewhere I might >>>>>>> have messed something up but i remember following the instructions in the >>>>>>> wiki.For example when i tried to execute the batch_uploads_imageuploader >>>>>>> perl script the line >>>>>>> >>>>>>> my $command = "$bin_dir/uploadNeuroDB/imaging_upload_file.pl " >>>>>>> >>>>>>> was pointing to an invalid path, for example: it was repeating a >>>>>>> pattern and had LORIS instead of loris ( i think the mistake was that in >>>>>>> the wiki it suggested to use $PROJECT for the name of the folder and i >>>>>>> choosed 'loris' https://github.com/aces/Loris/wiki/Imaging-Database >>>>>>> ). See also the in image for the modification i did. Also i didn't find the >>>>>>> section to config the $data_dir variable. >>>>>>> >>>>>>> Thank you, >>>>>>> >>>>>>> Sotirios >>>>>>> >>>>>>> >>>>>>> >>>>>>> ???? ???, 24 ???? 2019 ???? 6:05 ?.?., ?/? Cecile Madjar < >>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>> >>>>>>>> Hi Sotirios, >>>>>>>> >>>>>>>> It looks like you have one Config setting not set (based on the >>>>>>>> first error message you got when running it the first time). You need to >>>>>>>> set a value for "Loris-MRI Data Directory" in the Config module under the >>>>>>>> Imaging pipeline section (from what I can see, it would be >>>>>>>> /data/loris/data/ for your setup). >>>>>>>> >>>>>>>> The reason you got that error message the second time is that the >>>>>>>> pipeline checks if the DICOM study was already inserted in the tarchive >>>>>>>> table. Since you ran the study once already, the study is inserted into the >>>>>>>> tarchive, hence the message. However, the pipeline did not complete as the >>>>>>>> MINC files were not created and inserted into the files/parameter_file >>>>>>>> table. In order to continue the insertion, you will then need to run the >>>>>>>> tarchiveLoader command suggested in the terminal during the second run: >>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation >>>>>>>> -profile prod >>>>>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >>>>>>>> -verbose >>>>>>>> >>>>>>>> Finally, I am a bit confused by your setup. Where are the scripts >>>>>>>> located? In /data/loris/bin/mri or /data/loris/data? It feels like the >>>>>>>> Config setting "LORIS-MRI code" in the Paths section was set to >>>>>>>> "/data/LORIS/data" but it should have been set to the directory where the >>>>>>>> scripts are located (most probably /data/loris/bin/mri). >>>>>>>> >>>>>>>> Hope this helps, >>>>>>>> >>>>>>>> C?cile >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jul 24, 2019 at 10:55 AM Sotirios Nikoloutsopoulos < >>>>>>>> sotirisnik at gmail.com> wrote: >>>>>>>> >>>>>>>>> Ok first i execute this command: >>>>>>>>> >>>>>>>>> "lorisadmin at hbp:/data/loris/bin/mri$ >>>>>>>>> ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > log.txt >>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>> string at ./batch_uploads_imageuploader line 143. >>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>> string at ./batch_uploads_imageuploader line 143. >>>>>>>>> Use of uninitialized value $_ in pattern match (m//) at >>>>>>>>> ./batch_uploads_imageuploader line 147. >>>>>>>>> base is DCC0000_258024_V01 >>>>>>>>> path is /data/incoming/ >>>>>>>>> type is .tar.gz >>>>>>>>> fullpath is /data/incoming/DCC0000_258024_V01.tar.gz >>>>>>>>> >>>>>>>>> source /tmp/ImagingUpload-17-43-ZUEFjb >>>>>>>>> targetlocation /data/loris/data/tarchive >>>>>>>>> >>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256. >>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284. >>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>> string at /data/loris/bin/mri/uploadNeuroDB/tarchive_validation.pl >>>>>>>>> line 219. >>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>> string at /data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl >>>>>>>>> line 270. >>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711. >>>>>>>>> Can't exec "mail": No such file or directory at >>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721. >>>>>>>>> print() on closed filehandle MAIL at >>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722. >>>>>>>>> print() on closed filehandle MAIL at >>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723. >>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>>>>>>>> print() on closed filehandle MAIL at >>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>>>>>>>> print() on closed filehandle MAIL at >>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725. >>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735. >>>>>>>>> >>>>>>>>> No Mincs inserted >>>>>>>>> >>>>>>>>> >>>>>>>>> The tarchiveLoader insertion script has failed. >>>>>>>>> Running now the following command: /data/loris/data/uploadNeuroDB/ >>>>>>>>> imaging_upload_file.pl -profile prod -upload_id 3 >>>>>>>>> /data/incoming/DCC0000_258024_V01.tar.gz -verbose" >>>>>>>>> >>>>>>>>> I inserted the VisitLabel, but still the Mincs are not inserted >>>>>>>>> and i am getting this error >>>>>>>>> >>>>>>>>> "lorisadmin at hbp:/data/loris/bin/mri$ >>>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile >>>>>>>>> prod -upload_id 3 /data/incoming/DCC0000_258024_V01.tar.gz -verbose >>>>>>>>> >>>>>>>>> find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name '__MACOSX' >>>>>>>>> -delete >>>>>>>>> Spool message is: >>>>>>>>> The Scan for the uploadID 3 has already been run with tarchiveID: >>>>>>>>> 7. >>>>>>>>> To continue with the rest of the insertion pipeline, please run >>>>>>>>> tarchiveLoader from a terminal as follows: >>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod >>>>>>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >>>>>>>>> -verbose" >>>>>>>>> >>>>>>>>> Which files should i delete? I thought that deleting the files >>>>>>>>> from the tarchive was enough. And i executed the last proposed command to >>>>>>>>> continue to rest of the insertion pipeline ( check the image ). >>>>>>>>> >>>>>>>>> Thank you >>>>>>>>> >>>>>>>>> ???? ???, 24 ???? 2019 ???? 5:05 ?.?., ?/? Cecile Madjar < >>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>> >>>>>>>>>> Hi Sotirios, >>>>>>>>>> >>>>>>>>>> Sorry I was not clear before. Every visits of candidates are >>>>>>>>>> created in the session table with one entry per CandID/VisitLabel. So if >>>>>>>>>> the imaging pipeline created visits in the session table (that are not >>>>>>>>>> attached to instruments) and you want to clean your database completely, >>>>>>>>>> you can delete entries in that table as well. >>>>>>>>>> >>>>>>>>>> Regarding your point 2, the imaging pipeline can create the visit >>>>>>>>>> in the session table as long as the visit label is present in the >>>>>>>>>> Visit_Windows table (which means the visit label stored in the PatientName >>>>>>>>>> is a valid visit label). >>>>>>>>>> However, in your prod file, if the >>>>>>>>>> variable $subjectID{'createVisitLabel'} is set to 1 for candidates, the >>>>>>>>>> visit label should be created in the Visit_Windows table I believe. Here >>>>>>>>>> is the line of the prod file you would need to change >>>>>>>>>> >>>>>>>>>> . >>>>>>>>>> In general, we prefer populating the Visit_Windows table and set >>>>>>>>>> the $subjectID{'createVisitLabel'} to 0 for candidate data to avoid >>>>>>>>>> insertion of badly labelled MRI data that are complex to relabel. >>>>>>>>>> >>>>>>>>>> Finally, regarding the candidate creation, if you execute the >>>>>>>>>> following steps, the pipeline should be able to create candidates: >>>>>>>>>> - set in the Imaging Pipeline section of the config module the >>>>>>>>>> config setting "Upload creation of candidates" to Yes >>>>>>>>>> - manually transfer the scans to the LORIS server instead of >>>>>>>>>> uploading it via the imaging browser >>>>>>>>>> - run batch_imaging_upload.pl as you did until now. >>>>>>>>>> >>>>>>>>>> Hope this helps! >>>>>>>>>> Best, >>>>>>>>>> >>>>>>>>>> C?cile >>>>>>>>>> >>>>>>>>>> On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos < >>>>>>>>>> sotirisnik at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> About my second question i replaced the name of the patient in >>>>>>>>>>> the .dcm files and now i am getting an error that the visit label does not >>>>>>>>>>> exist. >>>>>>>>>>> I thought it was supposed to be automatically created. >>>>>>>>>>> >>>>>>>>>>> "Done adding archive info into database >>>>>>>>>>> >>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation >>>>>>>>>>> -profile prod >>>>>>>>>>> /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>>>>>>>> -verbose >>>>>>>>>>> md5sum >>>>>>>>>>> /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>> CandID id: 258024 >>>>>>>>>>> visit_label is: V1 >>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>> CandID id: 258024 >>>>>>>>>>> visit_label is: V1 >>>>>>>>>>> candidate id 258024 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> => No Visit labelVisit label does not exist >>>>>>>>>>> Set centerID = 1 >>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>> CandID id: 258024 >>>>>>>>>>> visit_label is: V1 >>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>> CandID id: 258024 >>>>>>>>>>> visit_label is: V1 >>>>>>>>>>> >>>>>>>>>>> Number of MINC files that will be considered for inserting into >>>>>>>>>>> the database: 1 >>>>>>>>>>> >>>>>>>>>>> log dir is /logs and log file is /logs/TarLoad-15-42-L6pocU.log >>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>> CandID id: 258024 >>>>>>>>>>> visit_label is: V1 >>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>> CandID id: 258024 >>>>>>>>>>> visit_label is: V1 >>>>>>>>>>> candidate id 258024 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> => No Visit label >>>>>>>>>>> Cleaning up temp files: rm -rf >>>>>>>>>>> /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*" >>>>>>>>>>> >>>>>>>>>>> ???? ???, 24 ???? 2019 ???? 3:38 ?.?., ?/? Sotirios >>>>>>>>>>> Nikoloutsopoulos ??????: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> 1) What are the records in the session table? >>>>>>>>>>>> >>>>>>>>>>>> 2) I tried to upload a non-phantom file like this >>>>>>>>>>>> >>>>>>>>>>>> input_file.txt: >>>>>>>>>>>> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1 >>>>>>>>>>>> >>>>>>>>>>>> Terminal: >>>>>>>>>>>> ./batch_uploads_imageuploader -profile prod < >>>>>>>>>>>> ~/Desktop/input.txt > log.txt >>>>>>>>>>>> >>>>>>>>>>>> and i get the following error in the log.txt file >>>>>>>>>>>> >>>>>>>>>>>> "Running now the following command: >>>>>>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile >>>>>>>>>>>> prod -upload_id 7 /data/incoming/DCC0000_258024_V1.tar.gz -verbose >>>>>>>>>>>> >>>>>>>>>>>> find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name >>>>>>>>>>>> '__MACOSX' -delete >>>>>>>>>>>> Spool message is: >>>>>>>>>>>> The PatientName read from the DICOM header does not start with >>>>>>>>>>>> DCC0000_258024_V1 from the mri_upload table >>>>>>>>>>>> " >>>>>>>>>>>> >>>>>>>>>>>> and the spool message is repeated for each .dcm file. Should i >>>>>>>>>>>> preprocess every .dcm file and change the PatientName to >>>>>>>>>>>> "DCC0000_258024_V1"? >>>>>>>>>>>> >>>>>>>>>>>> 3) As far as it concerns the candidate profile i have to create >>>>>>>>>>>> a new one each time before i try to upload a file? Is there already a way >>>>>>>>>>>> to create a candidate profile and get the DCCID and PSCID without the user >>>>>>>>>>>> interface? >>>>>>>>>>>> >>>>>>>>>>>> Thank you, >>>>>>>>>>>> >>>>>>>>>>>> Sotirios >>>>>>>>>>>> >>>>>>>>>>>> ???? ???, 18 ???? 2019 ???? 5:00 ?.?., ?/? Cecile Madjar < >>>>>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Sotirios, >>>>>>>>>>>>> >>>>>>>>>>>>> See answers to your questions below. >>>>>>>>>>>>> >>>>>>>>>>>>> About the 1st case we had a dicom with a blank studyUID, maybe >>>>>>>>>>>>>> someone accidentally removed it. >>>>>>>>>>>>>> >>>>>>>>>>>>> That is strange. Something must have been done to the DICOM >>>>>>>>>>>>> files. I have never seen a study with no StudyUID out of the scanner. >>>>>>>>>>>>> Usually, you need to manually erase it with some tool. However, DICOMs >>>>>>>>>>>>> never cease to surprise me... >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> Some other questions: >>>>>>>>>>>>>> 1) What are the steps to remove all of my patients and >>>>>>>>>>>>>> uploads?: because i want to try to reupload all the testing dicoms we had >>>>>>>>>>>>>> with a clean database. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Release 21.0 of LORIS-MRI will offer a script to remove all >>>>>>>>>>>>> the entries and files specific to an upload. However, this release is not >>>>>>>>>>>>> yet out. Hopefully in the next few weeks. >>>>>>>>>>>>> In the meantime, since your case is particular and that you >>>>>>>>>>>>> just want to start with a clean database, the following deletes should work >>>>>>>>>>>>> (hopefully I won't forget any tables): >>>>>>>>>>>>> delete from mri_violations_log; >>>>>>>>>>>>> delete from mri_protocol_violated_scans; >>>>>>>>>>>>> delete from MRICandidateErrors; >>>>>>>>>>>>> delete from parameter_file; >>>>>>>>>>>>> delete from files_qcstatus; # (if you played with the QC part >>>>>>>>>>>>> of the imaging browser for testing) >>>>>>>>>>>>> delete from feedback_mri_comments; # (if you played with the >>>>>>>>>>>>> QC part of the imaging browser for testing) >>>>>>>>>>>>> delete from tarchive_series; >>>>>>>>>>>>> delete from tarchive_files; >>>>>>>>>>>>> delete from mri_upload; >>>>>>>>>>>>> delete from tarchive; >>>>>>>>>>>>> delete from mri_scanner; >>>>>>>>>>>>> delete from candidate where PSCID="scanner"; >>>>>>>>>>>>> >>>>>>>>>>>>> I don't know if you want to also delete entries in the session >>>>>>>>>>>>> and candidate table too? >>>>>>>>>>>>> >>>>>>>>>>>>> 2) We do not need to create the profile of a patient ( are the >>>>>>>>>>>>>> patients stored in the candidate table? because i can't see the ghosts ), >>>>>>>>>>>>>> because will it automatically be created when importing a dicom? >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I am not sure I understand fully this question. All candidates >>>>>>>>>>>>> are stored in the candidate table. The phantom scans however are attached >>>>>>>>>>>>> to a scanner candidate depending on where the scan happened. It is a bit of >>>>>>>>>>>>> a weird concept that we have to redesign eventually but never got a chance >>>>>>>>>>>>> to get to it. >>>>>>>>>>>>> >>>>>>>>>>>>> Hopefully this answered your question, otherwise, don't >>>>>>>>>>>>> hesitate to let me know. >>>>>>>>>>>>> >>>>>>>>>>>>> 3) What is the difference between uploading a dicom as phantom >>>>>>>>>>>>>> instead of importing as PSCCID_DCCID_VisitLabel? >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> When you upload the scan as PSCID_DCCID_VisitLabel, there is a >>>>>>>>>>>>> check that makes sure the candidate IDs and visit label are valid at the >>>>>>>>>>>>> time of upload. >>>>>>>>>>>>> When you upload a scan as phantom, it expects that the DICOM >>>>>>>>>>>>> field PatientName and uploaded filename contains the string "phantom". We >>>>>>>>>>>>> enforced this behaviour on the imaging uploader side in recent releases but >>>>>>>>>>>>> I can't remember which one. Probably the upcoming 21.0 release. >>>>>>>>>>>>> >>>>>>>>>>>>> All those verifications might seem cumbersome but they are >>>>>>>>>>>>> here to ensure that the files inserted are all valid and labelled properly >>>>>>>>>>>>> as it is a bit messy to have to delete files that were wrongly labelled. At >>>>>>>>>>>>> least, the delete script present in release 21.0 will make this process >>>>>>>>>>>>> easier but it is still good practice to verify those things. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> With pleasure. Hopefully the answers to your questions will be >>>>>>>>>>>>> helpful. >>>>>>>>>>>>> >>>>>>>>>>>>> Best, >>>>>>>>>>>>> >>>>>>>>>>>>> C?cile >>>>>>>>>>>>> >>>>>>>>>>>>> PS: loris-dev mailing list: you will find the initial answer I >>>>>>>>>>>>> gave Sotirios below. I forgot to cc the loris-dev in my earlier reply... >>>>>>>>>>>>> >>>>>>>>>>>>> ???? ???, 16 ???? 2019 ???? 12:02 ?.?., ?/? Cecile Madjar < >>>>>>>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Dear Sotirios, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you for reaching out. Since your email was already >>>>>>>>>>>>>>> organized in points, I will reply directly below your questions below. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> C?cile >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1) What do we do when the StudyUID is blank on the header? >>>>>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md >>>>>>>>>>>>>>>> There solution provided at the table 3 does not guide us how to find the >>>>>>>>>>>>>>>> information ( is possible ). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The StudyUID is the field used in our pipeline at the moment >>>>>>>>>>>>>>> to check whether a DICOM study was already inserted into LORIS with that >>>>>>>>>>>>>>> same StudyUID since it is supposed to be unique for every single study. >>>>>>>>>>>>>>> Currently, we do not support insertion of DICOM studies if they do not have >>>>>>>>>>>>>>> a StudyUID associated with them. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Is there a specific reason why the StudyUID is blank in the >>>>>>>>>>>>>>> DICOM headers in your DICOM files? Did they go through some processes >>>>>>>>>>>>>>> before upload to LORIS? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2) When i am about to upload a non-phanton case i need a >>>>>>>>>>>>>>>> 'Visit Label'. Is there already a script that can create a new visit label >>>>>>>>>>>>>>>> for a specific Candidate, so i can provide it as input afterwards to the >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md ) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am not sure if the front-end allows for it but if you use >>>>>>>>>>>>>>> batch_uploads_imageuploader, the insertion scripts could create the visit >>>>>>>>>>>>>>> label for you. However, it does need to be included in the patient name in >>>>>>>>>>>>>>> order for the script to know which label it should use. >>>>>>>>>>>>>>> For example: you upload V02 for MTL0123 (DCCID: 456789) but >>>>>>>>>>>>>>> V02 was not yet created for MTL0123; the insertion scripts will create a >>>>>>>>>>>>>>> V02 visit for MTL0123 if the PatientName field for the dataset is >>>>>>>>>>>>>>> MTL0123_456789_V02 with Stage marked as "Not Started". >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> However, *note that for the script to create the visit for >>>>>>>>>>>>>>> the candidate, you have to make sure that all the visit label of your >>>>>>>>>>>>>>> projects were inserted in the Visit_Windows table* >>>>>>>>>>>>>>> (otherwise, the visit will never be created as they were not specified as >>>>>>>>>>>>>>> being part of the list of visit label to expect). So in the example above, >>>>>>>>>>>>>>> you should have one row in Visit_Windows with Visit_label="V02" set. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 3) Section 4.3 >>>>>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> What do we do when the folder contains multiple mnc files? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This is a case by case. Most of the time, the MINC files >>>>>>>>>>>>>>> that failed insertion into the imaging browser end up in the MRI violation >>>>>>>>>>>>>>> module where you can see what went wrong with the acquisitions not >>>>>>>>>>>>>>> inserted. If you notice that the MINC file should be inserted as a specific >>>>>>>>>>>>>>> protocol, you can force the insertion as explained in section 4.3. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> In your case, I have a feeling that you want to be able to >>>>>>>>>>>>>>> insert MINC files that did not go through the whole pipeline insertion >>>>>>>>>>>>>>> (DICOM archival, then dcm2mnc, then protocol identification, then >>>>>>>>>>>>>>> insertion...). In theory, you could call minc_insertion.pl >>>>>>>>>>>>>>> on any MINC file, provided you have at least the following information: >>>>>>>>>>>>>>> - path to the MINC file >>>>>>>>>>>>>>> - uploadID associated with the MINC files or DICOM archive >>>>>>>>>>>>>>> path from which those MINC files were created from >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Then the minc_insertion.pl script will do the subject >>>>>>>>>>>>>>> information and protocol validation etc... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If all the MINC files come from the same uploadID or >>>>>>>>>>>>>>> TarchiveID, then you could run a loop in bash calling the >>>>>>>>>>>>>>> minc_insertion.pl script like this for files deriving from >>>>>>>>>>>>>>> uploadID=1: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ls /path/to/mnc/folder/* | while read f; do >>>>>>>>>>>>>>> minc_insertion.pl -profile prod -uploadID 1 -mincPath $f; >>>>>>>>>>>>>>> done >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hope this helped! >>>>>>>>>>>>>>> >>>>>>>>>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.31.35 AM.png Type: image/png Size: 45707 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.32.50 AM.png Type: image/png Size: 134887 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.40.52 AM.png Type: image/png Size: 95814 bytes Desc: not available URL: From sotirisnik at gmail.com Tue Jul 30 09:50:57 2019 From: sotirisnik at gmail.com (Sotirios Nikoloutsopoulos) Date: Tue, 30 Jul 2019 16:50:57 +0300 Subject: [Loris-dev] Import mri - scripts In-Reply-To: References: Message-ID: Hi, there was no visit label 01 in the Visit_Windows table and i don't know how to insert it because the record requires lots of attributes. Also the system for a reason asked me to update my password and now i don't remember it and my account got suspended. Is there a way to reset it or set a simple password? I updated the password hash in the users table by using the hash from another LORIS setup we had and i still couldn't login ( also the attribute password is always NULL and the expiration date is 1999-01-01 for some reason ). Thanks ???? ???, 29 ???? 2019 ???? 5:21 ?.?., ?/? Cecile Madjar < cecile.madjar at mcin.ca> ??????: > Hi Sotirios, > > Can you try adding the visit label v01 to the Visit_Windows table? > > However, I noticed that in a previous dataset, the label was V1. Does that > refer to the same visit? If so, you might want to harmonize the visit > labels so they are all the same across datasets for a given timepoint, > meaning you will need to relabel scans when the visit label is different > but referring to the same visit label (v01 = V1 etc...). > > Hope this helps, > > C?cile > > On Mon, Jul 29, 2019 at 10:15 AM Sotirios Nikoloutsopoulos < > sotirisnik at gmail.com> wrote: > >> Hi, >> >> Could you help me with the "Visit label does not exist"? I had changed >> $subjectID{'createVisitLabel'} to 1 as you had suggested but it appears >> that this problem hasn't been resolved yet. >> >> lorisadmin at hbp:/data/loris/bin/mri$ cat >> /data/loris/data//logs/TarLoad-14-58-9Xog7H.log >> >> ==> Loading file from disk >> /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc >> >> --> mapping DICOM parameter for >> /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc >> >> --> using user-defined filterParameters for >> /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc >> >> Candidate Mismatch Error is Visit label does not exist >> -> WARNING: This candidate was invalid. Logging to >> MRICandidateErrors table with reason Visit label does not >> exist >> >> Thank you, >> >> Sotirios >> >> ???? ???, 26 ???? 2019 ???? 3:02 ?.?., ?/? Sotirios Nikoloutsopoulos < >> sotirisnik at gmail.com> ??????: >> >>> Hi Cecile, >>> >>> The Center error was fixed. Still the insertion of mincs fails. Please >>> see the output.txt file i attached. Also is there something else i need to >>> change at the paths configuration? >>> >>> Thank you, >>> >>> Sotirios >>> >>> ???? ???, 25 ???? 2019 ???? 6:55 ?.?., ?/? Cecile Madjar < >>> cecile.madjar at mcin.ca> ??????: >>> >>>> Hi Sotirios, >>>> >>>> We are making progress! >>>> >>>> I have a feeling this error comes from the fact that your psc table >>>> does not have the MRI_alias populated for the DCC site. Running the >>>> following query in the mysql database should fix this: >>>> UPDATE psc SET MRI_alias="DCC", Alias="DCC" WHERE Name="Data >>>> Coordinating Center"; >>>> Repeat the same with all your sites (make sure to change the Aliases >>>> between sites, they should all be unique). >>>> >>>> Let me know how it goes! >>>> Best, >>>> >>>> C?cile >>>> >>>> >>>> >>>> >>>> >>>> On Thu, Jul 25, 2019 at 11:43 AM Sotirios Nikoloutsopoulos < >>>> sotirisnik at gmail.com> wrote: >>>> >>>>> Dear Cecile, >>>>> You were right about your guesses. I did the modifications you >>>>> suggested, i still get the "no mincs inserted" error, now i am getting the >>>>> "No center found this candidate" error ( how do i fix that? from what i see >>>>> the .dcm files contains a 'Institution Name', is that the center? ), so i >>>>> guess that's why the validation fails, but i changed the force variable of >>>>> tarchiveLoader to 1, shouldn't it have inserted the mincs then? Also it is >>>>> optional to have a mail server, right? ( from what i see the mail error >>>>> does not affect the $valid_study, so i assume we are okay with that ) >>>>> Thank you a lot, >>>>> Sotirios >>>>> >>>>> ???? ???, 25 ???? 2019 ???? 5:48 ?.?., ?/? Cecile Madjar < >>>>> cecile.madjar at mcin.ca> ??????: >>>>> >>>>>> Hi Sotirios, >>>>>> >>>>>> From what I understand, I believe you just need to update a few >>>>>> config settings from the frontend in the Config module (under the Admin >>>>>> menu). >>>>>> [image: Screen Shot 2019-07-25 at 10.31.35 AM.png] >>>>>> >>>>>> Then go to the Paths section of the config module and update the >>>>>> "LORIS-MRI code" config setting to /data/loris/bin/mri (my guess is that >>>>>> currently, it is set to /data/LORIS/data) and click on the submit button. >>>>>> [image: Screen Shot 2019-07-25 at 10.32.50 AM.png] >>>>>> >>>>>> Then, go to Imaging Pipeline section of the Configuration module and >>>>>> update the "Loris-MRI Data Directory" to /data/loris/data/ (my guess is >>>>>> that it is empty at the moment in your database which is why $data_dir is >>>>>> not defined when you run the tarchiveLoader step of the insertion pipeline). >>>>>> [image: Screen Shot 2019-07-25 at 10.40.52 AM.png] >>>>>> >>>>>> Finally, I took a look at your environment file and it might be a >>>>>> good idea to update your PERL5LIB variable to the following (it has been >>>>>> added to the code a few releases ago as we realized this was needed for the >>>>>> libraries. Not sure which version of LORIS-MRI you are using but it does >>>>>> not hurt to update this variable to what I suggest below): >>>>>> export >>>>>> PERL5LIB=/data/loris/bin/mri/uploadNeuroDB:/data/loris/bin/mri/dicom-archive:$PERL5LIB >>>>>> >>>>>> Hope this helps. Sorry you are experiencing so much trouble setting >>>>>> things up. We probably need to update our WIKI. >>>>>> >>>>>> Best, >>>>>> >>>>>> C?cile >>>>>> >>>>>> On Thu, Jul 25, 2019 at 9:48 AM Sotirios Nikoloutsopoulos < >>>>>> sotirisnik at gmail.com> wrote: >>>>>> >>>>>>> The "Loris-MRI Data Directory" should be in the environment file? >>>>>>> >>>>>>> ???? ???, 25 ???? 2019 ???? 12:59 ?.?., ?/? Sotirios >>>>>>> Nikoloutsopoulos ??????: >>>>>>> >>>>>>>> The scripts are located in /data/loris/bin/mri/. Somewhere I might >>>>>>>> have messed something up but i remember following the instructions in the >>>>>>>> wiki.For example when i tried to execute the batch_uploads_imageuploader >>>>>>>> perl script the line >>>>>>>> >>>>>>>> my $command = "$bin_dir/uploadNeuroDB/imaging_upload_file.pl " >>>>>>>> >>>>>>>> was pointing to an invalid path, for example: it was repeating a >>>>>>>> pattern and had LORIS instead of loris ( i think the mistake was that in >>>>>>>> the wiki it suggested to use $PROJECT for the name of the folder and i >>>>>>>> choosed 'loris' >>>>>>>> https://github.com/aces/Loris/wiki/Imaging-Database ). See also >>>>>>>> the in image for the modification i did. Also i didn't find the section to >>>>>>>> config the $data_dir variable. >>>>>>>> >>>>>>>> Thank you, >>>>>>>> >>>>>>>> Sotirios >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ???? ???, 24 ???? 2019 ???? 6:05 ?.?., ?/? Cecile Madjar < >>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>> >>>>>>>>> Hi Sotirios, >>>>>>>>> >>>>>>>>> It looks like you have one Config setting not set (based on the >>>>>>>>> first error message you got when running it the first time). You need to >>>>>>>>> set a value for "Loris-MRI Data Directory" in the Config module under the >>>>>>>>> Imaging pipeline section (from what I can see, it would be >>>>>>>>> /data/loris/data/ for your setup). >>>>>>>>> >>>>>>>>> The reason you got that error message the second time is that the >>>>>>>>> pipeline checks if the DICOM study was already inserted in the tarchive >>>>>>>>> table. Since you ran the study once already, the study is inserted into the >>>>>>>>> tarchive, hence the message. However, the pipeline did not complete as the >>>>>>>>> MINC files were not created and inserted into the files/parameter_file >>>>>>>>> table. In order to continue the insertion, you will then need to run the >>>>>>>>> tarchiveLoader command suggested in the terminal during the second run: >>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation >>>>>>>>> -profile prod >>>>>>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >>>>>>>>> -verbose >>>>>>>>> >>>>>>>>> Finally, I am a bit confused by your setup. Where are the scripts >>>>>>>>> located? In /data/loris/bin/mri or /data/loris/data? It feels like the >>>>>>>>> Config setting "LORIS-MRI code" in the Paths section was set to >>>>>>>>> "/data/LORIS/data" but it should have been set to the directory where the >>>>>>>>> scripts are located (most probably /data/loris/bin/mri). >>>>>>>>> >>>>>>>>> Hope this helps, >>>>>>>>> >>>>>>>>> C?cile >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Jul 24, 2019 at 10:55 AM Sotirios Nikoloutsopoulos < >>>>>>>>> sotirisnik at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Ok first i execute this command: >>>>>>>>>> >>>>>>>>>> "lorisadmin at hbp:/data/loris/bin/mri$ >>>>>>>>>> ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > log.txt >>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>> string at ./batch_uploads_imageuploader line 143. >>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>> string at ./batch_uploads_imageuploader line 143. >>>>>>>>>> Use of uninitialized value $_ in pattern match (m//) at >>>>>>>>>> ./batch_uploads_imageuploader line 147. >>>>>>>>>> base is DCC0000_258024_V01 >>>>>>>>>> path is /data/incoming/ >>>>>>>>>> type is .tar.gz >>>>>>>>>> fullpath is /data/incoming/DCC0000_258024_V01.tar.gz >>>>>>>>>> >>>>>>>>>> source /tmp/ImagingUpload-17-43-ZUEFjb >>>>>>>>>> targetlocation /data/loris/data/tarchive >>>>>>>>>> >>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256. >>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284. >>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>> string at /data/loris/bin/mri/uploadNeuroDB/ >>>>>>>>>> tarchive_validation.pl line 219. >>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>> string at /data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl >>>>>>>>>> line 270. >>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711. >>>>>>>>>> Can't exec "mail": No such file or directory at >>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721. >>>>>>>>>> print() on closed filehandle MAIL at >>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722. >>>>>>>>>> print() on closed filehandle MAIL at >>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723. >>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>>>>>>>>> print() on closed filehandle MAIL at >>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>>>>>>>>> print() on closed filehandle MAIL at >>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725. >>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735. >>>>>>>>>> >>>>>>>>>> No Mincs inserted >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The tarchiveLoader insertion script has failed. >>>>>>>>>> Running now the following command: /data/loris/data/uploadNeuroDB/ >>>>>>>>>> imaging_upload_file.pl -profile prod -upload_id 3 >>>>>>>>>> /data/incoming/DCC0000_258024_V01.tar.gz -verbose" >>>>>>>>>> >>>>>>>>>> I inserted the VisitLabel, but still the Mincs are not inserted >>>>>>>>>> and i am getting this error >>>>>>>>>> >>>>>>>>>> "lorisadmin at hbp:/data/loris/bin/mri$ >>>>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile >>>>>>>>>> prod -upload_id 3 /data/incoming/DCC0000_258024_V01.tar.gz -verbose >>>>>>>>>> >>>>>>>>>> find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name '__MACOSX' >>>>>>>>>> -delete >>>>>>>>>> Spool message is: >>>>>>>>>> The Scan for the uploadID 3 has already been run with tarchiveID: >>>>>>>>>> 7. >>>>>>>>>> To continue with the rest of the insertion pipeline, please run >>>>>>>>>> tarchiveLoader from a terminal as follows: >>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod >>>>>>>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >>>>>>>>>> -verbose" >>>>>>>>>> >>>>>>>>>> Which files should i delete? I thought that deleting the files >>>>>>>>>> from the tarchive was enough. And i executed the last proposed command to >>>>>>>>>> continue to rest of the insertion pipeline ( check the image ). >>>>>>>>>> >>>>>>>>>> Thank you >>>>>>>>>> >>>>>>>>>> ???? ???, 24 ???? 2019 ???? 5:05 ?.?., ?/? Cecile Madjar < >>>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>>> >>>>>>>>>>> Hi Sotirios, >>>>>>>>>>> >>>>>>>>>>> Sorry I was not clear before. Every visits of candidates are >>>>>>>>>>> created in the session table with one entry per CandID/VisitLabel. So if >>>>>>>>>>> the imaging pipeline created visits in the session table (that are not >>>>>>>>>>> attached to instruments) and you want to clean your database completely, >>>>>>>>>>> you can delete entries in that table as well. >>>>>>>>>>> >>>>>>>>>>> Regarding your point 2, the imaging pipeline can create the >>>>>>>>>>> visit in the session table as long as the visit label is present in the >>>>>>>>>>> Visit_Windows table (which means the visit label stored in the PatientName >>>>>>>>>>> is a valid visit label). >>>>>>>>>>> However, in your prod file, if the >>>>>>>>>>> variable $subjectID{'createVisitLabel'} is set to 1 for candidates, the >>>>>>>>>>> visit label should be created in the Visit_Windows table I believe. Here >>>>>>>>>>> is the line of the prod file you would need to change >>>>>>>>>>> >>>>>>>>>>> . >>>>>>>>>>> In general, we prefer populating the Visit_Windows table and set >>>>>>>>>>> the $subjectID{'createVisitLabel'} to 0 for candidate data to avoid >>>>>>>>>>> insertion of badly labelled MRI data that are complex to relabel. >>>>>>>>>>> >>>>>>>>>>> Finally, regarding the candidate creation, if you execute the >>>>>>>>>>> following steps, the pipeline should be able to create candidates: >>>>>>>>>>> - set in the Imaging Pipeline section of the config module the >>>>>>>>>>> config setting "Upload creation of candidates" to Yes >>>>>>>>>>> - manually transfer the scans to the LORIS server instead of >>>>>>>>>>> uploading it via the imaging browser >>>>>>>>>>> - run batch_imaging_upload.pl as you did until now. >>>>>>>>>>> >>>>>>>>>>> Hope this helps! >>>>>>>>>>> Best, >>>>>>>>>>> >>>>>>>>>>> C?cile >>>>>>>>>>> >>>>>>>>>>> On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos < >>>>>>>>>>> sotirisnik at gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> About my second question i replaced the name of the patient in >>>>>>>>>>>> the .dcm files and now i am getting an error that the visit label does not >>>>>>>>>>>> exist. >>>>>>>>>>>> I thought it was supposed to be automatically created. >>>>>>>>>>>> >>>>>>>>>>>> "Done adding archive info into database >>>>>>>>>>>> >>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation >>>>>>>>>>>> -profile prod >>>>>>>>>>>> /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>>>>>>>>> -verbose >>>>>>>>>>>> md5sum >>>>>>>>>>>> /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>>> CandID id: 258024 >>>>>>>>>>>> visit_label is: V1 >>>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>>> CandID id: 258024 >>>>>>>>>>>> visit_label is: V1 >>>>>>>>>>>> candidate id 258024 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> => No Visit labelVisit label does not exist >>>>>>>>>>>> Set centerID = 1 >>>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>>> CandID id: 258024 >>>>>>>>>>>> visit_label is: V1 >>>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>>> CandID id: 258024 >>>>>>>>>>>> visit_label is: V1 >>>>>>>>>>>> >>>>>>>>>>>> Number of MINC files that will be considered for inserting into >>>>>>>>>>>> the database: 1 >>>>>>>>>>>> >>>>>>>>>>>> log dir is /logs and log file is /logs/TarLoad-15-42-L6pocU.log >>>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>>> CandID id: 258024 >>>>>>>>>>>> visit_label is: V1 >>>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>>> CandID id: 258024 >>>>>>>>>>>> visit_label is: V1 >>>>>>>>>>>> candidate id 258024 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> => No Visit label >>>>>>>>>>>> Cleaning up temp files: rm -rf >>>>>>>>>>>> /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*" >>>>>>>>>>>> >>>>>>>>>>>> ???? ???, 24 ???? 2019 ???? 3:38 ?.?., ?/? Sotirios >>>>>>>>>>>> Nikoloutsopoulos ??????: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> 1) What are the records in the session table? >>>>>>>>>>>>> >>>>>>>>>>>>> 2) I tried to upload a non-phantom file like this >>>>>>>>>>>>> >>>>>>>>>>>>> input_file.txt: >>>>>>>>>>>>> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1 >>>>>>>>>>>>> >>>>>>>>>>>>> Terminal: >>>>>>>>>>>>> ./batch_uploads_imageuploader -profile prod < >>>>>>>>>>>>> ~/Desktop/input.txt > log.txt >>>>>>>>>>>>> >>>>>>>>>>>>> and i get the following error in the log.txt file >>>>>>>>>>>>> >>>>>>>>>>>>> "Running now the following command: >>>>>>>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl >>>>>>>>>>>>> -profile prod -upload_id 7 /data/incoming/DCC0000_258024_V1.tar.gz -verbose >>>>>>>>>>>>> >>>>>>>>>>>>> find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name >>>>>>>>>>>>> '__MACOSX' -delete >>>>>>>>>>>>> Spool message is: >>>>>>>>>>>>> The PatientName read from the DICOM header does not start with >>>>>>>>>>>>> DCC0000_258024_V1 from the mri_upload table >>>>>>>>>>>>> " >>>>>>>>>>>>> >>>>>>>>>>>>> and the spool message is repeated for each .dcm file. Should i >>>>>>>>>>>>> preprocess every .dcm file and change the PatientName to >>>>>>>>>>>>> "DCC0000_258024_V1"? >>>>>>>>>>>>> >>>>>>>>>>>>> 3) As far as it concerns the candidate profile i have to >>>>>>>>>>>>> create a new one each time before i try to upload a file? Is there already >>>>>>>>>>>>> a way to create a candidate profile and get the DCCID and PSCID without the >>>>>>>>>>>>> user interface? >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you, >>>>>>>>>>>>> >>>>>>>>>>>>> Sotirios >>>>>>>>>>>>> >>>>>>>>>>>>> ???? ???, 18 ???? 2019 ???? 5:00 ?.?., ?/? Cecile Madjar < >>>>>>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Sotirios, >>>>>>>>>>>>>> >>>>>>>>>>>>>> See answers to your questions below. >>>>>>>>>>>>>> >>>>>>>>>>>>>> About the 1st case we had a dicom with a blank studyUID, >>>>>>>>>>>>>>> maybe someone accidentally removed it. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> That is strange. Something must have been done to the DICOM >>>>>>>>>>>>>> files. I have never seen a study with no StudyUID out of the scanner. >>>>>>>>>>>>>> Usually, you need to manually erase it with some tool. However, DICOMs >>>>>>>>>>>>>> never cease to surprise me... >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Some other questions: >>>>>>>>>>>>>>> 1) What are the steps to remove all of my patients and >>>>>>>>>>>>>>> uploads?: because i want to try to reupload all the testing dicoms we had >>>>>>>>>>>>>>> with a clean database. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Release 21.0 of LORIS-MRI will offer a script to remove all >>>>>>>>>>>>>> the entries and files specific to an upload. However, this release is not >>>>>>>>>>>>>> yet out. Hopefully in the next few weeks. >>>>>>>>>>>>>> In the meantime, since your case is particular and that you >>>>>>>>>>>>>> just want to start with a clean database, the following deletes should work >>>>>>>>>>>>>> (hopefully I won't forget any tables): >>>>>>>>>>>>>> delete from mri_violations_log; >>>>>>>>>>>>>> delete from mri_protocol_violated_scans; >>>>>>>>>>>>>> delete from MRICandidateErrors; >>>>>>>>>>>>>> delete from parameter_file; >>>>>>>>>>>>>> delete from files_qcstatus; # (if you played with the QC part >>>>>>>>>>>>>> of the imaging browser for testing) >>>>>>>>>>>>>> delete from feedback_mri_comments; # (if you played with the >>>>>>>>>>>>>> QC part of the imaging browser for testing) >>>>>>>>>>>>>> delete from tarchive_series; >>>>>>>>>>>>>> delete from tarchive_files; >>>>>>>>>>>>>> delete from mri_upload; >>>>>>>>>>>>>> delete from tarchive; >>>>>>>>>>>>>> delete from mri_scanner; >>>>>>>>>>>>>> delete from candidate where PSCID="scanner"; >>>>>>>>>>>>>> >>>>>>>>>>>>>> I don't know if you want to also delete entries in the >>>>>>>>>>>>>> session and candidate table too? >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2) We do not need to create the profile of a patient ( are >>>>>>>>>>>>>>> the patients stored in the candidate table? because i can't see the ghosts >>>>>>>>>>>>>>> ), because will it automatically be created when importing a dicom? >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I am not sure I understand fully this question. All >>>>>>>>>>>>>> candidates are stored in the candidate table. The phantom scans however are >>>>>>>>>>>>>> attached to a scanner candidate depending on where the scan happened. It is >>>>>>>>>>>>>> a bit of a weird concept that we have to redesign eventually but never got >>>>>>>>>>>>>> a chance to get to it. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hopefully this answered your question, otherwise, don't >>>>>>>>>>>>>> hesitate to let me know. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 3) What is the difference between uploading a dicom as >>>>>>>>>>>>>>> phantom instead of importing as PSCCID_DCCID_VisitLabel? >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> When you upload the scan as PSCID_DCCID_VisitLabel, there is >>>>>>>>>>>>>> a check that makes sure the candidate IDs and visit label are valid at the >>>>>>>>>>>>>> time of upload. >>>>>>>>>>>>>> When you upload a scan as phantom, it expects that the DICOM >>>>>>>>>>>>>> field PatientName and uploaded filename contains the string "phantom". We >>>>>>>>>>>>>> enforced this behaviour on the imaging uploader side in recent releases but >>>>>>>>>>>>>> I can't remember which one. Probably the upcoming 21.0 release. >>>>>>>>>>>>>> >>>>>>>>>>>>>> All those verifications might seem cumbersome but they are >>>>>>>>>>>>>> here to ensure that the files inserted are all valid and labelled properly >>>>>>>>>>>>>> as it is a bit messy to have to delete files that were wrongly labelled. At >>>>>>>>>>>>>> least, the delete script present in release 21.0 will make this process >>>>>>>>>>>>>> easier but it is still good practice to verify those things. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> With pleasure. Hopefully the answers to your questions will >>>>>>>>>>>>>> be helpful. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best, >>>>>>>>>>>>>> >>>>>>>>>>>>>> C?cile >>>>>>>>>>>>>> >>>>>>>>>>>>>> PS: loris-dev mailing list: you will find the initial answer >>>>>>>>>>>>>> I gave Sotirios below. I forgot to cc the loris-dev in my earlier reply... >>>>>>>>>>>>>> >>>>>>>>>>>>>> ???? ???, 16 ???? 2019 ???? 12:02 ?.?., ?/? Cecile Madjar < >>>>>>>>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Dear Sotirios, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thank you for reaching out. Since your email was already >>>>>>>>>>>>>>>> organized in points, I will reply directly below your questions below. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> C?cile >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1) What do we do when the StudyUID is blank on the header? >>>>>>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md >>>>>>>>>>>>>>>>> There solution provided at the table 3 does not guide us how to find the >>>>>>>>>>>>>>>>> information ( is possible ). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The StudyUID is the field used in our pipeline at the >>>>>>>>>>>>>>>> moment to check whether a DICOM study was already inserted into LORIS with >>>>>>>>>>>>>>>> that same StudyUID since it is supposed to be unique for every single >>>>>>>>>>>>>>>> study. Currently, we do not support insertion of DICOM studies if they do >>>>>>>>>>>>>>>> not have a StudyUID associated with them. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Is there a specific reason why the StudyUID is blank in the >>>>>>>>>>>>>>>> DICOM headers in your DICOM files? Did they go through some processes >>>>>>>>>>>>>>>> before upload to LORIS? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2) When i am about to upload a non-phanton case i need a >>>>>>>>>>>>>>>>> 'Visit Label'. Is there already a script that can create a new visit label >>>>>>>>>>>>>>>>> for a specific Candidate, so i can provide it as input afterwards to the >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md ) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I am not sure if the front-end allows for it but if you use >>>>>>>>>>>>>>>> batch_uploads_imageuploader, the insertion scripts could create the visit >>>>>>>>>>>>>>>> label for you. However, it does need to be included in the patient name in >>>>>>>>>>>>>>>> order for the script to know which label it should use. >>>>>>>>>>>>>>>> For example: you upload V02 for MTL0123 (DCCID: 456789) >>>>>>>>>>>>>>>> but V02 was not yet created for MTL0123; the insertion scripts will create >>>>>>>>>>>>>>>> a V02 visit for MTL0123 if the PatientName field for the dataset is >>>>>>>>>>>>>>>> MTL0123_456789_V02 with Stage marked as "Not Started". >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> However, *note that for the script to create the visit for >>>>>>>>>>>>>>>> the candidate, you have to make sure that all the visit label of your >>>>>>>>>>>>>>>> projects were inserted in the Visit_Windows table* >>>>>>>>>>>>>>>> (otherwise, the visit will never be created as they were not specified as >>>>>>>>>>>>>>>> being part of the list of visit label to expect). So in the example above, >>>>>>>>>>>>>>>> you should have one row in Visit_Windows with Visit_label="V02" set. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 3) Section 4.3 >>>>>>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> What do we do when the folder contains multiple mnc files? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> This is a case by case. Most of the time, the MINC files >>>>>>>>>>>>>>>> that failed insertion into the imaging browser end up in the MRI violation >>>>>>>>>>>>>>>> module where you can see what went wrong with the acquisitions not >>>>>>>>>>>>>>>> inserted. If you notice that the MINC file should be inserted as a specific >>>>>>>>>>>>>>>> protocol, you can force the insertion as explained in section 4.3. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> In your case, I have a feeling that you want to be able to >>>>>>>>>>>>>>>> insert MINC files that did not go through the whole pipeline insertion >>>>>>>>>>>>>>>> (DICOM archival, then dcm2mnc, then protocol identification, then >>>>>>>>>>>>>>>> insertion...). In theory, you could call minc_insertion.pl >>>>>>>>>>>>>>>> on any MINC file, provided you have at least the following information: >>>>>>>>>>>>>>>> - path to the MINC file >>>>>>>>>>>>>>>> - uploadID associated with the MINC files or DICOM archive >>>>>>>>>>>>>>>> path from which those MINC files were created from >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Then the minc_insertion.pl script will do the subject >>>>>>>>>>>>>>>> information and protocol validation etc... >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> If all the MINC files come from the same uploadID or >>>>>>>>>>>>>>>> TarchiveID, then you could run a loop in bash calling the >>>>>>>>>>>>>>>> minc_insertion.pl script like this for files deriving from >>>>>>>>>>>>>>>> uploadID=1: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ls /path/to/mnc/folder/* | while read f; do >>>>>>>>>>>>>>>> minc_insertion.pl -profile prod -uploadID 1 -mincPath $f; >>>>>>>>>>>>>>>> done >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hope this helped! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.31.35 AM.png Type: image/png Size: 45707 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.32.50 AM.png Type: image/png Size: 134887 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.40.52 AM.png Type: image/png Size: 95814 bytes Desc: not available URL: From sotirisnik at gmail.com Tue Jul 30 09:58:17 2019 From: sotirisnik at gmail.com (Sotirios Nikoloutsopoulos) Date: Tue, 30 Jul 2019 16:58:17 +0300 Subject: [Loris-dev] Import mri - scripts In-Reply-To: References: Message-ID: Ok I fixed the login problem by using a known hash and by deleting the rows from the login_history i was able to login. ???? ???, 30 ???? 2019 ???? 4:50 ?.?., ?/? Sotirios Nikoloutsopoulos < sotirisnik at gmail.com> ??????: > Hi, > > there was no visit label 01 in the Visit_Windows table and i don't know > how to insert it because the record requires lots of attributes. Also the > system for a reason asked me to update my password and now i don't remember > it and my account got suspended. Is there a way to reset it or set a simple > password? I updated the password hash in the users table by using the hash > from another LORIS setup we had and i still couldn't login ( also the > attribute password is always NULL and the expiration date is 1999-01-01 for > some reason ). > > Thanks > > > > ???? ???, 29 ???? 2019 ???? 5:21 ?.?., ?/? Cecile Madjar < > cecile.madjar at mcin.ca> ??????: > >> Hi Sotirios, >> >> Can you try adding the visit label v01 to the Visit_Windows table? >> >> However, I noticed that in a previous dataset, the label was V1. Does >> that refer to the same visit? If so, you might want to harmonize the visit >> labels so they are all the same across datasets for a given timepoint, >> meaning you will need to relabel scans when the visit label is different >> but referring to the same visit label (v01 = V1 etc...). >> >> Hope this helps, >> >> C?cile >> >> On Mon, Jul 29, 2019 at 10:15 AM Sotirios Nikoloutsopoulos < >> sotirisnik at gmail.com> wrote: >> >>> Hi, >>> >>> Could you help me with the "Visit label does not exist"? I had changed >>> $subjectID{'createVisitLabel'} to 1 as you had suggested but it appears >>> that this problem hasn't been resolved yet. >>> >>> lorisadmin at hbp:/data/loris/bin/mri$ cat >>> /data/loris/data//logs/TarLoad-14-58-9Xog7H.log >>> >>> ==> Loading file from disk >>> /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc >>> >>> --> mapping DICOM parameter for >>> /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc >>> >>> --> using user-defined filterParameters for >>> /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc >>> >>> Candidate Mismatch Error is Visit label does not exist >>> -> WARNING: This candidate was invalid. Logging to >>> MRICandidateErrors table with reason Visit label does not >>> exist >>> >>> Thank you, >>> >>> Sotirios >>> >>> ???? ???, 26 ???? 2019 ???? 3:02 ?.?., ?/? Sotirios Nikoloutsopoulos < >>> sotirisnik at gmail.com> ??????: >>> >>>> Hi Cecile, >>>> >>>> The Center error was fixed. Still the insertion of mincs fails. Please >>>> see the output.txt file i attached. Also is there something else i need to >>>> change at the paths configuration? >>>> >>>> Thank you, >>>> >>>> Sotirios >>>> >>>> ???? ???, 25 ???? 2019 ???? 6:55 ?.?., ?/? Cecile Madjar < >>>> cecile.madjar at mcin.ca> ??????: >>>> >>>>> Hi Sotirios, >>>>> >>>>> We are making progress! >>>>> >>>>> I have a feeling this error comes from the fact that your psc table >>>>> does not have the MRI_alias populated for the DCC site. Running the >>>>> following query in the mysql database should fix this: >>>>> UPDATE psc SET MRI_alias="DCC", Alias="DCC" WHERE Name="Data >>>>> Coordinating Center"; >>>>> Repeat the same with all your sites (make sure to change the Aliases >>>>> between sites, they should all be unique). >>>>> >>>>> Let me know how it goes! >>>>> Best, >>>>> >>>>> C?cile >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Jul 25, 2019 at 11:43 AM Sotirios Nikoloutsopoulos < >>>>> sotirisnik at gmail.com> wrote: >>>>> >>>>>> Dear Cecile, >>>>>> You were right about your guesses. I did the modifications you >>>>>> suggested, i still get the "no mincs inserted" error, now i am getting the >>>>>> "No center found this candidate" error ( how do i fix that? from what i see >>>>>> the .dcm files contains a 'Institution Name', is that the center? ), so i >>>>>> guess that's why the validation fails, but i changed the force variable of >>>>>> tarchiveLoader to 1, shouldn't it have inserted the mincs then? Also it is >>>>>> optional to have a mail server, right? ( from what i see the mail error >>>>>> does not affect the $valid_study, so i assume we are okay with that ) >>>>>> Thank you a lot, >>>>>> Sotirios >>>>>> >>>>>> ???? ???, 25 ???? 2019 ???? 5:48 ?.?., ?/? Cecile Madjar < >>>>>> cecile.madjar at mcin.ca> ??????: >>>>>> >>>>>>> Hi Sotirios, >>>>>>> >>>>>>> From what I understand, I believe you just need to update a few >>>>>>> config settings from the frontend in the Config module (under the Admin >>>>>>> menu). >>>>>>> [image: Screen Shot 2019-07-25 at 10.31.35 AM.png] >>>>>>> >>>>>>> Then go to the Paths section of the config module and update the >>>>>>> "LORIS-MRI code" config setting to /data/loris/bin/mri (my guess is that >>>>>>> currently, it is set to /data/LORIS/data) and click on the submit button. >>>>>>> [image: Screen Shot 2019-07-25 at 10.32.50 AM.png] >>>>>>> >>>>>>> Then, go to Imaging Pipeline section of the Configuration module and >>>>>>> update the "Loris-MRI Data Directory" to /data/loris/data/ (my guess is >>>>>>> that it is empty at the moment in your database which is why $data_dir is >>>>>>> not defined when you run the tarchiveLoader step of the insertion pipeline). >>>>>>> [image: Screen Shot 2019-07-25 at 10.40.52 AM.png] >>>>>>> >>>>>>> Finally, I took a look at your environment file and it might be a >>>>>>> good idea to update your PERL5LIB variable to the following (it has been >>>>>>> added to the code a few releases ago as we realized this was needed for the >>>>>>> libraries. Not sure which version of LORIS-MRI you are using but it does >>>>>>> not hurt to update this variable to what I suggest below): >>>>>>> export >>>>>>> PERL5LIB=/data/loris/bin/mri/uploadNeuroDB:/data/loris/bin/mri/dicom-archive:$PERL5LIB >>>>>>> >>>>>>> Hope this helps. Sorry you are experiencing so much trouble setting >>>>>>> things up. We probably need to update our WIKI. >>>>>>> >>>>>>> Best, >>>>>>> >>>>>>> C?cile >>>>>>> >>>>>>> On Thu, Jul 25, 2019 at 9:48 AM Sotirios Nikoloutsopoulos < >>>>>>> sotirisnik at gmail.com> wrote: >>>>>>> >>>>>>>> The "Loris-MRI Data Directory" should be in the environment file? >>>>>>>> >>>>>>>> ???? ???, 25 ???? 2019 ???? 12:59 ?.?., ?/? Sotirios >>>>>>>> Nikoloutsopoulos ??????: >>>>>>>> >>>>>>>>> The scripts are located in /data/loris/bin/mri/. Somewhere I >>>>>>>>> might have messed something up but i remember following the instructions in >>>>>>>>> the wiki.For example when i tried to execute the >>>>>>>>> batch_uploads_imageuploader perl script the line >>>>>>>>> >>>>>>>>> my $command = "$bin_dir/uploadNeuroDB/imaging_upload_file.pl " >>>>>>>>> >>>>>>>>> was pointing to an invalid path, for example: it was repeating a >>>>>>>>> pattern and had LORIS instead of loris ( i think the mistake was that in >>>>>>>>> the wiki it suggested to use $PROJECT for the name of the folder and i >>>>>>>>> choosed 'loris' >>>>>>>>> https://github.com/aces/Loris/wiki/Imaging-Database ). See also >>>>>>>>> the in image for the modification i did. Also i didn't find the section to >>>>>>>>> config the $data_dir variable. >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> >>>>>>>>> Sotirios >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> ???? ???, 24 ???? 2019 ???? 6:05 ?.?., ?/? Cecile Madjar < >>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>> >>>>>>>>>> Hi Sotirios, >>>>>>>>>> >>>>>>>>>> It looks like you have one Config setting not set (based on the >>>>>>>>>> first error message you got when running it the first time). You need to >>>>>>>>>> set a value for "Loris-MRI Data Directory" in the Config module under the >>>>>>>>>> Imaging pipeline section (from what I can see, it would be >>>>>>>>>> /data/loris/data/ for your setup). >>>>>>>>>> >>>>>>>>>> The reason you got that error message the second time is that the >>>>>>>>>> pipeline checks if the DICOM study was already inserted in the tarchive >>>>>>>>>> table. Since you ran the study once already, the study is inserted into the >>>>>>>>>> tarchive, hence the message. However, the pipeline did not complete as the >>>>>>>>>> MINC files were not created and inserted into the files/parameter_file >>>>>>>>>> table. In order to continue the insertion, you will then need to run the >>>>>>>>>> tarchiveLoader command suggested in the terminal during the second run: >>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation >>>>>>>>>> -profile prod >>>>>>>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >>>>>>>>>> -verbose >>>>>>>>>> >>>>>>>>>> Finally, I am a bit confused by your setup. Where are the scripts >>>>>>>>>> located? In /data/loris/bin/mri or /data/loris/data? It feels like the >>>>>>>>>> Config setting "LORIS-MRI code" in the Paths section was set to >>>>>>>>>> "/data/LORIS/data" but it should have been set to the directory where the >>>>>>>>>> scripts are located (most probably /data/loris/bin/mri). >>>>>>>>>> >>>>>>>>>> Hope this helps, >>>>>>>>>> >>>>>>>>>> C?cile >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Jul 24, 2019 at 10:55 AM Sotirios Nikoloutsopoulos < >>>>>>>>>> sotirisnik at gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Ok first i execute this command: >>>>>>>>>>> >>>>>>>>>>> "lorisadmin at hbp:/data/loris/bin/mri$ >>>>>>>>>>> ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > log.txt >>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>>> string at ./batch_uploads_imageuploader line 143. >>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>>> string at ./batch_uploads_imageuploader line 143. >>>>>>>>>>> Use of uninitialized value $_ in pattern match (m//) at >>>>>>>>>>> ./batch_uploads_imageuploader line 147. >>>>>>>>>>> base is DCC0000_258024_V01 >>>>>>>>>>> path is /data/incoming/ >>>>>>>>>>> type is .tar.gz >>>>>>>>>>> fullpath is /data/incoming/DCC0000_258024_V01.tar.gz >>>>>>>>>>> >>>>>>>>>>> source /tmp/ImagingUpload-17-43-ZUEFjb >>>>>>>>>>> targetlocation /data/loris/data/tarchive >>>>>>>>>>> >>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256. >>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284. >>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>>> string at /data/loris/bin/mri/uploadNeuroDB/ >>>>>>>>>>> tarchive_validation.pl line 219. >>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>>> string at /data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl >>>>>>>>>>> line 270. >>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711. >>>>>>>>>>> Can't exec "mail": No such file or directory at >>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721. >>>>>>>>>>> print() on closed filehandle MAIL at >>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722. >>>>>>>>>>> print() on closed filehandle MAIL at >>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723. >>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>>>>>>>>>> print() on closed filehandle MAIL at >>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>>>>>>>>>> print() on closed filehandle MAIL at >>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725. >>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735. >>>>>>>>>>> >>>>>>>>>>> No Mincs inserted >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The tarchiveLoader insertion script has failed. >>>>>>>>>>> Running now the following command: >>>>>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile >>>>>>>>>>> prod -upload_id 3 /data/incoming/DCC0000_258024_V01.tar.gz -verbose" >>>>>>>>>>> >>>>>>>>>>> I inserted the VisitLabel, but still the Mincs are not inserted >>>>>>>>>>> and i am getting this error >>>>>>>>>>> >>>>>>>>>>> "lorisadmin at hbp:/data/loris/bin/mri$ >>>>>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile >>>>>>>>>>> prod -upload_id 3 /data/incoming/DCC0000_258024_V01.tar.gz -verbose >>>>>>>>>>> >>>>>>>>>>> find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name >>>>>>>>>>> '__MACOSX' -delete >>>>>>>>>>> Spool message is: >>>>>>>>>>> The Scan for the uploadID 3 has already been run with >>>>>>>>>>> tarchiveID: 7. >>>>>>>>>>> To continue with the rest of the insertion pipeline, please run >>>>>>>>>>> tarchiveLoader from a terminal as follows: >>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod >>>>>>>>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >>>>>>>>>>> -verbose" >>>>>>>>>>> >>>>>>>>>>> Which files should i delete? I thought that deleting the files >>>>>>>>>>> from the tarchive was enough. And i executed the last proposed command to >>>>>>>>>>> continue to rest of the insertion pipeline ( check the image ). >>>>>>>>>>> >>>>>>>>>>> Thank you >>>>>>>>>>> >>>>>>>>>>> ???? ???, 24 ???? 2019 ???? 5:05 ?.?., ?/? Cecile Madjar < >>>>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>>>> >>>>>>>>>>>> Hi Sotirios, >>>>>>>>>>>> >>>>>>>>>>>> Sorry I was not clear before. Every visits of candidates are >>>>>>>>>>>> created in the session table with one entry per CandID/VisitLabel. So if >>>>>>>>>>>> the imaging pipeline created visits in the session table (that are not >>>>>>>>>>>> attached to instruments) and you want to clean your database completely, >>>>>>>>>>>> you can delete entries in that table as well. >>>>>>>>>>>> >>>>>>>>>>>> Regarding your point 2, the imaging pipeline can create the >>>>>>>>>>>> visit in the session table as long as the visit label is present in the >>>>>>>>>>>> Visit_Windows table (which means the visit label stored in the PatientName >>>>>>>>>>>> is a valid visit label). >>>>>>>>>>>> However, in your prod file, if the >>>>>>>>>>>> variable $subjectID{'createVisitLabel'} is set to 1 for candidates, the >>>>>>>>>>>> visit label should be created in the Visit_Windows table I believe. Here >>>>>>>>>>>> is the line of the prod file you would need to change >>>>>>>>>>>> >>>>>>>>>>>> . >>>>>>>>>>>> In general, we prefer populating the Visit_Windows table and >>>>>>>>>>>> set the $subjectID{'createVisitLabel'} to 0 for candidate data to avoid >>>>>>>>>>>> insertion of badly labelled MRI data that are complex to relabel. >>>>>>>>>>>> >>>>>>>>>>>> Finally, regarding the candidate creation, if you execute the >>>>>>>>>>>> following steps, the pipeline should be able to create candidates: >>>>>>>>>>>> - set in the Imaging Pipeline section of the config module the >>>>>>>>>>>> config setting "Upload creation of candidates" to Yes >>>>>>>>>>>> - manually transfer the scans to the LORIS server instead of >>>>>>>>>>>> uploading it via the imaging browser >>>>>>>>>>>> - run batch_imaging_upload.pl as you did until now. >>>>>>>>>>>> >>>>>>>>>>>> Hope this helps! >>>>>>>>>>>> Best, >>>>>>>>>>>> >>>>>>>>>>>> C?cile >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos < >>>>>>>>>>>> sotirisnik at gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> About my second question i replaced the name of the patient in >>>>>>>>>>>>> the .dcm files and now i am getting an error that the visit label does not >>>>>>>>>>>>> exist. >>>>>>>>>>>>> I thought it was supposed to be automatically created. >>>>>>>>>>>>> >>>>>>>>>>>>> "Done adding archive info into database >>>>>>>>>>>>> >>>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation >>>>>>>>>>>>> -profile prod >>>>>>>>>>>>> /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>>>>>>>>>> -verbose >>>>>>>>>>>>> md5sum >>>>>>>>>>>>> /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>>>> CandID id: 258024 >>>>>>>>>>>>> visit_label is: V1 >>>>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>>>> CandID id: 258024 >>>>>>>>>>>>> visit_label is: V1 >>>>>>>>>>>>> candidate id 258024 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> => No Visit labelVisit label does not exist >>>>>>>>>>>>> Set centerID = 1 >>>>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>>>> CandID id: 258024 >>>>>>>>>>>>> visit_label is: V1 >>>>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>>>> CandID id: 258024 >>>>>>>>>>>>> visit_label is: V1 >>>>>>>>>>>>> >>>>>>>>>>>>> Number of MINC files that will be considered for inserting >>>>>>>>>>>>> into the database: 1 >>>>>>>>>>>>> >>>>>>>>>>>>> log dir is /logs and log file is >>>>>>>>>>>>> /logs/TarLoad-15-42-L6pocU.log >>>>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>>>> CandID id: 258024 >>>>>>>>>>>>> visit_label is: V1 >>>>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>>>> CandID id: 258024 >>>>>>>>>>>>> visit_label is: V1 >>>>>>>>>>>>> candidate id 258024 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> => No Visit label >>>>>>>>>>>>> Cleaning up temp files: rm -rf >>>>>>>>>>>>> /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*" >>>>>>>>>>>>> >>>>>>>>>>>>> ???? ???, 24 ???? 2019 ???? 3:38 ?.?., ?/? Sotirios >>>>>>>>>>>>> Nikoloutsopoulos ??????: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1) What are the records in the session table? >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2) I tried to upload a non-phantom file like this >>>>>>>>>>>>>> >>>>>>>>>>>>>> input_file.txt: >>>>>>>>>>>>>> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Terminal: >>>>>>>>>>>>>> ./batch_uploads_imageuploader -profile prod < >>>>>>>>>>>>>> ~/Desktop/input.txt > log.txt >>>>>>>>>>>>>> >>>>>>>>>>>>>> and i get the following error in the log.txt file >>>>>>>>>>>>>> >>>>>>>>>>>>>> "Running now the following command: >>>>>>>>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl >>>>>>>>>>>>>> -profile prod -upload_id 7 /data/incoming/DCC0000_258024_V1.tar.gz -verbose >>>>>>>>>>>>>> >>>>>>>>>>>>>> find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name >>>>>>>>>>>>>> '__MACOSX' -delete >>>>>>>>>>>>>> Spool message is: >>>>>>>>>>>>>> The PatientName read from the DICOM header does not start >>>>>>>>>>>>>> with DCC0000_258024_V1 from the mri_upload table >>>>>>>>>>>>>> " >>>>>>>>>>>>>> >>>>>>>>>>>>>> and the spool message is repeated for each .dcm file. Should >>>>>>>>>>>>>> i preprocess every .dcm file and change the PatientName to >>>>>>>>>>>>>> "DCC0000_258024_V1"? >>>>>>>>>>>>>> >>>>>>>>>>>>>> 3) As far as it concerns the candidate profile i have to >>>>>>>>>>>>>> create a new one each time before i try to upload a file? Is there already >>>>>>>>>>>>>> a way to create a candidate profile and get the DCCID and PSCID without the >>>>>>>>>>>>>> user interface? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sotirios >>>>>>>>>>>>>> >>>>>>>>>>>>>> ???? ???, 18 ???? 2019 ???? 5:00 ?.?., ?/? Cecile Madjar < >>>>>>>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Sotirios, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> See answers to your questions below. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> About the 1st case we had a dicom with a blank studyUID, >>>>>>>>>>>>>>>> maybe someone accidentally removed it. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> That is strange. Something must have been done to the DICOM >>>>>>>>>>>>>>> files. I have never seen a study with no StudyUID out of the scanner. >>>>>>>>>>>>>>> Usually, you need to manually erase it with some tool. However, DICOMs >>>>>>>>>>>>>>> never cease to surprise me... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Some other questions: >>>>>>>>>>>>>>>> 1) What are the steps to remove all of my patients and >>>>>>>>>>>>>>>> uploads?: because i want to try to reupload all the testing dicoms we had >>>>>>>>>>>>>>>> with a clean database. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Release 21.0 of LORIS-MRI will offer a script to remove all >>>>>>>>>>>>>>> the entries and files specific to an upload. However, this release is not >>>>>>>>>>>>>>> yet out. Hopefully in the next few weeks. >>>>>>>>>>>>>>> In the meantime, since your case is particular and that you >>>>>>>>>>>>>>> just want to start with a clean database, the following deletes should work >>>>>>>>>>>>>>> (hopefully I won't forget any tables): >>>>>>>>>>>>>>> delete from mri_violations_log; >>>>>>>>>>>>>>> delete from mri_protocol_violated_scans; >>>>>>>>>>>>>>> delete from MRICandidateErrors; >>>>>>>>>>>>>>> delete from parameter_file; >>>>>>>>>>>>>>> delete from files_qcstatus; # (if you played with the QC >>>>>>>>>>>>>>> part of the imaging browser for testing) >>>>>>>>>>>>>>> delete from feedback_mri_comments; # (if you played with the >>>>>>>>>>>>>>> QC part of the imaging browser for testing) >>>>>>>>>>>>>>> delete from tarchive_series; >>>>>>>>>>>>>>> delete from tarchive_files; >>>>>>>>>>>>>>> delete from mri_upload; >>>>>>>>>>>>>>> delete from tarchive; >>>>>>>>>>>>>>> delete from mri_scanner; >>>>>>>>>>>>>>> delete from candidate where PSCID="scanner"; >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I don't know if you want to also delete entries in the >>>>>>>>>>>>>>> session and candidate table too? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2) We do not need to create the profile of a patient ( are >>>>>>>>>>>>>>>> the patients stored in the candidate table? because i can't see the ghosts >>>>>>>>>>>>>>>> ), because will it automatically be created when importing a dicom? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am not sure I understand fully this question. All >>>>>>>>>>>>>>> candidates are stored in the candidate table. The phantom scans however are >>>>>>>>>>>>>>> attached to a scanner candidate depending on where the scan happened. It is >>>>>>>>>>>>>>> a bit of a weird concept that we have to redesign eventually but never got >>>>>>>>>>>>>>> a chance to get to it. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hopefully this answered your question, otherwise, don't >>>>>>>>>>>>>>> hesitate to let me know. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 3) What is the difference between uploading a dicom as >>>>>>>>>>>>>>>> phantom instead of importing as PSCCID_DCCID_VisitLabel? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> When you upload the scan as PSCID_DCCID_VisitLabel, there is >>>>>>>>>>>>>>> a check that makes sure the candidate IDs and visit label are valid at the >>>>>>>>>>>>>>> time of upload. >>>>>>>>>>>>>>> When you upload a scan as phantom, it expects that the DICOM >>>>>>>>>>>>>>> field PatientName and uploaded filename contains the string "phantom". We >>>>>>>>>>>>>>> enforced this behaviour on the imaging uploader side in recent releases but >>>>>>>>>>>>>>> I can't remember which one. Probably the upcoming 21.0 release. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> All those verifications might seem cumbersome but they are >>>>>>>>>>>>>>> here to ensure that the files inserted are all valid and labelled properly >>>>>>>>>>>>>>> as it is a bit messy to have to delete files that were wrongly labelled. At >>>>>>>>>>>>>>> least, the delete script present in release 21.0 will make this process >>>>>>>>>>>>>>> easier but it is still good practice to verify those things. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> With pleasure. Hopefully the answers to your questions will >>>>>>>>>>>>>>> be helpful. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> C?cile >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> PS: loris-dev mailing list: you will find the initial answer >>>>>>>>>>>>>>> I gave Sotirios below. I forgot to cc the loris-dev in my earlier reply... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ???? ???, 16 ???? 2019 ???? 12:02 ?.?., ?/? Cecile Madjar < >>>>>>>>>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Dear Sotirios, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thank you for reaching out. Since your email was already >>>>>>>>>>>>>>>>> organized in points, I will reply directly below your questions below. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> C?cile >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1) What do we do when the StudyUID is blank on the header? >>>>>>>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md >>>>>>>>>>>>>>>>>> There solution provided at the table 3 does not guide us how to find the >>>>>>>>>>>>>>>>>> information ( is possible ). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The StudyUID is the field used in our pipeline at the >>>>>>>>>>>>>>>>> moment to check whether a DICOM study was already inserted into LORIS with >>>>>>>>>>>>>>>>> that same StudyUID since it is supposed to be unique for every single >>>>>>>>>>>>>>>>> study. Currently, we do not support insertion of DICOM studies if they do >>>>>>>>>>>>>>>>> not have a StudyUID associated with them. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Is there a specific reason why the StudyUID is blank in >>>>>>>>>>>>>>>>> the DICOM headers in your DICOM files? Did they go through some processes >>>>>>>>>>>>>>>>> before upload to LORIS? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2) When i am about to upload a non-phanton case i need a >>>>>>>>>>>>>>>>>> 'Visit Label'. Is there already a script that can create a new visit label >>>>>>>>>>>>>>>>>> for a specific Candidate, so i can provide it as input afterwards to the >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md ) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I am not sure if the front-end allows for it but if you >>>>>>>>>>>>>>>>> use batch_uploads_imageuploader, the insertion scripts could create the >>>>>>>>>>>>>>>>> visit label for you. However, it does need to be included in the patient >>>>>>>>>>>>>>>>> name in order for the script to know which label it should use. >>>>>>>>>>>>>>>>> For example: you upload V02 for MTL0123 (DCCID: 456789) >>>>>>>>>>>>>>>>> but V02 was not yet created for MTL0123; the insertion scripts will create >>>>>>>>>>>>>>>>> a V02 visit for MTL0123 if the PatientName field for the dataset is >>>>>>>>>>>>>>>>> MTL0123_456789_V02 with Stage marked as "Not Started". >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> However, *note that for the script to create the visit >>>>>>>>>>>>>>>>> for the candidate, you have to make sure that all the visit label of your >>>>>>>>>>>>>>>>> projects were inserted in the Visit_Windows table* >>>>>>>>>>>>>>>>> (otherwise, the visit will never be created as they were not specified as >>>>>>>>>>>>>>>>> being part of the list of visit label to expect). So in the example above, >>>>>>>>>>>>>>>>> you should have one row in Visit_Windows with Visit_label="V02" set. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 3) Section 4.3 >>>>>>>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> What do we do when the folder contains multiple mnc files? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> This is a case by case. Most of the time, the MINC files >>>>>>>>>>>>>>>>> that failed insertion into the imaging browser end up in the MRI violation >>>>>>>>>>>>>>>>> module where you can see what went wrong with the acquisitions not >>>>>>>>>>>>>>>>> inserted. If you notice that the MINC file should be inserted as a specific >>>>>>>>>>>>>>>>> protocol, you can force the insertion as explained in section 4.3. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> In your case, I have a feeling that you want to be able to >>>>>>>>>>>>>>>>> insert MINC files that did not go through the whole pipeline insertion >>>>>>>>>>>>>>>>> (DICOM archival, then dcm2mnc, then protocol identification, then >>>>>>>>>>>>>>>>> insertion...). In theory, you could call minc_insertion.pl >>>>>>>>>>>>>>>>> on any MINC file, provided you have at least the following information: >>>>>>>>>>>>>>>>> - path to the MINC file >>>>>>>>>>>>>>>>> - uploadID associated with the MINC files or DICOM archive >>>>>>>>>>>>>>>>> path from which those MINC files were created from >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Then the minc_insertion.pl script will do the subject >>>>>>>>>>>>>>>>> information and protocol validation etc... >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> If all the MINC files come from the same uploadID or >>>>>>>>>>>>>>>>> TarchiveID, then you could run a loop in bash calling the >>>>>>>>>>>>>>>>> minc_insertion.pl script like this for files deriving >>>>>>>>>>>>>>>>> from uploadID=1: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ls /path/to/mnc/folder/* | while read f; do >>>>>>>>>>>>>>>>> minc_insertion.pl -profile prod -uploadID 1 -mincPath $f; >>>>>>>>>>>>>>>>> done >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hope this helped! >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.31.35 AM.png Type: image/png Size: 45707 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.32.50 AM.png Type: image/png Size: 134887 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.40.52 AM.png Type: image/png Size: 95814 bytes Desc: not available URL: From sotirisnik at gmail.com Tue Jul 30 10:46:31 2019 From: sotirisnik at gmail.com (Sotirios Nikoloutsopoulos) Date: Tue, 30 Jul 2019 17:46:31 +0300 Subject: [Loris-dev] Import mri - scripts In-Reply-To: References: Message-ID: I hope this is sufficient as the error for the visit label disappeared. [image: image.png] Please see the attached file. Also something i found. lorisadmin at hbp:/data/loris/bin/mri$ cat /data/loris/data//logs/TarLoad-17-39-xubBdv.log ==> Loading file from disk /tmp/TarLoad-17-39-UyQcG2/dcc0000_258024_v1_20121205_102108_501e1_mri.mnc --> mapping DICOM parameter for /tmp/TarLoad-17-39-UyQcG2/dcc0000_258024_v1_20121205_102108_501e1_mri.mnc ==> computing md5 hash for MINC body. --> md5: 6b1202fd63e29de4c9be1cd9925888a5 ==> verifying acquisition protocol Acquisition protocol is unknown --> The minc file cannot be registered since the AcquisitionProtocol is unknown ???? ???, 30 ???? 2019 ???? 4:58 ?.?., ?/? Sotirios Nikoloutsopoulos < sotirisnik at gmail.com> ??????: > Ok I fixed the login problem by using a known hash and by deleting the > rows from the login_history i was able to login. > > ???? ???, 30 ???? 2019 ???? 4:50 ?.?., ?/? Sotirios Nikoloutsopoulos < > sotirisnik at gmail.com> ??????: > >> Hi, >> >> there was no visit label 01 in the Visit_Windows table and i don't know >> how to insert it because the record requires lots of attributes. Also the >> system for a reason asked me to update my password and now i don't remember >> it and my account got suspended. Is there a way to reset it or set a simple >> password? I updated the password hash in the users table by using the hash >> from another LORIS setup we had and i still couldn't login ( also the >> attribute password is always NULL and the expiration date is 1999-01-01 for >> some reason ). >> >> Thanks >> >> >> >> ???? ???, 29 ???? 2019 ???? 5:21 ?.?., ?/? Cecile Madjar < >> cecile.madjar at mcin.ca> ??????: >> >>> Hi Sotirios, >>> >>> Can you try adding the visit label v01 to the Visit_Windows table? >>> >>> However, I noticed that in a previous dataset, the label was V1. Does >>> that refer to the same visit? If so, you might want to harmonize the visit >>> labels so they are all the same across datasets for a given timepoint, >>> meaning you will need to relabel scans when the visit label is different >>> but referring to the same visit label (v01 = V1 etc...). >>> >>> Hope this helps, >>> >>> C?cile >>> >>> On Mon, Jul 29, 2019 at 10:15 AM Sotirios Nikoloutsopoulos < >>> sotirisnik at gmail.com> wrote: >>> >>>> Hi, >>>> >>>> Could you help me with the "Visit label does not exist"? I had changed >>>> $subjectID{'createVisitLabel'} to 1 as you had suggested but it appears >>>> that this problem hasn't been resolved yet. >>>> >>>> lorisadmin at hbp:/data/loris/bin/mri$ cat >>>> /data/loris/data//logs/TarLoad-14-58-9Xog7H.log >>>> >>>> ==> Loading file from disk >>>> /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc >>>> >>>> --> mapping DICOM parameter for >>>> /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc >>>> >>>> --> using user-defined filterParameters for >>>> /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc >>>> >>>> Candidate Mismatch Error is Visit label does not exist >>>> -> WARNING: This candidate was invalid. Logging to >>>> MRICandidateErrors table with reason Visit label does not >>>> exist >>>> >>>> Thank you, >>>> >>>> Sotirios >>>> >>>> ???? ???, 26 ???? 2019 ???? 3:02 ?.?., ?/? Sotirios Nikoloutsopoulos < >>>> sotirisnik at gmail.com> ??????: >>>> >>>>> Hi Cecile, >>>>> >>>>> The Center error was fixed. Still the insertion of mincs fails. Please >>>>> see the output.txt file i attached. Also is there something else i need to >>>>> change at the paths configuration? >>>>> >>>>> Thank you, >>>>> >>>>> Sotirios >>>>> >>>>> ???? ???, 25 ???? 2019 ???? 6:55 ?.?., ?/? Cecile Madjar < >>>>> cecile.madjar at mcin.ca> ??????: >>>>> >>>>>> Hi Sotirios, >>>>>> >>>>>> We are making progress! >>>>>> >>>>>> I have a feeling this error comes from the fact that your psc table >>>>>> does not have the MRI_alias populated for the DCC site. Running the >>>>>> following query in the mysql database should fix this: >>>>>> UPDATE psc SET MRI_alias="DCC", Alias="DCC" WHERE Name="Data >>>>>> Coordinating Center"; >>>>>> Repeat the same with all your sites (make sure to change the Aliases >>>>>> between sites, they should all be unique). >>>>>> >>>>>> Let me know how it goes! >>>>>> Best, >>>>>> >>>>>> C?cile >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Jul 25, 2019 at 11:43 AM Sotirios Nikoloutsopoulos < >>>>>> sotirisnik at gmail.com> wrote: >>>>>> >>>>>>> Dear Cecile, >>>>>>> You were right about your guesses. I did the modifications you >>>>>>> suggested, i still get the "no mincs inserted" error, now i am getting the >>>>>>> "No center found this candidate" error ( how do i fix that? from what i see >>>>>>> the .dcm files contains a 'Institution Name', is that the center? ), so i >>>>>>> guess that's why the validation fails, but i changed the force variable of >>>>>>> tarchiveLoader to 1, shouldn't it have inserted the mincs then? Also it is >>>>>>> optional to have a mail server, right? ( from what i see the mail error >>>>>>> does not affect the $valid_study, so i assume we are okay with that ) >>>>>>> Thank you a lot, >>>>>>> Sotirios >>>>>>> >>>>>>> ???? ???, 25 ???? 2019 ???? 5:48 ?.?., ?/? Cecile Madjar < >>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>> >>>>>>>> Hi Sotirios, >>>>>>>> >>>>>>>> From what I understand, I believe you just need to update a few >>>>>>>> config settings from the frontend in the Config module (under the Admin >>>>>>>> menu). >>>>>>>> [image: Screen Shot 2019-07-25 at 10.31.35 AM.png] >>>>>>>> >>>>>>>> Then go to the Paths section of the config module and update the >>>>>>>> "LORIS-MRI code" config setting to /data/loris/bin/mri (my guess is that >>>>>>>> currently, it is set to /data/LORIS/data) and click on the submit button. >>>>>>>> [image: Screen Shot 2019-07-25 at 10.32.50 AM.png] >>>>>>>> >>>>>>>> Then, go to Imaging Pipeline section of the Configuration module >>>>>>>> and update the "Loris-MRI Data Directory" to /data/loris/data/ (my guess is >>>>>>>> that it is empty at the moment in your database which is why $data_dir is >>>>>>>> not defined when you run the tarchiveLoader step of the insertion pipeline). >>>>>>>> [image: Screen Shot 2019-07-25 at 10.40.52 AM.png] >>>>>>>> >>>>>>>> Finally, I took a look at your environment file and it might be a >>>>>>>> good idea to update your PERL5LIB variable to the following (it has been >>>>>>>> added to the code a few releases ago as we realized this was needed for the >>>>>>>> libraries. Not sure which version of LORIS-MRI you are using but it does >>>>>>>> not hurt to update this variable to what I suggest below): >>>>>>>> export >>>>>>>> PERL5LIB=/data/loris/bin/mri/uploadNeuroDB:/data/loris/bin/mri/dicom-archive:$PERL5LIB >>>>>>>> >>>>>>>> Hope this helps. Sorry you are experiencing so much trouble setting >>>>>>>> things up. We probably need to update our WIKI. >>>>>>>> >>>>>>>> Best, >>>>>>>> >>>>>>>> C?cile >>>>>>>> >>>>>>>> On Thu, Jul 25, 2019 at 9:48 AM Sotirios Nikoloutsopoulos < >>>>>>>> sotirisnik at gmail.com> wrote: >>>>>>>> >>>>>>>>> The "Loris-MRI Data Directory" should be in the environment file? >>>>>>>>> >>>>>>>>> ???? ???, 25 ???? 2019 ???? 12:59 ?.?., ?/? Sotirios >>>>>>>>> Nikoloutsopoulos ??????: >>>>>>>>> >>>>>>>>>> The scripts are located in /data/loris/bin/mri/. Somewhere I >>>>>>>>>> might have messed something up but i remember following the instructions in >>>>>>>>>> the wiki.For example when i tried to execute the >>>>>>>>>> batch_uploads_imageuploader perl script the line >>>>>>>>>> >>>>>>>>>> my $command = "$bin_dir/uploadNeuroDB/imaging_upload_file.pl " >>>>>>>>>> >>>>>>>>>> was pointing to an invalid path, for example: it was repeating a >>>>>>>>>> pattern and had LORIS instead of loris ( i think the mistake was that in >>>>>>>>>> the wiki it suggested to use $PROJECT for the name of the folder and i >>>>>>>>>> choosed 'loris' >>>>>>>>>> https://github.com/aces/Loris/wiki/Imaging-Database ). See also >>>>>>>>>> the in image for the modification i did. Also i didn't find the section to >>>>>>>>>> config the $data_dir variable. >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> >>>>>>>>>> Sotirios >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ???? ???, 24 ???? 2019 ???? 6:05 ?.?., ?/? Cecile Madjar < >>>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>>> >>>>>>>>>>> Hi Sotirios, >>>>>>>>>>> >>>>>>>>>>> It looks like you have one Config setting not set (based on the >>>>>>>>>>> first error message you got when running it the first time). You need to >>>>>>>>>>> set a value for "Loris-MRI Data Directory" in the Config module under the >>>>>>>>>>> Imaging pipeline section (from what I can see, it would be >>>>>>>>>>> /data/loris/data/ for your setup). >>>>>>>>>>> >>>>>>>>>>> The reason you got that error message the second time is that >>>>>>>>>>> the pipeline checks if the DICOM study was already inserted in the tarchive >>>>>>>>>>> table. Since you ran the study once already, the study is inserted into the >>>>>>>>>>> tarchive, hence the message. However, the pipeline did not complete as the >>>>>>>>>>> MINC files were not created and inserted into the files/parameter_file >>>>>>>>>>> table. In order to continue the insertion, you will then need to run the >>>>>>>>>>> tarchiveLoader command suggested in the terminal during the second run: >>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation >>>>>>>>>>> -profile prod >>>>>>>>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >>>>>>>>>>> -verbose >>>>>>>>>>> >>>>>>>>>>> Finally, I am a bit confused by your setup. Where are the >>>>>>>>>>> scripts located? In /data/loris/bin/mri or /data/loris/data? It feels like >>>>>>>>>>> the Config setting "LORIS-MRI code" in the Paths section was set to >>>>>>>>>>> "/data/LORIS/data" but it should have been set to the directory where the >>>>>>>>>>> scripts are located (most probably /data/loris/bin/mri). >>>>>>>>>>> >>>>>>>>>>> Hope this helps, >>>>>>>>>>> >>>>>>>>>>> C?cile >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Jul 24, 2019 at 10:55 AM Sotirios Nikoloutsopoulos < >>>>>>>>>>> sotirisnik at gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Ok first i execute this command: >>>>>>>>>>>> >>>>>>>>>>>> "lorisadmin at hbp:/data/loris/bin/mri$ >>>>>>>>>>>> ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > log.txt >>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>>>> string at ./batch_uploads_imageuploader line 143. >>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>>>> string at ./batch_uploads_imageuploader line 143. >>>>>>>>>>>> Use of uninitialized value $_ in pattern match (m//) at >>>>>>>>>>>> ./batch_uploads_imageuploader line 147. >>>>>>>>>>>> base is DCC0000_258024_V01 >>>>>>>>>>>> path is /data/incoming/ >>>>>>>>>>>> type is .tar.gz >>>>>>>>>>>> fullpath is /data/incoming/DCC0000_258024_V01.tar.gz >>>>>>>>>>>> >>>>>>>>>>>> source /tmp/ImagingUpload-17-43-ZUEFjb >>>>>>>>>>>> targetlocation /data/loris/data/tarchive >>>>>>>>>>>> >>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256. >>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284. >>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>>>> string at /data/loris/bin/mri/uploadNeuroDB/ >>>>>>>>>>>> tarchive_validation.pl line 219. >>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>>>> string at /data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl >>>>>>>>>>>> line 270. >>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711. >>>>>>>>>>>> Can't exec "mail": No such file or directory at >>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721. >>>>>>>>>>>> print() on closed filehandle MAIL at >>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722. >>>>>>>>>>>> print() on closed filehandle MAIL at >>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723. >>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>>>>>>>>>>> print() on closed filehandle MAIL at >>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. >>>>>>>>>>>> print() on closed filehandle MAIL at >>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725. >>>>>>>>>>>> Use of uninitialized value $data_dir in concatenation (.) or >>>>>>>>>>>> string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735. >>>>>>>>>>>> >>>>>>>>>>>> No Mincs inserted >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> The tarchiveLoader insertion script has failed. >>>>>>>>>>>> Running now the following command: >>>>>>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile >>>>>>>>>>>> prod -upload_id 3 /data/incoming/DCC0000_258024_V01.tar.gz -verbose" >>>>>>>>>>>> >>>>>>>>>>>> I inserted the VisitLabel, but still the Mincs are not inserted >>>>>>>>>>>> and i am getting this error >>>>>>>>>>>> >>>>>>>>>>>> "lorisadmin at hbp:/data/loris/bin/mri$ >>>>>>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile >>>>>>>>>>>> prod -upload_id 3 /data/incoming/DCC0000_258024_V01.tar.gz -verbose >>>>>>>>>>>> >>>>>>>>>>>> find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name >>>>>>>>>>>> '__MACOSX' -delete >>>>>>>>>>>> Spool message is: >>>>>>>>>>>> The Scan for the uploadID 3 has already been run with >>>>>>>>>>>> tarchiveID: 7. >>>>>>>>>>>> To continue with the rest of the insertion pipeline, please run >>>>>>>>>>>> tarchiveLoader from a terminal as follows: >>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod >>>>>>>>>>>> /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar >>>>>>>>>>>> -verbose" >>>>>>>>>>>> >>>>>>>>>>>> Which files should i delete? I thought that deleting the files >>>>>>>>>>>> from the tarchive was enough. And i executed the last proposed command to >>>>>>>>>>>> continue to rest of the insertion pipeline ( check the image ). >>>>>>>>>>>> >>>>>>>>>>>> Thank you >>>>>>>>>>>> >>>>>>>>>>>> ???? ???, 24 ???? 2019 ???? 5:05 ?.?., ?/? Cecile Madjar < >>>>>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Sotirios, >>>>>>>>>>>>> >>>>>>>>>>>>> Sorry I was not clear before. Every visits of candidates are >>>>>>>>>>>>> created in the session table with one entry per CandID/VisitLabel. So if >>>>>>>>>>>>> the imaging pipeline created visits in the session table (that are not >>>>>>>>>>>>> attached to instruments) and you want to clean your database completely, >>>>>>>>>>>>> you can delete entries in that table as well. >>>>>>>>>>>>> >>>>>>>>>>>>> Regarding your point 2, the imaging pipeline can create the >>>>>>>>>>>>> visit in the session table as long as the visit label is present in the >>>>>>>>>>>>> Visit_Windows table (which means the visit label stored in the PatientName >>>>>>>>>>>>> is a valid visit label). >>>>>>>>>>>>> However, in your prod file, if the >>>>>>>>>>>>> variable $subjectID{'createVisitLabel'} is set to 1 for candidates, the >>>>>>>>>>>>> visit label should be created in the Visit_Windows table I believe. Here >>>>>>>>>>>>> is the line of the prod file you would need to change >>>>>>>>>>>>> >>>>>>>>>>>>> . >>>>>>>>>>>>> In general, we prefer populating the Visit_Windows table and >>>>>>>>>>>>> set the $subjectID{'createVisitLabel'} to 0 for candidate data to avoid >>>>>>>>>>>>> insertion of badly labelled MRI data that are complex to relabel. >>>>>>>>>>>>> >>>>>>>>>>>>> Finally, regarding the candidate creation, if you execute the >>>>>>>>>>>>> following steps, the pipeline should be able to create candidates: >>>>>>>>>>>>> - set in the Imaging Pipeline section of the config module the >>>>>>>>>>>>> config setting "Upload creation of candidates" to Yes >>>>>>>>>>>>> - manually transfer the scans to the LORIS server instead of >>>>>>>>>>>>> uploading it via the imaging browser >>>>>>>>>>>>> - run batch_imaging_upload.pl as you did until now. >>>>>>>>>>>>> >>>>>>>>>>>>> Hope this helps! >>>>>>>>>>>>> Best, >>>>>>>>>>>>> >>>>>>>>>>>>> C?cile >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos < >>>>>>>>>>>>> sotirisnik at gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> About my second question i replaced the name of the patient >>>>>>>>>>>>>> in the .dcm files and now i am getting an error that the visit label does >>>>>>>>>>>>>> not exist. >>>>>>>>>>>>>> I thought it was supposed to be automatically created. >>>>>>>>>>>>>> >>>>>>>>>>>>>> "Done adding archive info into database >>>>>>>>>>>>>> >>>>>>>>>>>>>> /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation >>>>>>>>>>>>>> -profile prod >>>>>>>>>>>>>> /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>>>>>>>>>>> -verbose >>>>>>>>>>>>>> md5sum >>>>>>>>>>>>>> /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar >>>>>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>>>>> CandID id: 258024 >>>>>>>>>>>>>> visit_label is: V1 >>>>>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>>>>> CandID id: 258024 >>>>>>>>>>>>>> visit_label is: V1 >>>>>>>>>>>>>> candidate id 258024 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> => No Visit labelVisit label does not exist >>>>>>>>>>>>>> Set centerID = 1 >>>>>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>>>>> CandID id: 258024 >>>>>>>>>>>>>> visit_label is: V1 >>>>>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>>>>> CandID id: 258024 >>>>>>>>>>>>>> visit_label is: V1 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Number of MINC files that will be considered for inserting >>>>>>>>>>>>>> into the database: 1 >>>>>>>>>>>>>> >>>>>>>>>>>>>> log dir is /logs and log file is >>>>>>>>>>>>>> /logs/TarLoad-15-42-L6pocU.log >>>>>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>>>>> CandID id: 258024 >>>>>>>>>>>>>> visit_label is: V1 >>>>>>>>>>>>>> PSCID is: DCC0000 >>>>>>>>>>>>>> CandID id: 258024 >>>>>>>>>>>>>> visit_label is: V1 >>>>>>>>>>>>>> candidate id 258024 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> => No Visit label >>>>>>>>>>>>>> Cleaning up temp files: rm -rf >>>>>>>>>>>>>> /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*" >>>>>>>>>>>>>> >>>>>>>>>>>>>> ???? ???, 24 ???? 2019 ???? 3:38 ?.?., ?/? Sotirios >>>>>>>>>>>>>> Nikoloutsopoulos ??????: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1) What are the records in the session table? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2) I tried to upload a non-phantom file like this >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> input_file.txt: >>>>>>>>>>>>>>> /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Terminal: >>>>>>>>>>>>>>> ./batch_uploads_imageuploader -profile prod < >>>>>>>>>>>>>>> ~/Desktop/input.txt > log.txt >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> and i get the following error in the log.txt file >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> "Running now the following command: >>>>>>>>>>>>>>> /data/loris/data/uploadNeuroDB/imaging_upload_file.pl >>>>>>>>>>>>>>> -profile prod -upload_id 7 /data/incoming/DCC0000_258024_V1.tar.gz -verbose >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name >>>>>>>>>>>>>>> '__MACOSX' -delete >>>>>>>>>>>>>>> Spool message is: >>>>>>>>>>>>>>> The PatientName read from the DICOM header does not start >>>>>>>>>>>>>>> with DCC0000_258024_V1 from the mri_upload table >>>>>>>>>>>>>>> " >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> and the spool message is repeated for each .dcm file. Should >>>>>>>>>>>>>>> i preprocess every .dcm file and change the PatientName to >>>>>>>>>>>>>>> "DCC0000_258024_V1"? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 3) As far as it concerns the candidate profile i have to >>>>>>>>>>>>>>> create a new one each time before i try to upload a file? Is there already >>>>>>>>>>>>>>> a way to create a candidate profile and get the DCCID and PSCID without the >>>>>>>>>>>>>>> user interface? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Sotirios >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ???? ???, 18 ???? 2019 ???? 5:00 ?.?., ?/? Cecile Madjar < >>>>>>>>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Sotirios, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> See answers to your questions below. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> About the 1st case we had a dicom with a blank studyUID, >>>>>>>>>>>>>>>>> maybe someone accidentally removed it. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> That is strange. Something must have been done to the DICOM >>>>>>>>>>>>>>>> files. I have never seen a study with no StudyUID out of the scanner. >>>>>>>>>>>>>>>> Usually, you need to manually erase it with some tool. However, DICOMs >>>>>>>>>>>>>>>> never cease to surprise me... >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Some other questions: >>>>>>>>>>>>>>>>> 1) What are the steps to remove all of my patients and >>>>>>>>>>>>>>>>> uploads?: because i want to try to reupload all the testing dicoms we had >>>>>>>>>>>>>>>>> with a clean database. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Release 21.0 of LORIS-MRI will offer a script to remove all >>>>>>>>>>>>>>>> the entries and files specific to an upload. However, this release is not >>>>>>>>>>>>>>>> yet out. Hopefully in the next few weeks. >>>>>>>>>>>>>>>> In the meantime, since your case is particular and that you >>>>>>>>>>>>>>>> just want to start with a clean database, the following deletes should work >>>>>>>>>>>>>>>> (hopefully I won't forget any tables): >>>>>>>>>>>>>>>> delete from mri_violations_log; >>>>>>>>>>>>>>>> delete from mri_protocol_violated_scans; >>>>>>>>>>>>>>>> delete from MRICandidateErrors; >>>>>>>>>>>>>>>> delete from parameter_file; >>>>>>>>>>>>>>>> delete from files_qcstatus; # (if you played with the QC >>>>>>>>>>>>>>>> part of the imaging browser for testing) >>>>>>>>>>>>>>>> delete from feedback_mri_comments; # (if you played with >>>>>>>>>>>>>>>> the QC part of the imaging browser for testing) >>>>>>>>>>>>>>>> delete from tarchive_series; >>>>>>>>>>>>>>>> delete from tarchive_files; >>>>>>>>>>>>>>>> delete from mri_upload; >>>>>>>>>>>>>>>> delete from tarchive; >>>>>>>>>>>>>>>> delete from mri_scanner; >>>>>>>>>>>>>>>> delete from candidate where PSCID="scanner"; >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I don't know if you want to also delete entries in the >>>>>>>>>>>>>>>> session and candidate table too? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2) We do not need to create the profile of a patient ( are >>>>>>>>>>>>>>>>> the patients stored in the candidate table? because i can't see the ghosts >>>>>>>>>>>>>>>>> ), because will it automatically be created when importing a dicom? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I am not sure I understand fully this question. All >>>>>>>>>>>>>>>> candidates are stored in the candidate table. The phantom scans however are >>>>>>>>>>>>>>>> attached to a scanner candidate depending on where the scan happened. It is >>>>>>>>>>>>>>>> a bit of a weird concept that we have to redesign eventually but never got >>>>>>>>>>>>>>>> a chance to get to it. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hopefully this answered your question, otherwise, don't >>>>>>>>>>>>>>>> hesitate to let me know. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 3) What is the difference between uploading a dicom as >>>>>>>>>>>>>>>>> phantom instead of importing as PSCCID_DCCID_VisitLabel? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> When you upload the scan as PSCID_DCCID_VisitLabel, there >>>>>>>>>>>>>>>> is a check that makes sure the candidate IDs and visit label are valid at >>>>>>>>>>>>>>>> the time of upload. >>>>>>>>>>>>>>>> When you upload a scan as phantom, it expects that the >>>>>>>>>>>>>>>> DICOM field PatientName and uploaded filename contains the string >>>>>>>>>>>>>>>> "phantom". We enforced this behaviour on the imaging uploader side in >>>>>>>>>>>>>>>> recent releases but I can't remember which one. Probably the upcoming 21.0 >>>>>>>>>>>>>>>> release. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> All those verifications might seem cumbersome but they are >>>>>>>>>>>>>>>> here to ensure that the files inserted are all valid and labelled properly >>>>>>>>>>>>>>>> as it is a bit messy to have to delete files that were wrongly labelled. At >>>>>>>>>>>>>>>> least, the delete script present in release 21.0 will make this process >>>>>>>>>>>>>>>> easier but it is still good practice to verify those things. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> With pleasure. Hopefully the answers to your questions will >>>>>>>>>>>>>>>> be helpful. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> C?cile >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> PS: loris-dev mailing list: you will find the initial >>>>>>>>>>>>>>>> answer I gave Sotirios below. I forgot to cc the loris-dev in my earlier >>>>>>>>>>>>>>>> reply... >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ???? ???, 16 ???? 2019 ???? 12:02 ?.?., ?/? Cecile Madjar < >>>>>>>>>>>>>>>>> cecile.madjar at mcin.ca> ??????: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Dear Sotirios, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thank you for reaching out. Since your email was already >>>>>>>>>>>>>>>>>> organized in points, I will reply directly below your questions below. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> C?cile >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 1) What do we do when the StudyUID is blank on the >>>>>>>>>>>>>>>>>>> header? >>>>>>>>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md >>>>>>>>>>>>>>>>>>> There solution provided at the table 3 does not guide us how to find the >>>>>>>>>>>>>>>>>>> information ( is possible ). >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The StudyUID is the field used in our pipeline at the >>>>>>>>>>>>>>>>>> moment to check whether a DICOM study was already inserted into LORIS with >>>>>>>>>>>>>>>>>> that same StudyUID since it is supposed to be unique for every single >>>>>>>>>>>>>>>>>> study. Currently, we do not support insertion of DICOM studies if they do >>>>>>>>>>>>>>>>>> not have a StudyUID associated with them. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Is there a specific reason why the StudyUID is blank in >>>>>>>>>>>>>>>>>> the DICOM headers in your DICOM files? Did they go through some processes >>>>>>>>>>>>>>>>>> before upload to LORIS? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 2) When i am about to upload a non-phanton case i need a >>>>>>>>>>>>>>>>>>> 'Visit Label'. Is there already a script that can create a new visit label >>>>>>>>>>>>>>>>>>> for a specific Candidate, so i can provide it as input afterwards to the >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md ) >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I am not sure if the front-end allows for it but if you >>>>>>>>>>>>>>>>>> use batch_uploads_imageuploader, the insertion scripts could create the >>>>>>>>>>>>>>>>>> visit label for you. However, it does need to be included in the patient >>>>>>>>>>>>>>>>>> name in order for the script to know which label it should use. >>>>>>>>>>>>>>>>>> For example: you upload V02 for MTL0123 (DCCID: 456789) >>>>>>>>>>>>>>>>>> but V02 was not yet created for MTL0123; the insertion scripts will create >>>>>>>>>>>>>>>>>> a V02 visit for MTL0123 if the PatientName field for the dataset is >>>>>>>>>>>>>>>>>> MTL0123_456789_V02 with Stage marked as "Not Started". >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> However, *note that for the script to create the visit >>>>>>>>>>>>>>>>>> for the candidate, you have to make sure that all the visit label of your >>>>>>>>>>>>>>>>>> projects were inserted in the Visit_Windows table* >>>>>>>>>>>>>>>>>> (otherwise, the visit will never be created as they were not specified as >>>>>>>>>>>>>>>>>> being part of the list of visit label to expect). So in the example above, >>>>>>>>>>>>>>>>>> you should have one row in Visit_Windows with Visit_label="V02" set. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 3) Section 4.3 >>>>>>>>>>>>>>>>>>> https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> What do we do when the folder contains multiple mnc files? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> This is a case by case. Most of the time, the MINC files >>>>>>>>>>>>>>>>>> that failed insertion into the imaging browser end up in the MRI violation >>>>>>>>>>>>>>>>>> module where you can see what went wrong with the acquisitions not >>>>>>>>>>>>>>>>>> inserted. If you notice that the MINC file should be inserted as a specific >>>>>>>>>>>>>>>>>> protocol, you can force the insertion as explained in section 4.3. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> In your case, I have a feeling that you want to be able >>>>>>>>>>>>>>>>>> to insert MINC files that did not go through the whole pipeline insertion >>>>>>>>>>>>>>>>>> (DICOM archival, then dcm2mnc, then protocol identification, then >>>>>>>>>>>>>>>>>> insertion...). In theory, you could call >>>>>>>>>>>>>>>>>> minc_insertion.pl on any MINC file, provided you have at >>>>>>>>>>>>>>>>>> least the following information: >>>>>>>>>>>>>>>>>> - path to the MINC file >>>>>>>>>>>>>>>>>> - uploadID associated with the MINC files or DICOM >>>>>>>>>>>>>>>>>> archive path from which those MINC files were created from >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Then the minc_insertion.pl script will do the subject >>>>>>>>>>>>>>>>>> information and protocol validation etc... >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> If all the MINC files come from the same uploadID or >>>>>>>>>>>>>>>>>> TarchiveID, then you could run a loop in bash calling the >>>>>>>>>>>>>>>>>> minc_insertion.pl script like this for files deriving >>>>>>>>>>>>>>>>>> from uploadID=1: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> ls /path/to/mnc/folder/* | while read f; do >>>>>>>>>>>>>>>>>> minc_insertion.pl -profile prod -uploadID 1 -mincPath >>>>>>>>>>>>>>>>>> $f; done >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hope this helped! >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.31.35 AM.png Type: image/png Size: 45707 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.32.50 AM.png Type: image/png Size: 134887 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.40.52 AM.png Type: image/png Size: 95814 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 27691 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: output_loris Type: application/octet-stream Size: 5906 bytes Desc: not available URL: From christine.rogers at mcgill.ca Tue Jul 30 13:29:04 2019 From: christine.rogers at mcgill.ca (Christine Rogers, Ms.) Date: Tue, 30 Jul 2019 17:29:04 +0000 Subject: [Loris-dev] Import mri - scripts In-Reply-To: References: Message-ID: Hi Sotirios, Glad to hear you're making progress with the visit labels. To follow up on your last few emails: a- The Wiki's Setup Guide contains Sample insert statements for the Visit Windows table. As Cecile mentioned, the consistency of visit label format is important: V01, v01, V1, 01 -- whichever you select to use is fine; we use V01 by default across all documentation. b- For Password reset, there is a script in the tools/ directory so that a back-end administrator can reset any front-end user password, including Admin. (See info on setting up / resetting User passwords in the Wiki's Setup Guide) c- The minc file not being registered -- due to AcquisitionProtocol unknown. This means the SeriesDescription (Acquisition Protocol) in you Dicom header was not matched to any scan listed in the mri_protocol database table. (In the online Loris-MRI Troubleshooting Guide, this is Table 3, 2nd-last item "no MINCs inserted"). The MRI Violations module (Readme here) will show you the details of this mismatch -- Go to the module (under the Imaging menu), find your scan and click the link "Could not identify scan type" -- in the next subpage you can compare the scan's actual parameters against the mri_protocol table. To resolve this, adjust your mri_protocol table values or perhaps ensure your DICOMs are labelled consistently. Let us know if this works for you -- we can also walk you through an example of non-identified scans on Demo.loris.ca (MRI Violations module). Best, Christine The LORIS team On Tue, Jul 30, 2019 at 10:47 AM Sotirios Nikoloutsopoulos > wrote: I hope this is sufficient as the error for the visit label disappeared. [image.png] Please see the attached file. Also something i found. lorisadmin at hbp:/data/loris/bin/mri$ cat /data/loris/data//logs/TarLoad-17-39-xubBdv.log ==> Loading file from disk /tmp/TarLoad-17-39-UyQcG2/dcc0000_258024_v1_20121205_102108_501e1_mri.mnc --> mapping DICOM parameter for /tmp/TarLoad-17-39-UyQcG2/dcc0000_258024_v1_20121205_102108_501e1_mri.mnc ==> computing md5 hash for MINC body. --> md5: 6b1202fd63e29de4c9be1cd9925888a5 ==> verifying acquisition protocol Acquisition protocol is unknown --> The minc file cannot be registered since the AcquisitionProtocol is unknown ???? ???, 30 ???? 2019 ???? 4:58 ?.?., ?/? Sotirios Nikoloutsopoulos > ??????: Ok I fixed the login problem by using a known hash and by deleting the rows from the login_history i was able to login. ???? ???, 30 ???? 2019 ???? 4:50 ?.?., ?/? Sotirios Nikoloutsopoulos > ??????: Hi, there was no visit label 01 in the Visit_Windows table and i don't know how to insert it because the record requires lots of attributes. Also the system for a reason asked me to update my password and now i don't remember it and my account got suspended. Is there a way to reset it or set a simple password? I updated the password hash in the users table by using the hash from another LORIS setup we had and i still couldn't login ( also the attribute password is always NULL and the expiration date is 1999-01-01 for some reason ). Thanks ???? ???, 29 ???? 2019 ???? 5:21 ?.?., ?/? Cecile Madjar > ??????: Hi Sotirios, Can you try adding the visit label v01 to the Visit_Windows table? However, I noticed that in a previous dataset, the label was V1. Does that refer to the same visit? If so, you might want to harmonize the visit labels so they are all the same across datasets for a given timepoint, meaning you will need to relabel scans when the visit label is different but referring to the same visit label (v01 = V1 etc...). Hope this helps, C?cile On Mon, Jul 29, 2019 at 10:15 AM Sotirios Nikoloutsopoulos > wrote: Hi, Could you help me with the "Visit label does not exist"? I had changed $subjectID{'createVisitLabel'} to 1 as you had suggested but it appears that this problem hasn't been resolved yet. lorisadmin at hbp:/data/loris/bin/mri$ cat /data/loris/data//logs/TarLoad-14-58-9Xog7H.log ==> Loading file from disk /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc --> mapping DICOM parameter for /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc --> using user-defined filterParameters for /tmp/TarLoad-14-58-5Z4hYX/dcc0000_258024_v01_20121205_102108_501e1_mri.mnc Candidate Mismatch Error is Visit label does not exist -> WARNING: This candidate was invalid. Logging to MRICandidateErrors table with reason Visit label does not exist Thank you, Sotirios ???? ???, 26 ???? 2019 ???? 3:02 ?.?., ?/? Sotirios Nikoloutsopoulos > ??????: Hi Cecile, The Center error was fixed. Still the insertion of mincs fails. Please see the output.txt file i attached. Also is there something else i need to change at the paths configuration? Thank you, Sotirios ???? ???, 25 ???? 2019 ???? 6:55 ?.?., ?/? Cecile Madjar > ??????: Hi Sotirios, We are making progress! I have a feeling this error comes from the fact that your psc table does not have the MRI_alias populated for the DCC site. Running the following query in the mysql database should fix this: UPDATE psc SET MRI_alias="DCC", Alias="DCC" WHERE Name="Data Coordinating Center"; Repeat the same with all your sites (make sure to change the Aliases between sites, they should all be unique). Let me know how it goes! Best, C?cile On Thu, Jul 25, 2019 at 11:43 AM Sotirios Nikoloutsopoulos > wrote: Dear Cecile, You were right about your guesses. I did the modifications you suggested, i still get the "no mincs inserted" error, now i am getting the "No center found this candidate" error ( how do i fix that? from what i see the .dcm files contains a 'Institution Name', is that the center? ), so i guess that's why the validation fails, but i changed the force variable of tarchiveLoader to 1, shouldn't it have inserted the mincs then? Also it is optional to have a mail server, right? ( from what i see the mail error does not affect the $valid_study, so i assume we are okay with that ) Thank you a lot, Sotirios ???? ???, 25 ???? 2019 ???? 5:48 ?.?., ?/? Cecile Madjar > ??????: Hi Sotirios, From what I understand, I believe you just need to update a few config settings from the frontend in the Config module (under the Admin menu). [Screen Shot 2019-07-25 at 10.31.35 AM.png] Then go to the Paths section of the config module and update the "LORIS-MRI code" config setting to /data/loris/bin/mri (my guess is that currently, it is set to /data/LORIS/data) and click on the submit button. [Screen Shot 2019-07-25 at 10.32.50 AM.png] Then, go to Imaging Pipeline section of the Configuration module and update the "Loris-MRI Data Directory" to /data/loris/data/ (my guess is that it is empty at the moment in your database which is why $data_dir is not defined when you run the tarchiveLoader step of the insertion pipeline). [Screen Shot 2019-07-25 at 10.40.52 AM.png] Finally, I took a look at your environment file and it might be a good idea to update your PERL5LIB variable to the following (it has been added to the code a few releases ago as we realized this was needed for the libraries. Not sure which version of LORIS-MRI you are using but it does not hurt to update this variable to what I suggest below): export PERL5LIB=/data/loris/bin/mri/uploadNeuroDB:/data/loris/bin/mri/dicom-archive:$PERL5LIB Hope this helps. Sorry you are experiencing so much trouble setting things up. We probably need to update our WIKI. Best, C?cile On Thu, Jul 25, 2019 at 9:48 AM Sotirios Nikoloutsopoulos > wrote: The "Loris-MRI Data Directory" should be in the environment file? ???? ???, 25 ???? 2019 ???? 12:59 ?.?., ?/? Sotirios Nikoloutsopoulos > ??????: The scripts are located in /data/loris/bin/mri/. Somewhere I might have messed something up but i remember following the instructions in the wiki.For example when i tried to execute the batch_uploads_imageuploader perl script the line my $command = "$bin_dir/uploadNeuroDB/imaging_upload_file.pl " was pointing to an invalid path, for example: it was repeating a pattern and had LORIS instead of loris ( i think the mistake was that in the wiki it suggested to use $PROJECT for the name of the folder and i choosed 'loris' https://github.com/aces/Loris/wiki/Imaging-Database ). See also the in image for the modification i did. Also i didn't find the section to config the $data_dir variable. Thank you, Sotirios ???? ???, 24 ???? 2019 ???? 6:05 ?.?., ?/? Cecile Madjar > ??????: Hi Sotirios, It looks like you have one Config setting not set (based on the first error message you got when running it the first time). You need to set a value for "Loris-MRI Data Directory" in the Config module under the Imaging pipeline section (from what I can see, it would be /data/loris/data/ for your setup). The reason you got that error message the second time is that the pipeline checks if the DICOM study was already inserted in the tarchive table. Since you ran the study once already, the study is inserted into the tarchive, hence the message. However, the pipeline did not complete as the MINC files were not created and inserted into the files/parameter_file table. In order to continue the insertion, you will then need to run the tarchiveLoader command suggested in the terminal during the second run: /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar -verbose Finally, I am a bit confused by your setup. Where are the scripts located? In /data/loris/bin/mri or /data/loris/data? It feels like the Config setting "LORIS-MRI code" in the Paths section was set to "/data/LORIS/data" but it should have been set to the directory where the scripts are located (most probably /data/loris/bin/mri). Hope this helps, C?cile On Wed, Jul 24, 2019 at 10:55 AM Sotirios Nikoloutsopoulos > wrote: Ok first i execute this command: "lorisadmin at hbp:/data/loris/bin/mri$ ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > log.txt Use of uninitialized value $data_dir in concatenation (.) or string at ./batch_uploads_imageuploader line 143. Use of uninitialized value $data_dir in concatenation (.) or string at ./batch_uploads_imageuploader line 143. Use of uninitialized value $_ in pattern match (m//) at ./batch_uploads_imageuploader line 147. base is DCC0000_258024_V01 path is /data/incoming/ type is .tar.gz fullpath is /data/incoming/DCC0000_258024_V01.tar.gz source /tmp/ImagingUpload-17-43-ZUEFjb targetlocation /data/loris/data/tarchive Use of uninitialized value $data_dir in concatenation (.) or string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 256. Use of uninitialized value $data_dir in concatenation (.) or string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 284. Use of uninitialized value $data_dir in concatenation (.) or string at /data/loris/bin/mri/uploadNeuroDB/tarchive_validation.pl line 219. Use of uninitialized value $data_dir in concatenation (.) or string at /data/loris/bin/mri/uploadNeuroDB/minc_insertion.pl line 270. Use of uninitialized value $data_dir in concatenation (.) or string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 711. Can't exec "mail": No such file or directory at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 721. print() on closed filehandle MAIL at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 722. print() on closed filehandle MAIL at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 723. Use of uninitialized value $data_dir in concatenation (.) or string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. print() on closed filehandle MAIL at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 724. print() on closed filehandle MAIL at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 725. Use of uninitialized value $data_dir in concatenation (.) or string at /data/LORIS/data//uploadNeuroDB/tarchiveLoader line 735. No Mincs inserted The tarchiveLoader insertion script has failed. Running now the following command: /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile prod -upload_id 3 /data/incoming/DCC0000_258024_V01.tar.gz -verbose" I inserted the VisitLabel, but still the Mincs are not inserted and i am getting this error "lorisadmin at hbp:/data/loris/bin/mri$ /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile prod -upload_id 3 /data/incoming/DCC0000_258024_V01.tar.gz -verbose find -path \/tmp\/ImagingUpload\-17\-45\-5MsWEd -name '__MACOSX' -delete Spool message is: The Scan for the uploadID 3 has already been run with tarchiveID: 7. To continue with the rest of the insertion pipeline, please run tarchiveLoader from a terminal as follows: /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod /data/loris/data/tarchive//2012/DCM_2012-12-05_ImagingUpload-17-43-ZUEFjb.tar -verbose" Which files should i delete? I thought that deleting the files from the tarchive was enough. And i executed the last proposed command to continue to rest of the insertion pipeline ( check the image ). Thank you ???? ???, 24 ???? 2019 ???? 5:05 ?.?., ?/? Cecile Madjar > ??????: Hi Sotirios, Sorry I was not clear before. Every visits of candidates are created in the session table with one entry per CandID/VisitLabel. So if the imaging pipeline created visits in the session table (that are not attached to instruments) and you want to clean your database completely, you can delete entries in that table as well. Regarding your point 2, the imaging pipeline can create the visit in the session table as long as the visit label is present in the Visit_Windows table (which means the visit label stored in the PatientName is a valid visit label). However, in your prod file, if the variable $subjectID{'createVisitLabel'} is set to 1 for candidates, the visit label should be created in the Visit_Windows table I believe. Here is the line of the prod file you would need to change. In general, we prefer populating the Visit_Windows table and set the $subjectID{'createVisitLabel'} to 0 for candidate data to avoid insertion of badly labelled MRI data that are complex to relabel. Finally, regarding the candidate creation, if you execute the following steps, the pipeline should be able to create candidates: - set in the Imaging Pipeline section of the config module the config setting "Upload creation of candidates" to Yes - manually transfer the scans to the LORIS server instead of uploading it via the imaging browser - run batch_imaging_upload.pl as you did until now. Hope this helps! Best, C?cile On Wed, Jul 24, 2019 at 8:47 AM Sotirios Nikoloutsopoulos > wrote: About my second question i replaced the name of the patient in the .dcm files and now i am getting an error that the visit label does not exist. I thought it was supposed to be automatically created. "Done adding archive info into database /data/LORIS/data//uploadNeuroDB/tarchiveLoader -globLocation -profile prod /data/loris/data/tarchive//DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar -verbose md5sum /data/loris/data/tarchive/DCM_2012-12-05_ImagingUpload-15-42-yUwQQV.tar PSCID is: DCC0000 CandID id: 258024 visit_label is: V1 PSCID is: DCC0000 CandID id: 258024 visit_label is: V1 candidate id 258024 => No Visit labelVisit label does not exist Set centerID = 1 PSCID is: DCC0000 CandID id: 258024 visit_label is: V1 PSCID is: DCC0000 CandID id: 258024 visit_label is: V1 Number of MINC files that will be considered for inserting into the database: 1 log dir is /logs and log file is /logs/TarLoad-15-42-L6pocU.log PSCID is: DCC0000 CandID id: 258024 visit_label is: V1 PSCID is: DCC0000 CandID id: 258024 visit_label is: V1 candidate id 258024 => No Visit label Cleaning up temp files: rm -rf /tmp/TarLoad-15-42-J2rQms/ImagingUpload-15-42-yUwQQV*" ???? ???, 24 ???? 2019 ???? 3:38 ?.?., ?/? Sotirios Nikoloutsopoulos > ??????: Hi, 1) What are the records in the session table? 2) I tried to upload a non-phantom file like this input_file.txt: /data/incoming/DCC0000_258024_V1.tar.gz N DCC0000_258024_V1 Terminal: ./batch_uploads_imageuploader -profile prod < ~/Desktop/input.txt > log.txt and i get the following error in the log.txt file "Running now the following command: /data/loris/data/uploadNeuroDB/imaging_upload_file.pl -profile prod -upload_id 7 /data/incoming/DCC0000_258024_V1.tar.gz -verbose find -path \/tmp\/ImagingUpload\-14\-56\-FeLCB4 -name '__MACOSX' -delete Spool message is: The PatientName read from the DICOM header does not start with DCC0000_258024_V1 from the mri_upload table " and the spool message is repeated for each .dcm file. Should i preprocess every .dcm file and change the PatientName to "DCC0000_258024_V1"? 3) As far as it concerns the candidate profile i have to create a new one each time before i try to upload a file? Is there already a way to create a candidate profile and get the DCCID and PSCID without the user interface? Thank you, Sotirios ???? ???, 18 ???? 2019 ???? 5:00 ?.?., ?/? Cecile Madjar > ??????: Hi Sotirios, See answers to your questions below. About the 1st case we had a dicom with a blank studyUID, maybe someone accidentally removed it. That is strange. Something must have been done to the DICOM files. I have never seen a study with no StudyUID out of the scanner. Usually, you need to manually erase it with some tool. However, DICOMs never cease to surprise me... Some other questions: 1) What are the steps to remove all of my patients and uploads?: because i want to try to reupload all the testing dicoms we had with a clean database. Release 21.0 of LORIS-MRI will offer a script to remove all the entries and files specific to an upload. However, this release is not yet out. Hopefully in the next few weeks. In the meantime, since your case is particular and that you just want to start with a clean database, the following deletes should work (hopefully I won't forget any tables): delete from mri_violations_log; delete from mri_protocol_violated_scans; delete from MRICandidateErrors; delete from parameter_file; delete from files_qcstatus; # (if you played with the QC part of the imaging browser for testing) delete from feedback_mri_comments; # (if you played with the QC part of the imaging browser for testing) delete from tarchive_series; delete from tarchive_files; delete from mri_upload; delete from tarchive; delete from mri_scanner; delete from candidate where PSCID="scanner"; I don't know if you want to also delete entries in the session and candidate table too? 2) We do not need to create the profile of a patient ( are the patients stored in the candidate table? because i can't see the ghosts ), because will it automatically be created when importing a dicom? I am not sure I understand fully this question. All candidates are stored in the candidate table. The phantom scans however are attached to a scanner candidate depending on where the scan happened. It is a bit of a weird concept that we have to redesign eventually but never got a chance to get to it. Hopefully this answered your question, otherwise, don't hesitate to let me know. 3) What is the difference between uploading a dicom as phantom instead of importing as PSCCID_DCCID_VisitLabel? When you upload the scan as PSCID_DCCID_VisitLabel, there is a check that makes sure the candidate IDs and visit label are valid at the time of upload. When you upload a scan as phantom, it expects that the DICOM field PatientName and uploaded filename contains the string "phantom". We enforced this behaviour on the imaging uploader side in recent releases but I can't remember which one. Probably the upcoming 21.0 release. All those verifications might seem cumbersome but they are here to ensure that the files inserted are all valid and labelled properly as it is a bit messy to have to delete files that were wrongly labelled. At least, the delete script present in release 21.0 will make this process easier but it is still good practice to verify those things. Thank you, With pleasure. Hopefully the answers to your questions will be helpful. Best, C?cile PS: loris-dev mailing list: you will find the initial answer I gave Sotirios below. I forgot to cc the loris-dev in my earlier reply... ???? ???, 16 ???? 2019 ???? 12:02 ?.?., ?/? Cecile Madjar > ??????: Dear Sotirios, Thank you for reaching out. Since your email was already organized in points, I will reply directly below your questions below. Best, C?cile 1) What do we do when the StudyUID is blank on the header? https://github.com/aces/Loris-MRI/blob/minor/docs/AppendixA-Troubleshooting_guideline.md There solution provided at the table 3 does not guide us how to find the information ( is possible ). The StudyUID is the field used in our pipeline at the moment to check whether a DICOM study was already inserted into LORIS with that same StudyUID since it is supposed to be unique for every single study. Currently, we do not support insertion of DICOM studies if they do not have a StudyUID associated with them. Is there a specific reason why the StudyUID is blank in the DICOM headers in your DICOM files? Did they go through some processes before upload to LORIS? 2) When i am about to upload a non-phanton case i need a 'Visit Label'. Is there already a script that can create a new visit label for a specific Candidate, so i can provide it as input afterwards to the batch_uploads_imageuploader ( See option 5.1.3 https://github.com/aces/Loris-MRI/blob/minor/docs/05-PipelineLaunchOptions.md ) I am not sure if the front-end allows for it but if you use batch_uploads_imageuploader, the insertion scripts could create the visit label for you. However, it does need to be included in the patient name in order for the script to know which label it should use. For example: you upload V02 for MTL0123 (DCCID: 456789) but V02 was not yet created for MTL0123; the insertion scripts will create a V02 visit for MTL0123 if the PatientName field for the dataset is MTL0123_456789_V02 with Stage marked as "Not Started". However, note that for the script to create the visit for the candidate, you have to make sure that all the visit label of your projects were inserted in the Visit_Windows table (otherwise, the visit will never be created as they were not specified as being part of the list of visit label to expect). So in the example above, you should have one row in Visit_Windows with Visit_label="V02" set. 3) Section 4.3 https://github.com/aces/Loris-MRI/blob/minor/docs/04-Scripts.md What do we do when the folder contains multiple mnc files? This is a case by case. Most of the time, the MINC files that failed insertion into the imaging browser end up in the MRI violation module where you can see what went wrong with the acquisitions not inserted. If you notice that the MINC file should be inserted as a specific protocol, you can force the insertion as explained in section 4.3. In your case, I have a feeling that you want to be able to insert MINC files that did not go through the whole pipeline insertion (DICOM archival, then dcm2mnc, then protocol identification, then insertion...). In theory, you could call minc_insertion.pl on any MINC file, provided you have at least the following information: - path to the MINC file - uploadID associated with the MINC files or DICOM archive path from which those MINC files were created from Then the minc_insertion.pl script will do the subject information and protocol validation etc... If all the MINC files come from the same uploadID or TarchiveID, then you could run a loop in bash calling the minc_insertion.pl script like this for files deriving from uploadID=1: ls /path/to/mnc/folder/* | while read f; do minc_insertion.pl -profile prod -uploadID 1 -mincPath $f; done Thank you, Hope this helped! _______________________________________________ Loris-dev mailing list Loris-dev at bic.mni.mcgill.ca https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev -- christine.rogers at mcgill.ca McGill Centre for Integrative Neuroscience | MCIN.ca Montreal Neurological Institute McGill University | Montreal | Canada -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.31.35 AM.png Type: image/png Size: 45707 bytes Desc: Screen Shot 2019-07-25 at 10.31.35 AM.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.32.50 AM.png Type: image/png Size: 134887 bytes Desc: Screen Shot 2019-07-25 at 10.32.50 AM.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Screen Shot 2019-07-25 at 10.40.52 AM.png Type: image/png Size: 95814 bytes Desc: Screen Shot 2019-07-25 at 10.40.52 AM.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 27691 bytes Desc: image.png URL: