From AMORALESPINZON at bwh.harvard.edu Mon Apr 6 17:44:09 2020 From: AMORALESPINZON at bwh.harvard.edu (Morales Pinzon, Alfredo) Date: Mon, 6 Apr 2020 21:44:09 +0000 Subject: [Loris-dev] Errors when using delete_imaging_upload.pl Message-ID: Dear LorisDev team, I successfully deleted 90% of the MRIs for a study using the script ?delete_imaging_upload.pl?. Now I am dealing with the 10% missing. Here are the 4 types errors that I am getting, could you help me figure out how to delete these entries: Error type 1: === Error! File /ABC/data/pic/979907/IPMSA_979907_m00_NA_001_182_check.jpg exists in the database but was not found on the file system. Aborting. === Error type 2: === Error! File /ABC/data/assembly/677803/m12/mri/native/IPMSA_677803_m12_NA_002.nii exists in the database but was not found on the file system. Aborting. === Error type 3: === /ABC/data/pic/819059/IPMSA_819059_m12_NA_006_3098_check.jpg /ABC/data/pic/819059/IPMSA_819059_m12_NA_007_3099_check.jpg /ABC/data/pic/819059/IPMSA_819059_m12_NA_008_3100_check.jpg /ABCdata/pic/819059/IPMSA_819059_m12_NA_009_3101_check.jpg /ABC/data/pic/819059/IPMSA_819059_m12_t1_001_3091_check.jpg /ABC/data/pic/819059/IPMSA_819059_m12_t1_002_3092_check.jpg /ABC/data/tarchive/2009/DCM_2009-12-10_ImagingUpload-7-57-r0hk9h.tar DBD::mysql::db do failed: Cannot delete or update a parent row: a foreign key constraint fails (`IPMSA_LORIS`.`mri_upload`, CONSTRAINT `FK_mriupload_TarchiveID` FOREIGN KEY (`TarchiveID`) REFERENCES `tarchive` (`TarchiveID`)) at /data_/ipmsa/loris_data/IPMSA/bin/mri/tools//delete_imaging_upload.pl line 1612. Issuing rollback() due to DESTROY without explicit disconnect() of DBD::mysql::db handle database=IPMSA_LORIS;host=132.216.133.50;port=3306; at /data_/ipmsa/loris_data/IPMSA/bin/mri/tools//delete_imaging_upload.pl line 1612. === Error type 4: === Cannot delete upload 1543: the MRI pipeline is currently processing it. === Looking forward to hearing from you. Regards, Alfredo. From cecile.madjar at mcin.ca Mon Apr 6 18:02:13 2020 From: cecile.madjar at mcin.ca (Cecile Madjar) Date: Mon, 6 Apr 2020 18:02:13 -0400 Subject: [Loris-dev] Errors when using delete_imaging_upload.pl In-Reply-To: References: Message-ID: Hi Alfredo, See answers below in blue. Error type 1: > === > Error! File /ABC/data/pic/979907/IPMSA_979907_m00_NA_001_182_check.jpg > exists in the database but was not found on the file system. > Aborting. > === > > Error type 2: > === > Error! File > /ABC/data/assembly/677803/m12/mri/native/IPMSA_677803_m12_NA_002.nii exists > in the database but was not found on the file system. > Aborting. > === > The first two errors are the same. The delete script always checks first if the file is on the filesystem. My guess is that when you ran the pipeline, the path to the NIfTI file was inserted into the database even when the NIfTI file was not created due to some mnc2nii conversion errors (known bug that was fixed in release 22.0). You can use the option -ignore to ignore the check between database and file system. > > Error type 3: > === > /ABC/data/pic/819059/IPMSA_819059_m12_NA_006_3098_check.jpg > /ABC/data/pic/819059/IPMSA_819059_m12_NA_007_3099_check.jpg > /ABC/data/pic/819059/IPMSA_819059_m12_NA_008_3100_check.jpg > /ABCdata/pic/819059/IPMSA_819059_m12_NA_009_3101_check.jpg > /ABC/data/pic/819059/IPMSA_819059_m12_t1_001_3091_check.jpg > /ABC/data/pic/819059/IPMSA_819059_m12_t1_002_3092_check.jpg > /ABC/data/tarchive/2009/DCM_2009-12-10_ImagingUpload-7-57-r0hk9h.tar > > DBD::mysql::db do failed: Cannot delete or update a parent row: a foreign > key constraint fails (`IPMSA_LORIS`.`mri_upload`, CONSTRAINT > `FK_mriupload_TarchiveID` FOREIGN KEY (`TarchiveID`) REFERENCES `tarchive` > (`TarchiveID`)) at /data_/ipmsa/loris_data/IPMSA/bin/mri/tools// > delete_imaging_upload.pl line 1612. > Issuing rollback() due to DESTROY without explicit disconnect() of > DBD::mysql::db handle database=IPMSA_LORIS;host=132.216.133.50;port=3306; > at /data_/ipmsa/loris_data/IPMSA/bin/mri/tools//delete_imaging_upload.pl > line 1612. > === > Not sure what is going on here. Could you point me to the file so I can check where the issue comes from? Or the release version you are using. Are there any processed output data for that upload ID? > > Error type 4: > === > Cannot delete upload 1543: the MRI pipeline is currently processing it. > === > If you are sure you are not running the pipeline, you will need to update the mri_upload table to reflect this. For some obscure reason, the insertion status fields of the mri_upload table were not updated for that UploadID. Maybe the pipeline got interrupted? UPDATE mri_upload SET Inserting=0, InsertionComplete=1 WHERE UploadID=1543; Then, rerun the delete script on that UploadID. Hope this helps. Best, C?cile > Looking forward to hearing from you. > > Regards, > Alfredo. > > _______________________________________________ > Loris-dev mailing list > Loris-dev at bic.mni.mcgill.ca > https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lingyun.ma at mcin.ca Mon Apr 6 18:09:47 2020 From: lingyun.ma at mcin.ca (Ling Ma) Date: Mon, 6 Apr 2020 18:09:47 -0400 Subject: [Loris-dev] Errors when using delete_imaging_upload.pl In-Reply-To: References: Message-ID: Hi Alfredo and Cecile The third question comes most likely from the fact the delete_imaging_upload.pl tries to delete from image_upload table, which references some record in the archive table. You need to check the related record in your DB to see what that record was referring to, then delete the record from archive if you are sure what you are doing. Possibly this is a bug of the delete_imaging_upload.pl, also possibly this bug has been fixed in the more recent version. Cecile knows more about that. Best regards, Ling Ma On Mon, Apr 6, 2020 at 6:02 PM Cecile Madjar wrote: > Hi Alfredo, > > See answers below in blue. > > Error type 1: >> === >> Error! File /ABC/data/pic/979907/IPMSA_979907_m00_NA_001_182_check.jpg >> exists in the database but was not found on the file system. >> Aborting. >> === >> >> Error type 2: >> === >> Error! File >> /ABC/data/assembly/677803/m12/mri/native/IPMSA_677803_m12_NA_002.nii exists >> in the database but was not found on the file system. >> Aborting. >> === >> > > The first two errors are the same. The delete script always checks first > if the file is on the filesystem. My guess is that when you ran the > pipeline, the path to the NIfTI file was inserted into the database even > when the NIfTI file was not created due to some mnc2nii conversion errors > (known bug that was fixed in release 22.0). You can use the option -ignore > to ignore the check between database and file system. > > >> >> Error type 3: >> === >> /ABC/data/pic/819059/IPMSA_819059_m12_NA_006_3098_check.jpg >> /ABC/data/pic/819059/IPMSA_819059_m12_NA_007_3099_check.jpg >> /ABC/data/pic/819059/IPMSA_819059_m12_NA_008_3100_check.jpg >> /ABCdata/pic/819059/IPMSA_819059_m12_NA_009_3101_check.jpg >> /ABC/data/pic/819059/IPMSA_819059_m12_t1_001_3091_check.jpg >> /ABC/data/pic/819059/IPMSA_819059_m12_t1_002_3092_check.jpg >> /ABC/data/tarchive/2009/DCM_2009-12-10_ImagingUpload-7-57-r0hk9h.tar >> >> DBD::mysql::db do failed: Cannot delete or update a parent row: a foreign >> key constraint fails (`IPMSA_LORIS`.`mri_upload`, CONSTRAINT >> `FK_mriupload_TarchiveID` FOREIGN KEY (`TarchiveID`) REFERENCES `tarchive` >> (`TarchiveID`)) at /data_/ipmsa/loris_data/IPMSA/bin/mri/tools// >> delete_imaging_upload.pl line 1612. >> Issuing rollback() due to DESTROY without explicit disconnect() of >> DBD::mysql::db handle database=IPMSA_LORIS;host=132.216.133.50;port=3306; >> at /data_/ipmsa/loris_data/IPMSA/bin/mri/tools//delete_imaging_upload.pl >> line 1612. >> === >> > > Not sure what is going on here. Could you point me to the file so I can > check where the issue comes from? Or the release version you are using. Are > there any processed output data for that upload ID? > > >> >> Error type 4: >> === >> Cannot delete upload 1543: the MRI pipeline is currently processing it. >> === >> > > If you are sure you are not running the pipeline, you will need to update > the mri_upload table to reflect this. For some obscure reason, the > insertion status fields of the mri_upload table were not updated for that > UploadID. Maybe the pipeline got interrupted? > UPDATE mri_upload SET Inserting=0, InsertionComplete=1 WHERE UploadID=1543; > Then, rerun the delete script on that UploadID. > > Hope this helps. Best, > > C?cile > > >> Looking forward to hearing from you. >> >> Regards, >> Alfredo. >> >> _______________________________________________ >> Loris-dev mailing list >> Loris-dev at bic.mni.mcgill.ca >> https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev >> > _______________________________________________ > Loris-dev mailing list > Loris-dev at bic.mni.mcgill.ca > https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lingyun.ma at mcin.ca Mon Apr 6 18:14:32 2020 From: lingyun.ma at mcin.ca (Ling Ma) Date: Mon, 6 Apr 2020 18:14:32 -0400 Subject: [Loris-dev] Errors when using delete_imaging_upload.pl In-Reply-To: References: Message-ID: *tarchive*, not archive, the email automatically corrected my writing. On Mon, Apr 6, 2020 at 6:09 PM Ling Ma wrote: > Hi Alfredo and Cecile > > The third question comes most likely from the fact the > delete_imaging_upload.pl tries to delete from image_upload table, which > references some record in the archive table. You need to check the related > record in your DB to see what that record was referring to, then delete the > record from archive if you are sure what you are doing. Possibly this is a > bug of the delete_imaging_upload.pl, also possibly this bug has been > fixed in the more recent version. Cecile knows more about that. > > Best regards, > Ling Ma > > On Mon, Apr 6, 2020 at 6:02 PM Cecile Madjar > wrote: > >> Hi Alfredo, >> >> See answers below in blue. >> >> Error type 1: >>> === >>> Error! File /ABC/data/pic/979907/IPMSA_979907_m00_NA_001_182_check.jpg >>> exists in the database but was not found on the file system. >>> Aborting. >>> === >>> >>> Error type 2: >>> === >>> Error! File >>> /ABC/data/assembly/677803/m12/mri/native/IPMSA_677803_m12_NA_002.nii exists >>> in the database but was not found on the file system. >>> Aborting. >>> === >>> >> >> The first two errors are the same. The delete script always checks first >> if the file is on the filesystem. My guess is that when you ran the >> pipeline, the path to the NIfTI file was inserted into the database even >> when the NIfTI file was not created due to some mnc2nii conversion errors >> (known bug that was fixed in release 22.0). You can use the option -ignore >> to ignore the check between database and file system. >> >> >>> >>> Error type 3: >>> === >>> /ABC/data/pic/819059/IPMSA_819059_m12_NA_006_3098_check.jpg >>> /ABC/data/pic/819059/IPMSA_819059_m12_NA_007_3099_check.jpg >>> /ABC/data/pic/819059/IPMSA_819059_m12_NA_008_3100_check.jpg >>> /ABCdata/pic/819059/IPMSA_819059_m12_NA_009_3101_check.jpg >>> /ABC/data/pic/819059/IPMSA_819059_m12_t1_001_3091_check.jpg >>> /ABC/data/pic/819059/IPMSA_819059_m12_t1_002_3092_check.jpg >>> /ABC/data/tarchive/2009/DCM_2009-12-10_ImagingUpload-7-57-r0hk9h.tar >>> >>> DBD::mysql::db do failed: Cannot delete or update a parent row: a >>> foreign key constraint fails (`IPMSA_LORIS`.`mri_upload`, CONSTRAINT >>> `FK_mriupload_TarchiveID` FOREIGN KEY (`TarchiveID`) REFERENCES `tarchive` >>> (`TarchiveID`)) at /data_/ipmsa/loris_data/IPMSA/bin/mri/tools// >>> delete_imaging_upload.pl line 1612. >>> Issuing rollback() due to DESTROY without explicit disconnect() of >>> DBD::mysql::db handle database=IPMSA_LORIS;host=132.216.133.50;port=3306; >>> at /data_/ipmsa/loris_data/IPMSA/bin/mri/tools//delete_imaging_upload.pl >>> line 1612. >>> === >>> >> >> Not sure what is going on here. Could you point me to the file so I can >> check where the issue comes from? Or the release version you are using. Are >> there any processed output data for that upload ID? >> >> >>> >>> Error type 4: >>> === >>> Cannot delete upload 1543: the MRI pipeline is currently processing it. >>> === >>> >> >> If you are sure you are not running the pipeline, you will need to update >> the mri_upload table to reflect this. For some obscure reason, the >> insertion status fields of the mri_upload table were not updated for that >> UploadID. Maybe the pipeline got interrupted? >> UPDATE mri_upload SET Inserting=0, InsertionComplete=1 WHERE >> UploadID=1543; >> Then, rerun the delete script on that UploadID. >> >> Hope this helps. Best, >> >> C?cile >> >> >>> Looking forward to hearing from you. >>> >>> Regards, >>> Alfredo. >>> >>> _______________________________________________ >>> Loris-dev mailing list >>> Loris-dev at bic.mni.mcgill.ca >>> https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev >>> >> _______________________________________________ >> Loris-dev mailing list >> Loris-dev at bic.mni.mcgill.ca >> https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lingyun.ma at mcin.ca Mon Apr 6 18:48:24 2020 From: lingyun.ma at mcin.ca (Ling Ma) Date: Mon, 6 Apr 2020 18:48:24 -0400 Subject: [Loris-dev] Errors when using delete_imaging_upload.pl In-Reply-To: References: Message-ID: Hi Alfredo and Cecile, Sorry for the erroneous information provided, I just checked, the program execution order seems to be correct, the order should be first to delete from image_upload, then tarchive table record. You need to check data to see what happened, possibly the first deletion was not successful in image_upload then the second deletion attempt broke out. Ling Ma On Mon, Apr 6, 2020 at 6:14 PM Ling Ma wrote: > *tarchive*, not archive, the email automatically corrected my writing. > > On Mon, Apr 6, 2020 at 6:09 PM Ling Ma wrote: > >> Hi Alfredo and Cecile >> >> The third question comes most likely from the fact the >> delete_imaging_upload.pl tries to delete from image_upload table, which >> references some record in the archive table. You need to check the related >> record in your DB to see what that record was referring to, then delete the >> record from archive if you are sure what you are doing. Possibly this is a >> bug of the delete_imaging_upload.pl, also possibly this bug has been >> fixed in the more recent version. Cecile knows more about that. >> >> Best regards, >> Ling Ma >> >> On Mon, Apr 6, 2020 at 6:02 PM Cecile Madjar >> wrote: >> >>> Hi Alfredo, >>> >>> See answers below in blue. >>> >>> Error type 1: >>>> === >>>> Error! File /ABC/data/pic/979907/IPMSA_979907_m00_NA_001_182_check.jpg >>>> exists in the database but was not found on the file system. >>>> Aborting. >>>> === >>>> >>>> Error type 2: >>>> === >>>> Error! File >>>> /ABC/data/assembly/677803/m12/mri/native/IPMSA_677803_m12_NA_002.nii exists >>>> in the database but was not found on the file system. >>>> Aborting. >>>> === >>>> >>> >>> The first two errors are the same. The delete script always checks first >>> if the file is on the filesystem. My guess is that when you ran the >>> pipeline, the path to the NIfTI file was inserted into the database even >>> when the NIfTI file was not created due to some mnc2nii conversion errors >>> (known bug that was fixed in release 22.0). You can use the option -ignore >>> to ignore the check between database and file system. >>> >>> >>>> >>>> Error type 3: >>>> === >>>> /ABC/data/pic/819059/IPMSA_819059_m12_NA_006_3098_check.jpg >>>> /ABC/data/pic/819059/IPMSA_819059_m12_NA_007_3099_check.jpg >>>> /ABC/data/pic/819059/IPMSA_819059_m12_NA_008_3100_check.jpg >>>> /ABCdata/pic/819059/IPMSA_819059_m12_NA_009_3101_check.jpg >>>> /ABC/data/pic/819059/IPMSA_819059_m12_t1_001_3091_check.jpg >>>> /ABC/data/pic/819059/IPMSA_819059_m12_t1_002_3092_check.jpg >>>> /ABC/data/tarchive/2009/DCM_2009-12-10_ImagingUpload-7-57-r0hk9h.tar >>>> >>>> DBD::mysql::db do failed: Cannot delete or update a parent row: a >>>> foreign key constraint fails (`IPMSA_LORIS`.`mri_upload`, CONSTRAINT >>>> `FK_mriupload_TarchiveID` FOREIGN KEY (`TarchiveID`) REFERENCES `tarchive` >>>> (`TarchiveID`)) at /data_/ipmsa/loris_data/IPMSA/bin/mri/tools// >>>> delete_imaging_upload.pl line 1612. >>>> Issuing rollback() due to DESTROY without explicit disconnect() of >>>> DBD::mysql::db handle database=IPMSA_LORIS;host=132.216.133.50;port=3306; >>>> at /data_/ipmsa/loris_data/IPMSA/bin/mri/tools// >>>> delete_imaging_upload.pl line 1612. >>>> === >>>> >>> >>> Not sure what is going on here. Could you point me to the file so I can >>> check where the issue comes from? Or the release version you are using. Are >>> there any processed output data for that upload ID? >>> >>> >>>> >>>> Error type 4: >>>> === >>>> Cannot delete upload 1543: the MRI pipeline is currently processing it. >>>> === >>>> >>> >>> If you are sure you are not running the pipeline, you will need to >>> update the mri_upload table to reflect this. For some obscure reason, the >>> insertion status fields of the mri_upload table were not updated for that >>> UploadID. Maybe the pipeline got interrupted? >>> UPDATE mri_upload SET Inserting=0, InsertionComplete=1 WHERE >>> UploadID=1543; >>> Then, rerun the delete script on that UploadID. >>> >>> Hope this helps. Best, >>> >>> C?cile >>> >>> >>>> Looking forward to hearing from you. >>>> >>>> Regards, >>>> Alfredo. >>>> >>>> _______________________________________________ >>>> Loris-dev mailing list >>>> Loris-dev at bic.mni.mcgill.ca >>>> https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev >>>> >>> _______________________________________________ >>> Loris-dev mailing list >>> Loris-dev at bic.mni.mcgill.ca >>> https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolasbrossard.mni at gmail.com Tue Apr 7 08:06:41 2020 From: nicolasbrossard.mni at gmail.com (Nicolas Brossard) Date: Tue, 7 Apr 2020 08:06:41 -0400 Subject: [Loris-dev] Errors when using delete_imaging_upload.pl In-Reply-To: References: Message-ID: Hi all, Regarding the foreign key constraint fail error, my guess is that you have two records in mri_uploads that are tied to the same archive and you are trying to delete one of them. What I am saying is that you have two uploads, say U1 and U2, which have the same tarchive ID, say A1. If you try to delete U1, the script will try to delete everything associated to it, including archive A1, which will fail because of the foreign key constraint still tying U2 to A1. This is unfortunately a bug of the delete script. You should be able to delete *all* the uploads that have the same tarchive ID though, but you'd have to do it in the same script invocation. So back to our example, you could delete U1 and U2 but you'd have to delete them at the same time by supplying both upload IDs in a comma-separated list via the -uploadID switch (see the perldoc for the delete script). I hope this makes sense. Let me know if I you need more information. Regards, Nicolas On Mon, Apr 6, 2020 at 6:02 PM Cecile Madjar wrote: > Hi Alfredo, > > See answers below in blue. > > Error type 1: >> === >> Error! File /ABC/data/pic/979907/IPMSA_979907_m00_NA_001_182_check.jpg >> exists in the database but was not found on the file system. >> Aborting. >> === >> >> Error type 2: >> === >> Error! File >> /ABC/data/assembly/677803/m12/mri/native/IPMSA_677803_m12_NA_002.nii exists >> in the database but was not found on the file system. >> Aborting. >> === >> > > The first two errors are the same. The delete script always checks first > if the file is on the filesystem. My guess is that when you ran the > pipeline, the path to the NIfTI file was inserted into the database even > when the NIfTI file was not created due to some mnc2nii conversion errors > (known bug that was fixed in release 22.0). You can use the option -ignore > to ignore the check between database and file system. > > >> >> Error type 3: >> === >> /ABC/data/pic/819059/IPMSA_819059_m12_NA_006_3098_check.jpg >> /ABC/data/pic/819059/IPMSA_819059_m12_NA_007_3099_check.jpg >> /ABC/data/pic/819059/IPMSA_819059_m12_NA_008_3100_check.jpg >> /ABCdata/pic/819059/IPMSA_819059_m12_NA_009_3101_check.jpg >> /ABC/data/pic/819059/IPMSA_819059_m12_t1_001_3091_check.jpg >> /ABC/data/pic/819059/IPMSA_819059_m12_t1_002_3092_check.jpg >> /ABC/data/tarchive/2009/DCM_2009-12-10_ImagingUpload-7-57-r0hk9h.tar >> >> DBD::mysql::db do failed: Cannot delete or update a parent row: a foreign >> key constraint fails (`IPMSA_LORIS`.`mri_upload`, CONSTRAINT >> `FK_mriupload_TarchiveID` FOREIGN KEY (`TarchiveID`) REFERENCES `tarchive` >> (`TarchiveID`)) at /data_/ipmsa/loris_data/IPMSA/bin/mri/tools// >> delete_imaging_upload.pl line 1612. >> Issuing rollback() due to DESTROY without explicit disconnect() of >> DBD::mysql::db handle database=IPMSA_LORIS;host=132.216.133.50;port=3306; >> at /data_/ipmsa/loris_data/IPMSA/bin/mri/tools//delete_imaging_upload.pl >> line 1612. >> === >> > > Not sure what is going on here. Could you point me to the file so I can > check where the issue comes from? Or the release version you are using. Are > there any processed output data for that upload ID? > > >> >> Error type 4: >> === >> Cannot delete upload 1543: the MRI pipeline is currently processing it. >> === >> > > If you are sure you are not running the pipeline, you will need to update > the mri_upload table to reflect this. For some obscure reason, the > insertion status fields of the mri_upload table were not updated for that > UploadID. Maybe the pipeline got interrupted? > UPDATE mri_upload SET Inserting=0, InsertionComplete=1 WHERE UploadID=1543; > Then, rerun the delete script on that UploadID. > > Hope this helps. Best, > > C?cile > > >> Looking forward to hearing from you. >> >> Regards, >> Alfredo. >> >> _______________________________________________ >> Loris-dev mailing list >> Loris-dev at bic.mni.mcgill.ca >> https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev >> > _______________________________________________ > Loris-dev mailing list > Loris-dev at bic.mni.mcgill.ca > https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From AMORALESPINZON at bwh.harvard.edu Mon Apr 13 20:03:33 2020 From: AMORALESPINZON at bwh.harvard.edu (Morales Pinzon, Alfredo) Date: Tue, 14 Apr 2020 00:03:33 +0000 Subject: [Loris-dev] Errors when using delete_imaging_upload.pl In-Reply-To: References: Message-ID: Dear C?cile, Ling ma, and Nicolas, I modified my scripts to execute the solutions you provided. @Cecile, the option -ignored worked perfect. @Nicolas, there were multiple entries in ?mri_upload? linked to the same TarchiveID indeed. Now all the UploadIDs linked to the study I am removing images from are now deleted. However, in the front-end I can still see 8 imaging visits! I will send an email explaining the issue. Thank you again. Regards, Alfredo. On Apr 7, 2020, at 8:06 AM, Nicolas Brossard > wrote: External Email - Use Caution Hi all, Regarding the foreign key constraint fail error, my guess is that you have two records in mri_uploads that are tied to the same archive and you are trying to delete one of them. What I am saying is that you have two uploads, say U1 and U2, which have the same tarchive ID, say A1. If you try to delete U1, the script will try to delete everything associated to it, including archive A1, which will fail because of the foreign key constraint still tying U2 to A1. This is unfortunately a bug of the delete script. You should be able to delete *all* the uploads that have the same tarchive ID though, but you'd have to do it in the same script invocation. So back to our example, you could delete U1 and U2 but you'd have to delete them at the same time by supplying both upload IDs in a comma-separated list via the -uploadID switch (see the perldoc for the delete script). I hope this makes sense. Let me know if I you need more information. Regards, Nicolas On Mon, Apr 6, 2020 at 6:02 PM Cecile Madjar > wrote: Hi Alfredo, See answers below in blue. Error type 1: === Error! File /ABC/data/pic/979907/IPMSA_979907_m00_NA_001_182_check.jpg exists in the database but was not found on the file system. Aborting. === Error type 2: === Error! File /ABC/data/assembly/677803/m12/mri/native/IPMSA_677803_m12_NA_002.nii exists in the database but was not found on the file system. Aborting. === The first two errors are the same. The delete script always checks first if the file is on the filesystem. My guess is that when you ran the pipeline, the path to the NIfTI file was inserted into the database even when the NIfTI file was not created due to some mnc2nii conversion errors (known bug that was fixed in release 22.0). You can use the option -ignore to ignore the check between database and file system. Error type 3: === /ABC/data/pic/819059/IPMSA_819059_m12_NA_006_3098_check.jpg /ABC/data/pic/819059/IPMSA_819059_m12_NA_007_3099_check.jpg /ABC/data/pic/819059/IPMSA_819059_m12_NA_008_3100_check.jpg /ABCdata/pic/819059/IPMSA_819059_m12_NA_009_3101_check.jpg /ABC/data/pic/819059/IPMSA_819059_m12_t1_001_3091_check.jpg /ABC/data/pic/819059/IPMSA_819059_m12_t1_002_3092_check.jpg /ABC/data/tarchive/2009/DCM_2009-12-10_ImagingUpload-7-57-r0hk9h.tar DBD::mysql::db do failed: Cannot delete or update a parent row: a foreign key constraint fails (`IPMSA_LORIS`.`mri_upload`, CONSTRAINT `FK_mriupload_TarchiveID` FOREIGN KEY (`TarchiveID`) REFERENCES `tarchive` (`TarchiveID`)) at /data_/ipmsa/loris_data/IPMSA/bin/mri/tools//delete_imaging_upload.pl line 1612. Issuing rollback() due to DESTROY without explicit disconnect() of DBD::mysql::db handle database=IPMSA_LORIS;host=132.216.133.50;port=3306; at /data_/ipmsa/loris_data/IPMSA/bin/mri/tools//delete_imaging_upload.pl line 1612. === Not sure what is going on here. Could you point me to the file so I can check where the issue comes from? Or the release version you are using. Are there any processed output data for that upload ID? Error type 4: === Cannot delete upload 1543: the MRI pipeline is currently processing it. === If you are sure you are not running the pipeline, you will need to update the mri_upload table to reflect this. For some obscure reason, the insertion status fields of the mri_upload table were not updated for that UploadID. Maybe the pipeline got interrupted? UPDATE mri_upload SET Inserting=0, InsertionComplete=1 WHERE UploadID=1543; Then, rerun the delete script on that UploadID. Hope this helps. Best, C?cile Looking forward to hearing from you. Regards, Alfredo. _______________________________________________ Loris-dev mailing list Loris-dev at bic.mni.mcgill.ca https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev _______________________________________________ Loris-dev mailing list Loris-dev at bic.mni.mcgill.ca https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From AMORALESPINZON at bwh.harvard.edu Mon Apr 13 20:16:24 2020 From: AMORALESPINZON at bwh.harvard.edu (Morales Pinzon, Alfredo) Date: Tue, 14 Apr 2020 00:16:24 +0000 Subject: [Loris-dev] Imaging visits in front-end but no UploadIDs in the database Message-ID: Dear LorisDev Team, I recently removed all the UploadIDs from a study using the script ?delete_imaging_upload.pl?. However, in the front-end I can see some imaging visits for some candidates of the study. Here is an example for one visit: Here is the imaging session for one the Candidates with the issue: === select * from session where CandID in (450030) \G; *************************** 1. row *************************** ID: 845 CandID: 450030 CenterID: 60 VisitNo: 1 Visit_label: m00 SubprojectID: 2 Submitted: N Current_stage: Not Started Date_stage_change: NULL Screening: NULL Date_screening: NULL Visit: NULL Date_visit: NULL Approval: NULL Date_approval: NULL Active: Y Date_active: 2017-11-10 RegisteredBy: 5216-ICR_dataman UserID: 5216-ICR_dataman Date_registered: 2017-11-10 Testdate: 2017-11-30 11:07:25 Hardcopy_request: - BVLQCStatus: NULL BVLQCType: NULL BVLQCExclusion: NULL QCd: NULL Scan_done: Y MRIQCStatus: MRIQCPending: N MRIQCFirstChangeTime: NULL MRIQCLastChangeTime: NULL MRICaveat: false === However, there is no TarchiveID associated with the session id in the mri_upload table === select TarchiveID from mri_upload where SessionID=845 \G; Empty set (0.00 sec) === Any ideas on how can I fix this. Thank you in advance for your help. Regards, Alfredo. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolasbrossard.mni at gmail.com Mon Apr 13 20:37:45 2020 From: nicolasbrossard.mni at gmail.com (Nicolas Brossard) Date: Mon, 13 Apr 2020 20:37:45 -0400 Subject: [Loris-dev] Imaging visits in front-end but no UploadIDs in the database In-Reply-To: References: Message-ID: Hi Alfredo, I am curious to see the actual records that the delete script removed from your mri_upload table. Since the delete script backs up what it deletes, you can have a look inside the backup file it created. By default, this file is named imaging_upload_backup.tar.gz. Inside, you'll find an SQL file that should contain all the entries in mri_upload that were deleted. Can you check if all of them have a sessionID set to 845 (they should)? Can you also check in the same file the UPDATE statement that was issued on the session table? We'll start with these checks first. Best, Nicolas On Mon, Apr 13, 2020 at 8:16 PM Morales Pinzon, Alfredo < AMORALESPINZON at bwh.harvard.edu> wrote: > Dear LorisDev Team, > > I recently removed all the UploadIDs from a study using the script ? > delete_imaging_upload.pl?. However, in the front-end I can see some imaging > visits for some candidates of the study. > > Here is an example for one visit: > > Here is the imaging session for one the Candidates with the issue: > === > select * from session where CandID in (450030) \G; > *************************** 1. row *************************** > ID: 845 > CandID: 450030 > CenterID: 60 > VisitNo: 1 > Visit_label: m00 > SubprojectID: 2 > Submitted: N > Current_stage: Not Started > Date_stage_change: NULL > Screening: NULL > Date_screening: NULL > Visit: NULL > Date_visit: NULL > Approval: NULL > Date_approval: NULL > Active: Y > Date_active: 2017-11-10 > RegisteredBy: 5216-ICR_dataman > UserID: 5216-ICR_dataman > Date_registered: 2017-11-10 > Testdate: 2017-11-30 11:07:25 > Hardcopy_request: - > BVLQCStatus: NULL > BVLQCType: NULL > BVLQCExclusion: NULL > QCd: NULL > Scan_done: Y > MRIQCStatus: > MRIQCPending: N > MRIQCFirstChangeTime: NULL > MRIQCLastChangeTime: NULL > MRICaveat: false > === > > However, there is no TarchiveID associated with the session id in the > mri_upload table > === > select TarchiveID from mri_upload where SessionID=845 \G; > Empty set (0.00 sec) > === > > Any ideas on how can I fix this. > > Thank you in advance for your help. > > Regards, > Alfredo. > > > _______________________________________________ > Loris-dev mailing list > Loris-dev at bic.mni.mcgill.ca > https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From AMORALESPINZON at bwh.harvard.edu Wed Apr 15 12:51:15 2020 From: AMORALESPINZON at bwh.harvard.edu (Morales Pinzon, Alfredo) Date: Wed, 15 Apr 2020 16:51:15 +0000 Subject: [Loris-dev] Imaging visits in front-end but no UploadIDs in the database In-Reply-To: References: Message-ID: <883EBD0A-AE8C-4887-A7A2-3D9FC04E1F85@bwh.harvard.edu> Hi Nicolas, I checked the records, for the 3200 deleted uploadId, and the sessionID was not in there. Also, the UPDATE statement was issues in the same table. Here is an example of INSERT and UPDATE commands in the SQL file: === INSERT INTO `mri_upload` VALUES (9,'lorisadmin\n','2017-11-28 16:30:47',?/ABC/AAB21935_880779_m00.tar.gz','/tmp/ImagingUpload-16-30-jIjf1G',1,0,'AAB21935_880779_m00',8,8,6,4,1,1,'N'); UPDATE session s SET Scan_done = 'Y' WHERE s.ID IN (4) AND (SELECT COUNT(*) FROM mri_upload m WHERE m.SessionID=s.ID) > 0; === Maybe the insertion process didn?t finish properly but the state of the database was updated? As you can see this images were uploaded in 2017 using LORIS v17.0.0 or v18.0.0. Let me know if you have more suggestions on what to check and how we can solve this. Best, Alfredo. On Apr 13, 2020, at 8:37 PM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, I am curious to see the actual records that the delete script removed from your mri_upload table. Since the delete script backs up what it deletes, you can have a look inside the backup file it created. By default, this file is named imaging_upload_backup.tar.gz. Inside, you'll find an SQL file that should contain all the entries in mri_upload that were deleted. Can you check if all of them have a sessionID set to 845 (they should)? Can you also check in the same file the UPDATE statement that was issued on the session table? We'll start with these checks first. Best, Nicolas On Mon, Apr 13, 2020 at 8:16 PM Morales Pinzon, Alfredo > wrote: Dear LorisDev Team, I recently removed all the UploadIDs from a study using the script ?delete_imaging_upload.pl?. However, in the front-end I can see some imaging visits for some candidates of the study. Here is an example for one visit: Here is the imaging session for one the Candidates with the issue: === select * from session where CandID in (450030) \G; *************************** 1. row *************************** ID: 845 CandID: 450030 CenterID: 60 VisitNo: 1 Visit_label: m00 SubprojectID: 2 Submitted: N Current_stage: Not Started Date_stage_change: NULL Screening: NULL Date_screening: NULL Visit: NULL Date_visit: NULL Approval: NULL Date_approval: NULL Active: Y Date_active: 2017-11-10 RegisteredBy: 5216-ICR_dataman UserID: 5216-ICR_dataman Date_registered: 2017-11-10 Testdate: 2017-11-30 11:07:25 Hardcopy_request: - BVLQCStatus: NULL BVLQCType: NULL BVLQCExclusion: NULL QCd: NULL Scan_done: Y MRIQCStatus: MRIQCPending: N MRIQCFirstChangeTime: NULL MRIQCLastChangeTime: NULL MRICaveat: false === However, there is no TarchiveID associated with the session id in the mri_upload table === select TarchiveID from mri_upload where SessionID=845 \G; Empty set (0.00 sec) === Any ideas on how can I fix this. Thank you in advance for your help. Regards, Alfredo. _______________________________________________ Loris-dev mailing list Loris-dev at bic.mni.mcgill.ca https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolasbrossard.mni at gmail.com Wed Apr 15 14:17:11 2020 From: nicolasbrossard.mni at gmail.com (Nicolas Brossard) Date: Wed, 15 Apr 2020 14:17:11 -0400 Subject: [Loris-dev] Imaging visits in front-end but no UploadIDs in the database In-Reply-To: <883EBD0A-AE8C-4887-A7A2-3D9FC04E1F85@bwh.harvard.edu> References: <883EBD0A-AE8C-4887-A7A2-3D9FC04E1F85@bwh.harvard.edu> Message-ID: Hi Alfredo, So what you are saying is that the script deleted a record in table mri_upload with uploadid = 3200 but there was no corresponding UPDATE session statement? In your previous email you were referring to a session with ID 845 that had scan_done set to 'Y' even though there were no archives associated to it. Was the mri_upload with uploadid 3200 tied to session 845? Are you able to see in the backup file all the mri_uploads tied to session 845 that were deleted? Nicolas On Wed, Apr 15, 2020 at 12:51 PM Morales Pinzon, Alfredo < AMORALESPINZON at bwh.harvard.edu> wrote: > Hi Nicolas, > > I checked the records, for the 3200 deleted uploadId, and the sessionID > was not in there. Also, the UPDATE statement was issues in the same table. > Here is an example of INSERT and UPDATE commands in the SQL file: > > === > INSERT INTO `mri_upload` VALUES (9,'lorisadmin\n','2017-11-28 > 16:30:47',?/ABC/AAB21935_880779_m00.tar.gz','/tmp/ImagingUpload-16-30-jIjf1G',1,0,'AAB21935_880779_m00',8,8,6, > 4,1,1,'N'); > UPDATE session s SET Scan_done = 'Y' WHERE s.ID IN (4) AND (SELECT > COUNT(*) FROM mri_upload m WHERE m.SessionID=s.ID) > 0; > === > > Maybe the insertion process didn?t finish properly but the state of the > database was updated? As you can see this images were uploaded in 2017 > using LORIS v17.0.0 or v18.0.0. Let me know if you have more suggestions on > what to check and how we can solve this. > > Best, > Alfredo. > > On Apr 13, 2020, at 8:37 PM, Nicolas Brossard < > nicolasbrossard.mni at gmail.com> wrote: > > External Email - Use Caution > > Hi Alfredo, > > I am curious to see the actual records that the delete script removed from > your mri_upload table. Since the delete script backs up what it deletes, > you can have a look inside the backup file it created. By default, this > file is named imaging_upload_backup.tar.gz. Inside, you'll find an SQL file > that should contain all the entries in mri_upload that were deleted. Can > you check if all of them have a sessionID set to 845 (they should)? Can you > also check in the same file the UPDATE statement that was issued on the > session table? We'll start with these checks first. > > > Best, > Nicolas > > > On Mon, Apr 13, 2020 at 8:16 PM Morales Pinzon, Alfredo < > AMORALESPINZON at bwh.harvard.edu> wrote: > >> Dear LorisDev Team, >> >> I recently removed all the UploadIDs from a study using the script ? >> delete_imaging_upload.pl?. However, in the front-end I can see some imaging >> visits for some candidates of the study. >> >> Here is an example for one visit: >> >> Here is the imaging session for one the Candidates with the issue: >> === >> select * from session where CandID in (450030) \G; >> *************************** 1. row *************************** >> ID: 845 >> CandID: 450030 >> CenterID: 60 >> VisitNo: 1 >> Visit_label: m00 >> SubprojectID: 2 >> Submitted: N >> Current_stage: Not Started >> Date_stage_change: NULL >> Screening: NULL >> Date_screening: NULL >> Visit: NULL >> Date_visit: NULL >> Approval: NULL >> Date_approval: NULL >> Active: Y >> Date_active: 2017-11-10 >> RegisteredBy: 5216-ICR_dataman >> UserID: 5216-ICR_dataman >> Date_registered: 2017-11-10 >> Testdate: 2017-11-30 11:07:25 >> Hardcopy_request: - >> BVLQCStatus: NULL >> BVLQCType: NULL >> BVLQCExclusion: NULL >> QCd: NULL >> Scan_done: Y >> MRIQCStatus: >> MRIQCPending: N >> MRIQCFirstChangeTime: NULL >> MRIQCLastChangeTime: NULL >> MRICaveat: false >> === >> >> However, there is no TarchiveID associated with the session id in the >> mri_upload table >> === >> select TarchiveID from mri_upload where SessionID=845 \G; >> Empty set (0.00 sec) >> === >> >> Any ideas on how can I fix this. >> >> Thank you in advance for your help. >> >> Regards, >> Alfredo. >> >> >> _______________________________________________ >> Loris-dev mailing list >> Loris-dev at bic.mni.mcgill.ca >> https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From AMORALESPINZON at bwh.harvard.edu Wed Apr 15 15:32:50 2020 From: AMORALESPINZON at bwh.harvard.edu (Morales Pinzon, Alfredo) Date: Wed, 15 Apr 2020 19:32:50 +0000 Subject: [Loris-dev] Imaging visits in front-end but no UploadIDs in the database In-Reply-To: References: <883EBD0A-AE8C-4887-A7A2-3D9FC04E1F85@bwh.harvard.edu> Message-ID: <0CE7151D-03B0-4048-ACBB-3F4C5A7078AE@bwh.harvard.edu> Hi Nicolas, Apologies for not being clear in my last email. I meant to say that after removing 3200 upload ids and didn?t find the session 845 in any of the backup SQL generated files. I have multiple back up files as I removed the 3200 ids one by one. Is it clear now? Best, Alfredo. On Apr 15, 2020, at 2:17 PM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, So what you are saying is that the script deleted a record in table mri_upload with uploadid = 3200 but there was no corresponding UPDATE session statement? In your previous email you were referring to a session with ID 845 that had scan_done set to 'Y' even though there were no archives associated to it. Was the mri_upload with uploadid 3200 tied to session 845? Are you able to see in the backup file all the mri_uploads tied to session 845 that were deleted? Nicolas On Wed, Apr 15, 2020 at 12:51 PM Morales Pinzon, Alfredo > wrote: Hi Nicolas, I checked the records, for the 3200 deleted uploadId, and the sessionID was not in there. Also, the UPDATE statement was issues in the same table. Here is an example of INSERT and UPDATE commands in the SQL file: === INSERT INTO `mri_upload` VALUES (9,'lorisadmin\n','2017-11-28 16:30:47',?/ABC/AAB21935_880779_m00.tar.gz','/tmp/ImagingUpload-16-30-jIjf1G',1,0,'AAB21935_880779_m00',8,8,6,4,1,1,'N'); UPDATE session s SET Scan_done = 'Y' WHERE s.ID IN (4) AND (SELECT COUNT(*) FROM mri_upload m WHERE m.SessionID=s.ID) > 0; === Maybe the insertion process didn?t finish properly but the state of the database was updated? As you can see this images were uploaded in 2017 using LORIS v17.0.0 or v18.0.0. Let me know if you have more suggestions on what to check and how we can solve this. Best, Alfredo. On Apr 13, 2020, at 8:37 PM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, I am curious to see the actual records that the delete script removed from your mri_upload table. Since the delete script backs up what it deletes, you can have a look inside the backup file it created. By default, this file is named imaging_upload_backup.tar.gz. Inside, you'll find an SQL file that should contain all the entries in mri_upload that were deleted. Can you check if all of them have a sessionID set to 845 (they should)? Can you also check in the same file the UPDATE statement that was issued on the session table? We'll start with these checks first. Best, Nicolas On Mon, Apr 13, 2020 at 8:16 PM Morales Pinzon, Alfredo > wrote: Dear LorisDev Team, I recently removed all the UploadIDs from a study using the script ?delete_imaging_upload.pl?. However, in the front-end I can see some imaging visits for some candidates of the study. Here is an example for one visit: Here is the imaging session for one the Candidates with the issue: === select * from session where CandID in (450030) \G; *************************** 1. row *************************** ID: 845 CandID: 450030 CenterID: 60 VisitNo: 1 Visit_label: m00 SubprojectID: 2 Submitted: N Current_stage: Not Started Date_stage_change: NULL Screening: NULL Date_screening: NULL Visit: NULL Date_visit: NULL Approval: NULL Date_approval: NULL Active: Y Date_active: 2017-11-10 RegisteredBy: 5216-ICR_dataman UserID: 5216-ICR_dataman Date_registered: 2017-11-10 Testdate: 2017-11-30 11:07:25 Hardcopy_request: - BVLQCStatus: NULL BVLQCType: NULL BVLQCExclusion: NULL QCd: NULL Scan_done: Y MRIQCStatus: MRIQCPending: N MRIQCFirstChangeTime: NULL MRIQCLastChangeTime: NULL MRICaveat: false === However, there is no TarchiveID associated with the session id in the mri_upload table === select TarchiveID from mri_upload where SessionID=845 \G; Empty set (0.00 sec) === Any ideas on how can I fix this. Thank you in advance for your help. Regards, Alfredo. _______________________________________________ Loris-dev mailing list Loris-dev at bic.mni.mcgill.ca https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolasbrossard.mni at gmail.com Wed Apr 15 15:42:01 2020 From: nicolasbrossard.mni at gmail.com (Nicolas Brossard) Date: Wed, 15 Apr 2020 15:42:01 -0400 Subject: [Loris-dev] Imaging visits in front-end but no UploadIDs in the database In-Reply-To: <0CE7151D-03B0-4048-ACBB-3F4C5A7078AE@bwh.harvard.edu> References: <883EBD0A-AE8C-4887-A7A2-3D9FC04E1F85@bwh.harvard.edu> <0CE7151D-03B0-4048-ACBB-3F4C5A7078AE@bwh.harvard.edu> Message-ID: Hi Alfredo, Yep, all clear. Tables mri_upload and tarchive both have a column SessionID. You said you did not find in your backup files an UPDATE statement for session 845. Can you check in your backup files the INSERT statements for tables mri_upload and tarchive? Do any of these statements have a SessionID set to 845? Also, are you sure you did not manually delete records from mri_upload after the delete script was run? Best, Nicolas On Wed, Apr 15, 2020 at 3:32 PM Morales Pinzon, Alfredo < AMORALESPINZON at bwh.harvard.edu> wrote: > Hi Nicolas, > > Apologies for not being clear in my last email. I meant to say that after > removing 3200 upload ids and didn?t find the session 845 in any of the > backup SQL generated files. I have multiple back up files as I removed the > 3200 ids one by one. Is it clear now? > > Best, > Alfredo. > > On Apr 15, 2020, at 2:17 PM, Nicolas Brossard < > nicolasbrossard.mni at gmail.com> wrote: > > External Email - Use Caution > > Hi Alfredo, > > So what you are saying is that the script deleted a record in table > mri_upload with uploadid = 3200 but there was no corresponding UPDATE > session statement? In your previous email you were referring to a session > with ID 845 that had scan_done set to 'Y' even though there were no > archives associated to it. Was the mri_upload with uploadid 3200 tied to > session 845? Are you able to see in the backup file all the mri_uploads > tied to session 845 that were deleted? > > > Nicolas > > On Wed, Apr 15, 2020 at 12:51 PM Morales Pinzon, Alfredo < > AMORALESPINZON at bwh.harvard.edu> wrote: > >> Hi Nicolas, >> >> I checked the records, for the 3200 deleted uploadId, and the sessionID >> was not in there. Also, the UPDATE statement was issues in the same table. >> Here is an example of INSERT and UPDATE commands in the SQL file: >> >> === >> INSERT INTO `mri_upload` VALUES (9,'lorisadmin\n','2017-11-28 >> 16:30:47',?/ABC/AAB21935_880779_m00.tar.gz','/tmp/ImagingUpload-16-30-jIjf1G',1,0,'AAB21935_880779_m00',8,8,6, >> 4,1,1,'N'); >> UPDATE session s SET Scan_done = 'Y' WHERE s.ID IN (4) AND (SELECT >> COUNT(*) FROM mri_upload m WHERE m.SessionID=s.ID) > 0; >> === >> >> Maybe the insertion process didn?t finish properly but the state of the >> database was updated? As you can see this images were uploaded in 2017 >> using LORIS v17.0.0 or v18.0.0. Let me know if you have more suggestions on >> what to check and how we can solve this. >> >> Best, >> Alfredo. >> >> On Apr 13, 2020, at 8:37 PM, Nicolas Brossard < >> nicolasbrossard.mni at gmail.com> wrote: >> >> External Email - Use Caution >> >> Hi Alfredo, >> >> I am curious to see the actual records that the delete script removed >> from your mri_upload table. Since the delete script backs up what it >> deletes, you can have a look inside the backup file it created. By default, >> this file is named imaging_upload_backup.tar.gz. Inside, you'll find an SQL >> file that should contain all the entries in mri_upload that were deleted. >> Can you check if all of them have a sessionID set to 845 (they should)? Can >> you also check in the same file the UPDATE statement that was issued on the >> session table? We'll start with these checks first. >> >> >> Best, >> Nicolas >> >> >> On Mon, Apr 13, 2020 at 8:16 PM Morales Pinzon, Alfredo < >> AMORALESPINZON at bwh.harvard.edu> wrote: >> >>> Dear LorisDev Team, >>> >>> I recently removed all the UploadIDs from a study using the script ? >>> delete_imaging_upload.pl?. However, in the front-end I can see some imaging >>> visits for some candidates of the study. >>> >>> Here is an example for one visit: >>> >>> Here is the imaging session for one the Candidates with the issue: >>> === >>> select * from session where CandID in (450030) \G; >>> *************************** 1. row *************************** >>> ID: 845 >>> CandID: 450030 >>> CenterID: 60 >>> VisitNo: 1 >>> Visit_label: m00 >>> SubprojectID: 2 >>> Submitted: N >>> Current_stage: Not Started >>> Date_stage_change: NULL >>> Screening: NULL >>> Date_screening: NULL >>> Visit: NULL >>> Date_visit: NULL >>> Approval: NULL >>> Date_approval: NULL >>> Active: Y >>> Date_active: 2017-11-10 >>> RegisteredBy: 5216-ICR_dataman >>> UserID: 5216-ICR_dataman >>> Date_registered: 2017-11-10 >>> Testdate: 2017-11-30 11:07:25 >>> Hardcopy_request: - >>> BVLQCStatus: NULL >>> BVLQCType: NULL >>> BVLQCExclusion: NULL >>> QCd: NULL >>> Scan_done: Y >>> MRIQCStatus: >>> MRIQCPending: N >>> MRIQCFirstChangeTime: NULL >>> MRIQCLastChangeTime: NULL >>> MRICaveat: false >>> === >>> >>> However, there is no TarchiveID associated with the session id in the >>> mri_upload table >>> === >>> select TarchiveID from mri_upload where SessionID=845 \G; >>> Empty set (0.00 sec) >>> === >>> >>> Any ideas on how can I fix this. >>> >>> Thank you in advance for your help. >>> >>> Regards, >>> Alfredo. >>> >>> >>> _______________________________________________ >>> Loris-dev mailing list >>> Loris-dev at bic.mni.mcgill.ca >>> https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev >>> >> >> > The information in this e-mail is intended only for the person to whom it > is > addressed. If you believe this e-mail was sent to you in error and the > e-mail > contains patient information, please contact the Partners Compliance > HelpLine at > http://www.partners.org/complianceline . If the e-mail was sent to you in > error > but does not contain patient information, please contact the sender and > properly > dispose of the e-mail. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From AMORALESPINZON at bwh.harvard.edu Thu Apr 16 00:04:28 2020 From: AMORALESPINZON at bwh.harvard.edu (Morales Pinzon, Alfredo) Date: Thu, 16 Apr 2020 04:04:28 +0000 Subject: [Loris-dev] Imaging visits in front-end but no UploadIDs in the database In-Reply-To: References: <883EBD0A-AE8C-4887-A7A2-3D9FC04E1F85@bwh.harvard.edu> <0CE7151D-03B0-4048-ACBB-3F4C5A7078AE@bwh.harvard.edu> Message-ID: Hi Nicolas, I checked the INSERT statements but no one has a SessionID set to 845. Also, are you sure you did not manually delete records from mri_upload after the delete script was run? Yes, I?ve only used the script. However, I recall that during the insertion of this study I had issues with the insertion of the images, for instance I had errors in the middle of the pipeline. Let me know if a call can be helpful to figure this out faster. Best, Alfredo. On Apr 15, 2020, at 3:42 PM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, Yep, all clear. Tables mri_upload and tarchive both have a column SessionID. You said you did not find in your backup files an UPDATE statement for session 845. Can you check in your backup files the INSERT statements for tables mri_upload and tarchive? Do any of these statements have a SessionID set to 845? Also, are you sure you did not manually delete records from mri_upload after the delete script was run? Best, Nicolas On Wed, Apr 15, 2020 at 3:32 PM Morales Pinzon, Alfredo > wrote: Hi Nicolas, Apologies for not being clear in my last email. I meant to say that after removing 3200 upload ids and didn?t find the session 845 in any of the backup SQL generated files. I have multiple back up files as I removed the 3200 ids one by one. Is it clear now? Best, Alfredo. On Apr 15, 2020, at 2:17 PM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, So what you are saying is that the script deleted a record in table mri_upload with uploadid = 3200 but there was no corresponding UPDATE session statement? In your previous email you were referring to a session with ID 845 that had scan_done set to 'Y' even though there were no archives associated to it. Was the mri_upload with uploadid 3200 tied to session 845? Are you able to see in the backup file all the mri_uploads tied to session 845 that were deleted? Nicolas On Wed, Apr 15, 2020 at 12:51 PM Morales Pinzon, Alfredo > wrote: Hi Nicolas, I checked the records, for the 3200 deleted uploadId, and the sessionID was not in there. Also, the UPDATE statement was issues in the same table. Here is an example of INSERT and UPDATE commands in the SQL file: === INSERT INTO `mri_upload` VALUES (9,'lorisadmin\n','2017-11-28 16:30:47',?/ABC/AAB21935_880779_m00.tar.gz','/tmp/ImagingUpload-16-30-jIjf1G',1,0,'AAB21935_880779_m00',8,8,6,4,1,1,'N'); UPDATE session s SET Scan_done = 'Y' WHERE s.ID IN (4) AND (SELECT COUNT(*) FROM mri_upload m WHERE m.SessionID=s.ID) > 0; === Maybe the insertion process didn?t finish properly but the state of the database was updated? As you can see this images were uploaded in 2017 using LORIS v17.0.0 or v18.0.0. Let me know if you have more suggestions on what to check and how we can solve this. Best, Alfredo. On Apr 13, 2020, at 8:37 PM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, I am curious to see the actual records that the delete script removed from your mri_upload table. Since the delete script backs up what it deletes, you can have a look inside the backup file it created. By default, this file is named imaging_upload_backup.tar.gz. Inside, you'll find an SQL file that should contain all the entries in mri_upload that were deleted. Can you check if all of them have a sessionID set to 845 (they should)? Can you also check in the same file the UPDATE statement that was issued on the session table? We'll start with these checks first. Best, Nicolas On Mon, Apr 13, 2020 at 8:16 PM Morales Pinzon, Alfredo > wrote: Dear LorisDev Team, I recently removed all the UploadIDs from a study using the script ?delete_imaging_upload.pl?. However, in the front-end I can see some imaging visits for some candidates of the study. Here is an example for one visit: Here is the imaging session for one the Candidates with the issue: === select * from session where CandID in (450030) \G; *************************** 1. row *************************** ID: 845 CandID: 450030 CenterID: 60 VisitNo: 1 Visit_label: m00 SubprojectID: 2 Submitted: N Current_stage: Not Started Date_stage_change: NULL Screening: NULL Date_screening: NULL Visit: NULL Date_visit: NULL Approval: NULL Date_approval: NULL Active: Y Date_active: 2017-11-10 RegisteredBy: 5216-ICR_dataman UserID: 5216-ICR_dataman Date_registered: 2017-11-10 Testdate: 2017-11-30 11:07:25 Hardcopy_request: - BVLQCStatus: NULL BVLQCType: NULL BVLQCExclusion: NULL QCd: NULL Scan_done: Y MRIQCStatus: MRIQCPending: N MRIQCFirstChangeTime: NULL MRIQCLastChangeTime: NULL MRICaveat: false === However, there is no TarchiveID associated with the session id in the mri_upload table === select TarchiveID from mri_upload where SessionID=845 \G; Empty set (0.00 sec) === Any ideas on how can I fix this. Thank you in advance for your help. Regards, Alfredo. _______________________________________________ Loris-dev mailing list Loris-dev at bic.mni.mcgill.ca https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolasbrossard.mni at gmail.com Thu Apr 16 09:05:32 2020 From: nicolasbrossard.mni at gmail.com (Nicolas Brossard) Date: Thu, 16 Apr 2020 09:05:32 -0400 Subject: [Loris-dev] Imaging visits in front-end but no UploadIDs in the database In-Reply-To: References: <883EBD0A-AE8C-4887-A7A2-3D9FC04E1F85@bwh.harvard.edu> <0CE7151D-03B0-4048-ACBB-3F4C5A7078AE@bwh.harvard.edu> Message-ID: Hi Alfredo, If none of the INSERT statements have a session ID of 845, then that session was not related in any way to the uploads you deleted. Consequently, I think that the behaviour of the delete script is totally unrelated to the fact that session 845 has scan_done = 'Y' even though it is not tied to any archive. When you start a visit, you have to specify whether a scan was done or not and the system will not immediately "force" you to upload scans if you indicated that a scan was done. It is thus entirely possible that: 1. Session 845 was started and the user who started it indicated that a scan was done. 2. No scans were uploaded afterwards for that session.... I personally think this behaviour is not desirable and the Scan_done flag should only be set to 'Yes' if the session is tied to an existing archive. If you also agree, you can just identify the sessions that do not fill this requirement and set their Scan_Done flag to 'No'. Hope that help! Best, Nicolas On Thu, Apr 16, 2020 at 12:04 AM Morales Pinzon, Alfredo < AMORALESPINZON at bwh.harvard.edu> wrote: > Hi Nicolas, > > I checked the INSERT statements but no one has a SessionID set to 845. > > Also, are you sure you did not manually delete records from mri_upload > after the delete script was run? > > > Yes, I?ve only used the script. However, I recall that during the > insertion of this study I had issues with the insertion of the images, for > instance I had errors in the middle of the pipeline. > > Let me know if a call can be helpful to figure this out faster. > > Best, > Alfredo. > > On Apr 15, 2020, at 3:42 PM, Nicolas Brossard < > nicolasbrossard.mni at gmail.com> wrote: > > External Email - Use Caution > > Hi Alfredo, > > Yep, all clear. Tables mri_upload and tarchive both have a column > SessionID. You said you did not find in your backup files an UPDATE > statement for session 845. Can you check in your backup files the INSERT > statements for tables mri_upload and tarchive? Do any of these statements > have a SessionID set to 845? > > Also, are you sure you did not manually delete records from mri_upload > after the delete script was run? > > > Best, > Nicolas > > On Wed, Apr 15, 2020 at 3:32 PM Morales Pinzon, Alfredo < > AMORALESPINZON at bwh.harvard.edu> wrote: > >> Hi Nicolas, >> >> Apologies for not being clear in my last email. I meant to say that after >> removing 3200 upload ids and didn?t find the session 845 in any of the >> backup SQL generated files. I have multiple back up files as I removed the >> 3200 ids one by one. Is it clear now? >> >> Best, >> Alfredo. >> >> On Apr 15, 2020, at 2:17 PM, Nicolas Brossard < >> nicolasbrossard.mni at gmail.com> wrote: >> >> External Email - Use Caution >> >> Hi Alfredo, >> >> So what you are saying is that the script deleted a record in table >> mri_upload with uploadid = 3200 but there was no corresponding UPDATE >> session statement? In your previous email you were referring to a session >> with ID 845 that had scan_done set to 'Y' even though there were no >> archives associated to it. Was the mri_upload with uploadid 3200 tied to >> session 845? Are you able to see in the backup file all the mri_uploads >> tied to session 845 that were deleted? >> >> >> Nicolas >> >> On Wed, Apr 15, 2020 at 12:51 PM Morales Pinzon, Alfredo < >> AMORALESPINZON at bwh.harvard.edu> wrote: >> >>> Hi Nicolas, >>> >>> I checked the records, for the 3200 deleted uploadId, and the sessionID >>> was not in there. Also, the UPDATE statement was issues in the same table. >>> Here is an example of INSERT and UPDATE commands in the SQL file: >>> >>> === >>> INSERT INTO `mri_upload` VALUES (9,'lorisadmin\n','2017-11-28 >>> 16:30:47',?/ABC/AAB21935_880779_m00.tar.gz','/tmp/ImagingUpload-16-30-jIjf1G',1,0,'AAB21935_880779_m00',8,8,6, >>> 4,1,1,'N'); >>> UPDATE session s SET Scan_done = 'Y' WHERE s.ID IN (4) AND (SELECT >>> COUNT(*) FROM mri_upload m WHERE m.SessionID=s.ID) > 0; >>> === >>> >>> Maybe the insertion process didn?t finish properly but the state of the >>> database was updated? As you can see this images were uploaded in 2017 >>> using LORIS v17.0.0 or v18.0.0. Let me know if you have more suggestions on >>> what to check and how we can solve this. >>> >>> Best, >>> Alfredo. >>> >>> On Apr 13, 2020, at 8:37 PM, Nicolas Brossard < >>> nicolasbrossard.mni at gmail.com> wrote: >>> >>> External Email - Use Caution >>> >>> Hi Alfredo, >>> >>> I am curious to see the actual records that the delete script removed >>> from your mri_upload table. Since the delete script backs up what it >>> deletes, you can have a look inside the backup file it created. By default, >>> this file is named imaging_upload_backup.tar.gz. Inside, you'll find an SQL >>> file that should contain all the entries in mri_upload that were deleted. >>> Can you check if all of them have a sessionID set to 845 (they should)? Can >>> you also check in the same file the UPDATE statement that was issued on the >>> session table? We'll start with these checks first. >>> >>> >>> Best, >>> Nicolas >>> >>> >>> On Mon, Apr 13, 2020 at 8:16 PM Morales Pinzon, Alfredo < >>> AMORALESPINZON at bwh.harvard.edu> wrote: >>> >>>> Dear LorisDev Team, >>>> >>>> I recently removed all the UploadIDs from a study using the script ? >>>> delete_imaging_upload.pl?. However, in the front-end I can see some imaging >>>> visits for some candidates of the study. >>>> >>>> Here is an example for one visit: >>>> >>>> Here is the imaging session for one the Candidates with the issue: >>>> === >>>> select * from session where CandID in (450030) \G; >>>> *************************** 1. row *************************** >>>> ID: 845 >>>> CandID: 450030 >>>> CenterID: 60 >>>> VisitNo: 1 >>>> Visit_label: m00 >>>> SubprojectID: 2 >>>> Submitted: N >>>> Current_stage: Not Started >>>> Date_stage_change: NULL >>>> Screening: NULL >>>> Date_screening: NULL >>>> Visit: NULL >>>> Date_visit: NULL >>>> Approval: NULL >>>> Date_approval: NULL >>>> Active: Y >>>> Date_active: 2017-11-10 >>>> RegisteredBy: 5216-ICR_dataman >>>> UserID: 5216-ICR_dataman >>>> Date_registered: 2017-11-10 >>>> Testdate: 2017-11-30 11:07:25 >>>> Hardcopy_request: - >>>> BVLQCStatus: NULL >>>> BVLQCType: NULL >>>> BVLQCExclusion: NULL >>>> QCd: NULL >>>> Scan_done: Y >>>> MRIQCStatus: >>>> MRIQCPending: N >>>> MRIQCFirstChangeTime: NULL >>>> MRIQCLastChangeTime: NULL >>>> MRICaveat: false >>>> === >>>> >>>> However, there is no TarchiveID associated with the session id in the >>>> mri_upload table >>>> === >>>> select TarchiveID from mri_upload where SessionID=845 \G; >>>> Empty set (0.00 sec) >>>> === >>>> >>>> Any ideas on how can I fix this. >>>> >>>> Thank you in advance for your help. >>>> >>>> Regards, >>>> Alfredo. >>>> >>>> >>>> _______________________________________________ >>>> Loris-dev mailing list >>>> Loris-dev at bic.mni.mcgill.ca >>>> https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev >>>> >>> >>> >> >> The information in this e-mail is intended only for the person to whom it >> is >> addressed. If you believe this e-mail was sent to you in error and the >> e-mail >> contains patient information, please contact the Partners Compliance >> HelpLine at >> http://www.partners.org/complianceline . If the e-mail was sent to you >> in error >> but does not contain patient information, please contact the sender and >> properly >> dispose of the e-mail. >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From AMORALESPINZON at bwh.harvard.edu Thu Apr 16 16:03:18 2020 From: AMORALESPINZON at bwh.harvard.edu (Morales Pinzon, Alfredo) Date: Thu, 16 Apr 2020 20:03:18 +0000 Subject: [Loris-dev] Imaging visits in front-end but no UploadIDs in the database In-Reply-To: References: <883EBD0A-AE8C-4887-A7A2-3D9FC04E1F85@bwh.harvard.edu> <0CE7151D-03B0-4048-ACBB-3F4C5A7078AE@bwh.harvard.edu> Message-ID: <08CFA709-F0C7-4F63-9B7C-526CB7F50DD7@bwh.harvard.edu> Hi Nicolas, It makes sense. I?ll identity these caes and flag the Scan_Done to ?No?. Thank you. Best, Alfredo. On Apr 16, 2020, at 9:05 AM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, If none of the INSERT statements have a session ID of 845, then that session was not related in any way to the uploads you deleted. Consequently, I think that the behaviour of the delete script is totally unrelated to the fact that session 845 has scan_done = 'Y' even though it is not tied to any archive. When you start a visit, you have to specify whether a scan was done or not and the system will not immediately "force" you to upload scans if you indicated that a scan was done. It is thus entirely possible that: 1. Session 845 was started and the user who started it indicated that a scan was done. 2. No scans were uploaded afterwards for that session.... I personally think this behaviour is not desirable and the Scan_done flag should only be set to 'Yes' if the session is tied to an existing archive. If you also agree, you can just identify the sessions that do not fill this requirement and set their Scan_Done flag to 'No'. Hope that help! Best, Nicolas On Thu, Apr 16, 2020 at 12:04 AM Morales Pinzon, Alfredo > wrote: Hi Nicolas, I checked the INSERT statements but no one has a SessionID set to 845. Also, are you sure you did not manually delete records from mri_upload after the delete script was run? Yes, I?ve only used the script. However, I recall that during the insertion of this study I had issues with the insertion of the images, for instance I had errors in the middle of the pipeline. Let me know if a call can be helpful to figure this out faster. Best, Alfredo. On Apr 15, 2020, at 3:42 PM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, Yep, all clear. Tables mri_upload and tarchive both have a column SessionID. You said you did not find in your backup files an UPDATE statement for session 845. Can you check in your backup files the INSERT statements for tables mri_upload and tarchive? Do any of these statements have a SessionID set to 845? Also, are you sure you did not manually delete records from mri_upload after the delete script was run? Best, Nicolas On Wed, Apr 15, 2020 at 3:32 PM Morales Pinzon, Alfredo > wrote: Hi Nicolas, Apologies for not being clear in my last email. I meant to say that after removing 3200 upload ids and didn?t find the session 845 in any of the backup SQL generated files. I have multiple back up files as I removed the 3200 ids one by one. Is it clear now? Best, Alfredo. On Apr 15, 2020, at 2:17 PM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, So what you are saying is that the script deleted a record in table mri_upload with uploadid = 3200 but there was no corresponding UPDATE session statement? In your previous email you were referring to a session with ID 845 that had scan_done set to 'Y' even though there were no archives associated to it. Was the mri_upload with uploadid 3200 tied to session 845? Are you able to see in the backup file all the mri_uploads tied to session 845 that were deleted? Nicolas On Wed, Apr 15, 2020 at 12:51 PM Morales Pinzon, Alfredo > wrote: Hi Nicolas, I checked the records, for the 3200 deleted uploadId, and the sessionID was not in there. Also, the UPDATE statement was issues in the same table. Here is an example of INSERT and UPDATE commands in the SQL file: === INSERT INTO `mri_upload` VALUES (9,'lorisadmin\n','2017-11-28 16:30:47',?/ABC/AAB21935_880779_m00.tar.gz','/tmp/ImagingUpload-16-30-jIjf1G',1,0,'AAB21935_880779_m00',8,8,6,4,1,1,'N'); UPDATE session s SET Scan_done = 'Y' WHERE s.ID IN (4) AND (SELECT COUNT(*) FROM mri_upload m WHERE m.SessionID=s.ID) > 0; === Maybe the insertion process didn?t finish properly but the state of the database was updated? As you can see this images were uploaded in 2017 using LORIS v17.0.0 or v18.0.0. Let me know if you have more suggestions on what to check and how we can solve this. Best, Alfredo. On Apr 13, 2020, at 8:37 PM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, I am curious to see the actual records that the delete script removed from your mri_upload table. Since the delete script backs up what it deletes, you can have a look inside the backup file it created. By default, this file is named imaging_upload_backup.tar.gz. Inside, you'll find an SQL file that should contain all the entries in mri_upload that were deleted. Can you check if all of them have a sessionID set to 845 (they should)? Can you also check in the same file the UPDATE statement that was issued on the session table? We'll start with these checks first. Best, Nicolas On Mon, Apr 13, 2020 at 8:16 PM Morales Pinzon, Alfredo > wrote: Dear LorisDev Team, I recently removed all the UploadIDs from a study using the script ?delete_imaging_upload.pl?. However, in the front-end I can see some imaging visits for some candidates of the study. Here is an example for one visit: Here is the imaging session for one the Candidates with the issue: === select * from session where CandID in (450030) \G; *************************** 1. row *************************** ID: 845 CandID: 450030 CenterID: 60 VisitNo: 1 Visit_label: m00 SubprojectID: 2 Submitted: N Current_stage: Not Started Date_stage_change: NULL Screening: NULL Date_screening: NULL Visit: NULL Date_visit: NULL Approval: NULL Date_approval: NULL Active: Y Date_active: 2017-11-10 RegisteredBy: 5216-ICR_dataman UserID: 5216-ICR_dataman Date_registered: 2017-11-10 Testdate: 2017-11-30 11:07:25 Hardcopy_request: - BVLQCStatus: NULL BVLQCType: NULL BVLQCExclusion: NULL QCd: NULL Scan_done: Y MRIQCStatus: MRIQCPending: N MRIQCFirstChangeTime: NULL MRIQCLastChangeTime: NULL MRICaveat: false === However, there is no TarchiveID associated with the session id in the mri_upload table === select TarchiveID from mri_upload where SessionID=845 \G; Empty set (0.00 sec) === Any ideas on how can I fix this. Thank you in advance for your help. Regards, Alfredo. _______________________________________________ Loris-dev mailing list Loris-dev at bic.mni.mcgill.ca https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From AMORALESPINZON at bwh.harvard.edu Mon Apr 20 15:55:29 2020 From: AMORALESPINZON at bwh.harvard.edu (Morales Pinzon, Alfredo) Date: Mon, 20 Apr 2020 19:55:29 +0000 Subject: [Loris-dev] Imaging visits in front-end but no UploadIDs in the database In-Reply-To: References: <883EBD0A-AE8C-4887-A7A2-3D9FC04E1F85@bwh.harvard.edu> <0CE7151D-03B0-4048-ACBB-3F4C5A7078AE@bwh.harvard.edu> Message-ID: <2A85469E-2683-4041-BB3D-710862DCCB67@bwh.harvard.edu> Hi Nicolas, I change the Scan_Done value from ?Y? to ?N? but I still can see these visits and links, which seem to be broken links, in the front-end. Any ideas? We can arrange a Zoom call if this helps us to figure things out. Best, Alfredo. On Apr 16, 2020, at 9:05 AM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, If none of the INSERT statements have a session ID of 845, then that session was not related in any way to the uploads you deleted. Consequently, I think that the behaviour of the delete script is totally unrelated to the fact that session 845 has scan_done = 'Y' even though it is not tied to any archive. When you start a visit, you have to specify whether a scan was done or not and the system will not immediately "force" you to upload scans if you indicated that a scan was done. It is thus entirely possible that: 1. Session 845 was started and the user who started it indicated that a scan was done. 2. No scans were uploaded afterwards for that session.... I personally think this behaviour is not desirable and the Scan_done flag should only be set to 'Yes' if the session is tied to an existing archive. If you also agree, you can just identify the sessions that do not fill this requirement and set their Scan_Done flag to 'No'. Hope that help! Best, Nicolas On Thu, Apr 16, 2020 at 12:04 AM Morales Pinzon, Alfredo > wrote: Hi Nicolas, I checked the INSERT statements but no one has a SessionID set to 845. Also, are you sure you did not manually delete records from mri_upload after the delete script was run? Yes, I?ve only used the script. However, I recall that during the insertion of this study I had issues with the insertion of the images, for instance I had errors in the middle of the pipeline. Let me know if a call can be helpful to figure this out faster. Best, Alfredo. On Apr 15, 2020, at 3:42 PM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, Yep, all clear. Tables mri_upload and tarchive both have a column SessionID. You said you did not find in your backup files an UPDATE statement for session 845. Can you check in your backup files the INSERT statements for tables mri_upload and tarchive? Do any of these statements have a SessionID set to 845? Also, are you sure you did not manually delete records from mri_upload after the delete script was run? Best, Nicolas On Wed, Apr 15, 2020 at 3:32 PM Morales Pinzon, Alfredo > wrote: Hi Nicolas, Apologies for not being clear in my last email. I meant to say that after removing 3200 upload ids and didn?t find the session 845 in any of the backup SQL generated files. I have multiple back up files as I removed the 3200 ids one by one. Is it clear now? Best, Alfredo. On Apr 15, 2020, at 2:17 PM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, So what you are saying is that the script deleted a record in table mri_upload with uploadid = 3200 but there was no corresponding UPDATE session statement? In your previous email you were referring to a session with ID 845 that had scan_done set to 'Y' even though there were no archives associated to it. Was the mri_upload with uploadid 3200 tied to session 845? Are you able to see in the backup file all the mri_uploads tied to session 845 that were deleted? Nicolas On Wed, Apr 15, 2020 at 12:51 PM Morales Pinzon, Alfredo > wrote: Hi Nicolas, I checked the records, for the 3200 deleted uploadId, and the sessionID was not in there. Also, the UPDATE statement was issues in the same table. Here is an example of INSERT and UPDATE commands in the SQL file: === INSERT INTO `mri_upload` VALUES (9,'lorisadmin\n','2017-11-28 16:30:47',?/ABC/AAB21935_880779_m00.tar.gz','/tmp/ImagingUpload-16-30-jIjf1G',1,0,'AAB21935_880779_m00',8,8,6,4,1,1,'N'); UPDATE session s SET Scan_done = 'Y' WHERE s.ID IN (4) AND (SELECT COUNT(*) FROM mri_upload m WHERE m.SessionID=s.ID) > 0; === Maybe the insertion process didn?t finish properly but the state of the database was updated? As you can see this images were uploaded in 2017 using LORIS v17.0.0 or v18.0.0. Let me know if you have more suggestions on what to check and how we can solve this. Best, Alfredo. On Apr 13, 2020, at 8:37 PM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, I am curious to see the actual records that the delete script removed from your mri_upload table. Since the delete script backs up what it deletes, you can have a look inside the backup file it created. By default, this file is named imaging_upload_backup.tar.gz. Inside, you'll find an SQL file that should contain all the entries in mri_upload that were deleted. Can you check if all of them have a sessionID set to 845 (they should)? Can you also check in the same file the UPDATE statement that was issued on the session table? We'll start with these checks first. Best, Nicolas On Mon, Apr 13, 2020 at 8:16 PM Morales Pinzon, Alfredo > wrote: Dear LorisDev Team, I recently removed all the UploadIDs from a study using the script ?delete_imaging_upload.pl?. However, in the front-end I can see some imaging visits for some candidates of the study. Here is an example for one visit: Here is the imaging session for one the Candidates with the issue: === select * from session where CandID in (450030) \G; *************************** 1. row *************************** ID: 845 CandID: 450030 CenterID: 60 VisitNo: 1 Visit_label: m00 SubprojectID: 2 Submitted: N Current_stage: Not Started Date_stage_change: NULL Screening: NULL Date_screening: NULL Visit: NULL Date_visit: NULL Approval: NULL Date_approval: NULL Active: Y Date_active: 2017-11-10 RegisteredBy: 5216-ICR_dataman UserID: 5216-ICR_dataman Date_registered: 2017-11-10 Testdate: 2017-11-30 11:07:25 Hardcopy_request: - BVLQCStatus: NULL BVLQCType: NULL BVLQCExclusion: NULL QCd: NULL Scan_done: Y MRIQCStatus: MRIQCPending: N MRIQCFirstChangeTime: NULL MRIQCLastChangeTime: NULL MRICaveat: false === However, there is no TarchiveID associated with the session id in the mri_upload table === select TarchiveID from mri_upload where SessionID=845 \G; Empty set (0.00 sec) === Any ideas on how can I fix this. Thank you in advance for your help. Regards, Alfredo. _______________________________________________ Loris-dev mailing list Loris-dev at bic.mni.mcgill.ca https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolasbrossard.mni at gmail.com Tue Apr 21 09:48:39 2020 From: nicolasbrossard.mni at gmail.com (Nicolas Brossard) Date: Tue, 21 Apr 2020 09:48:39 -0400 Subject: [Loris-dev] Imaging visits in front-end but no UploadIDs in the database In-Reply-To: <2A85469E-2683-4041-BB3D-710862DCCB67@bwh.harvard.edu> References: <883EBD0A-AE8C-4887-A7A2-3D9FC04E1F85@bwh.harvard.edu> <0CE7151D-03B0-4048-ACBB-3F4C5A7078AE@bwh.harvard.edu> <2A85469E-2683-4041-BB3D-710862DCCB67@bwh.harvard.edu> Message-ID: Hi Alfredo, When you say that you can still see those visits, I assume that you are saying that they are still listed on the time point list page for a given candidate even though you were not expecting them to be. Is that correct? And you are also saying that there is still a clickable 'Yes' in the 'Imaging Scan Done' column for that visit that leads nowhere (i.e broken link), right? Just trying to make sure I understand the observed behaviour. I have been very unsuccessful running both Zoom and slack phone call on my computer. My laptop has has some kind of hardware problem that makes it crash very often when I run video intensive softwares like these (it tends to shutdown after 5-10 mins. or so, as I have often experienced during the remote LORIS meetings....) We could talk via slack (no video) if you like. Let me know what you think is best. Cheers, Nicolas On Mon, Apr 20, 2020 at 3:55 PM Morales Pinzon, Alfredo < AMORALESPINZON at bwh.harvard.edu> wrote: > Hi Nicolas, > > I change the Scan_Done value from ?Y? to ?N? but I still can see these > visits and links, which seem to be broken links, in the front-end. Any > ideas? We can arrange a Zoom call if this helps us to figure things out. > > Best, > Alfredo. > > On Apr 16, 2020, at 9:05 AM, Nicolas Brossard < > nicolasbrossard.mni at gmail.com> wrote: > > External Email - Use Caution > > Hi Alfredo, > > If none of the INSERT statements have a session ID of 845, then that > session was not related in any way to the uploads you deleted. > Consequently, I think that the behaviour of the delete script is totally > unrelated to the fact that session 845 has scan_done = 'Y' even though it > is not tied to any archive. When you start a visit, you have to specify > whether a scan was done or not and the system will not immediately "force" > you to upload scans if you indicated that a scan was done. It is thus > entirely possible that: > > 1. Session 845 was started and the user who started it indicated that a > scan was done. > 2. No scans were uploaded afterwards for that session.... > > I personally think this behaviour is not desirable and the Scan_done flag > should only be set to 'Yes' if the session is tied to an existing archive. > If you also agree, you can just identify the sessions that do not fill this > requirement and set their Scan_Done flag to 'No'. > > Hope that help! > > > Best, > Nicolas > > > On Thu, Apr 16, 2020 at 12:04 AM Morales Pinzon, Alfredo < > AMORALESPINZON at bwh.harvard.edu> wrote: > >> Hi Nicolas, >> >> I checked the INSERT statements but no one has a SessionID set to 845. >> >> Also, are you sure you did not manually delete records from mri_upload >> after the delete script was run? >> >> >> Yes, I?ve only used the script. However, I recall that during the >> insertion of this study I had issues with the insertion of the images, for >> instance I had errors in the middle of the pipeline. >> >> Let me know if a call can be helpful to figure this out faster. >> >> Best, >> Alfredo. >> >> On Apr 15, 2020, at 3:42 PM, Nicolas Brossard < >> nicolasbrossard.mni at gmail.com> wrote: >> >> External Email - Use Caution >> >> Hi Alfredo, >> >> Yep, all clear. Tables mri_upload and tarchive both have a column >> SessionID. You said you did not find in your backup files an UPDATE >> statement for session 845. Can you check in your backup files the INSERT >> statements for tables mri_upload and tarchive? Do any of these statements >> have a SessionID set to 845? >> >> Also, are you sure you did not manually delete records from mri_upload >> after the delete script was run? >> >> >> Best, >> Nicolas >> >> On Wed, Apr 15, 2020 at 3:32 PM Morales Pinzon, Alfredo < >> AMORALESPINZON at bwh.harvard.edu> wrote: >> >>> Hi Nicolas, >>> >>> Apologies for not being clear in my last email. I meant to say that >>> after removing 3200 upload ids and didn?t find the session 845 in any of >>> the backup SQL generated files. I have multiple back up files as I removed >>> the 3200 ids one by one. Is it clear now? >>> >>> Best, >>> Alfredo. >>> >>> On Apr 15, 2020, at 2:17 PM, Nicolas Brossard < >>> nicolasbrossard.mni at gmail.com> wrote: >>> >>> External Email - Use Caution >>> >>> Hi Alfredo, >>> >>> So what you are saying is that the script deleted a record in table >>> mri_upload with uploadid = 3200 but there was no corresponding UPDATE >>> session statement? In your previous email you were referring to a session >>> with ID 845 that had scan_done set to 'Y' even though there were no >>> archives associated to it. Was the mri_upload with uploadid 3200 tied to >>> session 845? Are you able to see in the backup file all the mri_uploads >>> tied to session 845 that were deleted? >>> >>> >>> Nicolas >>> >>> On Wed, Apr 15, 2020 at 12:51 PM Morales Pinzon, Alfredo < >>> AMORALESPINZON at bwh.harvard.edu> wrote: >>> >>>> Hi Nicolas, >>>> >>>> I checked the records, for the 3200 deleted uploadId, and the sessionID >>>> was not in there. Also, the UPDATE statement was issues in the same table. >>>> Here is an example of INSERT and UPDATE commands in the SQL file: >>>> >>>> === >>>> INSERT INTO `mri_upload` VALUES (9,'lorisadmin\n','2017-11-28 >>>> 16:30:47',?/ABC/AAB21935_880779_m00.tar.gz','/tmp/ImagingUpload-16-30-jIjf1G',1,0,'AAB21935_880779_m00',8,8,6, >>>> 4,1,1,'N'); >>>> UPDATE session s SET Scan_done = 'Y' WHERE s.ID IN (4) AND (SELECT >>>> COUNT(*) FROM mri_upload m WHERE m.SessionID=s.ID) > 0; >>>> === >>>> >>>> Maybe the insertion process didn?t finish properly but the state of the >>>> database was updated? As you can see this images were uploaded in 2017 >>>> using LORIS v17.0.0 or v18.0.0. Let me know if you have more suggestions on >>>> what to check and how we can solve this. >>>> >>>> Best, >>>> Alfredo. >>>> >>>> On Apr 13, 2020, at 8:37 PM, Nicolas Brossard < >>>> nicolasbrossard.mni at gmail.com> wrote: >>>> >>>> External Email - Use Caution >>>> >>>> Hi Alfredo, >>>> >>>> I am curious to see the actual records that the delete script removed >>>> from your mri_upload table. Since the delete script backs up what it >>>> deletes, you can have a look inside the backup file it created. By default, >>>> this file is named imaging_upload_backup.tar.gz. Inside, you'll find an SQL >>>> file that should contain all the entries in mri_upload that were deleted. >>>> Can you check if all of them have a sessionID set to 845 (they should)? Can >>>> you also check in the same file the UPDATE statement that was issued on the >>>> session table? We'll start with these checks first. >>>> >>>> >>>> Best, >>>> Nicolas >>>> >>>> >>>> On Mon, Apr 13, 2020 at 8:16 PM Morales Pinzon, Alfredo < >>>> AMORALESPINZON at bwh.harvard.edu> wrote: >>>> >>>>> Dear LorisDev Team, >>>>> >>>>> I recently removed all the UploadIDs from a study using the script ? >>>>> delete_imaging_upload.pl?. However, in the front-end I can see some imaging >>>>> visits for some candidates of the study. >>>>> >>>>> Here is an example for one visit: >>>>> >>>>> Here is the imaging session for one the Candidates with the issue: >>>>> === >>>>> select * from session where CandID in (450030) \G; >>>>> *************************** 1. row *************************** >>>>> ID: 845 >>>>> CandID: 450030 >>>>> CenterID: 60 >>>>> VisitNo: 1 >>>>> Visit_label: m00 >>>>> SubprojectID: 2 >>>>> Submitted: N >>>>> Current_stage: Not Started >>>>> Date_stage_change: NULL >>>>> Screening: NULL >>>>> Date_screening: NULL >>>>> Visit: NULL >>>>> Date_visit: NULL >>>>> Approval: NULL >>>>> Date_approval: NULL >>>>> Active: Y >>>>> Date_active: 2017-11-10 >>>>> RegisteredBy: 5216-ICR_dataman >>>>> UserID: 5216-ICR_dataman >>>>> Date_registered: 2017-11-10 >>>>> Testdate: 2017-11-30 11:07:25 >>>>> Hardcopy_request: - >>>>> BVLQCStatus: NULL >>>>> BVLQCType: NULL >>>>> BVLQCExclusion: NULL >>>>> QCd: NULL >>>>> Scan_done: Y >>>>> MRIQCStatus: >>>>> MRIQCPending: N >>>>> MRIQCFirstChangeTime: NULL >>>>> MRIQCLastChangeTime: NULL >>>>> MRICaveat: false >>>>> === >>>>> >>>>> However, there is no TarchiveID associated with the session id in the >>>>> mri_upload table >>>>> === >>>>> select TarchiveID from mri_upload where SessionID=845 \G; >>>>> Empty set (0.00 sec) >>>>> === >>>>> >>>>> Any ideas on how can I fix this. >>>>> >>>>> Thank you in advance for your help. >>>>> >>>>> Regards, >>>>> Alfredo. >>>>> >>>>> >>>>> _______________________________________________ >>>>> Loris-dev mailing list >>>>> Loris-dev at bic.mni.mcgill.ca >>>>> https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev >>>>> >>>> >>>> >>> >>> The information in this e-mail is intended only for the person to whom >>> it is >>> addressed. If you believe this e-mail was sent to you in error and the >>> e-mail >>> contains patient information, please contact the Partners Compliance >>> HelpLine at >>> http://www.partners.org/complianceline . If the e-mail was sent to you >>> in error >>> but does not contain patient information, please contact the sender and >>> properly >>> dispose of the e-mail. >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From AMORALESPINZON at bwh.harvard.edu Wed Apr 22 08:45:29 2020 From: AMORALESPINZON at bwh.harvard.edu (Morales Pinzon, Alfredo) Date: Wed, 22 Apr 2020 12:45:29 +0000 Subject: [Loris-dev] Imaging visits in front-end but no UploadIDs in the database In-Reply-To: References: <883EBD0A-AE8C-4887-A7A2-3D9FC04E1F85@bwh.harvard.edu> <0CE7151D-03B0-4048-ACBB-3F4C5A7078AE@bwh.harvard.edu> <2A85469E-2683-4041-BB3D-710862DCCB67@bwh.harvard.edu> Message-ID: <5CD0A69D-4127-43DD-BB09-CA3A4B7FC081@bwh.harvard.edu> Hi Nicolas, Thank you for your answer. I will prepare some screenshots before our meeting. What slack group should I join? Best, Alfredo. On Apr 21, 2020, at 9:48 AM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, When you say that you can still see those visits, I assume that you are saying that they are still listed on the time point list page for a given candidate even though you were not expecting them to be. Is that correct? And you are also saying that there is still a clickable 'Yes' in the 'Imaging Scan Done' column for that visit that leads nowhere (i.e broken link), right? Just trying to make sure I understand the observed behaviour. I have been very unsuccessful running both Zoom and slack phone call on my computer. My laptop has has some kind of hardware problem that makes it crash very often when I run video intensive softwares like these (it tends to shutdown after 5-10 mins. or so, as I have often experienced during the remote LORIS meetings....) We could talk via slack (no video) if you like. Let me know what you think is best. Cheers, Nicolas On Mon, Apr 20, 2020 at 3:55 PM Morales Pinzon, Alfredo > wrote: Hi Nicolas, I change the Scan_Done value from ?Y? to ?N? but I still can see these visits and links, which seem to be broken links, in the front-end. Any ideas? We can arrange a Zoom call if this helps us to figure things out. Best, Alfredo. On Apr 16, 2020, at 9:05 AM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, If none of the INSERT statements have a session ID of 845, then that session was not related in any way to the uploads you deleted. Consequently, I think that the behaviour of the delete script is totally unrelated to the fact that session 845 has scan_done = 'Y' even though it is not tied to any archive. When you start a visit, you have to specify whether a scan was done or not and the system will not immediately "force" you to upload scans if you indicated that a scan was done. It is thus entirely possible that: 1. Session 845 was started and the user who started it indicated that a scan was done. 2. No scans were uploaded afterwards for that session.... I personally think this behaviour is not desirable and the Scan_done flag should only be set to 'Yes' if the session is tied to an existing archive. If you also agree, you can just identify the sessions that do not fill this requirement and set their Scan_Done flag to 'No'. Hope that help! Best, Nicolas On Thu, Apr 16, 2020 at 12:04 AM Morales Pinzon, Alfredo > wrote: Hi Nicolas, I checked the INSERT statements but no one has a SessionID set to 845. Also, are you sure you did not manually delete records from mri_upload after the delete script was run? Yes, I?ve only used the script. However, I recall that during the insertion of this study I had issues with the insertion of the images, for instance I had errors in the middle of the pipeline. Let me know if a call can be helpful to figure this out faster. Best, Alfredo. On Apr 15, 2020, at 3:42 PM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, Yep, all clear. Tables mri_upload and tarchive both have a column SessionID. You said you did not find in your backup files an UPDATE statement for session 845. Can you check in your backup files the INSERT statements for tables mri_upload and tarchive? Do any of these statements have a SessionID set to 845? Also, are you sure you did not manually delete records from mri_upload after the delete script was run? Best, Nicolas On Wed, Apr 15, 2020 at 3:32 PM Morales Pinzon, Alfredo > wrote: Hi Nicolas, Apologies for not being clear in my last email. I meant to say that after removing 3200 upload ids and didn?t find the session 845 in any of the backup SQL generated files. I have multiple back up files as I removed the 3200 ids one by one. Is it clear now? Best, Alfredo. On Apr 15, 2020, at 2:17 PM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, So what you are saying is that the script deleted a record in table mri_upload with uploadid = 3200 but there was no corresponding UPDATE session statement? In your previous email you were referring to a session with ID 845 that had scan_done set to 'Y' even though there were no archives associated to it. Was the mri_upload with uploadid 3200 tied to session 845? Are you able to see in the backup file all the mri_uploads tied to session 845 that were deleted? Nicolas On Wed, Apr 15, 2020 at 12:51 PM Morales Pinzon, Alfredo > wrote: Hi Nicolas, I checked the records, for the 3200 deleted uploadId, and the sessionID was not in there. Also, the UPDATE statement was issues in the same table. Here is an example of INSERT and UPDATE commands in the SQL file: === INSERT INTO `mri_upload` VALUES (9,'lorisadmin\n','2017-11-28 16:30:47',?/ABC/AAB21935_880779_m00.tar.gz','/tmp/ImagingUpload-16-30-jIjf1G',1,0,'AAB21935_880779_m00',8,8,6,4,1,1,'N'); UPDATE session s SET Scan_done = 'Y' WHERE s.ID IN (4) AND (SELECT COUNT(*) FROM mri_upload m WHERE m.SessionID=s.ID) > 0; === Maybe the insertion process didn?t finish properly but the state of the database was updated? As you can see this images were uploaded in 2017 using LORIS v17.0.0 or v18.0.0. Let me know if you have more suggestions on what to check and how we can solve this. Best, Alfredo. On Apr 13, 2020, at 8:37 PM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, I am curious to see the actual records that the delete script removed from your mri_upload table. Since the delete script backs up what it deletes, you can have a look inside the backup file it created. By default, this file is named imaging_upload_backup.tar.gz. Inside, you'll find an SQL file that should contain all the entries in mri_upload that were deleted. Can you check if all of them have a sessionID set to 845 (they should)? Can you also check in the same file the UPDATE statement that was issued on the session table? We'll start with these checks first. Best, Nicolas On Mon, Apr 13, 2020 at 8:16 PM Morales Pinzon, Alfredo > wrote: Dear LorisDev Team, I recently removed all the UploadIDs from a study using the script ?delete_imaging_upload.pl?. However, in the front-end I can see some imaging visits for some candidates of the study. Here is an example for one visit: Here is the imaging session for one the Candidates with the issue: === select * from session where CandID in (450030) \G; *************************** 1. row *************************** ID: 845 CandID: 450030 CenterID: 60 VisitNo: 1 Visit_label: m00 SubprojectID: 2 Submitted: N Current_stage: Not Started Date_stage_change: NULL Screening: NULL Date_screening: NULL Visit: NULL Date_visit: NULL Approval: NULL Date_approval: NULL Active: Y Date_active: 2017-11-10 RegisteredBy: 5216-ICR_dataman UserID: 5216-ICR_dataman Date_registered: 2017-11-10 Testdate: 2017-11-30 11:07:25 Hardcopy_request: - BVLQCStatus: NULL BVLQCType: NULL BVLQCExclusion: NULL QCd: NULL Scan_done: Y MRIQCStatus: MRIQCPending: N MRIQCFirstChangeTime: NULL MRIQCLastChangeTime: NULL MRICaveat: false === However, there is no TarchiveID associated with the session id in the mri_upload table === select TarchiveID from mri_upload where SessionID=845 \G; Empty set (0.00 sec) === Any ideas on how can I fix this. Thank you in advance for your help. Regards, Alfredo. _______________________________________________ Loris-dev mailing list Loris-dev at bic.mni.mcgill.ca https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolasbrossard.mni at gmail.com Wed Apr 22 10:10:19 2020 From: nicolasbrossard.mni at gmail.com (Nicolas Brossard) Date: Wed, 22 Apr 2020 10:10:19 -0400 Subject: [Loris-dev] Imaging visits in front-end but no UploadIDs in the database In-Reply-To: <5CD0A69D-4127-43DD-BB09-CA3A4B7FC081@bwh.harvard.edu> References: <883EBD0A-AE8C-4887-A7A2-3D9FC04E1F85@bwh.harvard.edu> <0CE7151D-03B0-4048-ACBB-3F4C5A7078AE@bwh.harvard.edu> <2A85469E-2683-4041-BB3D-710862DCCB67@bwh.harvard.edu> <5CD0A69D-4127-43DD-BB09-CA3A4B7FC081@bwh.harvard.edu> Message-ID: Hi Alfredo, I was told you were given access to MCIN's loris-mri channel. Let me know if you can login successfully. Nicolas On Wed, Apr 22, 2020 at 8:45 AM Morales Pinzon, Alfredo < AMORALESPINZON at bwh.harvard.edu> wrote: > Hi Nicolas, > > Thank you for your answer. I will prepare some screenshots before our > meeting. What slack group should I join? > > Best, > Alfredo. > > On Apr 21, 2020, at 9:48 AM, Nicolas Brossard < > nicolasbrossard.mni at gmail.com> wrote: > > External Email - Use Caution > > Hi Alfredo, > > When you say that you can still see those visits, I assume that you are > saying that they are still listed on the time point list page for a given > candidate even though you were not expecting them to be. Is that correct? > And you are also saying that there is still a clickable 'Yes' in the > 'Imaging Scan Done' column for that visit that leads nowhere (i.e broken > link), right? Just trying to make sure I understand the observed behaviour. > > I have been very unsuccessful running both Zoom and slack phone call on my > computer. My laptop has has some kind of hardware problem that makes it > crash very often when I run video intensive softwares like these (it tends > to shutdown after 5-10 mins. or so, as I have often experienced during the > remote LORIS meetings....) We could talk via slack (no video) if you like. > Let me know what you think is best. > > > Cheers, > Nicolas > > > On Mon, Apr 20, 2020 at 3:55 PM Morales Pinzon, Alfredo < > AMORALESPINZON at bwh.harvard.edu> wrote: > >> Hi Nicolas, >> >> I change the Scan_Done value from ?Y? to ?N? but I still can see these >> visits and links, which seem to be broken links, in the front-end. Any >> ideas? We can arrange a Zoom call if this helps us to figure things out. >> >> Best, >> Alfredo. >> >> On Apr 16, 2020, at 9:05 AM, Nicolas Brossard < >> nicolasbrossard.mni at gmail.com> wrote: >> >> External Email - Use Caution >> >> Hi Alfredo, >> >> If none of the INSERT statements have a session ID of 845, then that >> session was not related in any way to the uploads you deleted. >> Consequently, I think that the behaviour of the delete script is totally >> unrelated to the fact that session 845 has scan_done = 'Y' even though it >> is not tied to any archive. When you start a visit, you have to specify >> whether a scan was done or not and the system will not immediately "force" >> you to upload scans if you indicated that a scan was done. It is thus >> entirely possible that: >> >> 1. Session 845 was started and the user who started it indicated that a >> scan was done. >> 2. No scans were uploaded afterwards for that session.... >> >> I personally think this behaviour is not desirable and the Scan_done flag >> should only be set to 'Yes' if the session is tied to an existing archive. >> If you also agree, you can just identify the sessions that do not fill this >> requirement and set their Scan_Done flag to 'No'. >> >> Hope that help! >> >> >> Best, >> Nicolas >> >> >> On Thu, Apr 16, 2020 at 12:04 AM Morales Pinzon, Alfredo < >> AMORALESPINZON at bwh.harvard.edu> wrote: >> >>> Hi Nicolas, >>> >>> I checked the INSERT statements but no one has a SessionID set to 845. >>> >>> Also, are you sure you did not manually delete records from mri_upload >>> after the delete script was run? >>> >>> >>> Yes, I?ve only used the script. However, I recall that during the >>> insertion of this study I had issues with the insertion of the images, for >>> instance I had errors in the middle of the pipeline. >>> >>> Let me know if a call can be helpful to figure this out faster. >>> >>> Best, >>> Alfredo. >>> >>> On Apr 15, 2020, at 3:42 PM, Nicolas Brossard < >>> nicolasbrossard.mni at gmail.com> wrote: >>> >>> External Email - Use Caution >>> >>> Hi Alfredo, >>> >>> Yep, all clear. Tables mri_upload and tarchive both have a column >>> SessionID. You said you did not find in your backup files an UPDATE >>> statement for session 845. Can you check in your backup files the INSERT >>> statements for tables mri_upload and tarchive? Do any of these statements >>> have a SessionID set to 845? >>> >>> Also, are you sure you did not manually delete records from mri_upload >>> after the delete script was run? >>> >>> >>> Best, >>> Nicolas >>> >>> On Wed, Apr 15, 2020 at 3:32 PM Morales Pinzon, Alfredo < >>> AMORALESPINZON at bwh.harvard.edu> wrote: >>> >>>> Hi Nicolas, >>>> >>>> Apologies for not being clear in my last email. I meant to say that >>>> after removing 3200 upload ids and didn?t find the session 845 in any of >>>> the backup SQL generated files. I have multiple back up files as I removed >>>> the 3200 ids one by one. Is it clear now? >>>> >>>> Best, >>>> Alfredo. >>>> >>>> On Apr 15, 2020, at 2:17 PM, Nicolas Brossard < >>>> nicolasbrossard.mni at gmail.com> wrote: >>>> >>>> External Email - Use Caution >>>> >>>> Hi Alfredo, >>>> >>>> So what you are saying is that the script deleted a record in table >>>> mri_upload with uploadid = 3200 but there was no corresponding UPDATE >>>> session statement? In your previous email you were referring to a session >>>> with ID 845 that had scan_done set to 'Y' even though there were no >>>> archives associated to it. Was the mri_upload with uploadid 3200 tied to >>>> session 845? Are you able to see in the backup file all the mri_uploads >>>> tied to session 845 that were deleted? >>>> >>>> >>>> Nicolas >>>> >>>> On Wed, Apr 15, 2020 at 12:51 PM Morales Pinzon, Alfredo < >>>> AMORALESPINZON at bwh.harvard.edu> wrote: >>>> >>>>> Hi Nicolas, >>>>> >>>>> I checked the records, for the 3200 deleted uploadId, and the >>>>> sessionID was not in there. Also, the UPDATE statement was issues in the >>>>> same table. Here is an example of INSERT and UPDATE commands in the SQL >>>>> file: >>>>> >>>>> === >>>>> INSERT INTO `mri_upload` VALUES (9,'lorisadmin\n','2017-11-28 >>>>> 16:30:47',?/ABC/AAB21935_880779_m00.tar.gz','/tmp/ImagingUpload-16-30-jIjf1G',1,0,'AAB21935_880779_m00',8,8,6, >>>>> 4,1,1,'N'); >>>>> UPDATE session s SET Scan_done = 'Y' WHERE s.ID IN (4) AND (SELECT >>>>> COUNT(*) FROM mri_upload m WHERE m.SessionID=s.ID) > 0; >>>>> === >>>>> >>>>> Maybe the insertion process didn?t finish properly but the state of >>>>> the database was updated? As you can see this images were uploaded in 2017 >>>>> using LORIS v17.0.0 or v18.0.0. Let me know if you have more suggestions on >>>>> what to check and how we can solve this. >>>>> >>>>> Best, >>>>> Alfredo. >>>>> >>>>> On Apr 13, 2020, at 8:37 PM, Nicolas Brossard < >>>>> nicolasbrossard.mni at gmail.com> wrote: >>>>> >>>>> External Email - Use Caution >>>>> >>>>> Hi Alfredo, >>>>> >>>>> I am curious to see the actual records that the delete script removed >>>>> from your mri_upload table. Since the delete script backs up what it >>>>> deletes, you can have a look inside the backup file it created. By default, >>>>> this file is named imaging_upload_backup.tar.gz. Inside, you'll find an SQL >>>>> file that should contain all the entries in mri_upload that were deleted. >>>>> Can you check if all of them have a sessionID set to 845 (they should)? Can >>>>> you also check in the same file the UPDATE statement that was issued on the >>>>> session table? We'll start with these checks first. >>>>> >>>>> >>>>> Best, >>>>> Nicolas >>>>> >>>>> >>>>> On Mon, Apr 13, 2020 at 8:16 PM Morales Pinzon, Alfredo < >>>>> AMORALESPINZON at bwh.harvard.edu> wrote: >>>>> >>>>>> Dear LorisDev Team, >>>>>> >>>>>> I recently removed all the UploadIDs from a study using the script ? >>>>>> delete_imaging_upload.pl?. However, in the front-end I can see some imaging >>>>>> visits for some candidates of the study. >>>>>> >>>>>> Here is an example for one visit: >>>>>> >>>>>> Here is the imaging session for one the Candidates with the issue: >>>>>> === >>>>>> select * from session where CandID in (450030) \G; >>>>>> *************************** 1. row *************************** >>>>>> ID: 845 >>>>>> CandID: 450030 >>>>>> CenterID: 60 >>>>>> VisitNo: 1 >>>>>> Visit_label: m00 >>>>>> SubprojectID: 2 >>>>>> Submitted: N >>>>>> Current_stage: Not Started >>>>>> Date_stage_change: NULL >>>>>> Screening: NULL >>>>>> Date_screening: NULL >>>>>> Visit: NULL >>>>>> Date_visit: NULL >>>>>> Approval: NULL >>>>>> Date_approval: NULL >>>>>> Active: Y >>>>>> Date_active: 2017-11-10 >>>>>> RegisteredBy: 5216-ICR_dataman >>>>>> UserID: 5216-ICR_dataman >>>>>> Date_registered: 2017-11-10 >>>>>> Testdate: 2017-11-30 11:07:25 >>>>>> Hardcopy_request: - >>>>>> BVLQCStatus: NULL >>>>>> BVLQCType: NULL >>>>>> BVLQCExclusion: NULL >>>>>> QCd: NULL >>>>>> Scan_done: Y >>>>>> MRIQCStatus: >>>>>> MRIQCPending: N >>>>>> MRIQCFirstChangeTime: NULL >>>>>> MRIQCLastChangeTime: NULL >>>>>> MRICaveat: false >>>>>> === >>>>>> >>>>>> However, there is no TarchiveID associated with the session id in the >>>>>> mri_upload table >>>>>> === >>>>>> select TarchiveID from mri_upload where SessionID=845 \G; >>>>>> Empty set (0.00 sec) >>>>>> === >>>>>> >>>>>> Any ideas on how can I fix this. >>>>>> >>>>>> Thank you in advance for your help. >>>>>> >>>>>> Regards, >>>>>> Alfredo. >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Loris-dev mailing list >>>>>> Loris-dev at bic.mni.mcgill.ca >>>>>> https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev >>>>>> >>>>> >>>>> >>>> >>>> The information in this e-mail is intended only for the person to whom >>>> it is >>>> addressed. If you believe this e-mail was sent to you in error and the >>>> e-mail >>>> contains patient information, please contact the Partners Compliance >>>> HelpLine at >>>> http://www.partners.org/complianceline . If the e-mail was sent to you >>>> in error >>>> but does not contain patient information, please contact the sender and >>>> properly >>>> dispose of the e-mail. >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From AMORALESPINZON at bwh.harvard.edu Wed Apr 22 16:42:36 2020 From: AMORALESPINZON at bwh.harvard.edu (Morales Pinzon, Alfredo) Date: Wed, 22 Apr 2020 20:42:36 +0000 Subject: [Loris-dev] Imaging visits in front-end but no UploadIDs in the database In-Reply-To: References: <883EBD0A-AE8C-4887-A7A2-3D9FC04E1F85@bwh.harvard.edu> <0CE7151D-03B0-4048-ACBB-3F4C5A7078AE@bwh.harvard.edu> <2A85469E-2683-4041-BB3D-710862DCCB67@bwh.harvard.edu> <5CD0A69D-4127-43DD-BB09-CA3A4B7FC081@bwh.harvard.edu> Message-ID: <8BB9FAF6-B4CD-4093-A988-AAE5B342BA40@bwh.harvard.edu> Hi Nicolas, Yes, I?m in the channel. Best, Alfredo. On Apr 22, 2020, at 10:10 AM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, I was told you were given access to MCIN's loris-mri channel. Let me know if you can login successfully. Nicolas On Wed, Apr 22, 2020 at 8:45 AM Morales Pinzon, Alfredo > wrote: Hi Nicolas, Thank you for your answer. I will prepare some screenshots before our meeting. What slack group should I join? Best, Alfredo. On Apr 21, 2020, at 9:48 AM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, When you say that you can still see those visits, I assume that you are saying that they are still listed on the time point list page for a given candidate even though you were not expecting them to be. Is that correct? And you are also saying that there is still a clickable 'Yes' in the 'Imaging Scan Done' column for that visit that leads nowhere (i.e broken link), right? Just trying to make sure I understand the observed behaviour. I have been very unsuccessful running both Zoom and slack phone call on my computer. My laptop has has some kind of hardware problem that makes it crash very often when I run video intensive softwares like these (it tends to shutdown after 5-10 mins. or so, as I have often experienced during the remote LORIS meetings....) We could talk via slack (no video) if you like. Let me know what you think is best. Cheers, Nicolas On Mon, Apr 20, 2020 at 3:55 PM Morales Pinzon, Alfredo > wrote: Hi Nicolas, I change the Scan_Done value from ?Y? to ?N? but I still can see these visits and links, which seem to be broken links, in the front-end. Any ideas? We can arrange a Zoom call if this helps us to figure things out. Best, Alfredo. On Apr 16, 2020, at 9:05 AM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, If none of the INSERT statements have a session ID of 845, then that session was not related in any way to the uploads you deleted. Consequently, I think that the behaviour of the delete script is totally unrelated to the fact that session 845 has scan_done = 'Y' even though it is not tied to any archive. When you start a visit, you have to specify whether a scan was done or not and the system will not immediately "force" you to upload scans if you indicated that a scan was done. It is thus entirely possible that: 1. Session 845 was started and the user who started it indicated that a scan was done. 2. No scans were uploaded afterwards for that session.... I personally think this behaviour is not desirable and the Scan_done flag should only be set to 'Yes' if the session is tied to an existing archive. If you also agree, you can just identify the sessions that do not fill this requirement and set their Scan_Done flag to 'No'. Hope that help! Best, Nicolas On Thu, Apr 16, 2020 at 12:04 AM Morales Pinzon, Alfredo > wrote: Hi Nicolas, I checked the INSERT statements but no one has a SessionID set to 845. Also, are you sure you did not manually delete records from mri_upload after the delete script was run? Yes, I?ve only used the script. However, I recall that during the insertion of this study I had issues with the insertion of the images, for instance I had errors in the middle of the pipeline. Let me know if a call can be helpful to figure this out faster. Best, Alfredo. On Apr 15, 2020, at 3:42 PM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, Yep, all clear. Tables mri_upload and tarchive both have a column SessionID. You said you did not find in your backup files an UPDATE statement for session 845. Can you check in your backup files the INSERT statements for tables mri_upload and tarchive? Do any of these statements have a SessionID set to 845? Also, are you sure you did not manually delete records from mri_upload after the delete script was run? Best, Nicolas On Wed, Apr 15, 2020 at 3:32 PM Morales Pinzon, Alfredo > wrote: Hi Nicolas, Apologies for not being clear in my last email. I meant to say that after removing 3200 upload ids and didn?t find the session 845 in any of the backup SQL generated files. I have multiple back up files as I removed the 3200 ids one by one. Is it clear now? Best, Alfredo. On Apr 15, 2020, at 2:17 PM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, So what you are saying is that the script deleted a record in table mri_upload with uploadid = 3200 but there was no corresponding UPDATE session statement? In your previous email you were referring to a session with ID 845 that had scan_done set to 'Y' even though there were no archives associated to it. Was the mri_upload with uploadid 3200 tied to session 845? Are you able to see in the backup file all the mri_uploads tied to session 845 that were deleted? Nicolas On Wed, Apr 15, 2020 at 12:51 PM Morales Pinzon, Alfredo > wrote: Hi Nicolas, I checked the records, for the 3200 deleted uploadId, and the sessionID was not in there. Also, the UPDATE statement was issues in the same table. Here is an example of INSERT and UPDATE commands in the SQL file: === INSERT INTO `mri_upload` VALUES (9,'lorisadmin\n','2017-11-28 16:30:47',?/ABC/AAB21935_880779_m00.tar.gz','/tmp/ImagingUpload-16-30-jIjf1G',1,0,'AAB21935_880779_m00',8,8,6,4,1,1,'N'); UPDATE session s SET Scan_done = 'Y' WHERE s.ID IN (4) AND (SELECT COUNT(*) FROM mri_upload m WHERE m.SessionID=s.ID) > 0; === Maybe the insertion process didn?t finish properly but the state of the database was updated? As you can see this images were uploaded in 2017 using LORIS v17.0.0 or v18.0.0. Let me know if you have more suggestions on what to check and how we can solve this. Best, Alfredo. On Apr 13, 2020, at 8:37 PM, Nicolas Brossard > wrote: External Email - Use Caution Hi Alfredo, I am curious to see the actual records that the delete script removed from your mri_upload table. Since the delete script backs up what it deletes, you can have a look inside the backup file it created. By default, this file is named imaging_upload_backup.tar.gz. Inside, you'll find an SQL file that should contain all the entries in mri_upload that were deleted. Can you check if all of them have a sessionID set to 845 (they should)? Can you also check in the same file the UPDATE statement that was issued on the session table? We'll start with these checks first. Best, Nicolas On Mon, Apr 13, 2020 at 8:16 PM Morales Pinzon, Alfredo > wrote: Dear LorisDev Team, I recently removed all the UploadIDs from a study using the script ?delete_imaging_upload.pl?. However, in the front-end I can see some imaging visits for some candidates of the study. Here is an example for one visit: Here is the imaging session for one the Candidates with the issue: === select * from session where CandID in (450030) \G; *************************** 1. row *************************** ID: 845 CandID: 450030 CenterID: 60 VisitNo: 1 Visit_label: m00 SubprojectID: 2 Submitted: N Current_stage: Not Started Date_stage_change: NULL Screening: NULL Date_screening: NULL Visit: NULL Date_visit: NULL Approval: NULL Date_approval: NULL Active: Y Date_active: 2017-11-10 RegisteredBy: 5216-ICR_dataman UserID: 5216-ICR_dataman Date_registered: 2017-11-10 Testdate: 2017-11-30 11:07:25 Hardcopy_request: - BVLQCStatus: NULL BVLQCType: NULL BVLQCExclusion: NULL QCd: NULL Scan_done: Y MRIQCStatus: MRIQCPending: N MRIQCFirstChangeTime: NULL MRIQCLastChangeTime: NULL MRICaveat: false === However, there is no TarchiveID associated with the session id in the mri_upload table === select TarchiveID from mri_upload where SessionID=845 \G; Empty set (0.00 sec) === Any ideas on how can I fix this. Thank you in advance for your help. Regards, Alfredo. _______________________________________________ Loris-dev mailing list Loris-dev at bic.mni.mcgill.ca https://mailman.bic.mni.mcgill.ca/mailman/listinfo/loris-dev The information in this e-mail is intended only for the person to whom it is addressed. If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Partners Compliance HelpLine at http://www.partners.org/complianceline . If the e-mail was sent to you in error but does not contain patient information, please contact the sender and properly dispose of the e-mail. -------------- next part -------------- An HTML attachment was scrubbed... URL: