From e.schlemm at uke.de Sat Oct 26 08:59:20 2019 From: e.schlemm at uke.de (Schlemm, Eckhard) Date: Sat, 26 Oct 2019 12:59:20 +0000 Subject: [MINC-users] volume_stats: Assertion failed in CachedArray.cc AND seg fault in spline_smooth Message-ID: <9062BBF748531F4A957F9217463264DC28069FD8@EXCMBX-4.mail.uke.ads> Hello, in the context of pre-processing some rs-fMRI data I am running freesurfer on an HPC. However, I am receiving segmentation faults in mri_convert which I have been able to track to a problem with volume_stats and smooth_spline: >From the mri folder of my subject's freesurfer tree I run /opt/freesurfer/mni/bin/volume_stats -all ./tmp.mri_nu_correct.mni.15919/0/nu0__.mnc -debug -verbose which results in Reading volume /work/test2/01387fs/mri/tmp.mri_nu_correct.mni.15919/0/nu0__.mncDone C1:32609792:32768 C2:31554272:32768 Created blocks: 32609792 31554272 Assertion failed at line 742 in file templates/CachedArray.cc Line 742 reads assert(_s.is_open()); Interestingly, when I run volume_stats with the -noache flag, the error disappears, namely Reading volume /work/test2/01387fs/mri/tmp.mri_nu_correct.mni.15919/0/nu0__.mncDone Scanning ./tmp.mri_nu_correct.mni.15919/0/nu0__.mnc.................................................................Done Creating histogram with 256 bins, bin width 1...Done File: ./tmp.mri_nu_correct.mni.15919/0/nu0__.mnc # voxels: 274625 % of total: 100 Volume (mm3): 1.7576e+07 Min: 0 Max: 255 Sum: 7.07616e+06 Sum2: 8.35459e+08 Mean: 25.7666 Variance: 2378.26 Stddev: 48.7674 Median: 2 Majority: 0 BiModalT: 59 PctT (-1.79769e+308): -0.5 255.5 Entropy: 4.42151 CoM_voxel: xspace:31.9172 yspace:35.3371 zspace:29.0955 CoM_world: zspace:-47.3955 yspace:-16.2662 xspace:-3.46566 In https://github.com/BIC-MNI/N3/blob/master/src/VolumeStats/volumeStats.cc, the only difference caused by the -nocache flag appears to be the use of CachedArray instead of SimpleArray. To test the functionality of CachedArray I compiled test.cc from https://github.com/BIC-MNI/EBTKS/tree/master/src which runs on my cluster without error. When I patch freesurfer so as to use the -nocache option in volume_stats as explained on https://mailman.bic.mni.mcgill.ca/pipermail/minc-users/2008-April/001996.html, the program exits at /opt/freesurfer/mni/bin/spline_smooth -clobber -verbose -full_support -distance 50 -b_spline -lambda 1.0e-7 -subsample 1 -mask ./tmp.mri_nu_correct.mni.27342/0/nu1_mask.mnc ./tmp.mri_nu_correct.mni.27342/0/nu1_temp.mnc ./tmp.mri_nu_correct.mni.27342/0/nu1_residue.mnc nu_estimate_np_and_em: crashed while running spline_smooth (termination status=11) Repeating the latest command gives a simple Segmentation Fault without further information. So there is 1) A problem related to CachedArray and asserting the successful opening of a temporary file 2) a seg fault in spline_smooth which I am unable to track down further. All commands can be run successfully with the same input files on my local computer. One of the hardware peculiarities on the cluster I am aware of is the inability to create hard links. Soft links are permitted. This problem appears to have popped up now and again in various mailing lists, but so far no definitive solution has transpired. Thanks! -- _____________________________________________________________________ Universit?tsklinikum Hamburg-Eppendorf; K?rperschaft des ?ffentlichen Rechts; Gerichtsstand: Hamburg | www.uke.de Vorstandsmitglieder: Prof. Dr. Burkhard G?ke (Vorsitzender), Prof. Dr. Dr. Uwe Koch-Gromus, Joachim Pr?l?, Marya Verdel _____________________________________________________________________ SAVE PAPER - THINK BEFORE PRINTING From gabriel.devenyi at mcgill.ca Sat Oct 26 09:12:07 2019 From: gabriel.devenyi at mcgill.ca (Gabriel Allan Devenyi, Dr) Date: Sat, 26 Oct 2019 13:12:07 +0000 Subject: [MINC-users] volume_stats: Assertion failed in CachedArray.cc AND seg fault in spline_smooth In-Reply-To: <9c65a99f78264c68996017f10bce6d5d@YT1PR01MB2860.CANPRD01.PROD.OUTLOOK.COM> References: <9c65a99f78264c68996017f10bce6d5d@YT1PR01MB2860.CANPRD01.PROD.OUTLOOK.COM> Message-ID: Hi, Can you please open an issue with the same details on our GitHub so we can track it there? On Sat., Oct. 26, 2019, 08:59 Schlemm, Eckhard, > wrote: Hello, in the context of pre-processing some rs-fMRI data I am running freesurfer on an HPC. However, I am receiving segmentation faults in mri_convert which I have been able to track to a problem with volume_stats and smooth_spline: From the mri folder of my subject's freesurfer tree I run /opt/freesurfer/mni/bin/volume_stats -all ./tmp.mri_nu_correct.mni.15919/0/nu0__.mnc -debug -verbose which results in Reading volume /work/test2/01387fs/mri/tmp.mri_nu_correct.mni.15919/0/nu0__.mncDone C1:32609792:32768 C2:31554272:32768 Created blocks: 32609792 31554272 Assertion failed at line 742 in file templates/CachedArray.cc Line 742 reads assert(_s.is_open()); Interestingly, when I run volume_stats with the -noache flag, the error disappears, namely Reading volume /work/test2/01387fs/mri/tmp.mri_nu_correct.mni.15919/0/nu0__.mncDone Scanning ./tmp.mri_nu_correct.mni.15919/0/nu0__.mnc.................................................................Done Creating histogram with 256 bins, bin width 1...Done File: ./tmp.mri_nu_correct.mni.15919/0/nu0__.mnc # voxels: 274625 % of total: 100 Volume (mm3): 1.7576e+07 Min: 0 Max: 255 Sum: 7.07616e+06 Sum2: 8.35459e+08 Mean: 25.7666 Variance: 2378.26 Stddev: 48.7674 Median: 2 Majority: 0 BiModalT: 59 PctT (-1.79769e+308): -0.5 255.5 Entropy: 4.42151 CoM_voxel: xspace:31.9172 yspace:35.3371 zspace:29.0955 CoM_world: zspace:-47.3955 yspace:-16.2662 xspace:-3.46566 In https://github.com/BIC-MNI/N3/blob/master/src/VolumeStats/volumeStats.cc, the only difference caused by the -nocache flag appears to be the use of CachedArray instead of SimpleArray. To test the functionality of CachedArray I compiled test.cc from https://github.com/BIC-MNI/EBTKS/tree/master/src which runs on my cluster without error. When I patch freesurfer so as to use the -nocache option in volume_stats as explained on https://mailman.bic.mni.mcgill.ca/pipermail/minc-users/2008-April/001996.html, the program exits at /opt/freesurfer/mni/bin/spline_smooth -clobber -verbose -full_support -distance 50 -b_spline -lambda 1.0e-7 -subsample 1 -mask ./tmp.mri_nu_correct.mni.27342/0/nu1_mask.mnc ./tmp.mri_nu_correct.mni.27342/0/nu1_temp.mnc ./tmp.mri_nu_correct.mni.27342/0/nu1_residue.mnc nu_estimate_np_and_em: crashed while running spline_smooth (termination status=11) Repeating the latest command gives a simple Segmentation Fault without further information. So there is 1) A problem related to CachedArray and asserting the successful opening of a temporary file 2) a seg fault in spline_smooth which I am unable to track down further. All commands can be run successfully with the same input files on my local computer. One of the hardware peculiarities on the cluster I am aware of is the inability to create hard links. Soft links are permitted. This problem appears to have popped up now and again in various mailing lists, but so far no definitive solution has transpired. Thanks! -- _____________________________________________________________________ Universit?tsklinikum Hamburg-Eppendorf; K?rperschaft des ?ffentlichen Rechts; Gerichtsstand: Hamburg | www.uke.de Vorstandsmitglieder: Prof. Dr. Burkhard G?ke (Vorsitzender), Prof. Dr. Dr. Uwe Koch-Gromus, Joachim Pr?l?, Marya Verdel _____________________________________________________________________ SAVE PAPER - THINK BEFORE PRINTING _______________________________________________ MINC-users at bic.mni.mcgill.ca https://mailman.bic.mni.mcgill.ca/mailman/listinfo/minc-users From claude at bic.mni.mcgill.ca Mon Oct 28 17:11:57 2019 From: claude at bic.mni.mcgill.ca (Claude LEPAGE) Date: Mon, 28 Oct 2019 17:11:57 -0400 Subject: [MINC-users] volume_stats: Assertion failed in CachedArray.cc AND seg fault in spline_smooth In-Reply-To: <9062BBF748531F4A957F9217463264DC28069FD8@EXCMBX-4.mail.uke.ads> Message-ID: <201910282111.x9SLBvNg027961@login2.bic.mni.mcgill.ca> Hi, I'll make a guess about the N3 crash. > /opt/freesurfer/mni/bin/spline_smooth -clobber -verbose -full_support -distance 50 -b_spline -lambda 1.0e-7 -subsample 1 -mask ./tmp.mri_nu_correct.mni.27342/0/nu1_mask.mnc ./tmp.mri_nu_correct.mni.27342/0/nu1_temp.mnc ./tmp.mri_nu_correct.mni.27342/0/nu1_residue.mnc Try with a slightly larger damping coefficient, for example -lambda 1.0e-6. The bsplines are obtained by solving the normal equations (instead of solving A*x=b, you solve transpose(A)*A*x = transpose(A)*b). This is because A arises from a least-squares matrix and may not be square. However, A*A doubles the condition number of the system. lambda is a damping coefficient needed for stabilization. If too small, your system becomes ill-conditioned and N3 crashes in an ugly fashion without giving you any information. In general, you want to keep lambda as small as possible, but there are some rare exceptions that need more damping (the scan looks good, but the splines fail). Of course, the problem may be due to another cause. However, this quick test may inform you if the problem is with damping or not. Claude > > Hello, > in the context of pre-processing some rs-fMRI data I am running freesurfer on an HPC. However, I am receiving segmentation faults in mri_convert which I have been able to track to a problem with volume_stats and smooth_spline: > > >From the mri folder of my subject's freesurfer tree I run > > /opt/freesurfer/mni/bin/volume_stats -all ./tmp.mri_nu_correct.mni.15919/0/nu0__.mnc -debug -verbose > > which results in > > Reading volume /work/test2/01387fs/mri/tmp.mri_nu_correct.mni.15919/0/nu0__.mncDone > C1:32609792:32768 C2:31554272:32768 > Created blocks: > 32609792 > 31554272 > Assertion failed at line 742 in file templates/CachedArray.cc > > Line 742 reads > > assert(_s.is_open()); > > Interestingly, when I run volume_stats with the -noache flag, the error disappears, namely > > Reading volume /work/test2/01387fs/mri/tmp.mri_nu_correct.mni.15919/0/nu0__.mncDone > Scanning ./tmp.mri_nu_correct.mni.15919/0/nu0__.mnc.................................................................Done > Creating histogram with 256 bins, bin width 1...Done > File: ./tmp.mri_nu_correct.mni.15919/0/nu0__.mnc > # voxels: 274625 > % of total: 100 > Volume (mm3): 1.7576e+07 > Min: 0 > Max: 255 > Sum: 7.07616e+06 > Sum2: 8.35459e+08 > Mean: 25.7666 > Variance: 2378.26 > Stddev: 48.7674 > Median: 2 > Majority: 0 > BiModalT: 59 > PctT (-1.79769e+308): -0.5 255.5 > Entropy: 4.42151 > CoM_voxel: xspace:31.9172 yspace:35.3371 zspace:29.0955 > CoM_world: zspace:-47.3955 yspace:-16.2662 xspace:-3.46566 > > In https://github.com/BIC-MNI/N3/blob/master/src/VolumeStats/volumeStats.cc, the only difference caused by the -nocache flag appears to be the use of CachedArray instead of SimpleArray. To test the functionality of CachedArray I compiled test.cc from https://github.com/BIC-MNI/EBTKS/tree/master/src which runs on my cluster without error. > > When I patch freesurfer so as to use the -nocache option in volume_stats as explained on https://mailman.bic.mni.mcgill.ca/pipermail/minc-users/2008-April/001996.html, the program exits at > > /opt/freesurfer/mni/bin/spline_smooth -clobber -verbose -full_support -distance 50 -b_spline -lambda 1.0e-7 -subsample 1 -mask ./tmp.mri_nu_correct.mni.27342/0/nu1_mask.mnc ./tmp.mri_nu_correct.mni.27342/0/nu1_temp.mnc ./tmp.mri_nu_correct.mni.27342/0/nu1_residue.mnc > nu_estimate_np_and_em: crashed while running spline_smooth (termination status=11) > > > Repeating the latest command gives a simple Segmentation Fault without further information. > > So there is > 1) A problem related to CachedArray and asserting the successful opening of a temporary file > 2) a seg fault in spline_smooth which I am unable to track down further. > > All commands can be run successfully with the same input files on my local computer. One of the hardware peculiarities on the cluster I am aware of is the inability to create hard links. Soft links are permitted. > > This problem appears to have popped up now and again in various mailing lists, but so far no definitive solution has transpired. > Thanks! > -- > > _____________________________________________________________________ > > Universit?tsklinikum Hamburg-Eppendorf; K?rperschaft des ?ffentlichen Rechts; Gerichtsstand: Hamburg | www.uke.de > Vorstandsmitglieder: Prof. Dr. Burkhard G?ke (Vorsitzender), Prof. Dr. Dr. Uwe Koch-Gromus, Joachim Pr?l?, Marya Verdel > _____________________________________________________________________ > > SAVE PAPER - THINK BEFORE PRINTING > _______________________________________________ > MINC-users at bic.mni.mcgill.ca > https://mailman.bic.mni.mcgill.ca/mailman/listinfo/minc-users >