From gabriel.devenyi at mcgill.ca Mon Dec 5 13:09:25 2016 From: gabriel.devenyi at mcgill.ca (Gabriel A. Devenyi) Date: Mon, 5 Dec 2016 13:09:25 -0500 Subject: [MINC-users] TIFF to MINC? Message-ID: I've got my hands on some autofluorescence data from a two-photon microscope that I'd like to convert into a MINC volume. Right now I have a stack aligned TIFF files. Has anyone tried something like this before and can share their process, otherwise, I'll try and document the final conversion process I figure out. Thanks. -- Gabriel A. Devenyi B.Eng. Ph.D. Research Computing Associate Computational Brain Anatomy Laboratory Cerebral Imaging Center Douglas Mental Health University Institute McGill University t: 514.761.6131x4781 e: gabriel.devenyi at mcgill.ca From sulantha.s at gmail.com Mon Dec 5 14:11:46 2016 From: sulantha.s at gmail.com (Sulantha Sanjeewa) Date: Mon, 5 Dec 2016 14:11:46 -0500 Subject: [MINC-users] TIFF to MINC? In-Reply-To: References: Message-ID: I haven?t done it but probably pyminc would be able to help. Opena dummy MINC file. Replace the internal data structure from the new matrix made from your images. Write back Should work (theoretically at least) Sulantha. On December 5, 2016 at 1:11:42 PM, Gabriel A. Devenyi ( gabriel.devenyi at mcgill.ca) wrote: I've got my hands on some autofluorescence data from a two-photon microscope that I'd like to convert into a MINC volume. Right now I have a stack aligned TIFF files. Has anyone tried something like this before and can share their process, otherwise, I'll try and document the final conversion process I figure out. Thanks. -- Gabriel A. Devenyi B.Eng. Ph.D. Research Computing Associate Computational Brain Anatomy Laboratory Cerebral Imaging Center Douglas Mental Health University Institute McGill University t: 514.761.6131x4781 e: gabriel.devenyi at mcgill.ca _______________________________________________ MINC-users at bic.mni.mcgill.ca http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users From gabriel.devenyi at mcgill.ca Mon Dec 5 14:14:19 2016 From: gabriel.devenyi at mcgill.ca (Gabriel A. Devenyi) Date: Mon, 5 Dec 2016 14:14:19 -0500 Subject: [MINC-users] TIFF to MINC? In-Reply-To: References: Message-ID: That's a totally better idea than hacking on rawtominc. Thanks! -- Gabriel A. Devenyi B.Eng. Ph.D. Research Computing Associate Computational Brain Anatomy Laboratory Cerebral Imaging Center Douglas Mental Health University Institute McGill University t: 514.761.6131x4781 e: gabriel.devenyi at mcgill.ca On Mon, Dec 5, 2016 at 2:11 PM, Sulantha Sanjeewa wrote: > I haven?t done it but probably pyminc would be able to help. > Opena dummy MINC file. > Replace the internal data structure from the new matrix made from your > images. > Write back > > Should work (theoretically at least) > Sulantha. > > > On December 5, 2016 at 1:11:42 PM, Gabriel A. Devenyi ( > gabriel.devenyi at mcgill.ca) wrote: > > I've got my hands on some autofluorescence data from a two-photon > microscope that I'd like to convert into a MINC volume. Right now I have a > stack aligned TIFF files. > > Has anyone tried something like this before and can share their process, > otherwise, I'll try and document the final conversion process I figure > out. > > Thanks. > > -- > Gabriel A. Devenyi B.Eng. Ph.D. > Research Computing Associate > Computational Brain Anatomy Laboratory > Cerebral Imaging Center > Douglas Mental Health University Institute > McGill University > t: 514.761.6131x4781 <(514)%20761-6131> > e: gabriel.devenyi at mcgill.ca > _______________________________________________ > MINC-users at bic.mni.mcgill.ca > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users > > From robert.d.vincent at mcgill.ca Mon Dec 5 14:15:29 2016 From: robert.d.vincent at mcgill.ca (Robert D. Vincent) Date: Mon, 5 Dec 2016 14:15:29 -0500 Subject: [MINC-users] TIFF to MINC? In-Reply-To: References: Message-ID: Gabriel, I would try extracting the TIFF pixels using something like imagemagick, then use rawtominc to convert to minc. -bert On Mon, Dec 5, 2016 at 2:11 PM, Sulantha Sanjeewa wrote: > I haven?t done it but probably pyminc would be able to help. > Opena dummy MINC file. > Replace the internal data structure from the new matrix made from your > images. > Write back > > Should work (theoretically at least) > Sulantha. > > > On December 5, 2016 at 1:11:42 PM, Gabriel A. Devenyi ( > gabriel.devenyi at mcgill.ca) wrote: > > I've got my hands on some autofluorescence data from a two-photon > microscope that I'd like to convert into a MINC volume. Right now I have a > stack aligned TIFF files. > > Has anyone tried something like this before and can share their process, > otherwise, I'll try and document the final conversion process I figure out. > > Thanks. > > -- > Gabriel A. Devenyi B.Eng. Ph.D. > Research Computing Associate > Computational Brain Anatomy Laboratory > Cerebral Imaging Center > Douglas Mental Health University Institute > McGill University > t: 514.761.6131x4781 > e: gabriel.devenyi at mcgill.ca > _______________________________________________ > MINC-users at bic.mni.mcgill.ca > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users > _______________________________________________ > MINC-users at bic.mni.mcgill.ca > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users > From gabriel.devenyi at mcgill.ca Mon Dec 5 14:18:35 2016 From: gabriel.devenyi at mcgill.ca (Gabriel A. Devenyi) Date: Mon, 5 Dec 2016 14:18:35 -0500 Subject: [MINC-users] TIFF to MINC? In-Reply-To: References: Message-ID: Thanks Bert, That's the other idea I'm working with. -- Gabriel A. Devenyi B.Eng. Ph.D. Research Computing Associate Computational Brain Anatomy Laboratory Cerebral Imaging Center Douglas Mental Health University Institute McGill University t: 514.761.6131x4781 e: gabriel.devenyi at mcgill.ca On Mon, Dec 5, 2016 at 2:15 PM, Robert Durham Vincent, Mr < robert.d.vincent at mcgill.ca> wrote: > Gabriel, > > I would try extracting the TIFF pixels using something like imagemagick, > then use rawtominc to convert to minc. > > -bert > > On Mon, Dec 5, 2016 at 2:11 PM, Sulantha Sanjeewa > wrote: > >> I haven?t done it but probably pyminc would be able to help. >> Opena dummy MINC file. >> Replace the internal data structure from the new matrix made from your >> images. >> Write back >> >> Should work (theoretically at least) >> Sulantha. >> >> >> On December 5, 2016 at 1:11:42 PM, Gabriel A. Devenyi ( >> gabriel.devenyi at mcgill.ca) wrote: >> >> I've got my hands on some autofluorescence data from a two-photon >> microscope that I'd like to convert into a MINC volume. Right now I have a >> stack aligned TIFF files. >> >> Has anyone tried something like this before and can share their process, >> otherwise, I'll try and document the final conversion process I figure >> out. >> >> Thanks. >> >> -- >> Gabriel A. Devenyi B.Eng. Ph.D. >> Research Computing Associate >> Computational Brain Anatomy Laboratory >> Cerebral Imaging Center >> Douglas Mental Health University Institute >> McGill University >> t: 514.761.6131x4781 >> e: gabriel.devenyi at mcgill.ca >> _______________________________________________ >> MINC-users at bic.mni.mcgill.ca >> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users >> _______________________________________________ >> MINC-users at bic.mni.mcgill.ca >> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users >> > > From vladimir.fonov at gmail.com Mon Dec 5 14:30:04 2016 From: vladimir.fonov at gmail.com (Vladimir S. FONOV) Date: Mon, 5 Dec 2016 14:30:04 -0500 Subject: [MINC-users] TIFF to MINC? In-Reply-To: References: Message-ID: <2f80058a-cb9d-177e-088b-956c9bee8b2f@gmail.com> c2d input.tiff output.mnc (c2d from minc-toolkit-v2) On 2016-12-05 02:18 PM, Gabriel A. Devenyi wrote: > Thanks Bert, > > That's the other idea I'm working with. > > -- > Gabriel A. Devenyi B.Eng. Ph.D. > Research Computing Associate > Computational Brain Anatomy Laboratory > Cerebral Imaging Center > Douglas Mental Health University Institute > McGill University > t: 514.761.6131x4781 > e: gabriel.devenyi at mcgill.ca > > On Mon, Dec 5, 2016 at 2:15 PM, Robert Durham Vincent, Mr < > robert.d.vincent at mcgill.ca> wrote: > >> Gabriel, >> >> I would try extracting the TIFF pixels using something like imagemagick, >> then use rawtominc to convert to minc. >> >> -bert >> >> On Mon, Dec 5, 2016 at 2:11 PM, Sulantha Sanjeewa >> wrote: >> >>> I haven?t done it but probably pyminc would be able to help. >>> Opena dummy MINC file. >>> Replace the internal data structure from the new matrix made from your >>> images. >>> Write back >>> >>> Should work (theoretically at least) >>> Sulantha. >>> >>> >>> On December 5, 2016 at 1:11:42 PM, Gabriel A. Devenyi ( >>> gabriel.devenyi at mcgill.ca) wrote: >>> >>> I've got my hands on some autofluorescence data from a two-photon >>> microscope that I'd like to convert into a MINC volume. Right now I have a >>> stack aligned TIFF files. >>> >>> Has anyone tried something like this before and can share their process, >>> otherwise, I'll try and document the final conversion process I figure >>> out. >>> >>> Thanks. >>> >>> -- >>> Gabriel A. Devenyi B.Eng. Ph.D. >>> Research Computing Associate >>> Computational Brain Anatomy Laboratory >>> Cerebral Imaging Center >>> Douglas Mental Health University Institute >>> McGill University >>> t: 514.761.6131x4781 >>> e: gabriel.devenyi at mcgill.ca >>> _______________________________________________ >>> MINC-users at bic.mni.mcgill.ca >>> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users >>> _______________________________________________ >>> MINC-users at bic.mni.mcgill.ca >>> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users >>> >> >> > _______________________________________________ > MINC-users at bic.mni.mcgill.ca > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users > -- Best regards, Vladimir S. FONOV ~ vladimir.fonov gmail.com From vladimir.fonov at gmail.com Mon Dec 5 14:33:54 2016 From: vladimir.fonov at gmail.com (Vladimir S. FONOV) Date: Mon, 5 Dec 2016 14:33:54 -0500 Subject: [MINC-users] TIFF to MINC? In-Reply-To: References: Message-ID: <34b6d080-8495-8d53-8c57-037e73c71058@gmail.com> c3d input*.tiff -tile z -o output.mnc will work too. On 2016-12-05 02:18 PM, Gabriel A. Devenyi wrote: > Thanks Bert, > > That's the other idea I'm working with. > > -- > Gabriel A. Devenyi B.Eng. Ph.D. > Research Computing Associate > Computational Brain Anatomy Laboratory > Cerebral Imaging Center > Douglas Mental Health University Institute > McGill University > t: 514.761.6131x4781 > e: gabriel.devenyi at mcgill.ca > > On Mon, Dec 5, 2016 at 2:15 PM, Robert Durham Vincent, Mr < > robert.d.vincent at mcgill.ca> wrote: > >> Gabriel, >> >> I would try extracting the TIFF pixels using something like imagemagick, >> then use rawtominc to convert to minc. >> >> -bert >> >> On Mon, Dec 5, 2016 at 2:11 PM, Sulantha Sanjeewa >> wrote: >> >>> I haven?t done it but probably pyminc would be able to help. >>> Opena dummy MINC file. >>> Replace the internal data structure from the new matrix made from your >>> images. >>> Write back >>> >>> Should work (theoretically at least) >>> Sulantha. >>> >>> >>> On December 5, 2016 at 1:11:42 PM, Gabriel A. Devenyi ( >>> gabriel.devenyi at mcgill.ca) wrote: >>> >>> I've got my hands on some autofluorescence data from a two-photon >>> microscope that I'd like to convert into a MINC volume. Right now I have a >>> stack aligned TIFF files. >>> >>> Has anyone tried something like this before and can share their process, >>> otherwise, I'll try and document the final conversion process I figure >>> out. >>> >>> Thanks. >>> >>> -- >>> Gabriel A. Devenyi B.Eng. Ph.D. >>> Research Computing Associate >>> Computational Brain Anatomy Laboratory >>> Cerebral Imaging Center >>> Douglas Mental Health University Institute >>> McGill University >>> t: 514.761.6131x4781 >>> e: gabriel.devenyi at mcgill.ca >>> _______________________________________________ >>> MINC-users at bic.mni.mcgill.ca >>> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users >>> _______________________________________________ >>> MINC-users at bic.mni.mcgill.ca >>> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users >>> >> >> > _______________________________________________ > MINC-users at bic.mni.mcgill.ca > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users > -- Best regards, Vladimir S. FONOV ~ vladimir.fonov gmail.com From a.janke at gmail.com Mon Dec 5 15:05:32 2016 From: a.janke at gmail.com (Andrew Janke) Date: Tue, 6 Dec 2016 06:05:32 +1000 Subject: [MINC-users] TIFF to MINC? In-Reply-To: References: Message-ID: Hi Gabriel, If these are relatively simple tiff files, have a look at img2mnc. https://github.com/BIC-MNI/minc-widgets/blob/master/img2mnc/img2mnc it's essentially mincpik turned upside down. However in the microscopy world tiff can take a number of meanings, OME TIFF is probably the most common in which a large amount of extra metadata is included. These are generated by the bioformats library (Omero), it's java based so you need to get familiar with it, the tool you want is bfconvert. Think of it as imagemagick for microscopy. OME Tiff files and other TIFF files in microscopy are often also tiled or what is called a bigtiff, these won't be read by traditional tools that read tiff files or perhaps they will but you'll only get the first frame/layer/chanel, for these you can either use bfconvert to get them to PNG or there is a small C tool that Claude wrote for bigtiff to minc. I have the code for this but haven't put it on github yet. let me know if you need it. On this note bioformats does have rudimentary read support for MINC, I currently have a project underway with the Omero group to incorporate better HDF5/MINC support within bioformats/omero. 3D data support is not done well in OMERO so I'm working with them to make MINC/HDF5 the answer to this. ta a On 6 December 2016 at 04:09, Gabriel A. Devenyi wrote: > I've got my hands on some autofluorescence data from a two-photon > microscope that I'd like to convert into a MINC volume. Right now I have a > stack aligned TIFF files. > > Has anyone tried something like this before and can share their process, > otherwise, I'll try and document the final conversion process I figure out. > > Thanks. > > -- > Gabriel A. Devenyi B.Eng. Ph.D. > Research Computing Associate > Computational Brain Anatomy Laboratory > Cerebral Imaging Center > Douglas Mental Health University Institute > McGill University > t: 514.761.6131x4781 > e: gabriel.devenyi at mcgill.ca > _______________________________________________ > MINC-users at bic.mni.mcgill.ca > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users From matthijs.vaneede at sickkids.ca Wed Dec 7 17:38:53 2016 From: matthijs.vaneede at sickkids.ca (Matthijs van Eede) Date: Wed, 7 Dec 2016 22:38:53 +0000 Subject: [MINC-users] Understanding memory usage in MINC(average) Message-ID: <960852518D084247B51C83A707C3EF3E53D49F1E@SKMBXX03.sickkids.ca> Hi All, We've had some issues with a couple of programs that use the MINC libraries in terms of their memory usage. On our old Ubuntu 12.04 system, some of our tools use a lot less memory than with a newer minc-toolkit-v2 on Ubuntu 16.04. It's a bit hard to get a clean test for this, but I've found that using mincaverage on both those systems also shows very different memory usage, and was wondering whether that's to be expected or not. Size of the input files to mincaverage: image: unsigned short 0 to 65535 image dimensions: zspace yspace xspace dimension name length step start -------------- ------ ---- ----- zspace 241 0.04 -4.2 yspace 478 0.04 -8.19 xspace 315 0.04 -6.27 On Ubuntu 12.04: mincaverage -version program: 2.3.00 libminc: 2.3.00 netcdf : 4.3.0 of Jan 15 2015 14:31:52 $ HDF5 : 1.8.10 Call: mincaverage [371 files] average.mnc Walltime (m:s): 32:29 Max memory usage: 0.6G Page size (bytes): 4096 On Ubuntu 16.04 (original minc-toolkit): mincaverage -version program: 2.3.01 libminc: 2.3.01 netcdf : 4.3.3 of Dec 7 2016 11:52:30 $ HDF5 : 1.8.15 Call: mincaverage [371 files] average.mnc Walltime (m:s): 5:48 Max memory usage: 7G Page size (bytes): 4096 On Ubuntu 16.04 (minc-toolkit-v2): mincaverage -version program: 2.4.02 libminc: 2.4.02 netcdf : 4.3.3.1 of Dec 7 2016 11:38:26 $ HDF5 : 1.8.17 Call: mincaverage [371 files] average.mnc Walltime (m:s): 6:30 Max memory usage: 7G Page size (bytes): 4096 The memory usage on 16.04 for both the original as well as the v2 toolkit is more than 10 fold that of the minc-toolkit on 12.04. At the same time the newer versions run 6 times faster. Both are set to use -max_buffer_size_in_kb 4096 which I thought would determine the total amount of memory used. Is that not a correct assumption? Happy with any insight! Matthijs ________________________________ This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies. From a.janke at gmail.com Wed Dec 7 20:18:10 2016 From: a.janke at gmail.com (Andrew Janke) Date: Thu, 8 Dec 2016 11:18:10 +1000 Subject: [MINC-users] Understanding memory usage in MINC(average) In-Reply-To: <960852518D084247B51C83A707C3EF3E53D49F1E@SKMBXX03.sickkids.ca> References: <960852518D084247B51C83A707C3EF3E53D49F1E@SKMBXX03.sickkids.ca> Message-ID: Hi Matthijs, A few things don't seem equal in your tests (HDF versions), but I'm going to take a punt as to the reason why. I'll note first that mincaverage is the "worst" at this as it has the capacity to open your 371 files at once. This could also be a MINC2 compression thing if your input files are compressed. Have a look at the thread on this commit: https://github.com/BIC-MNI/libminc/commit/08c0124648ad1411c8cbba6ab52862db86a770a5#commitcomment-13500107 The upshot being a new ENV var being parsed: MINC_MAX_MEMORY_KB As this dramatically speeds up such operations. This may also be dependent on the max number of open files that has changed over time, both in HDF5 and within MINC. Remember that max_buffer_size_in_kb only sets the buffer size within voxel_loop within mincxxx operations that use it, it has no effect on if your volumes are compressed (and thus might be caching) or the number of files that are open at once. It's for all of these reasons that I wrote mincbigaverage: https://github.com/BIC-MNI/minc-widgets/blob/master/mincbigaverage/mincbigaverage And tend to use it in pipelines in which I know there will be either a large number of input files or a number of large files. I find it faster than a call to mincaverage. Note that it also supports a -batch argument. This helps a lot in model generation pipelines in which the averaging step is a bottleneck. a On 8 December 2016 at 08:38, Matthijs van Eede wrote: > Hi All, > > We've had some issues with a couple of programs that use the MINC libraries in terms of their memory usage. On our old Ubuntu 12.04 system, some of our tools use a lot less memory than with a newer minc-toolkit-v2 on Ubuntu 16.04. It's a bit hard to get a clean test for this, but I've found that using mincaverage on both those systems also shows very different memory usage, and was wondering whether that's to be expected or not. > > Size of the input files to mincaverage: > image: unsigned short 0 to 65535 > image dimensions: zspace yspace xspace > dimension name length step start > -------------- ------ ---- ----- > zspace 241 0.04 -4.2 > yspace 478 0.04 -8.19 > xspace 315 0.04 -6.27 > > On Ubuntu 12.04: > mincaverage -version > program: 2.3.00 > libminc: 2.3.00 > netcdf : 4.3.0 of Jan 15 2015 14:31:52 $ > HDF5 : 1.8.10 > Call: mincaverage [371 files] average.mnc > Walltime (m:s): 32:29 > Max memory usage: 0.6G > Page size (bytes): 4096 > > On Ubuntu 16.04 (original minc-toolkit): > mincaverage -version > program: 2.3.01 > libminc: 2.3.01 > netcdf : 4.3.3 of Dec 7 2016 11:52:30 $ > HDF5 : 1.8.15 > Call: mincaverage [371 files] average.mnc > Walltime (m:s): 5:48 > Max memory usage: 7G > Page size (bytes): 4096 > > On Ubuntu 16.04 (minc-toolkit-v2): > mincaverage -version > program: 2.4.02 > libminc: 2.4.02 > netcdf : 4.3.3.1 of Dec 7 2016 11:38:26 $ > HDF5 : 1.8.17 > Call: mincaverage [371 files] average.mnc > Walltime (m:s): 6:30 > Max memory usage: 7G > Page size (bytes): 4096 > > The memory usage on 16.04 for both the original as well as the v2 toolkit is more than 10 fold that of the minc-toolkit on 12.04. At the same time the newer versions run 6 times faster. Both are set to use -max_buffer_size_in_kb 4096 which I thought would determine the total amount of memory used. Is that not a correct assumption? > > Happy with any insight! > Matthijs > > ________________________________ > > This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies. > _______________________________________________ > MINC-users at bic.mni.mcgill.ca > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users From a.janke at gmail.com Wed Dec 7 20:25:07 2016 From: a.janke at gmail.com (Andrew Janke) Date: Thu, 8 Dec 2016 11:25:07 +1000 Subject: [MINC-users] Understanding memory usage in MINC(average) In-Reply-To: References: <960852518D084247B51C83A707C3EF3E53D49F1E@SKMBXX03.sickkids.ca> Message-ID: For more history on this have a look at this thread: http://www.bic.mni.mcgill.ca/pipermail/minc-users/2015-June/004092.html and the resulting line in minc here: https://github.com/BIC-MNI/libminc/blob/95cd5374bd2864d0d927fa1ee0d0c203e7c7e0ff/libsrc/minc.h#L409 Changelog: https://github.com/BIC-MNI/libminc/blob/95cd5374bd2864d0d927fa1ee0d0c203e7c7e0ff/ChangeLog#L16 The main reason for this change was speed, with the previous value of 32 it meant that files were opened and closed multiple times for each voxel_loop chunk IIRC. Bert/Vlad may remember the details. a a On 8 December 2016 at 11:18, Andrew Janke wrote: > Hi Matthijs, > > A few things don't seem equal in your tests (HDF versions), but I'm > going to take a punt as to the reason why. I'll note first that > mincaverage is the "worst" at this as it has the capacity to open your > 371 files at once. This could also be a MINC2 compression thing if > your input files are compressed. Have a look at the thread on this > commit: > > https://github.com/BIC-MNI/libminc/commit/08c0124648ad1411c8cbba6ab52862db86a770a5#commitcomment-13500107 > > The upshot being a new ENV var being parsed: > > MINC_MAX_MEMORY_KB > > As this dramatically speeds up such operations. This may also be > dependent on the max number of open files that has changed over time, > both in HDF5 and within MINC. Remember that max_buffer_size_in_kb only > sets the buffer size within voxel_loop within mincxxx operations that > use it, it has no effect on if your volumes are compressed (and thus > might be caching) or the number of files that are open at once. > > It's for all of these reasons that I wrote mincbigaverage: > > https://github.com/BIC-MNI/minc-widgets/blob/master/mincbigaverage/mincbigaverage > > And tend to use it in pipelines in which I know there will be either a > large number of input files or a number of large files. I find it > faster than a call to mincaverage. Note that it also supports a -batch > argument. This helps a lot in model generation pipelines in which the > averaging step is a bottleneck. > > > a > > > On 8 December 2016 at 08:38, Matthijs van Eede > wrote: >> Hi All, >> >> We've had some issues with a couple of programs that use the MINC libraries in terms of their memory usage. On our old Ubuntu 12.04 system, some of our tools use a lot less memory than with a newer minc-toolkit-v2 on Ubuntu 16.04. It's a bit hard to get a clean test for this, but I've found that using mincaverage on both those systems also shows very different memory usage, and was wondering whether that's to be expected or not. >> >> Size of the input files to mincaverage: >> image: unsigned short 0 to 65535 >> image dimensions: zspace yspace xspace >> dimension name length step start >> -------------- ------ ---- ----- >> zspace 241 0.04 -4.2 >> yspace 478 0.04 -8.19 >> xspace 315 0.04 -6.27 >> >> On Ubuntu 12.04: >> mincaverage -version >> program: 2.3.00 >> libminc: 2.3.00 >> netcdf : 4.3.0 of Jan 15 2015 14:31:52 $ >> HDF5 : 1.8.10 >> Call: mincaverage [371 files] average.mnc >> Walltime (m:s): 32:29 >> Max memory usage: 0.6G >> Page size (bytes): 4096 >> >> On Ubuntu 16.04 (original minc-toolkit): >> mincaverage -version >> program: 2.3.01 >> libminc: 2.3.01 >> netcdf : 4.3.3 of Dec 7 2016 11:52:30 $ >> HDF5 : 1.8.15 >> Call: mincaverage [371 files] average.mnc >> Walltime (m:s): 5:48 >> Max memory usage: 7G >> Page size (bytes): 4096 >> >> On Ubuntu 16.04 (minc-toolkit-v2): >> mincaverage -version >> program: 2.4.02 >> libminc: 2.4.02 >> netcdf : 4.3.3.1 of Dec 7 2016 11:38:26 $ >> HDF5 : 1.8.17 >> Call: mincaverage [371 files] average.mnc >> Walltime (m:s): 6:30 >> Max memory usage: 7G >> Page size (bytes): 4096 >> >> The memory usage on 16.04 for both the original as well as the v2 toolkit is more than 10 fold that of the minc-toolkit on 12.04. At the same time the newer versions run 6 times faster. Both are set to use -max_buffer_size_in_kb 4096 which I thought would determine the total amount of memory used. Is that not a correct assumption? >> >> Happy with any insight! >> Matthijs >> >> ________________________________ >> >> This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies. >> _______________________________________________ >> MINC-users at bic.mni.mcgill.ca >> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users From matthijs.vaneede at sickkids.ca Thu Dec 8 12:02:20 2016 From: matthijs.vaneede at sickkids.ca (Matthijs van Eede) Date: Thu, 8 Dec 2016 17:02:20 +0000 Subject: [MINC-users] Understanding memory usage in MINC(average) In-Reply-To: References: <960852518D084247B51C83A707C3EF3E53D49F1E@SKMBXX03.sickkids.ca>, Message-ID: <960852518D084247B51C83A707C3EF3E53D4A05D@SKMBXX03.sickkids.ca> Hi Andrew, Thank you for the explanation! We have MINC_COMPRESS set to 4 by default, so all files produced will have some level of compression. This also perfectly mimics Vlad's comment about the speed-up together with the 10 fold memory increase. As per that thread, the environment variable to use is: MINC_FILE_CACHE_MB Which sets the maximum amount of memory to use for caching per file. MINC_MAX_MEMORY_KB has no effect on the memory usage of mincaverage. Interestingly, mincaverage ran in about the same time with MINC_FILE_CACHE_MB set to 100 (default) and 50, but only using half of the amount of memory in the latter case. I'll put a switch in our pydpiper code to use mincbigaverage when large number of files are used for a pipeline. Cheers, Matthijs ________________________________________ From: minc-users-bounces at bic.mni.mcgill.ca [minc-users-bounces at bic.mni.mcgill.ca] on behalf of Andrew Janke [a.janke at gmail.com] Sent: Wednesday, December 07, 2016 8:18 PM To: MINC users mailing list Subject: Re: [MINC-users] Understanding memory usage in MINC(average) Hi Matthijs, A few things don't seem equal in your tests (HDF versions), but I'm going to take a punt as to the reason why. I'll note first that mincaverage is the "worst" at this as it has the capacity to open your 371 files at once. This could also be a MINC2 compression thing if your input files are compressed. Have a look at the thread on this commit: https://github.com/BIC-MNI/libminc/commit/08c0124648ad1411c8cbba6ab52862db86a770a5#commitcomment-13500107 The upshot being a new ENV var being parsed: MINC_MAX_MEMORY_KB As this dramatically speeds up such operations. This may also be dependent on the max number of open files that has changed over time, both in HDF5 and within MINC. Remember that max_buffer_size_in_kb only sets the buffer size within voxel_loop within mincxxx operations that use it, it has no effect on if your volumes are compressed (and thus might be caching) or the number of files that are open at once. It's for all of these reasons that I wrote mincbigaverage: https://github.com/BIC-MNI/minc-widgets/blob/master/mincbigaverage/mincbigaverage And tend to use it in pipelines in which I know there will be either a large number of input files or a number of large files. I find it faster than a call to mincaverage. Note that it also supports a -batch argument. This helps a lot in model generation pipelines in which the averaging step is a bottleneck. a On 8 December 2016 at 08:38, Matthijs van Eede wrote: > Hi All, > > We've had some issues with a couple of programs that use the MINC libraries in terms of their memory usage. On our old Ubuntu 12.04 system, some of our tools use a lot less memory than with a newer minc-toolkit-v2 on Ubuntu 16.04. It's a bit hard to get a clean test for this, but I've found that using mincaverage on both those systems also shows very different memory usage, and was wondering whether that's to be expected or not. > > Size of the input files to mincaverage: > image: unsigned short 0 to 65535 > image dimensions: zspace yspace xspace > dimension name length step start > -------------- ------ ---- ----- > zspace 241 0.04 -4.2 > yspace 478 0.04 -8.19 > xspace 315 0.04 -6.27 > > On Ubuntu 12.04: > mincaverage -version > program: 2.3.00 > libminc: 2.3.00 > netcdf : 4.3.0 of Jan 15 2015 14:31:52 $ > HDF5 : 1.8.10 > Call: mincaverage [371 files] average.mnc > Walltime (m:s): 32:29 > Max memory usage: 0.6G > Page size (bytes): 4096 > > On Ubuntu 16.04 (original minc-toolkit): > mincaverage -version > program: 2.3.01 > libminc: 2.3.01 > netcdf : 4.3.3 of Dec 7 2016 11:52:30 $ > HDF5 : 1.8.15 > Call: mincaverage [371 files] average.mnc > Walltime (m:s): 5:48 > Max memory usage: 7G > Page size (bytes): 4096 > > On Ubuntu 16.04 (minc-toolkit-v2): > mincaverage -version > program: 2.4.02 > libminc: 2.4.02 > netcdf : 4.3.3.1 of Dec 7 2016 11:38:26 $ > HDF5 : 1.8.17 > Call: mincaverage [371 files] average.mnc > Walltime (m:s): 6:30 > Max memory usage: 7G > Page size (bytes): 4096 > > The memory usage on 16.04 for both the original as well as the v2 toolkit is more than 10 fold that of the minc-toolkit on 12.04. At the same time the newer versions run 6 times faster. Both are set to use -max_buffer_size_in_kb 4096 which I thought would determine the total amount of memory used. Is that not a correct assumption? > > Happy with any insight! > Matthijs > > ________________________________ > > This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies. > _______________________________________________ > MINC-users at bic.mni.mcgill.ca > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users _______________________________________________ MINC-users at bic.mni.mcgill.ca http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users ________________________________ This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies. From matthijs.vaneede at sickkids.ca Thu Dec 8 12:14:12 2016 From: matthijs.vaneede at sickkids.ca (Matthijs van Eede) Date: Thu, 8 Dec 2016 17:14:12 +0000 Subject: [MINC-users] Understanding memory usage in MINC(average) In-Reply-To: References: <960852518D084247B51C83A707C3EF3E53D49F1E@SKMBXX03.sickkids.ca> , Message-ID: <960852518D084247B51C83A707C3EF3E53D4A06E@SKMBXX03.sickkids.ca> In response to this, we're currently running pipelines with 1500 input files. Would we have to adjust MI_MAX_NUM_ICV in our case? Should we set it to a number that is larger than the number of MINC files we are likely to open at the same time? (Apart from mincaverage, some of the RMINC code performs calculations on a single slice of each input file in its call.) Thanks, Matthijs ________________________________________ From: minc-users-bounces at bic.mni.mcgill.ca [minc-users-bounces at bic.mni.mcgill.ca] on behalf of Andrew Janke [a.janke at gmail.com] Sent: Wednesday, December 07, 2016 8:25 PM To: MINC users mailing list Subject: Re: [MINC-users] Understanding memory usage in MINC(average) For more history on this have a look at this thread: http://www.bic.mni.mcgill.ca/pipermail/minc-users/2015-June/004092.html and the resulting line in minc here: https://github.com/BIC-MNI/libminc/blob/95cd5374bd2864d0d927fa1ee0d0c203e7c7e0ff/libsrc/minc.h#L409 Changelog: https://github.com/BIC-MNI/libminc/blob/95cd5374bd2864d0d927fa1ee0d0c203e7c7e0ff/ChangeLog#L16 The main reason for this change was speed, with the previous value of 32 it meant that files were opened and closed multiple times for each voxel_loop chunk IIRC. Bert/Vlad may remember the details. a a On 8 December 2016 at 11:18, Andrew Janke wrote: > Hi Matthijs, > > A few things don't seem equal in your tests (HDF versions), but I'm > going to take a punt as to the reason why. I'll note first that > mincaverage is the "worst" at this as it has the capacity to open your > 371 files at once. This could also be a MINC2 compression thing if > your input files are compressed. Have a look at the thread on this > commit: > > https://github.com/BIC-MNI/libminc/commit/08c0124648ad1411c8cbba6ab52862db86a770a5#commitcomment-13500107 > > The upshot being a new ENV var being parsed: > > MINC_MAX_MEMORY_KB > > As this dramatically speeds up such operations. This may also be > dependent on the max number of open files that has changed over time, > both in HDF5 and within MINC. Remember that max_buffer_size_in_kb only > sets the buffer size within voxel_loop within mincxxx operations that > use it, it has no effect on if your volumes are compressed (and thus > might be caching) or the number of files that are open at once. > > It's for all of these reasons that I wrote mincbigaverage: > > https://github.com/BIC-MNI/minc-widgets/blob/master/mincbigaverage/mincbigaverage > > And tend to use it in pipelines in which I know there will be either a > large number of input files or a number of large files. I find it > faster than a call to mincaverage. Note that it also supports a -batch > argument. This helps a lot in model generation pipelines in which the > averaging step is a bottleneck. > > > a > > > On 8 December 2016 at 08:38, Matthijs van Eede > wrote: >> Hi All, >> >> We've had some issues with a couple of programs that use the MINC libraries in terms of their memory usage. On our old Ubuntu 12.04 system, some of our tools use a lot less memory than with a newer minc-toolkit-v2 on Ubuntu 16.04. It's a bit hard to get a clean test for this, but I've found that using mincaverage on both those systems also shows very different memory usage, and was wondering whether that's to be expected or not. >> >> Size of the input files to mincaverage: >> image: unsigned short 0 to 65535 >> image dimensions: zspace yspace xspace >> dimension name length step start >> -------------- ------ ---- ----- >> zspace 241 0.04 -4.2 >> yspace 478 0.04 -8.19 >> xspace 315 0.04 -6.27 >> >> On Ubuntu 12.04: >> mincaverage -version >> program: 2.3.00 >> libminc: 2.3.00 >> netcdf : 4.3.0 of Jan 15 2015 14:31:52 $ >> HDF5 : 1.8.10 >> Call: mincaverage [371 files] average.mnc >> Walltime (m:s): 32:29 >> Max memory usage: 0.6G >> Page size (bytes): 4096 >> >> On Ubuntu 16.04 (original minc-toolkit): >> mincaverage -version >> program: 2.3.01 >> libminc: 2.3.01 >> netcdf : 4.3.3 of Dec 7 2016 11:52:30 $ >> HDF5 : 1.8.15 >> Call: mincaverage [371 files] average.mnc >> Walltime (m:s): 5:48 >> Max memory usage: 7G >> Page size (bytes): 4096 >> >> On Ubuntu 16.04 (minc-toolkit-v2): >> mincaverage -version >> program: 2.4.02 >> libminc: 2.4.02 >> netcdf : 4.3.3.1 of Dec 7 2016 11:38:26 $ >> HDF5 : 1.8.17 >> Call: mincaverage [371 files] average.mnc >> Walltime (m:s): 6:30 >> Max memory usage: 7G >> Page size (bytes): 4096 >> >> The memory usage on 16.04 for both the original as well as the v2 toolkit is more than 10 fold that of the minc-toolkit on 12.04. At the same time the newer versions run 6 times faster. Both are set to use -max_buffer_size_in_kb 4096 which I thought would determine the total amount of memory used. Is that not a correct assumption? >> >> Happy with any insight! >> Matthijs >> >> ________________________________ >> >> This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies. >> _______________________________________________ >> MINC-users at bic.mni.mcgill.ca >> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users _______________________________________________ MINC-users at bic.mni.mcgill.ca http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users ________________________________ This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies. From vladimir.fonov at gmail.com Thu Dec 8 12:45:55 2016 From: vladimir.fonov at gmail.com (Vladimir S. Fonov) Date: Thu, 8 Dec 2016 12:45:55 -0500 Subject: [MINC-users] Understanding memory usage in MINC(average) In-Reply-To: <960852518D084247B51C83A707C3EF3E53D4A05D@SKMBXX03.sickkids.ca> References: <960852518D084247B51C83A707C3EF3E53D49F1E@SKMBXX03.sickkids.ca> <960852518D084247B51C83A707C3EF3E53D4A05D@SKMBXX03.sickkids.ca> Message-ID: <8f94b3c3-75d9-ad97-5f71-08b0018763d0@gmail.com> Hello, this setting (and file caching) affects only io with MINC2 files, by setting https://support.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetCache by default (if MINC_FILE_CACHE_MB is not set) , it is set to MI_MAX_VAR_BUFFER_SIZE*10 , and MI_MAX_VAR_BUFFER_SIZE=1000000 see libsrc/hdf_convenience.c:2223 and it is used per file (i.e opening 100 files will use 100*1000000*10 bytes) On 2016-12-08 12:02 PM, Matthijs van Eede wrote: > Hi Andrew, > > Thank you for the explanation! We have MINC_COMPRESS set to 4 by default, so all files produced will have some level of compression. This also perfectly mimics Vlad's comment about the speed-up together with the 10 fold memory increase. As per that thread, the environment variable to use is: > > MINC_FILE_CACHE_MB > > Which sets the maximum amount of memory to use for caching per file. MINC_MAX_MEMORY_KB has no effect on the memory usage of mincaverage. Interestingly, mincaverage ran in about the same time with MINC_FILE_CACHE_MB set to 100 (default) and 50, but only using half of the amount of memory in the latter case. > > I'll put a switch in our pydpiper code to use mincbigaverage when large number of files are used for a pipeline. > > Cheers, > Matthijs > > ________________________________________ > From: minc-users-bounces at bic.mni.mcgill.ca [minc-users-bounces at bic.mni.mcgill.ca] on behalf of Andrew Janke [a.janke at gmail.com] > Sent: Wednesday, December 07, 2016 8:18 PM > To: MINC users mailing list > Subject: Re: [MINC-users] Understanding memory usage in MINC(average) > > Hi Matthijs, > > A few things don't seem equal in your tests (HDF versions), but I'm > going to take a punt as to the reason why. I'll note first that > mincaverage is the "worst" at this as it has the capacity to open your > 371 files at once. This could also be a MINC2 compression thing if > your input files are compressed. Have a look at the thread on this > commit: > > https://github.com/BIC-MNI/libminc/commit/08c0124648ad1411c8cbba6ab52862db86a770a5#commitcomment-13500107 > > The upshot being a new ENV var being parsed: > > MINC_MAX_MEMORY_KB > > As this dramatically speeds up such operations. This may also be > dependent on the max number of open files that has changed over time, > both in HDF5 and within MINC. Remember that max_buffer_size_in_kb only > sets the buffer size within voxel_loop within mincxxx operations that > use it, it has no effect on if your volumes are compressed (and thus > might be caching) or the number of files that are open at once. > > It's for all of these reasons that I wrote mincbigaverage: > > https://github.com/BIC-MNI/minc-widgets/blob/master/mincbigaverage/mincbigaverage > > And tend to use it in pipelines in which I know there will be either a > large number of input files or a number of large files. I find it > faster than a call to mincaverage. Note that it also supports a -batch > argument. This helps a lot in model generation pipelines in which the > averaging step is a bottleneck. > > > a > > > On 8 December 2016 at 08:38, Matthijs van Eede > wrote: >> Hi All, >> >> We've had some issues with a couple of programs that use the MINC libraries in terms of their memory usage. On our old Ubuntu 12.04 system, some of our tools use a lot less memory than with a newer minc-toolkit-v2 on Ubuntu 16.04. It's a bit hard to get a clean test for this, but I've found that using mincaverage on both those systems also shows very different memory usage, and was wondering whether that's to be expected or not. >> >> Size of the input files to mincaverage: >> image: unsigned short 0 to 65535 >> image dimensions: zspace yspace xspace >> dimension name length step start >> -------------- ------ ---- ----- >> zspace 241 0.04 -4.2 >> yspace 478 0.04 -8.19 >> xspace 315 0.04 -6.27 >> >> On Ubuntu 12.04: >> mincaverage -version >> program: 2.3.00 >> libminc: 2.3.00 >> netcdf : 4.3.0 of Jan 15 2015 14:31:52 $ >> HDF5 : 1.8.10 >> Call: mincaverage [371 files] average.mnc >> Walltime (m:s): 32:29 >> Max memory usage: 0.6G >> Page size (bytes): 4096 >> >> On Ubuntu 16.04 (original minc-toolkit): >> mincaverage -version >> program: 2.3.01 >> libminc: 2.3.01 >> netcdf : 4.3.3 of Dec 7 2016 11:52:30 $ >> HDF5 : 1.8.15 >> Call: mincaverage [371 files] average.mnc >> Walltime (m:s): 5:48 >> Max memory usage: 7G >> Page size (bytes): 4096 >> >> On Ubuntu 16.04 (minc-toolkit-v2): >> mincaverage -version >> program: 2.4.02 >> libminc: 2.4.02 >> netcdf : 4.3.3.1 of Dec 7 2016 11:38:26 $ >> HDF5 : 1.8.17 >> Call: mincaverage [371 files] average.mnc >> Walltime (m:s): 6:30 >> Max memory usage: 7G >> Page size (bytes): 4096 >> >> The memory usage on 16.04 for both the original as well as the v2 toolkit is more than 10 fold that of the minc-toolkit on 12.04. At the same time the newer versions run 6 times faster. Both are set to use -max_buffer_size_in_kb 4096 which I thought would determine the total amount of memory used. Is that not a correct assumption? >> >> Happy with any insight! >> Matthijs >> >> ________________________________ >> >> This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies. >> _______________________________________________ >> MINC-users at bic.mni.mcgill.ca >> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users > _______________________________________________ > MINC-users at bic.mni.mcgill.ca > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users > > ________________________________ > > This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies. > _______________________________________________ > MINC-users at bic.mni.mcgill.ca > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users > From a.janke at gmail.com Thu Dec 8 17:11:26 2016 From: a.janke at gmail.com (Andrew Janke) Date: Fri, 9 Dec 2016 08:11:26 +1000 Subject: [MINC-users] Understanding memory usage in MINC(average) In-Reply-To: <960852518D084247B51C83A707C3EF3E53D4A06E@SKMBXX03.sickkids.ca> References: <960852518D084247B51C83A707C3EF3E53D49F1E@SKMBXX03.sickkids.ca> <960852518D084247B51C83A707C3EF3E53D4A06E@SKMBXX03.sickkids.ca> Message-ID: Hi Matthijs, This one can't be answered for all cases. If your files are all 100kb in size, then yes, open them all at once, presuming HDF5 doesn't run out of file handles. If they are all 100GB with compression on then nope.. For 1500 files, I'd say leaving it at the default (1000) will be fine, it's only a few extra operations per voxel-loop chunk so should be OK. There will be some complex heuristic that will relate number of files, size of files, level of compression, size of voxel loop cache and number of ICV's, and it will likely differ from machine to machine given differing CPU cache sizes, memory bandwidth, CPU speed and disk I/O. It's for this reason that we chose to keep things simple with the current ENV vars as there was no easy way to determine the number of files that would be opened within libminc or to determine the value of the current vars based upon this. I recall a conversation about this on minc-dev but can't find it right now. TL;DR: use mincbigaverage and atomise other operations where possible. a On 9 December 2016 at 03:14, Matthijs van Eede wrote: > In response to this, we're currently running pipelines with 1500 input files. Would we have to adjust MI_MAX_NUM_ICV in our case? Should we set it to a number that is larger than the number of MINC files we are likely to open at the same time? (Apart from mincaverage, some of the RMINC code performs calculations on a single slice of each input file in its call.) From vladimir.fonov at gmail.com Thu Dec 8 17:14:14 2016 From: vladimir.fonov at gmail.com (Vladimir S. FONOV) Date: Thu, 8 Dec 2016 17:14:14 -0500 Subject: [MINC-users] Understanding memory usage in MINC(average) In-Reply-To: References: <960852518D084247B51C83A707C3EF3E53D49F1E@SKMBXX03.sickkids.ca> <960852518D084247B51C83A707C3EF3E53D4A06E@SKMBXX03.sickkids.ca> Message-ID: <30ca05ee-7834-06d4-9b0f-4a352e6d2e71@gmail.com> Hello, RMINC is using MINC2 API calls, they are not affected by this issue (i.e it uses default caching value from HDF5). On 2016-12-08 05:11 PM, Andrew Janke wrote: > Hi Matthijs, > > This one can't be answered for all cases. If your files are all 100kb > in size, then yes, open them all at once, presuming HDF5 doesn't run > out of file handles. If they are all 100GB with compression on then > nope.. > > For 1500 files, I'd say leaving it at the default (1000) will be fine, > it's only a few extra operations per voxel-loop chunk so should be OK. > There will be some complex heuristic that will relate number of files, > size of files, level of compression, size of voxel loop cache and > number of ICV's, and it will likely differ from machine to machine > given differing CPU cache sizes, memory bandwidth, CPU speed and disk > I/O. > > It's for this reason that we chose to keep things simple with the > current ENV vars as there was no easy way to determine the number of > files that would be opened within libminc or to determine the value of > the current vars based upon this. > > I recall a conversation about this on minc-dev but can't find it right now. > > TL;DR: use mincbigaverage and atomise other operations where possible. > > > a > > > > On 9 December 2016 at 03:14, Matthijs van Eede > wrote: >> In response to this, we're currently running pipelines with 1500 input files. Would we have to adjust MI_MAX_NUM_ICV in our case? Should we set it to a number that is larger than the number of MINC files we are likely to open at the same time? (Apart from mincaverage, some of the RMINC code performs calculations on a single slice of each input file in its call.) > _______________________________________________ > MINC-users at bic.mni.mcgill.ca > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users > -- Best regards, Vladimir S. FONOV ~ vladimir.fonov gmail.com From sorench at gmail.com Tue Dec 13 23:52:00 2016 From: sorench at gmail.com (Soren Christensen) Date: Tue, 13 Dec 2016 20:52:00 -0800 Subject: [MINC-users] minc-toolkit v2 build error Message-ID: Hi, I am building minc toolkit V2 using cmake 3.7.1 and getting this: [ 71%] Linking C executable volume_cog lib_minctracc.a(init_params.c.o): In function `vol_cog': init_params.c:(.text+0x0): multiple definition of `vol_cog' CMakeFiles/volume_cog.dir/Extra_progs/volume_cog.c.o:volume_cog.c:(.text+0x0): first defined here collect2: error: ld returned 1 exit status make[2]: *** [mni_autoreg/minctracc/volume_cog] Error 1 make[1]: *** [mni_autoreg/minctracc/CMakeFiles/volume_cog.dir/all] Error 2 make: *** [all] Error 2 Ideas appreciated! Thanks, Soren From ptcougopinto at gmail.com Wed Dec 21 11:34:26 2016 From: ptcougopinto at gmail.com (Pedro) Date: Wed, 21 Dec 2016 14:34:26 -0200 Subject: [MINC-users] Generate image file with label Message-ID: <4F2CBF49-E9D4-483B-9A67-80C598D5D421@gmail.com> Hi How can I generate an image file (JPEG, NPG etc) of a slice combined with a corresponding ROI. Mincpik doesn?t seem to give that option, and I can?t find out how to do it with Display. Thanks! Pedro From vladimir.fonov at gmail.com Wed Dec 21 11:38:43 2016 From: vladimir.fonov at gmail.com (Vladimir S. FONOV) Date: Wed, 21 Dec 2016 11:38:43 -0500 Subject: [MINC-users] Generate image file with label In-Reply-To: <4F2CBF49-E9D4-483B-9A67-80C598D5D421@gmail.com> References: <4F2CBF49-E9D4-483B-9A67-80C598D5D421@gmail.com> Message-ID: <397f4499-e42c-1217-6df1-d3dc6de6e262@gmail.com> See https://github.com/BIC-MNI/EZminc/blob/ITK4/scripts/minc_pretty_pic.pl https://github.com/BIC-MNI/EZminc/blob/ITK4/scripts/minc_qc.pl On 2016-12-21 11:34 AM, Pedro wrote: > Hi > > How can I generate an image file (JPEG, NPG etc) of a slice combined with a corresponding ROI. Mincpik doesn?t seem to give that option, and I can?t find out how to do it with Display. > > Thanks! > Pedro > _______________________________________________ > MINC-users at bic.mni.mcgill.ca > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users > -- Best regards, Vladimir S. FONOV ~ vladimir.fonov gmail.com From robert.d.vincent at mcgill.ca Wed Dec 21 11:47:52 2016 From: robert.d.vincent at mcgill.ca (Robert D. Vincent) Date: Wed, 21 Dec 2016 11:47:52 -0500 Subject: [MINC-users] Generate image file with label In-Reply-To: <4F2CBF49-E9D4-483B-9A67-80C598D5D421@gmail.com> References: <4F2CBF49-E9D4-483B-9A67-80C598D5D421@gmail.com> Message-ID: If you want to do it in Display, you should be able to select the "File" menu, then the command "Save Slice Image". The mouse cursor has to be over the slice image you want to save. On Wed, Dec 21, 2016 at 11:34 AM, Pedro wrote: > Hi > > How can I generate an image file (JPEG, NPG etc) of a slice combined with > a corresponding ROI. Mincpik doesn?t seem to give that option, and I can?t > find out how to do it with Display. > > Thanks! > Pedro > _______________________________________________ > MINC-users at bic.mni.mcgill.ca > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users > From zijdenbos at gmail.com Wed Dec 21 15:22:16 2016 From: zijdenbos at gmail.com (Alex Zijdenbos) Date: Wed, 21 Dec 2016 15:22:16 -0500 Subject: [MINC-users] Generate image file with label In-Reply-To: References: <4F2CBF49-E9D4-483B-9A67-80C598D5D421@gmail.com> Message-ID: ILT's create_verify_image will do this nicely as well On Wed, Dec 21, 2016 at 11:47 AM, Robert D. Vincent < robert.d.vincent at mcgill.ca> wrote: > If you want to do it in Display, you should be able to select the "File" > menu, then the command "Save Slice Image". The mouse cursor has to be over > the slice image you want to save. > > On Wed, Dec 21, 2016 at 11:34 AM, Pedro wrote: > > > Hi > > > > How can I generate an image file (JPEG, NPG etc) of a slice combined with > > a corresponding ROI. Mincpik doesn?t seem to give that option, and I > can?t > > find out how to do it with Display. > > > > Thanks! > > Pedro > > _______________________________________________ > > MINC-users at bic.mni.mcgill.ca > > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users > > > _______________________________________________ > MINC-users at bic.mni.mcgill.ca > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users > From vladimir.fonov at gmail.com Wed Dec 21 15:26:01 2016 From: vladimir.fonov at gmail.com (Vladimir S. FONOV) Date: Wed, 21 Dec 2016 15:26:01 -0500 Subject: [MINC-users] Generate image file with label In-Reply-To: References: <4F2CBF49-E9D4-483B-9A67-80C598D5D421@gmail.com> Message-ID: <54ffeebe-8f01-7765-44cf-1616de07a9fc@gmail.com> By the way , ILT is waiting for a loving hand to fix it: https://github.com/BIC-MNI/ILT/issues/4 On 2016-12-21 03:22 PM, Alex Zijdenbos wrote: > ILT's create_verify_image will do this nicely as well > > On Wed, Dec 21, 2016 at 11:47 AM, Robert D. Vincent < > robert.d.vincent at mcgill.ca> wrote: > >> If you want to do it in Display, you should be able to select the "File" >> menu, then the command "Save Slice Image". The mouse cursor has to be over >> the slice image you want to save. >> >> On Wed, Dec 21, 2016 at 11:34 AM, Pedro wrote: >> >>> Hi >>> >>> How can I generate an image file (JPEG, NPG etc) of a slice combined with >>> a corresponding ROI. Mincpik doesn?t seem to give that option, and I >> can?t >>> find out how to do it with Display. >>> >>> Thanks! >>> Pedro >>> _______________________________________________ >>> MINC-users at bic.mni.mcgill.ca >>> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users >>> >> _______________________________________________ >> MINC-users at bic.mni.mcgill.ca >> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users >> > _______________________________________________ > MINC-users at bic.mni.mcgill.ca > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users > -- Best regards, Vladimir S. FONOV ~ vladimir.fonov gmail.com