[MINC-users] Understanding memory usage in MINC(average)

Thu Dec 8 12:02:20 EST 2016

Hi Andrew,

Thank you for the explanation! We have MINC_COMPRESS set to 4 by default, so all files produced will have some level of compression. This also perfectly mimics Vlad's comment about the speed-up together with the 10 fold memory increase. As per that thread, the environment variable to use is:

MINC_FILE_CACHE_MB

Which sets the maximum amount of memory to use for caching per file. MINC_MAX_MEMORY_KB has no effect on the memory usage of mincaverage. Interestingly, mincaverage ran in about the same time with MINC_FILE_CACHE_MB set to 100 (default) and 50, but only using half of the amount of memory in the latter case.

I'll put a switch in our pydpiper code to use mincbigaverage when large number of files are used for a pipeline.

Cheers,
Matthijs

________________________________________
From: minc-users-bounces at bic.mni.mcgill.ca [minc-users-bounces at bic.mni.mcgill.ca] on behalf of Andrew Janke [a.janke at gmail.com]
Sent: Wednesday, December 07, 2016 8:18 PM
To: MINC users mailing list
Subject: Re: [MINC-users] Understanding memory usage in MINC(average)

Hi Matthijs,

A few things don't seem equal in your tests (HDF versions), but I'm
going to take a punt as to the reason why. I'll note first that
mincaverage is the "worst" at this as it has the capacity to open your
371 files at once.  This could also be a MINC2 compression thing if
your input files are compressed. Have a look at the thread on this
commit:

   https://github.com/BIC-MNI/libminc/commit/08c0124648ad1411c8cbba6ab52862db86a770a5#commitcomment-13500107

The upshot being a new ENV var being parsed:

   MINC_MAX_MEMORY_KB

As this dramatically speeds up such operations. This may also be
dependent on the max number of open files that has changed over time,
both in HDF5 and within MINC. Remember that max_buffer_size_in_kb only
sets the buffer size within voxel_loop within mincxxx operations that
use it, it has no effect on if your volumes are compressed (and thus
might be caching) or the number of files that are open at once.

 It's for all of these reasons that I wrote mincbigaverage:

   https://github.com/BIC-MNI/minc-widgets/blob/master/mincbigaverage/mincbigaverage

And tend to use it in pipelines in which I know there will be either a
large number of input files or a number of large files. I find it
faster than a call to mincaverage. Note that it also supports a -batch
argument. This helps a lot in model generation pipelines in which the
averaging step is a bottleneck.

a

On 8 December 2016 at 08:38, Matthijs van Eede
<matthijs.vaneede at sickkids.ca> wrote:
> Hi All,
>
> We've had some issues with a couple of programs that use the MINC libraries in terms of their memory usage. On our old Ubuntu 12.04 system, some of our tools use a lot less memory than with a newer minc-toolkit-v2 on Ubuntu 16.04. It's a bit hard to get a clean test for this, but I've found that using mincaverage on both those systems also shows very different memory usage, and was wondering whether that's to be expected or not.
>
> Size of the input files to mincaverage:
> image: unsigned short 0 to 65535
> image dimensions: zspace yspace xspace
>     dimension name         length         step        start
>     --------------         ------         ----        -----
>     zspace                    241         0.04         -4.2
>     yspace                    478         0.04        -8.19
>     xspace                    315         0.04        -6.27
>
> On Ubuntu 12.04:
> mincaverage  -version
> program: 2.3.00
> libminc: 2.3.00
> netcdf : 4.3.0 of Jan 15 2015 14:31:52 $
> HDF5   : 1.8.10
> Call: mincaverage [371 files] average.mnc
> Walltime (m:s): 32:29
> Max memory usage: 0.6G
> Page size (bytes): 4096
>
> On Ubuntu 16.04 (original minc-toolkit):
> mincaverage -version
> program: 2.3.01
> libminc: 2.3.01
> netcdf : 4.3.3 of Dec  7 2016 11:52:30 $
> HDF5   : 1.8.15
> Call: mincaverage [371 files] average.mnc
> Walltime (m:s): 5:48
> Max memory usage: 7G
> Page size (bytes): 4096
>
> On Ubuntu 16.04 (minc-toolkit-v2):
> mincaverage  -version
> program: 2.4.02
> libminc: 2.4.02
> netcdf : 4.3.3.1 of Dec  7 2016 11:38:26 $
> HDF5   : 1.8.17
> Call: mincaverage [371 files] average.mnc
> Walltime (m:s): 6:30
> Max memory usage: 7G
> Page size (bytes): 4096
>
> The memory usage on 16.04 for both the original as well as the v2 toolkit is more than 10 fold that of the minc-toolkit on 12.04. At the same time the newer versions run 6 times faster. Both are set to use -max_buffer_size_in_kb 4096 which I thought would determine the total amount of memory used. Is that not a correct assumption?
>
> Happy with any insight!
> Matthijs
>
> ________________________________
>
> This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies.
> _______________________________________________
> MINC-users at bic.mni.mcgill.ca
> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
_______________________________________________
MINC-users at bic.mni.mcgill.ca
http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users

________________________________

This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies.