[MINC-users] Understanding memory usage in MINC(average)

[Andrew Janke]

For more history on this have a look at this thread:

   http://www.bic.mni.mcgill.ca/pipermail/minc-users/2015-June/004092.html

and the resulting line in minc here:

   https://github.com/BIC-MNI/libminc/blob/95cd5374bd2864d0d927fa1ee0d0c203e7c7e0ff/libsrc/minc.h#L409

Changelog:

  https://github.com/BIC-MNI/libminc/blob/95cd5374bd2864d0d927fa1ee0d0c203e7c7e0ff/ChangeLog#L16

The main reason for this change was speed, with the previous value of
32 it meant that files were opened and closed multiple times for each
voxel_loop chunk IIRC.  Bert/Vlad may remember the details.


a

a

On 8 December 2016 at 11:18, Andrew Janke <a.janke at gmail.com> wrote:
> Hi Matthijs,
>
> A few things don't seem equal in your tests (HDF versions), but I'm
> going to take a punt as to the reason why. I'll note first that
> mincaverage is the "worst" at this as it has the capacity to open your
> 371 files at once.  This could also be a MINC2 compression thing if
> your input files are compressed. Have a look at the thread on this
> commit:
>
>    https://github.com/BIC-MNI/libminc/commit/08c0124648ad1411c8cbba6ab52862db86a770a5#commitcomment-13500107
>
> The upshot being a new ENV var being parsed:
>
>    MINC_MAX_MEMORY_KB
>
> As this dramatically speeds up such operations. This may also be
> dependent on the max number of open files that has changed over time,
> both in HDF5 and within MINC. Remember that max_buffer_size_in_kb only
> sets the buffer size within voxel_loop within mincxxx operations that
> use it, it has no effect on if your volumes are compressed (and thus
> might be caching) or the number of files that are open at once.
>
>  It's for all of these reasons that I wrote mincbigaverage:
>
>    https://github.com/BIC-MNI/minc-widgets/blob/master/mincbigaverage/mincbigaverage
>
> And tend to use it in pipelines in which I know there will be either a
> large number of input files or a number of large files. I find it
> faster than a call to mincaverage. Note that it also supports a -batch
> argument. This helps a lot in model generation pipelines in which the
> averaging step is a bottleneck.
>
>
> a
>
>
> On 8 December 2016 at 08:38, Matthijs van Eede
> <matthijs.vaneede at sickkids.ca> wrote:
>> Hi All,
>>
>> We've had some issues with a couple of programs that use the MINC libraries in terms of their memory usage. On our old Ubuntu 12.04 system, some of our tools use a lot less memory than with a newer minc-toolkit-v2 on Ubuntu 16.04. It's a bit hard to get a clean test for this, but I've found that using mincaverage on both those systems also shows very different memory usage, and was wondering whether that's to be expected or not.
>>
>> Size of the input files to mincaverage:
>> image: unsigned short 0 to 65535
>> image dimensions: zspace yspace xspace
>>     dimension name         length         step        start
>>     --------------         ------         ----        -----
>>     zspace                    241         0.04         -4.2
>>     yspace                    478         0.04        -8.19
>>     xspace                    315         0.04        -6.27
>>
>> On Ubuntu 12.04:
>> mincaverage  -version
>> program: 2.3.00
>> libminc: 2.3.00
>> netcdf : 4.3.0 of Jan 15 2015 14:31:52 $
>> HDF5   : 1.8.10
>> Call: mincaverage [371 files] average.mnc
>> Walltime (m:s): 32:29
>> Max memory usage: 0.6G
>> Page size (bytes): 4096
>>
>> On Ubuntu 16.04 (original minc-toolkit):
>> mincaverage -version
>> program: 2.3.01
>> libminc: 2.3.01
>> netcdf : 4.3.3 of Dec  7 2016 11:52:30 $
>> HDF5   : 1.8.15
>> Call: mincaverage [371 files] average.mnc
>> Walltime (m:s): 5:48
>> Max memory usage: 7G
>> Page size (bytes): 4096
>>
>> On Ubuntu 16.04 (minc-toolkit-v2):
>> mincaverage  -version
>> program: 2.4.02
>> libminc: 2.4.02
>> netcdf : 4.3.3.1 of Dec  7 2016 11:38:26 $
>> HDF5   : 1.8.17
>> Call: mincaverage [371 files] average.mnc
>> Walltime (m:s): 6:30
>> Max memory usage: 7G
>> Page size (bytes): 4096
>>
>> The memory usage on 16.04 for both the original as well as the v2 toolkit is more than 10 fold that of the minc-toolkit on 12.04. At the same time the newer versions run 6 times faster. Both are set to use -max_buffer_size_in_kb 4096 which I thought would determine the total amount of memory used. Is that not a correct assumption?
>>
>> Happy with any insight!
>> Matthijs
>>
>> ________________________________
>>
>> This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies.
>> _______________________________________________
>> MINC-users at bic.mni.mcgill.ca
>> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users