[MINC-users] Understanding memory usage in MINC(average)

[Andrew Janke]

Hi Matthijs,

This one can't be answered for all cases. If your files are all 100kb
in size, then yes, open them all at once, presuming HDF5 doesn't run
out of file handles. If they are all 100GB with compression on then
nope..

For 1500 files, I'd say leaving it at the default (1000) will be fine,
it's only a few extra operations per voxel-loop chunk so should be OK.
There will be some complex heuristic that will relate number of files,
size of files, level of compression, size of voxel loop cache and
number of ICV's, and it will likely differ from machine to machine
given differing CPU cache sizes, memory bandwidth, CPU speed and disk
I/O.

It's for this reason that we chose to keep things simple with the
current ENV vars as there was no easy way to determine the number of
files that would be opened within libminc or to determine the value of
the current vars based upon this.

I recall a conversation about this on minc-dev but can't find it right now.

TL;DR: use mincbigaverage and atomise other operations where possible.


a



On 9 December 2016 at 03:14, Matthijs van Eede
<matthijs.vaneede at sickkids.ca> wrote:
> In response to this, we're currently running pipelines with 1500 input files. Would we have to adjust MI_MAX_NUM_ICV in our case? Should we set it to a number that is larger than the number of MINC files we are likely to open at the same time? (Apart from mincaverage, some of the RMINC code performs calculations on a single slice of each input file in its call.)