[MINC-users] Understanding memory usage in MINC(average)

[Vladimir S. FONOV]

Hello,

RMINC is using MINC2 API calls, they are not affected by this issue (i.e 
it uses default caching value from HDF5).

On 2016-12-08 05:11 PM, Andrew Janke wrote:
> Hi Matthijs,
>
> This one can't be answered for all cases. If your files are all 100kb
> in size, then yes, open them all at once, presuming HDF5 doesn't run
> out of file handles. If they are all 100GB with compression on then
> nope..
>
> For 1500 files, I'd say leaving it at the default (1000) will be fine,
> it's only a few extra operations per voxel-loop chunk so should be OK.
> There will be some complex heuristic that will relate number of files,
> size of files, level of compression, size of voxel loop cache and
> number of ICV's, and it will likely differ from machine to machine
> given differing CPU cache sizes, memory bandwidth, CPU speed and disk
> I/O.
>
> It's for this reason that we chose to keep things simple with the
> current ENV vars as there was no easy way to determine the number of
> files that would be opened within libminc or to determine the value of
> the current vars based upon this.
>
> I recall a conversation about this on minc-dev but can't find it right now.
>
> TL;DR: use mincbigaverage and atomise other operations where possible.
>
>
> a
>
>
>
> On 9 December 2016 at 03:14, Matthijs van Eede
> <matthijs.vaneede at sickkids.ca> wrote:
>> In response to this, we're currently running pipelines with 1500 input files. Would we have to adjust MI_MAX_NUM_ICV in our case? Should we set it to a number that is larger than the number of MINC files we are likely to open at the same time? (Apart from mincaverage, some of the RMINC code performs calculations on a single slice of each input file in its call.)
> _______________________________________________
> MINC-users at bic.mni.mcgill.ca
> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>


-- 
Best regards,

  Vladimir S. FONOV ~ vladimir.fonov <at> gmail.com