[MINC-development] max_buffer_size_in_kb

[Andrew Janke]

>> This would have implications though, (going to swap). What this means
>> is that you want two env vars:
>>
>>    export MINC_MAX_MEMORY 40000000     # all of your RAM
>>    export MINC_MAX_FILE_BUFFER 4096     # buffer per file
>>
>> You'd then have to check (in voxel_loop) that MINC_MAX_FILE_BUFFER * #
>> input + output files is not larger than MINC_MAX_MEMORY.
>
> While you're at it, this would be a great change to implement for
> voxel_loop. MINC_MAX_FILE_BUFFER should be 1MB (like current buffer)
> and should be maxed out as long as #files * MINC_MAX_FILE_BUFFER
> is less than MINC_MAX_MEMORY. Safely, MINC_MAX_MEMORY could be
> 2Gb or 4Gb.

You sure about 1MB there? The current default is 4MB (in both
voxel_loop.c and mincprogs).

> io_buffer_per_file = min( MINC_MAX_MEMORY/number_files, MINC_MAX_FILE_BUFFER );

Agreed.

> I think this implementation would be dynamic enough to do all what
> you want without the environment variable and command line switch.

I hope so, but will still be adding the ENV vars and adding C/L
switches to all appropriate tools.

So is there consensus on what we call these two new ENV vars?  We
currently have this list:

   MINC_FORCE_V2 = {1, 0}
   MINC_COMPRESS = {0..9}-
   MINC_CHUNKING = {0..N}
   MINC_LOGFILE = [+]<pathname>|stdout|-
   MINC_LOGLEVEL = 0-4

So do:

   MINC_MAX_MEMORY
   MINC_MAX_FILE_BUFFER

fit in well with this?  Perhaps MINC_MEMMAX and MINC_FILE_MEMMAX?
whatever we choose there should be a nice easy to understand link
between -max_buffer_size_in_kb and MINC_MAX_FILE_BUFFER.

If I hear nothing else, I will press ahead with MINC_MAX_MEMORY +
MINC_MAX_FILE_BUFFER tonight.



a