[MINC-development] max_buffer_size_in_kb

[Claude LEPAGE]

Andrew,

> Its for all files (see line 1793 of libsrc/voxel_loop.c):
> 
> https://github.com/andrewjanke/minc/blob/f272c25a588f503c90fea16c972b149ca68d3194/libsrc/voxel_loop.c#L1793
> 
> You are setting loop_options->total_copy_space which is then divided
> by num_input_buffers (ie: the number of input files + output files +
> extra buffers).  Peter wrote this no doubt concerned about total
> memory usage, not usage per file.
> 
> > Maybe we have to change the meaning of the buffer to be per file,
> > because 1MB is "optimized" for one file.
> 
> This would have implications though, (going to swap). What this means
> is that you want two env vars:
> 
>    export MINC_MAX_MEMORY 40000000     # all of your RAM
>    export MINC_MAX_FILE_BUFFER 4096     # buffer per file
> 
> You'd then have to check (in voxel_loop) that MINC_MAX_FILE_BUFFER * #
> input + output files is not larger than MINC_MAX_MEMORY.

While you're at it, this would be a great change to implement for
voxel_loop. MINC_MAX_FILE_BUFFER should be 1MB (like current buffer)
and should be maxed out as long as #files * MINC_MAX_FILE_BUFFER 
is less than MINC_MAX_MEMORY. Safely, MINC_MAX_MEMORY could be
2Gb or 4Gb. 

io_buffer_per_file = max( MINC_MAX_MEMORY/number_files, MINC_MAX_FILE_BUFFER );

I think this implementation would be dynamic enough to do all what
you want without the environment variable and command line switch.
A static compilation with 4Gb/1Mb would go a long way without having
to modify all the code.

Just double-check this i/o buffer size with the chunk buffer for
internal compression.

I think we are converging towards an excellent solution!

Claude