[MINC-development] max_buffer_size_in_kb

[Andrew Janke]

> Please remind me: is this buffer per file or in total for all files?
> 600 MRIs at each 1MB buffer = 600MB.
> 600 MRIs at each 1MB/600=1.7Kb = 1MB.
> Now we are in trouble at 1.7Kb per file. I agree.

Its for all files (see line 1793 of libsrc/voxel_loop.c):

https://github.com/andrewjanke/minc/blob/f272c25a588f503c90fea16c972b149ca68d3194/libsrc/voxel_loop.c#L1793

You are setting loop_options->total_copy_space which is then divided
by num_input_buffers (ie: the number of input files + output files +
extra buffers).  Peter wrote this no doubt concerned about total
memory usage, not usage per file.

> Maybe we have to change the meaning of the buffer to be per file,
> because 1MB is "optimized" for one file.

This would have implications though, (going to swap). What this means
is that you want two env vars:

   export MINC_MAX_MEMORY 40000000     # all of your RAM
   export MINC_MAX_FILE_BUFFER 4096     # buffer per file

You'd then have to check (in voxel_loop) that MINC_MAX_FILE_BUFFER * #
input + output files is not larger than MINC_MAX_MEMORY.

> minccalc is kind of harmless. Do you ever use it with lots of files?

All the time, use it to do weighted averages (remove outliers) when
mincaverage wont do or won't work. eg: in one case I have 9000 input
files so break the averaging over multiple runs of minccalc -double.

> mincaverage and mincconcat are probably more problematic.

correct, for the average punter.

> The design to use a constant (#define) in minc is bad.

meaning what I have done or what was done in the past?

> In the context of object-oriented code (C++), it could be defined in the minc class when
> opening a minc file. Each file could have its own buffer size. wow!

Or not. ;)


a