[MINC-development] max_buffer_size_in_kb

[Claude LEPAGE]

Hi Andrew,

I was the last one to increase the default buffer size from
something like 10K to 1Mb. At the time, I ran many tests to
find a suitable value that would be optimal for speed for
the most typical application: a MRI in stereotaxic space at
1mm (minccalc, mincresample, mincreshape, etc). The file size 
is about 10Mb (that's minc2 compressed) for unsigned short. 
Remember also that there is a buffer for internal compression 
(which might be the same as the I/O buffer - beware). Operating 
on large compressed blocks slows down processing.

> For now I think it will be simpler (for me) to skip the ENV var and
> just increase the default. Anyone have any particular preference for
> how big? In my case I usually just set it to "massive", but I suspect
> this is not the best idea...
> 
> My initial thought is to set it to about 200MB (~50x the current).
> 1024*4*64 =3D 262144kb
> 
> Thoughts?

I think 200MB is too large for most typical MRI applications. I would
hesitate to go above 4MB as a default. However, I'm all in favour of
providing a user-friendly option to change the run-time default. The
environment variable sounds like a good work-around. Each user can
set it up in his (or her!) environment or in individual job scripts.

Also, keep in mind that I am processing hi-res histology slices 
(250Mb per slice/file) with a 1MB buffer and that the performance 
is quite good. 

So make sure you run lots of tests for speed and plot graphs for
execution time vs buffer size for a wide range of applications
(minccalc, mincresample, minctracc) and inputs. There might be
applications for which 200MB is optimal, but these will not be
the typical applications by most users. Whoever requires 200MB
can use an environment variable or recompile with a default of
200MB.

Claude