[MINC-development] max_buffer_size_in_kb

[Andrew Janke]

Hi Claude,

> I was the last one to increase the default buffer size from
> something like 10K to 1Mb. At the time, I ran many tests to
> find a suitable value that would be optimal for speed for
> the most typical application: a MRI in stereotaxic space at
> 1mm (minccalc, mincresample, mincreshape, etc). The file size
> is about 10Mb (that's minc2 compressed) for unsigned short.
> Remember also that there is a buffer for internal compression

True, and no doubt there are pathological cases of this.

> environment variable sounds like a good work-around. Each user can
> set it up in his (or her!) environment or in individual job scripts.

> Also, keep in mind that I am processing hi-res histology slices
> (250Mb per slice/file) with a 1MB buffer and that the performance
> is quite good.

To counter this, I have seen "catastrophic" performance in minccalc
when doing weighted averages over 19 800MB mouse brains and also on
600+ 10MB human head MRI's.  So I'll take your comments about keeping
things stable and leave things are they are for now but add the ENV
var so that those of us who like to live on the edge can wind up the
wick a bit.

Let me know what you think of this commit:

   https://github.com/andrewjanke/minc/commit/f272c25a588f503c90fea16c972b149ca68d3194

Note that I have only changed the logic of minccalc so far, if others
agree with this approach I will go and make the changes to all the
other minc tools that use voxel_loop. In some cases this will mean
adding the -max_buffer_in_kb argument (minclookup for one).


a