[MINC-development] max_buffer_size_in_kb

[Claude LEPAGE]

Andrew,

> True, and no doubt there are pathological cases of this.
> 
> > environment variable sounds like a good work-around. Each user can
> > set it up in his (or her!) environment or in individual job scripts.
> 
> > Also, keep in mind that I am processing hi-res histology slices
> > (250Mb per slice/file) with a 1MB buffer and that the performance
> > is quite good.
> 
> To counter this, I have seen "catastrophic" performance in minccalc
> when doing weighted averages over 19 800MB mouse brains and also on
> 600+ 10MB human head MRI's.  So I'll take your comments about keeping
> things stable and leave things are they are for now but add the ENV
> var so that those of us who like to live on the edge can wind up the
> wick a bit.

Please remind me: is this buffer per file or in total for all files?
600 MRIs at each 1MB buffer = 600MB. 
600 MRIs at each 1MB/600=1.7Kb = 1MB.
Now we are in trouble at 1.7Kb per file. I agree.

Maybe we have to change the meaning of the buffer to be per file, 
because 1MB is "optimized" for one file. 

> Note that I have only changed the logic of minccalc so far, if others
> agree with this approach I will go and make the changes to all the
> other minc tools that use voxel_loop. In some cases this will mean
> adding the -max_buffer_in_kb argument (minclookup for one).

minccalc is kind of harmless. Do you ever use it with lots of files?
mincaverage and mincconcat are probably more problematic. 

The design to use a constant (#define) in minc is bad. This value should
be in a global variable, thus could be changed at run-time. In the context
of object-oriented code (C++), it could be defined in the minc class when
opening a minc file. Each file could have its own buffer size. wow!

Go ahead with your changes. I'm not too picky about the implementation.
It's the concept that's important.

Claude