[MINC-development] Summary of January meeting

[John G. Sled]

As many of you are aware, we had a meeting in January to discuss the
design of MINC 2.0.  The meeting proved productive and covered a large
number of issues.  The following is a summary of the meeting.

The major decision at the meeting was that we pursue HDF5 and not
NetCDF as the low level file management library.  While this decision
is not yet set in stone, the consensus was that the features that HDF5
provides as well as its overall vitality make it a compelling choice.
Specifically, it supports large files with arbitrary layout, it
implements blocked disk layout (chunking in HDF5 terminology), and it
has a mechanism to plug in compression algorithms for compressed
chunks.  Bert and Leila have been investigating whether these benefits
can be realized for this project.  The most complex issue at present
is how to implement the fixed point integer translations.

Another issue that was discussed in detail was how to handle backwards
compatibility.  The conclusion was that the minimum acceptable level
of compatibility could be met by having all tools transparently run a
file format conversion program to generate a temporary file in the new
format.  The reverse process could optionally be invoked when files
are written to return them to the old format.  It was unclear from the
discussion whether a more efficient method of achieving backward
compatibility could be implemented without compromising the
functionality of the new library.  There was also some discussion on
what the default behaviour in terms of writing files should be if the
input file was in MINC 1.x format.  The conversion program would also be
available as a stand alone application.

With respect to multiresolution, the multi-thumbnail approach to
multiple resolutions was agreed to be the simpler and more practical
solution.  It appears that HDF5 allows new data organizations to added
in a manner that is transparent to the API.  This mechanism could
later be used to add a wavelet implementation of multiresolution data
layout if desired.

There was much discussion of the handling of data types at the
meeting.  It was agreed that there should be three types of scalar
data: integer, real, and enumerated.  Any of these can be represented
by either a integer or floating point format.  For example, a real
number with an integer representation would use the familiar voxel and
real range mapping.  It was not immediately clear what granularity is
appropriate for these mappings.  For example, the old strategy of one
mapping per slice could be used or perhaps one mapping per block in
block format files.  The latter may be possible to implement
transparently using the filter mechanism in HDF5.  Volumes with
enumerated data type would include a dictionary in the header that mapped
strings to integer values.

It was also proposed that the record data type in HDF5 be used to
implement non-scalar types such as complex numbers and vectors.  The
consensus was that the interpretation of these types should be
application specific rather than in the core library.  So, for example,
even though a variety of types might be specified it would be up
to specific applications to implement such operations as multiply and
divide.  A copy operation would be provided for all types.

With respect to structuring MINC 2.0, the functionality of libminc
would greatly increased such that for applications that do their own
memory management, libminc would be the prefered choice instead of
volume_io.  In order to maintain the functionality of existing
applications, voxel_loop and volume_io would be revised to use the new
libminc without changing their API.  It was also agreed that a rewrite
of the file access functionality in EMMA would be needed in order that
the MatLab based data analysis continue to function.  The plan
is also to create a successor to VolumeIO that would provide memory
management.  It was not clear at the meeting what form this new
library should take.

At present Bert, John C., Leila and I are working on a design document
based on these ideas.  Once a draft is ready, this will be circulated to
the list.  Please feel free to comment on any of the issues I've mentioned.

cheers,

John