[MINC-development] Fwd: [netCDF #AVF-863193]: Re: netCDF-4 compatibility with netCDF-3

[Andrew Janke]

An interesting email from Russ Rew (Ucar man) regarding some
HDF/Netcdf things I have been emailing him about in an attempt to sort
out if in MINC 2 we finally dump Netcdf and go all with HDF or perhaps
just go back to netcdf 1.4

My current feeling is that we just go back to netcdf only :)  it seems
to do all we need and it would simplify building MINC monstrously if
we only had to deal with netcdf and not the two.

FWIW, there is no point only using HDF as my understanding is that
netcdf is a pre-requisite for it....


a


---------- Forwarded message ----------
From: Unidata netCDF Support <support-netcdf at unidata.ucar.edu>
Date: 16 October 2010 04:07
Subject: [netCDF #AVF-863193]: Re: netCDF-4 compatibility with netCDF-3
To: a.janke at gmail.com
Cc: support-netcdf at unidata.ucar.edu


Hi Andrew,

> Thanks for taking the time to send a personal note about this. We
> (MINC) have been netcdf consumers for the past 15 or so years and HDF5
> for the past few. I suspect the misunderstandings in this post (to
> minc-development) was more a bit of frustration over the varying
> binary builds of HDF/netcdf that exist on various platforms.
>
> We build MINC across Windows (Cygwin), OSX and various flavours of
> things ending in ux and ix.
>
> One of the things that tripped us up a lot was the builds of HDF 1.8.x
> in Debian/Ubuntu with specific 1.6x compatibility flags but we are now
> back on top of it. I HOPE!
>
> The main issue is that we now support both netcdf and hdf5 in MINC
> (1.x and 2.x) and need a transition path out of this. This is
> something that I would be most grateful to have your advice about.

Due to a major refactoring of netCDF dispatch to support an OPeNDAP
client for remote access, the variants of netCDF formats for classic
and HDF5 storage, code to read some other formats data through the
netCDF API, and a desire to support in-memory netCDF "files", we've
temporarily broken the Windows Visual Studio port, however everything
still builds with cygwin.  We have plans to fix the Visual Studio
port:

 http://www.unidata.ucar.edu/netcdf/docs/faq.html#windows_netcdf4_1

That may be a problem for you, and you may want to wait until the
Visual Studio build works, if you use that.

> My current feeling from what you have said is that we might be best to
> drop HDF5 and instead use netcdf1.4 only. Our reasons for changing to
> hdf5 was to take advantage of chunking, block compression and
> arbitrary (and apparent) dimension ordering.
>
> From my quick reading this is all posible under netcdf1.4?

NetCDF 4.1 supports chunking and block compression.  I'm not sure what
you mean by "arbitrary (and apparent) dimension ordering".  NetCDF 4.1
supports multiple unlimited (extendable) dimensions, not just in
different groups, but also for the same variable, so a
multidimensional variable can grow incrementally and efficiently along
multiple axes without copying data.

We also got the HDF5 developers to add support for persistence of
dimension creation order, so you can get things back in the same order
you created them, if desired.

However, I'm not recommending dropping HDF5.  It still has features
netCDF-4 doesn't support, and you would have to evaluate whether the
power those features give you (even potentially if you aren't using
them now) are worth the trouble of supporting multiple data formats.

We have interoperability plans to increase support for HDF5 features,
but we will probably always support only a subset of the features we
regard as most useful, to keep the data model and API simple enough to
understand.  If we supported everything in HDF5, there would be little
reason to continue to develop netCDF ...

--Russ

Russ Rew                                         UCAR Unidata Program
russ at unidata.ucar.edu                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: AVF-863193
Department: Support netCDF
Priority: Normal
Status: Closed