From a.janke at gmail.com  Mon Nov 29 06:53:33 2010
From: a.janke at gmail.com (Andrew Janke)
Date: Mon, 29 Nov 2010 22:53:33 +1100
Subject: [MINC-development] Fwd: [netCDF #AVF-863193]: Re: netCDF-4
	compatibility with netCDF-3
In-Reply-To: <laccx4.42dkv8@www.unidata.ucar.edu>
References: <laccx4.42dkv8@www.unidata.ucar.edu>
Message-ID: <AANLkTimuohmj=-fv9YtZkQuGXD1zmCWf-ymztoU4cxtt@mail.gmail.com>

An interesting email from Russ Rew (Ucar man) regarding some
HDF/Netcdf things I have been emailing him about in an attempt to sort
out if in MINC 2 we finally dump Netcdf and go all with HDF or perhaps
just go back to netcdf 1.4

My current feeling is that we just go back to netcdf only :)  it seems
to do all we need and it would simplify building MINC monstrously if
we only had to deal with netcdf and not the two.

FWIW, there is no point only using HDF as my understanding is that
netcdf is a pre-requisite for it....


a


---------- Forwarded message ----------
From: Unidata netCDF Support <support-netcdf at unidata.ucar.edu>
Date: 16 October 2010 04:07
Subject: [netCDF #AVF-863193]: Re: netCDF-4 compatibility with netCDF-3
To: a.janke at gmail.com
Cc: support-netcdf at unidata.ucar.edu


Hi Andrew,

> Thanks for taking the time to send a personal note about this. We
> (MINC) have been netcdf consumers for the past 15 or so years and HDF5
> for the past few. I suspect the misunderstandings in this post (to
> minc-development) was more a bit of frustration over the varying
> binary builds of HDF/netcdf that exist on various platforms.
>
> We build MINC across Windows (Cygwin), OSX and various flavours of
> things ending in ux and ix.
>
> One of the things that tripped us up a lot was the builds of HDF 1.8.x
> in Debian/Ubuntu with specific 1.6x compatibility flags but we are now
> back on top of it. I HOPE!
>
> The main issue is that we now support both netcdf and hdf5 in MINC
> (1.x and 2.x) and need a transition path out of this. This is
> something that I would be most grateful to have your advice about.

Due to a major refactoring of netCDF dispatch to support an OPeNDAP
client for remote access, the variants of netCDF formats for classic
and HDF5 storage, code to read some other formats data through the
netCDF API, and a desire to support in-memory netCDF "files", we've
temporarily broken the Windows Visual Studio port, however everything
still builds with cygwin. ?We have plans to fix the Visual Studio
port:

?http://www.unidata.ucar.edu/netcdf/docs/faq.html#windows_netcdf4_1

That may be a problem for you, and you may want to wait until the
Visual Studio build works, if you use that.

> My current feeling from what you have said is that we might be best to
> drop HDF5 and instead use netcdf1.4 only. Our reasons for changing to
> hdf5 was to take advantage of chunking, block compression and
> arbitrary (and apparent) dimension ordering.
>
> From my quick reading this is all posible under netcdf1.4?

NetCDF 4.1 supports chunking and block compression. ?I'm not sure what
you mean by "arbitrary (and apparent) dimension ordering". ?NetCDF 4.1
supports multiple unlimited (extendable) dimensions, not just in
different groups, but also for the same variable, so a
multidimensional variable can grow incrementally and efficiently along
multiple axes without copying data.

We also got the HDF5 developers to add support for persistence of
dimension creation order, so you can get things back in the same order
you created them, if desired.

However, I'm not recommending dropping HDF5. ?It still has features
netCDF-4 doesn't support, and you would have to evaluate whether the
power those features give you (even potentially if you aren't using
them now) are worth the trouble of supporting multiple data formats.

We have interoperability plans to increase support for HDF5 features,
but we will probably always support only a subset of the features we
regard as most useful, to keep the data model and API simple enough to
understand. ?If we supported everything in HDF5, there would be little
reason to continue to develop netCDF ...

--Russ

Russ Rew ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? UCAR Unidata Program
russ at unidata.ucar.edu ? ? ? ? ? ? ? ? ? ? ?http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: AVF-863193
Department: Support netCDF
Priority: Normal
Status: Closed

From jason at phenogenomics.ca  Mon Nov 29 14:21:39 2010
From: jason at phenogenomics.ca (Jason Lerch)
Date: Mon, 29 Nov 2010 14:21:39 -0500
Subject: [MINC-development] Fwd: [netCDF #AVF-863193]: Re: netCDF-4
	compatibility with netCDF-3
In-Reply-To: <AANLkTimuohmj=-fv9YtZkQuGXD1zmCWf-ymztoU4cxtt@mail.gmail.com>
References: <laccx4.42dkv8@www.unidata.ucar.edu>
Message-ID: <733EF7CA-AB2E-4A8A-B999-178187C9D659@phenogenomics.ca>

Except for the irritation of at times needing out of date versions of either, what's the big support issue? I've not found building MINC to be particularly challenging (at least compared to brain-view or N3 with different C++ compilers!).

Jason

On 2010-11-29, at 6:53 AM, Andrew Janke wrote:

> An interesting email from Russ Rew (Ucar man) regarding some
> HDF/Netcdf things I have been emailing him about in an attempt to sort
> out if in MINC 2 we finally dump Netcdf and go all with HDF or perhaps
> just go back to netcdf 1.4
> 
> My current feeling is that we just go back to netcdf only :)  it seems
> to do all we need and it would simplify building MINC monstrously if
> we only had to deal with netcdf and not the two.
> 
> FWIW, there is no point only using HDF as my understanding is that
> netcdf is a pre-requisite for it....
> 
> 
> a
> 
> 
> ---------- Forwarded message ----------
> From: Unidata netCDF Support <support-netcdf at unidata.ucar.edu>
> Date: 16 October 2010 04:07
> Subject: [netCDF #AVF-863193]: Re: netCDF-4 compatibility with netCDF-3
> To: a.janke at gmail.com
> Cc: support-netcdf at unidata.ucar.edu
> 
> 
> Hi Andrew,
> 
>> Thanks for taking the time to send a personal note about this. We
>> (MINC) have been netcdf consumers for the past 15 or so years and HDF5
>> for the past few. I suspect the misunderstandings in this post (to
>> minc-development) was more a bit of frustration over the varying
>> binary builds of HDF/netcdf that exist on various platforms.
>> 
>> We build MINC across Windows (Cygwin), OSX and various flavours of
>> things ending in ux and ix.
>> 
>> One of the things that tripped us up a lot was the builds of HDF 1.8.x
>> in Debian/Ubuntu with specific 1.6x compatibility flags but we are now
>> back on top of it. I HOPE!
>> 
>> The main issue is that we now support both netcdf and hdf5 in MINC
>> (1.x and 2.x) and need a transition path out of this. This is
>> something that I would be most grateful to have your advice about.
> 
> Due to a major refactoring of netCDF dispatch to support an OPeNDAP
> client for remote access, the variants of netCDF formats for classic
> and HDF5 storage, code to read some other formats data through the
> netCDF API, and a desire to support in-memory netCDF "files", we've
> temporarily broken the Windows Visual Studio port, however everything
> still builds with cygwin.  We have plans to fix the Visual Studio
> port:
> 
>  http://www.unidata.ucar.edu/netcdf/docs/faq.html#windows_netcdf4_1
> 
> That may be a problem for you, and you may want to wait until the
> Visual Studio build works, if you use that.
> 
>> My current feeling from what you have said is that we might be best to
>> drop HDF5 and instead use netcdf1.4 only. Our reasons for changing to
>> hdf5 was to take advantage of chunking, block compression and
>> arbitrary (and apparent) dimension ordering.
>> 
>> From my quick reading this is all posible under netcdf1.4?
> 
> NetCDF 4.1 supports chunking and block compression.  I'm not sure what
> you mean by "arbitrary (and apparent) dimension ordering".  NetCDF 4.1
> supports multiple unlimited (extendable) dimensions, not just in
> different groups, but also for the same variable, so a
> multidimensional variable can grow incrementally and efficiently along
> multiple axes without copying data.
> 
> We also got the HDF5 developers to add support for persistence of
> dimension creation order, so you can get things back in the same order
> you created them, if desired.
> 
> However, I'm not recommending dropping HDF5.  It still has features
> netCDF-4 doesn't support, and you would have to evaluate whether the
> power those features give you (even potentially if you aren't using
> them now) are worth the trouble of supporting multiple data formats.
> 
> We have interoperability plans to increase support for HDF5 features,
> but we will probably always support only a subset of the features we
> regard as most useful, to keep the data model and API simple enough to
> understand.  If we supported everything in HDF5, there would be little
> reason to continue to develop netCDF ...
> 
> --Russ
> 
> Russ Rew                                         UCAR Unidata Program
> russ at unidata.ucar.edu                      http://www.unidata.ucar.edu
> 
> 
> 
> Ticket Details
> ===================
> Ticket ID: AVF-863193
> Department: Support netCDF
> Priority: Normal
> Status: Closed
> _______________________________________________
> MINC-development mailing list
> MINC-development at bic.mni.mcgill.ca
> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-development