From steve at sumost.ca  Sun Dec 11 02:15:59 2011
From: steve at sumost.ca (Steve M. Robbins)
Date: Sun, 11 Dec 2011 01:15:59 -0600
Subject: [MINC-development] [michael.hanke@gmail.com: Re: Bug#585676:
 minc-tools: Cannot use cmdline tools (file not found)]
Message-ID: <20111211071559.GB29922@sumost.ca>

Hello folks,

With minc 2.1.00, I see an error when running "mincinfo" on a gzipped
minc file such as from

  http://packages.bic.mni.mcgill.ca/tgz/mni-models_icbm152-nl-2009-1.0.tar.gz

If I gunzip the file first, mincinfo works.

Known issue?

Thanks,
-Steve


----- Forwarded message from Michael Hanke <michael.hanke at gmail.com> -----

Date: Mon, 14 Jun 2010 07:04:21 -0400
From: Michael Hanke <michael.hanke at gmail.com>
To: "Steve M. Robbins" <steve at sumost.ca>
Cc: 585676 at bugs.debian.org
Subject: Re: Bug#585676: minc-tools: Cannot use cmdline tools (file not found)

Hi,

On Mon, Jun 14, 2010 at 01:50:08AM -0500, Steve M. Robbins wrote:
> On Sat, Jun 12, 2010 at 07:16:28PM -0400, Michael Hanke wrote:
> > michael at meiner ...an/data/mni-models_icbm152-nl-2009-1.0 % mincinfo -minc_version mni_icbm152_t1_tal_nlin_sym_09a.mnc.gz
> > (from miopen): Unable to open file '/tmp/minc-5twjtq'
> > mincinfo: Error opening file "/tmp/minc-5twjtq"
> 
> I'd like to reproduce this.  Can you provide me with the
> file?  From the name, I'd guess it's one of the MNI models -- if
> so, the URL will suffice.

The tarball with the images is here:

  http://packages.bic.mni.mcgill.ca/tgz/mni-models_icbm152-nl-2009-1.0.tar.gz

IIRC 22 out of 24 images in this tarball cause this error.


Thanks,

Michael

-- 
GPG key:  1024D/3144BE0F Michael Hanke
http://mih.voxindeserto.de




----- End forwarded message -----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://www.bic.mni.mcgill.ca/pipermail/minc-development/attachments/20111211/0e6e512e/attachment.pgp>

From a.janke at gmail.com  Tue Dec 13 21:00:41 2011
From: a.janke at gmail.com (Andrew Janke)
Date: Wed, 14 Dec 2011 12:00:41 +1000
Subject: [MINC-development] [michael.hanke@gmail.com: Re: Bug#585676:
 minc-tools: Cannot use cmdline tools (file not found)]
In-Reply-To: <20111211071559.GB29922@sumost.ca>
References: <20111211071559.GB29922@sumost.ca>
Message-ID: <CAG0CB23nfeXnOnh5PPaiMicdGPyDoQnTgh=Zyumdogwd_qfFqQ@mail.gmail.com>

Hi Steve,

On 11 December 2011 17:15, Steve M. Robbins <steve at sumost.ca> wrote:
> With minc 2.1.00, I see an error when running "mincinfo" on a gzipped
> minc file such as from
>
> ?http://packages.bic.mni.mcgill.ca/tgz/mni-models_icbm152-nl-2009-1.0.tar.gz
>
> If I gunzip the file first, mincinfo works.
>
> Known issue?

Nope, not that I am aware of. A little bit of investigation would tell
me that mincinfo is fine with compressed MINC1 files (NetCDF) but
fails on compressed MINC2 (HDF) files.  it would seem that the
unzipped tmp file is empty.

wierd.



a

From a.janke at gmail.com  Tue Dec 13 21:23:16 2011
From: a.janke at gmail.com (Andrew Janke)
Date: Wed, 14 Dec 2011 12:23:16 +1000
Subject: [MINC-development] [michael.hanke@gmail.com: Re: Bug#585676:
 minc-tools: Cannot use cmdline tools (file not found)]
In-Reply-To: <CAG0CB23nfeXnOnh5PPaiMicdGPyDoQnTgh=Zyumdogwd_qfFqQ@mail.gmail.com>
References: <20111211071559.GB29922@sumost.ca>
	<CAG0CB23nfeXnOnh5PPaiMicdGPyDoQnTgh=Zyumdogwd_qfFqQ@mail.gmail.com>
Message-ID: <CAG0CB20mzS7H8dg2vRdftCdMBFHZjFr7NQBOB3nwmm0SGESWfw@mail.gmail.com>

>> If I gunzip the file first, mincinfo works.
>>
>> Known issue?
>
> Nope, not that I am aware of. A little bit of investigation would tell
> me that mincinfo is fine with compressed MINC1 files (NetCDF) but
> fails on compressed MINC2 (HDF) files. ?it would seem that the
> unzipped tmp file is empty.

OK, more digging reveals that the problem is here:

   https://github.com/andrewjanke/minc/blob/master/libsrc/netcdf_convenience.c#L244

There is a hard-coded "hack" in there:

   #define BYTES_PER_OPEN (1024*64)

That seems to work for netcdf files. In effect to just get the
"header_only" of a NetCDF file you just keep unzipping and reading in
chunks of this size and wait until you get a successful read.

Seems that there is some MINC2 specific code in there but it doesn't
appear to test if a HDF file can be opened,  just accessed. If you
add:

   header_only = 0

to about line 266, everything works nicely.

So the question now is should we support this behaviour or just revert
to uncompressing the whole thing? I say "support" because users should
be using HDF's internal compression scheme not external zipping.


a

From a.janke at gmail.com  Tue Dec 13 21:37:03 2011
From: a.janke at gmail.com (Andrew Janke)
Date: Wed, 14 Dec 2011 12:37:03 +1000
Subject: [MINC-development] [michael.hanke@gmail.com: Re: Bug#585676:
 minc-tools: Cannot use cmdline tools (file not found)]
In-Reply-To: <CAG0CB20mzS7H8dg2vRdftCdMBFHZjFr7NQBOB3nwmm0SGESWfw@mail.gmail.com>
References: <20111211071559.GB29922@sumost.ca>
	<CAG0CB23nfeXnOnh5PPaiMicdGPyDoQnTgh=Zyumdogwd_qfFqQ@mail.gmail.com>
	<CAG0CB20mzS7H8dg2vRdftCdMBFHZjFr7NQBOB3nwmm0SGESWfw@mail.gmail.com>
Message-ID: <CAG0CB22a2JT3EU2aZ_LB0ecmQ8JwgxTjszSipJua=cw=NqJXuQ@mail.gmail.com>

On 14 December 2011 12:23, Andrew Janke <a.janke at gmail.com> wrote:
> Seems that there is some MINC2 specific code in there but it doesn't
> appear to test if a HDF file can be opened, ?just accessed. If you
> add:
>
> ? header_only = 0
>
> to about line 266, everything works nicely.
>
> So the question now is should we support this behaviour or just revert
> to uncompressing the whole thing? I say "support" because users should
> be using HDF's internal compression scheme not external zipping.

And even more digging..  The call used in MINC to "validate" a HDF
file is H5Fis_hdf5():

   https://github.com/andrewjanke/minc/blob/master/libsrc/hdf_convenience.c#L2371

This doesn't do what we'd like, despite appearances:

   http://davis.lbl.gov/Manuals/HDF5-1.4.3/RM_H5F.html#File-IsHDF5

All it does is search for the HDF5 signature in a file.  From digging
through the HDF5 API documentation I can find no function to do what
we want/need. Meaning the "best" fix at this stage is to just unzip
the whole thing.


a

From steve at sumost.ca  Sun Dec 18 22:02:34 2011
From: steve at sumost.ca (Steve M. Robbins)
Date: Sun, 18 Dec 2011 21:02:34 -0600
Subject: [MINC-development] [michael.hanke@gmail.com: Re: Bug#585676:
 minc-tools: Cannot use cmdline tools (file not found)]
In-Reply-To: <CAG0CB22a2JT3EU2aZ_LB0ecmQ8JwgxTjszSipJua=cw=NqJXuQ@mail.gmail.com>
References: <20111211071559.GB29922@sumost.ca>
	<CAG0CB23nfeXnOnh5PPaiMicdGPyDoQnTgh=Zyumdogwd_qfFqQ@mail.gmail.com>
	<CAG0CB20mzS7H8dg2vRdftCdMBFHZjFr7NQBOB3nwmm0SGESWfw@mail.gmail.com>
	<CAG0CB22a2JT3EU2aZ_LB0ecmQ8JwgxTjszSipJua=cw=NqJXuQ@mail.gmail.com>
Message-ID: <20111219030233.GN5015@sumost.ca>

Andrew,

Thanks for digging into this.  I added the attached patch for the Debian packages,
which forces uncompressing the entire file as you suggested.


Incidentally, is the canonical source location still in CVS at the BIC or has it
moved to your github project?

Cheers,
-Steve
-------------- next part --------------
A non-text attachment was scrubbed...
Name: decompress-whole-file.diff
Type: text/x-diff
Size: 990 bytes
Desc: not available
URL: <http://www.bic.mni.mcgill.ca/pipermail/minc-development/attachments/20111218/0acfe0fe/attachment.diff>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://www.bic.mni.mcgill.ca/pipermail/minc-development/attachments/20111218/0acfe0fe/attachment.pgp>

From a.janke at gmail.com  Mon Dec 19 21:57:34 2011
From: a.janke at gmail.com (Andrew Janke)
Date: Tue, 20 Dec 2011 12:57:34 +1000
Subject: [MINC-development] [michael.hanke@gmail.com: Re: Bug#585676:
 minc-tools: Cannot use cmdline tools (file not found)]
In-Reply-To: <20111219030233.GN5015@sumost.ca>
References: <20111211071559.GB29922@sumost.ca>
	<CAG0CB23nfeXnOnh5PPaiMicdGPyDoQnTgh=Zyumdogwd_qfFqQ@mail.gmail.com>
	<CAG0CB20mzS7H8dg2vRdftCdMBFHZjFr7NQBOB3nwmm0SGESWfw@mail.gmail.com>
	<CAG0CB22a2JT3EU2aZ_LB0ecmQ8JwgxTjszSipJua=cw=NqJXuQ@mail.gmail.com>
	<20111219030233.GN5015@sumost.ca>
Message-ID: <CAG0CB20X0eeFUsz1Se4sc_UqLgQwb3RxYBWWCWyQf9GnKm7_gA@mail.gmail.com>

Hi Steve,

> Thanks for digging into this. ?I added the attached patch for the Debian packages,
> which forces uncompressing the entire file as you suggested.

I took a bit more of a scorched earth approach:

   https://github.com/andrewjanke/minc/commit/8367d0cdf7afc6eef26574be2fede7ebdf917e3e

> Incidentally, is the canonical source location still in CVS at the BIC or has it
> moved to your github project?

definitely github. releases will still be made at
packages.bic.mni.mcgill.ca/tgz and I do attempt to merge the git
changes to cvs from time to time. That said, it's currently not up to
date.

There are a bunch of people who are actively commiting things to the
github version so I will be sticking with it. For example I have just
merged vlads ezminc (CMake/itk) branch to the master. It's a great bit
of work reminiscent of volume_io that can be seen here:

   https://github.com/andrewjanke/minc/tree/master/minc4itk

   https://github.com/andrewjanke/minc/tree/master/ezminc




a