[MINC-users] minccomplete's completeness

[Robert D. Vincent]

Hi all,

I have just checked in changes to volume_io, voxel_loop and the following
minc tools:

mincaverage, minccalc, minccmp, mincconcat, mincconvert, minccopy,
mincextract, minclookup, mincmakescalar, mincmakevector, mincmath,
mincresample, mincreshape, mincsample, mincstats, minctoraw, mincwindow.

These tools will still behave as before, in that they will run to
completion with a corrupt file. However, they will report the error as a
non-zero exit code (EXIT_FAILURE), in addition to whatever messages are
currently displayed by the libraries.

Two existing tools, mincinfo and mincdump, seem to already have correct
handling of read errors. At least two others, mincblob and mincmorph,
should pick up the changes via volume_io.

This covers the majority of the command line tools. It may also make sense
to reflect these changes in some of the basic scripts (e.g. mincpik,
mincdiff) - let me know if there are any particularly salient cases we
should take care of.

    -bert

On Sat, May 14, 2016 at 5:04 PM, Alex Zijdenbos <zijdenbos at gmail.com> wrote:

> On this note: nii2mnc is one of the tools that doesn't appear to set the
> image:complete attribute. I haven't found any others yet but since I
> started running minccomplete on anything I generate, I may come up with
> more :)
>
> On Sat, May 14, 2016 at 6:53 AM, Vladimir S. FONOV <
> vladimir.fonov at gmail.com
> > wrote:
>
> > Hello,
> >
> > the only thing minccomplete does is read the contents of the
> image:complete
> > attribute. Which is not even respected by all minc tools.
> > One real way to check the consistency of the minc file would be to enable
> > checksum , integrated into HDF5 file format.
> >
> > Currently it can be enabled by setting the environment variable
> > MINC_CHECKSUM to 1 , at the time when file is created. Obviously, this
> > works only with MINC2 file format and it was not tested very well yet.
> >
> > On Fri, May 13, 2016 at 3:35 PM, Alex Zijdenbos <zijdenbos at gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I managed to generate a large number of broken MINC files;
> > possibly/likely
> > > due to a filesystem problem. The processes that created them (e.g.,
> > > mincaverage) did not produce any warnings and completed successfully;
> in
> > > addition, minccomplete tells me that the files are complete.
> > >
> > > Unfortunately, trying to read these files throws HDF5 and miicv errors
> > (see
> > > below) and they are obviously corrupt.
> > >
> > > I am thinking that it would be useful to complete minccomplete by
> having
> > it
> > > actually test-read the data, such that it would report on file
> integrity?
> > > This would make it easy to find these kinds of corruptions - and could
> > even
> > > tack that end the end of scripts to make sure outputs are intact. I'm
> > > currently using 'mincstats -quiet -min' to locate them, but it seems
> the
> > > natural place for this test would actually be minccomplete.
> > >
> > > -- A
> > >
> > > HDF5-DIAG: Error detected in HDF5 (1.8.9) thread 0:
> > >   #000: H5Dio.c line 174 in H5Dread(): can't read data
> > >     major: Dataset
> > >     minor: Read failed
> > >   #001: H5Dio.c line 449 in H5D_read(): can't read data
> > >     major: Dataset
> > >     minor: Read failed
> > >   #002: H5Dchunk.c line 1729 in H5D_chunk_read(): unable to read raw
> data
> > > chunk
> > >     major: Low-level I/O
> > >     minor: Read failed
> > >   #003: H5Dchunk.c line 2760 in H5D_chunk_lock(): data pipeline read
> > failed
> > >     major: Data filters
> > >     minor: Filter operation failed
> > >   #004: H5Z.c line 1120 in H5Z_pipeline(): filter returned failure
> during
> > > read
> > >     major: Data filters
> > >     minor: Read failed
> > >   #005: H5Zdeflate.c line 125 in H5Z_filter_deflate(): inflate() failed
> > >     major: Data filters
> > >     minor: Unable to initialize object
> > > mincstats (from miicv_get): Can't read dataset /minc-2.0/image/0/image
> > > _______________________________________________
> > > MINC-users at bic.mni.mcgill.ca
> > > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> >  Vladimir S. Fonov ~ vladimir <dot> fonov <at> gmail <dot> com
> > _______________________________________________
> > MINC-users at bic.mni.mcgill.ca
> > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
> >
> >
> _______________________________________________
> MINC-users at bic.mni.mcgill.ca
> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>