[MINC-users] minccomplete's completeness

Wed Jun 1 10:10:32 EDT 2016

Thanks, Bert!

I actually worked in explicit MINC integrity tests in the central sub I
have that spawns processes (similar to MNI::Spawn), so that every MINC
volume that is present in a spawned command line, is explicitly checked.
Besides nii2mnc, the only thing it has turned up so far is pyminc. But
there are likely a few more things out there.

-- A

On Wed, Jun 1, 2016 at 12:25 AM, Robert D. Vincent <
robert.d.vincent at mcgill.ca> wrote:

> Hi Alex,
>
> I just checked in a fix for nii2mnc. I am sure there are other tools that
> do not set the attribute - I'll try to find and fix them.
>
>     -bert
>
> On Mon, May 30, 2016 at 11:37 AM, Alex Zijdenbos <zijdenbos at gmail.com>
> wrote:
>
> > Nice work, Bert!
> >
> > I have since built in some paranoid checking in all of my scripts, and
> one
> > thing that keeps turning up is that nii2mnc doesn't actually set the
> > 'complete' flag. Iow, anything created by nii2mnc fails minccomplete. Any
> > chance you fixed that as well?
> >
> > Thanks,
> >
> > -- A
> >
> > On Mon, May 16, 2016 at 12:38 PM, Robert D. Vincent <
> > robert.d.vincent at mcgill.ca> wrote:
> >
> > > Hi all,
> > >
> > > I have just checked in changes to volume_io, voxel_loop and the
> following
> > > minc tools:
> > >
> > > mincaverage, minccalc, minccmp, mincconcat, mincconvert, minccopy,
> > > mincextract, minclookup, mincmakescalar, mincmakevector, mincmath,
> > > mincresample, mincreshape, mincsample, mincstats, minctoraw,
> mincwindow.
> > >
> > > These tools will still behave as before, in that they will run to
> > > completion with a corrupt file. However, they will report the error as
> a
> > > non-zero exit code (EXIT_FAILURE), in addition to whatever messages are
> > > currently displayed by the libraries.
> > >
> > > Two existing tools, mincinfo and mincdump, seem to already have correct
> > > handling of read errors. At least two others, mincblob and mincmorph,
> > > should pick up the changes via volume_io.
> > >
> > > This covers the majority of the command line tools. It may also make
> > sense
> > > to reflect these changes in some of the basic scripts (e.g. mincpik,
> > > mincdiff) - let me know if there are any particularly salient cases we
> > > should take care of.
> > >
> > >     -bert
> > >
> > > On Sat, May 14, 2016 at 5:04 PM, Alex Zijdenbos <zijdenbos at gmail.com>
> > > wrote:
> > >
> > > > On this note: nii2mnc is one of the tools that doesn't appear to set
> > the
> > > > image:complete attribute. I haven't found any others yet but since I
> > > > started running minccomplete on anything I generate, I may come up
> with
> > > > more :)
> > > >
> > > > On Sat, May 14, 2016 at 6:53 AM, Vladimir S. FONOV <
> > > > vladimir.fonov at gmail.com
> > > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > the only thing minccomplete does is read the contents of the
> > > > image:complete
> > > > > attribute. Which is not even respected by all minc tools.
> > > > > One real way to check the consistency of the minc file would be to
> > > enable
> > > > > checksum , integrated into HDF5 file format.
> > > > >
> > > > > Currently it can be enabled by setting the environment variable
> > > > > MINC_CHECKSUM to 1 , at the time when file is created. Obviously,
> > this
> > > > > works only with MINC2 file format and it was not tested very well
> > yet.
> > > > >
> > > > > On Fri, May 13, 2016 at 3:35 PM, Alex Zijdenbos <
> zijdenbos at gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I managed to generate a large number of broken MINC files;
> > > > > possibly/likely
> > > > > > due to a filesystem problem. The processes that created them
> (e.g.,
> > > > > > mincaverage) did not produce any warnings and completed
> > successfully;
> > > > in
> > > > > > addition, minccomplete tells me that the files are complete.
> > > > > >
> > > > > > Unfortunately, trying to read these files throws HDF5 and miicv
> > > errors
> > > > > (see
> > > > > > below) and they are obviously corrupt.
> > > > > >
> > > > > > I am thinking that it would be useful to complete minccomplete by
> > > > having
> > > > > it
> > > > > > actually test-read the data, such that it would report on file
> > > > integrity?
> > > > > > This would make it easy to find these kinds of corruptions - and
> > > could
> > > > > even
> > > > > > tack that end the end of scripts to make sure outputs are intact.
> > I'm
> > > > > > currently using 'mincstats -quiet -min' to locate them, but it
> > seems
> > > > the
> > > > > > natural place for this test would actually be minccomplete.
> > > > > >
> > > > > > -- A
> > > > > >
> > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.9) thread 0:
> > > > > >   #000: H5Dio.c line 174 in H5Dread(): can't read data
> > > > > >     major: Dataset
> > > > > >     minor: Read failed
> > > > > >   #001: H5Dio.c line 449 in H5D_read(): can't read data
> > > > > >     major: Dataset
> > > > > >     minor: Read failed
> > > > > >   #002: H5Dchunk.c line 1729 in H5D_chunk_read(): unable to read
> > raw
> > > > data
> > > > > > chunk
> > > > > >     major: Low-level I/O
> > > > > >     minor: Read failed
> > > > > >   #003: H5Dchunk.c line 2760 in H5D_chunk_lock(): data pipeline
> > read
> > > > > failed
> > > > > >     major: Data filters
> > > > > >     minor: Filter operation failed
> > > > > >   #004: H5Z.c line 1120 in H5Z_pipeline(): filter returned
> failure
> > > > during
> > > > > > read
> > > > > >     major: Data filters
> > > > > >     minor: Read failed
> > > > > >   #005: H5Zdeflate.c line 125 in H5Z_filter_deflate(): inflate()
> > > failed
> > > > > >     major: Data filters
> > > > > >     minor: Unable to initialize object
> > > > > > mincstats (from miicv_get): Can't read dataset
> > > /minc-2.0/image/0/image
> > > > > > _______________________________________________
> > > > > > MINC-users at bic.mni.mcgill.ca
> > > > > > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > >
> > > > >  Vladimir S. Fonov ~ vladimir <dot> fonov <at> gmail <dot> com
> > > > > _______________________________________________
> > > > > MINC-users at bic.mni.mcgill.ca
> > > > > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
> > > > >
> > > > >
> > > > _______________________________________________
> > > > MINC-users at bic.mni.mcgill.ca
> > > > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
> > > >
> > > _______________________________________________
> > > MINC-users at bic.mni.mcgill.ca
> > > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
> > >
> > >
> > _______________________________________________
> > MINC-users at bic.mni.mcgill.ca
> > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
> >
> _______________________________________________
> MINC-users at bic.mni.mcgill.ca
> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>
>