[MINC-users] minccomplete's completeness

[Alex Zijdenbos]

That sounds promising (and if it works an easy solution) for completing
minccomplete!

However, what concerns me the most about all of this is that a "bad" volume
may go entirely unnoticed as the standard MINC tools do not seem to care
about HDF5 errors. The volumes that I have are happily read by the MINC
tools, ignoring the HDF5 errors, and subsequently will produce output that
is "good" (see the the snapshot I sent). Ergo, if for whatever reason a bad
volume is created somewhere in a processing pipeline (which might even be a
temporary volume), it will be "fixed" by the downstream steps but you will
end up with bad data downstream. Unless visual QC picks it up or you
explicitly check all the logs for HDF5 warnings, you'd never know. This is
a very real problem IMHO that may have been affecting any of us...

-- A

On Sat, May 14, 2016 at 6:59 AM, Andrew Janke <a.janke at gmail.com> wrote:

> And that sounds even better.  Never realised it was part of HDF5.
>
>
> a
>
> On 14 May 2016 at 18:53, Vladimir S. FONOV <vladimir.fonov at gmail.com>
> wrote:
> > Hello,
> >
> > the only thing minccomplete does is read the contents of the
> image:complete
> > attribute. Which is not even respected by all minc tools.
> > One real way to check the consistency of the minc file would be to enable
> > checksum , integrated into HDF5 file format.
> >
> > Currently it can be enabled by setting the environment variable
> > MINC_CHECKSUM to 1 , at the time when file is created. Obviously, this
> > works only with MINC2 file format and it was not tested very well yet.
> >
> > On Fri, May 13, 2016 at 3:35 PM, Alex Zijdenbos <zijdenbos at gmail.com>
> wrote:
> >
> >> Hi all,
> >>
> >> I managed to generate a large number of broken MINC files;
> possibly/likely
> >> due to a filesystem problem. The processes that created them (e.g.,
> >> mincaverage) did not produce any warnings and completed successfully; in
> >> addition, minccomplete tells me that the files are complete.
> >>
> >> Unfortunately, trying to read these files throws HDF5 and miicv errors
> (see
> >> below) and they are obviously corrupt.
> >>
> >> I am thinking that it would be useful to complete minccomplete by
> having it
> >> actually test-read the data, such that it would report on file
> integrity?
> >> This would make it easy to find these kinds of corruptions - and could
> even
> >> tack that end the end of scripts to make sure outputs are intact. I'm
> >> currently using 'mincstats -quiet -min' to locate them, but it seems the
> >> natural place for this test would actually be minccomplete.
> >>
> >> -- A
> >>
> >> HDF5-DIAG: Error detected in HDF5 (1.8.9) thread 0:
> >>   #000: H5Dio.c line 174 in H5Dread(): can't read data
> >>     major: Dataset
> >>     minor: Read failed
> >>   #001: H5Dio.c line 449 in H5D_read(): can't read data
> >>     major: Dataset
> >>     minor: Read failed
> >>   #002: H5Dchunk.c line 1729 in H5D_chunk_read(): unable to read raw
> data
> >> chunk
> >>     major: Low-level I/O
> >>     minor: Read failed
> >>   #003: H5Dchunk.c line 2760 in H5D_chunk_lock(): data pipeline read
> failed
> >>     major: Data filters
> >>     minor: Filter operation failed
> >>   #004: H5Z.c line 1120 in H5Z_pipeline(): filter returned failure
> during
> >> read
> >>     major: Data filters
> >>     minor: Read failed
> >>   #005: H5Zdeflate.c line 125 in H5Z_filter_deflate(): inflate() failed
> >>     major: Data filters
> >>     minor: Unable to initialize object
> >> mincstats (from miicv_get): Can't read dataset /minc-2.0/image/0/image
> >> _______________________________________________
> >> MINC-users at bic.mni.mcgill.ca
> >> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
> >>
> >
> >
> >
> > --
> > Best regards,
> >
> >  Vladimir S. Fonov ~ vladimir <dot> fonov <at> gmail <dot> com
> > _______________________________________________
> > MINC-users at bic.mni.mcgill.ca
> > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
> _______________________________________________
> MINC-users at bic.mni.mcgill.ca
> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>
>