[MINC-users] minccomplete's completeness

[Alex Zijdenbos]

Ah - Bert was on the same page - thanks!!

On Sat, May 14, 2016 at 7:13 AM, Alex Zijdenbos <zijdenbos at gmail.com> wrote:

> That sounds promising (and if it works an easy solution) for completing
> minccomplete!
>
> However, what concerns me the most about all of this is that a "bad"
> volume may go entirely unnoticed as the standard MINC tools do not seem to
> care about HDF5 errors. The volumes that I have are happily read by the
> MINC tools, ignoring the HDF5 errors, and subsequently will produce output
> that is "good" (see the the snapshot I sent). Ergo, if for whatever reason
> a bad volume is created somewhere in a processing pipeline (which might
> even be a temporary volume), it will be "fixed" by the downstream steps but
> you will end up with bad data downstream. Unless visual QC picks it up or
> you explicitly check all the logs for HDF5 warnings, you'd never know. This
> is a very real problem IMHO that may have been affecting any of us...
>
> -- A
>
> On Sat, May 14, 2016 at 6:59 AM, Andrew Janke <a.janke at gmail.com> wrote:
>
>> And that sounds even better.  Never realised it was part of HDF5.
>>
>>
>> a
>>
>> On 14 May 2016 at 18:53, Vladimir S. FONOV <vladimir.fonov at gmail.com>
>> wrote:
>> > Hello,
>> >
>> > the only thing minccomplete does is read the contents of the
>> image:complete
>> > attribute. Which is not even respected by all minc tools.
>> > One real way to check the consistency of the minc file would be to
>> enable
>> > checksum , integrated into HDF5 file format.
>> >
>> > Currently it can be enabled by setting the environment variable
>> > MINC_CHECKSUM to 1 , at the time when file is created. Obviously, this
>> > works only with MINC2 file format and it was not tested very well yet.
>> >
>> > On Fri, May 13, 2016 at 3:35 PM, Alex Zijdenbos <zijdenbos at gmail.com>
>> wrote:
>> >
>> >> Hi all,
>> >>
>> >> I managed to generate a large number of broken MINC files;
>> possibly/likely
>> >> due to a filesystem problem. The processes that created them (e.g.,
>> >> mincaverage) did not produce any warnings and completed successfully;
>> in
>> >> addition, minccomplete tells me that the files are complete.
>> >>
>> >> Unfortunately, trying to read these files throws HDF5 and miicv errors
>> (see
>> >> below) and they are obviously corrupt.
>> >>
>> >> I am thinking that it would be useful to complete minccomplete by
>> having it
>> >> actually test-read the data, such that it would report on file
>> integrity?
>> >> This would make it easy to find these kinds of corruptions - and could
>> even
>> >> tack that end the end of scripts to make sure outputs are intact. I'm
>> >> currently using 'mincstats -quiet -min' to locate them, but it seems
>> the
>> >> natural place for this test would actually be minccomplete.
>> >>
>> >> -- A
>> >>
>> >> HDF5-DIAG: Error detected in HDF5 (1.8.9) thread 0:
>> >>   #000: H5Dio.c line 174 in H5Dread(): can't read data
>> >>     major: Dataset
>> >>     minor: Read failed
>> >>   #001: H5Dio.c line 449 in H5D_read(): can't read data
>> >>     major: Dataset
>> >>     minor: Read failed
>> >>   #002: H5Dchunk.c line 1729 in H5D_chunk_read(): unable to read raw
>> data
>> >> chunk
>> >>     major: Low-level I/O
>> >>     minor: Read failed
>> >>   #003: H5Dchunk.c line 2760 in H5D_chunk_lock(): data pipeline read
>> failed
>> >>     major: Data filters
>> >>     minor: Filter operation failed
>> >>   #004: H5Z.c line 1120 in H5Z_pipeline(): filter returned failure
>> during
>> >> read
>> >>     major: Data filters
>> >>     minor: Read failed
>> >>   #005: H5Zdeflate.c line 125 in H5Z_filter_deflate(): inflate() failed
>> >>     major: Data filters
>> >>     minor: Unable to initialize object
>> >> mincstats (from miicv_get): Can't read dataset /minc-2.0/image/0/image
>> >> _______________________________________________
>> >> MINC-users at bic.mni.mcgill.ca
>> >> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>> >>
>> >
>> >
>> >
>> > --
>> > Best regards,
>> >
>> >  Vladimir S. Fonov ~ vladimir <dot> fonov <at> gmail <dot> com
>> > _______________________________________________
>> > MINC-users at bic.mni.mcgill.ca
>> > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>> _______________________________________________
>> MINC-users at bic.mni.mcgill.ca
>> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>>
>>
>