[MINC-users] minccomplete's completeness

[Alex Zijdenbos]

PS I should also note that most, if not all, standard MINC tools are happy
to process these files. They will print these HDF5 errors, but they will
generate an output file and produce an image (that is of course bad). For
example:

mincmath -add -const 0 corrupt.mnc corrupt2.mnc

results in these two images:
​
 corrupt.png
<https://drive.google.com/file/d/0B5fOtqpIs4sKUlotMGg4XzdQN28/view?usp=drive_web>
​
and, unless you troll through log files, a processing pipeline may happily
complete without any explicit failures.

I'd say that HDF5 read errors should at the very least trigger a non-zero
exit status in all MINC tools.

Since somebody might ask, here is this example file (I have a few hundred
more):
​
 corrupt.mnc
<https://drive.google.com/file/d/0B5fOtqpIs4sKdVphRGtfTVF0SWM/view?usp=drive_web>
​
-- A


On Fri, May 13, 2016 at 3:35 PM, Alex Zijdenbos <zijdenbos at gmail.com> wrote:

> Hi all,
>
> I managed to generate a large number of broken MINC files; possibly/likely
> due to a filesystem problem. The processes that created them (e.g.,
> mincaverage) did not produce any warnings and completed successfully; in
> addition, minccomplete tells me that the files are complete.
>
> Unfortunately, trying to read these files throws HDF5 and miicv errors
> (see below) and they are obviously corrupt.
>
> I am thinking that it would be useful to complete minccomplete by having
> it actually test-read the data, such that it would report on file
> integrity? This would make it easy to find these kinds of corruptions - and
> could even tack that end the end of scripts to make sure outputs are
> intact. I'm currently using 'mincstats -quiet -min' to locate them, but it
> seems the natural place for this test would actually be minccomplete.
>
> -- A
>
> HDF5-DIAG: Error detected in HDF5 (1.8.9) thread 0:
>   #000: H5Dio.c line 174 in H5Dread(): can't read data
>     major: Dataset
>     minor: Read failed
>   #001: H5Dio.c line 449 in H5D_read(): can't read data
>     major: Dataset
>     minor: Read failed
>   #002: H5Dchunk.c line 1729 in H5D_chunk_read(): unable to read raw data
> chunk
>     major: Low-level I/O
>     minor: Read failed
>   #003: H5Dchunk.c line 2760 in H5D_chunk_lock(): data pipeline read failed
>     major: Data filters
>     minor: Filter operation failed
>   #004: H5Z.c line 1120 in H5Z_pipeline(): filter returned failure during
> read
>     major: Data filters
>     minor: Read failed
>   #005: H5Zdeflate.c line 125 in H5Z_filter_deflate(): inflate() failed
>     major: Data filters
>     minor: Unable to initialize object
> mincstats (from miicv_get): Can't read dataset /minc-2.0/image/0/image
>
>
>