[MINC-users] minccomplete's completeness

[Alex Zijdenbos]

Hi Claude,

That's a useful idea; but this specific test doesn't actually find the
errors in the volumes I have
so still incomplete :-)

-- A

On Fri, May 13, 2016 at 4:25 PM, Claude LEPAGE <claude at bic.mni.mcgill.ca>
wrote:

> Alex,
>
> Here is what I've been doing to check for completeness of minc2 files
> (for BigBrain 7404 slices):
>
> sub minc_valid {
>
>   my $input = shift;
>
>   my $ret = `h5ls -r $input`;
>   return( 0 ) if( !( $ret =~ m/\/minc-2.0\/info/ ) );
>   return( 0 ) if( !( $ret =~ m/\/minc-2.0\/image\/0\/image/ ) );
>   return( 0 ) if( !( $ret =~ m/\/minc-2.0\/image\/0\/image-min/ ) );
>   return( 0 ) if( !( $ret =~ m/\/minc-2.0\/image\/0\/image-max/ ) );
>
>   return( 1 );
> }
>
> h5ls is a HDF5 tool.
>
> Claude
>
> >
> > Hi all,
> >
> > I managed to generate a large number of broken MINC files;
> possibly/likely
> > due to a filesystem problem. The processes that created them (e.g.,
> > mincaverage) did not produce any warnings and completed successfully; in
> > addition, minccomplete tells me that the files are complete.
> >
> > Unfortunately, trying to read these files throws HDF5 and miicv errors
> (see
> > below) and they are obviously corrupt.
> >
> > I am thinking that it would be useful to complete minccomplete by having
> it
> > actually test-read the data, such that it would report on file integrity?
> > This would make it easy to find these kinds of corruptions - and could
> even
> > tack that end the end of scripts to make sure outputs are intact. I'm
> > currently using 'mincstats -quiet -min' to locate them, but it seems the
> > natural place for this test would actually be minccomplete.
> >
> > -- A
> >
> > HDF5-DIAG: Error detected in HDF5 (1.8.9) thread 0:
> >   #000: H5Dio.c line 174 in H5Dread(): can't read data
> >     major: Dataset
> >     minor: Read failed
> >   #001: H5Dio.c line 449 in H5D_read(): can't read data
> >     major: Dataset
> >     minor: Read failed
> >   #002: H5Dchunk.c line 1729 in H5D_chunk_read(): unable to read raw data
> > chunk
> >     major: Low-level I/O
> >     minor: Read failed
> >   #003: H5Dchunk.c line 2760 in H5D_chunk_lock(): data pipeline read
> failed
> >     major: Data filters
> >     minor: Filter operation failed
> >   #004: H5Z.c line 1120 in H5Z_pipeline(): filter returned failure during
> > read
> >     major: Data filters
> >     minor: Read failed
> >   #005: H5Zdeflate.c line 125 in H5Z_filter_deflate(): inflate() failed
> >     major: Data filters
> >     minor: Unable to initialize object
> > mincstats (from miicv_get): Can't read dataset /minc-2.0/image/0/image
> > _______________________________________________
> > MINC-users at bic.mni.mcgill.ca
> > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
> >
> _______________________________________________
> MINC-users at bic.mni.mcgill.ca
> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>
>