[MINC-users] minccomplete's completeness

Matthijs van Eede matthijs.vaneede at sickkids.ca
Fri Jun 17 14:57:37 EDT 2016 

Hi All,

We have a fix for pyminc, which will soon be in the master branch, but also a question about how you should set the image:complete flag in a MINC file. The first thing we tried was:

miset_attr_values(self.volPointer, MI_TYPE_STRING, "image", "complete", len("true_"), "true_")

which creates a new entry in the MINC file header like this:

int image ;
    image:vartype = "group________" ;
    image:complete = "true_" ;
    image:signtype = "signed__" ;

But this is not the entry that minccomplete is after. Then we used the following which does work, but which seems a bit sketchy:

miset_attribute(self.volPointer, "/minc-2.0/image/0/image", "complete", MI_TYPE_STRING, len("true_"), "true_")

Does anyone know why you can't set the image:complete attribute using the first method?

Thanks,
Matthijs

________________________________________
From: minc-users-bounces at bic.mni.mcgill.ca [minc-users-bounces at bic.mni.mcgill.ca] on behalf of Alex Zijdenbos [zijdenbos at gmail.com]
Sent: Wednesday, June 01, 2016 10:10 AM
To: MINC users mailing list
Subject: Re: [MINC-users] minccomplete's completeness

Thanks, Bert!

I actually worked in explicit MINC integrity tests in the central sub I
have that spawns processes (similar to MNI::Spawn), so that every MINC
volume that is present in a spawned command line, is explicitly checked.
Besides nii2mnc, the only thing it has turned up so far is pyminc. But
there are likely a few more things out there.

-- A

On Wed, Jun 1, 2016 at 12:25 AM, Robert D. Vincent <
robert.d.vincent at mcgill.ca> wrote:

> Hi Alex,
>
> I just checked in a fix for nii2mnc. I am sure there are other tools that
> do not set the attribute - I'll try to find and fix them.
>
>     -bert
>
> On Mon, May 30, 2016 at 11:37 AM, Alex Zijdenbos <zijdenbos at gmail.com>
> wrote:
>
> > Nice work, Bert!
> >
> > I have since built in some paranoid checking in all of my scripts, and
> one
> > thing that keeps turning up is that nii2mnc doesn't actually set the
> > 'complete' flag. Iow, anything created by nii2mnc fails minccomplete. Any
> > chance you fixed that as well?
> >
> > Thanks,
> >
> > -- A
> >
> > On Mon, May 16, 2016 at 12:38 PM, Robert D. Vincent <
> > robert.d.vincent at mcgill.ca> wrote:
> >
> > > Hi all,
> > >
> > > I have just checked in changes to volume_io, voxel_loop and the
> following
> > > minc tools:
> > >
> > > mincaverage, minccalc, minccmp, mincconcat, mincconvert, minccopy,
> > > mincextract, minclookup, mincmakescalar, mincmakevector, mincmath,
> > > mincresample, mincreshape, mincsample, mincstats, minctoraw,
> mincwindow.
> > >
> > > These tools will still behave as before, in that they will run to
> > > completion with a corrupt file. However, they will report the error as
> a
> > > non-zero exit code (EXIT_FAILURE), in addition to whatever messages are
> > > currently displayed by the libraries.
> > >
> > > Two existing tools, mincinfo and mincdump, seem to already have correct
> > > handling of read errors. At least two others, mincblob and mincmorph,
> > > should pick up the changes via volume_io.
> > >
> > > This covers the majority of the command line tools. It may also make
> > sense
> > > to reflect these changes in some of the basic scripts (e.g. mincpik,
> > > mincdiff) - let me know if there are any particularly salient cases we
> > > should take care of.
> > >
> > >     -bert
> > >
> > > On Sat, May 14, 2016 at 5:04 PM, Alex Zijdenbos <zijdenbos at gmail.com>
> > > wrote:
> > >
> > > > On this note: nii2mnc is one of the tools that doesn't appear to set
> > the
> > > > image:complete attribute. I haven't found any others yet but since I
> > > > started running minccomplete on anything I generate, I may come up
> with
> > > > more :)
> > > >
> > > > On Sat, May 14, 2016 at 6:53 AM, Vladimir S. FONOV <
> > > > vladimir.fonov at gmail.com
> > > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > the only thing minccomplete does is read the contents of the
> > > > image:complete
> > > > > attribute. Which is not even respected by all minc tools.
> > > > > One real way to check the consistency of the minc file would be to
> > > enable
> > > > > checksum , integrated into HDF5 file format.
> > > > >
> > > > > Currently it can be enabled by setting the environment variable
> > > > > MINC_CHECKSUM to 1 , at the time when file is created. Obviously,
> > this
> > > > > works only with MINC2 file format and it was not tested very well
> > yet.
> > > > >
> > > > > On Fri, May 13, 2016 at 3:35 PM, Alex Zijdenbos <
> zijdenbos at gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I managed to generate a large number of broken MINC files;
> > > > > possibly/likely
> > > > > > due to a filesystem problem. The processes that created them
> (e.g.,
> > > > > > mincaverage) did not produce any warnings and completed
> > successfully;
> > > > in
> > > > > > addition, minccomplete tells me that the files are complete.
> > > > > >
> > > > > > Unfortunately, trying to read these files throws HDF5 and miicv
> > > errors
> > > > > (see
> > > > > > below) and they are obviously corrupt.
> > > > > >
> > > > > > I am thinking that it would be useful to complete minccomplete by
> > > > having
> > > > > it
> > > > > > actually test-read the data, such that it would report on file
> > > > integrity?
> > > > > > This would make it easy to find these kinds of corruptions - and
> > > could
> > > > > even
> > > > > > tack that end the end of scripts to make sure outputs are intact.
> > I'm
> > > > > > currently using 'mincstats -quiet -min' to locate them, but it
> > seems
> > > > the
> > > > > > natural place for this test would actually be minccomplete.
> > > > > >
> > > > > > -- A
> > > > > >
> > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.9) thread 0:
> > > > > >   #000: H5Dio.c line 174 in H5Dread(): can't read data
> > > > > >     major: Dataset
> > > > > >     minor: Read failed
> > > > > >   #001: H5Dio.c line 449 in H5D_read(): can't read data
> > > > > >     major: Dataset
> > > > > >     minor: Read failed
> > > > > >   #002: H5Dchunk.c line 1729 in H5D_chunk_read(): unable to read
> > raw
> > > > data
> > > > > > chunk
> > > > > >     major: Low-level I/O
> > > > > >     minor: Read failed
> > > > > >   #003: H5Dchunk.c line 2760 in H5D_chunk_lock(): data pipeline
> > read
> > > > > failed
> > > > > >     major: Data filters
> > > > > >     minor: Filter operation failed
> > > > > >   #004: H5Z.c line 1120 in H5Z_pipeline(): filter returned
> failure
> > > > during
> > > > > > read
> > > > > >     major: Data filters
> > > > > >     minor: Read failed
> > > > > >   #005: H5Zdeflate.c line 125 in H5Z_filter_deflate(): inflate()
> > > failed
> > > > > >     major: Data filters
> > > > > >     minor: Unable to initialize object
> > > > > > mincstats (from miicv_get): Can't read dataset
> > > /minc-2.0/image/0/image
> > > > > > _______________________________________________
> > > > > > MINC-users at bic.mni.mcgill.ca
> > > > > > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > >
> > > > >  Vladimir S. Fonov ~ vladimir <dot> fonov <at> gmail <dot> com
> > > > > _______________________________________________
> > > > > MINC-users at bic.mni.mcgill.ca
> > > > > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
> > > > >
> > > > >
> > > > _______________________________________________
> > > > MINC-users at bic.mni.mcgill.ca
> > > > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
> > > >
> > > _______________________________________________
> > > MINC-users at bic.mni.mcgill.ca
> > > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
> > >
> > >
> > _______________________________________________
> > MINC-users at bic.mni.mcgill.ca
> > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
> >
> _______________________________________________
> MINC-users at bic.mni.mcgill.ca
> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>
>
_______________________________________________
MINC-users at bic.mni.mcgill.ca
http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users

________________________________

This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies.

More information about the MINC-users mailing list