[MINC-users] mincaverage error

Robert D. Vincent robert.d.vincent at mcgill.ca
Thu Jun 11 10:07:26 EDT 2015 

Hi all,

I have a partial fix for this issue. The voxel_loop code enforces a maximum
number of open files based on the constant MI_MAX_NUM_ICV, which is
currently set to 32. The way the code is written, if it cannot open all of
the files, it opens them one at a time. This causes some serious overhead
that seems to lead to an out-of-resource condition in the HDF5 library.

I will continue to investigate whether we can fix the resource leak, but in
the short run I will increase the number allowable open files to something
more realistic, like 1000. This can be done by changing the value of
MI_MAX_NUM_ICV in minc.h.

    -bert

On Thu, Jun 11, 2015 at 10:06 AM, Robert D. Vincent <bert at phalarope.com>
wrote:

> Hi all,
>
> I have a partial fix for this issue. The voxel_loop code enforces a
> maximum number of open files based on the constant MI_MAX_NUM_ICV, which is
> currently set to 32. The way the code is written, if it cannot open all of
> the files, it opens them one at a time. This causes some serious overhead
> that seems to lead to an out-of-resource condition in the HDF5 library.
>
> I will continue to investigate whether we can fix the resource leak, but
> in the short run I will increase the number allowable open files to
> something more realistic, like 1000. This can be done by changing the value
> of MI_MAX_NUM_ICV in minc.h.
>
>     -bert
>
> On Tue, Jun 9, 2015 at 4:49 AM, Andrew Janke <a.janke at gmail.com> wrote:
>
>> that's the idea. it is in mine...
>>
>> a
>> On 09/06/2015 6:48 PM, "Simon Eskildsen" <eskild at gmail.com> wrote:
>>
>> > Does it mean I can safely replace mincaverage in xfmavg with
>> > mincbigaverage? Why hasn't already?
>> >
>> > On Tue, Jun 9, 2015 at 10:37 AM, Andrew Janke <a.janke at gmail.com>
>> wrote:
>> >
>> > > supports most options yes. Will be identical apart from very small
>> > rounding
>> > > errors.
>> > >
>> > > take note of the options for defining the number to average at once.
>> > >
>> > > a
>> > > On 09/06/2015 6:25 PM, "Simon Eskildsen" <eskild at gmail.com> wrote:
>> > >
>> > > > Thanks. I'm trying this now. I assume same behaviour as mincaverage,
>> > > right?
>> > > >
>> > > > On Tue, Jun 9, 2015 at 10:12 AM, Andrew Janke <a.janke at gmail.com>
>> > wrote:
>> > > >
>> > > > > If it's memory, try mincbigaverage.  google knows where it is.
>> > > > >
>> > > > > a
>> > > > > On 09/06/2015 5:15 PM, "Simon Eskildsen" <eskild at gmail.com>
>> wrote:
>> > > > >
>> > > > > > Hi Robert,
>> > > > > >
>> > > > > > I'm averaging 38 deformation fields. Here's the info on one of
>> > them:
>> > > > > >
>> > > > > > $ mincinfo 21/stx2_0004_20131113_091044_t1.mnc.21.i_grid_0.mnc
>> > > > > > file: 21/stx2_0004_20131113_091044_t1.mnc.21.i_grid_0.mnc
>> > > > > > image: signed__ float -10.166246414184570312 to
>> > 11.006509780883789062
>> > > > > > image dimensions: xspace yspace zspace vector_dimension
>> > > > > >     dimension name         length         step        start
>> > > > > >     --------------         ------         ----        -----
>> > > > > >     xspace                    388          0.5        -96.5
>> > > > > >     yspace                    460          0.5       -132.5
>> > > > > >     zspace                    388          0.5        -78.5
>> > > > > >     vector_dimension            3      unknown      unknown
>> > > > > >
>> > > > > > The files are quite big (~800 Mb each). Could this be a memory
>> > > problem?
>> > > > > >
>> > > > > > I used mincaverage without any flags/options. Just a list of
>> files
>> > in
>> > > > > and a
>> > > > > > file out.
>> > > > > >
>> > > > > > Simon
>> > > > > >
>> > > > > > On Mon, Jun 8, 2015 at 3:46 PM, Robert D. Vincent <
>> > > > > > robert.d.vincent at mcgill.ca> wrote:
>> > > > > >
>> > > > > > > Hi Simon,
>> > > > > > >
>> > > > > > > It looks like HDF5 might be running out of resource ids, which
>> > > seems
>> > > > > > > unlikely, but there could be a resource leak somewhere.
>> > > > > > >
>> > > > > > > Can you give me the details of the failing case (e.g. the
>> files
>> > > > you're
>> > > > > > > using)?
>> > > > > > >
>> > > > > > >     -bert
>> > > > > > >
>> > > > > > >
>> > > > > > > On Mon, Jun 8, 2015 at 9:33 AM, Simon Eskildsen <
>> > eskild at gmail.com>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi all,
>> > > > > > > >
>> > > > > > > > Averaging a bunch of deformation fields using mincaverage, I
>> > got
>> > > > this
>> > > > > > > > error:
>> > > > > > > >
>> > > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.14) thread 0:
>> > > > > > > >   #000: /users/fristed/git/minc-toolkit/build/HDF5/src/H5A.c
>> > line
>> > > > 679
>> > > > > > in
>> > > > > > > > H5Aget_space(): can't get space ID of attribute
>> > > > > > > >     major: Invalid arguments to routine
>> > > > > > > >     minor: Can't get value
>> > > > > > > >   #001:
>> /users/fristed/git/minc-toolkit/build/HDF5/src/H5Aint.c
>> > > > line
>> > > > > > 719
>> > > > > > > in
>> > > > > > > > H5A_get_space(): unable to register dataspace atom
>> > > > > > > >     major: Object atom
>> > > > > > > >     minor: Unable to register new atom
>> > > > > > > >   #002: /users/fristed/git/minc-toolkit/build/HDF5/src/H5I.c
>> > line
>> > > > 902
>> > > > > > in
>> > > > > > > > H5I_register(): no IDs available in type
>> > > > > > > >     major: Object atom
>> > > > > > > >     minor: Out of IDs for group
>> > > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.14) thread 0:
>> > > > > > > >   #000: /users/fristed/git/minc-toolkit/build/HDF5/src/H5S.c
>> > line
>> > > > > 1383
>> > > > > > in
>> > > > > > > > H5Screate_simple(): unable to register dataspace ID
>> > > > > > > >     major: Object atom
>> > > > > > > >     minor: Unable to register new atom
>> > > > > > > >   #001: /users/fristed/git/minc-toolkit/build/HDF5/src/H5I.c
>> > line
>> > > > 902
>> > > > > > in
>> > > > > > > > H5I_register(): no IDs available in type
>> > > > > > > >     major: Object atom
>> > > > > > > >     minor: Out of IDs for group
>> > > > > > > > (from miicv_get): Should not happen!
>> > > > > > > >
>> > > > > > > > Setting -debug flag does not provide further information.
>> > > > > > > >
>> > > > > > > > Ideas for debugging this are very welcome.
>> > > > > > > >
>> > > > > > > > Simon
>> > > > > > > > _______________________________________________
>> > > > > > > > MINC-users at bic.mni.mcgill.ca
>> > > > > > > > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>> > > > > > > >
>> > > > > > > _______________________________________________
>> > > > > > > MINC-users at bic.mni.mcgill.ca
>> > > > > > > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>> > > > > > >
>> > > > > > _______________________________________________
>> > > > > > MINC-users at bic.mni.mcgill.ca
>> > > > > > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>> > > > > >
>> > > > > _______________________________________________
>> > > > > MINC-users at bic.mni.mcgill.ca
>> > > > > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>> > > > >
>> > > > _______________________________________________
>> > > > MINC-users at bic.mni.mcgill.ca
>> > > > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>> > > >
>> > > _______________________________________________
>> > > MINC-users at bic.mni.mcgill.ca
>> > > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>> > >
>> > _______________________________________________
>> > MINC-users at bic.mni.mcgill.ca
>> > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>> >
>> _______________________________________________
>> MINC-users at bic.mni.mcgill.ca
>> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>>
>
>

More information about the MINC-users mailing list