[MINC-users] mincaverage error

Thu Jun 11 10:38:51 EDT 2015

Hello,

well, this is interesting. I can imagine that we will have to do 
something about it to make MINC even remotely attractive to the masses.

One of the attractive features was ability to handle very large datasets 
that wouldn't fit into memory. Otherwise - it's very difficult to 
explain why even bother with MINC these days.

On 15-06-11 10:07 AM, Robert D. Vincent wrote:
> Hi all,
>
> I have a partial fix for this issue. The voxel_loop code enforces a maximum
> number of open files based on the constant MI_MAX_NUM_ICV, which is
> currently set to 32. The way the code is written, if it cannot open all of
> the files, it opens them one at a time. This causes some serious overhead
> that seems to lead to an out-of-resource condition in the HDF5 library.
>
> I will continue to investigate whether we can fix the resource leak, but in
> the short run I will increase the number allowable open files to something
> more realistic, like 1000. This can be done by changing the value of
> MI_MAX_NUM_ICV in minc.h.
>
>      -bert
>
> On Thu, Jun 11, 2015 at 10:06 AM, Robert D. Vincent <bert at phalarope.com>
> wrote:
>
>> Hi all,
>>
>> I have a partial fix for this issue. The voxel_loop code enforces a
>> maximum number of open files based on the constant MI_MAX_NUM_ICV, which is
>> currently set to 32. The way the code is written, if it cannot open all of
>> the files, it opens them one at a time. This causes some serious overhead
>> that seems to lead to an out-of-resource condition in the HDF5 library.
>>
>> I will continue to investigate whether we can fix the resource leak, but
>> in the short run I will increase the number allowable open files to
>> something more realistic, like 1000. This can be done by changing the value
>> of MI_MAX_NUM_ICV in minc.h.
>>
>>      -bert
>>
>> On Tue, Jun 9, 2015 at 4:49 AM, Andrew Janke <a.janke at gmail.com> wrote:
>>
>>> that's the idea. it is in mine...
>>>
>>> a
>>> On 09/06/2015 6:48 PM, "Simon Eskildsen" <eskild at gmail.com> wrote:
>>>
>>>> Does it mean I can safely replace mincaverage in xfmavg with
>>>> mincbigaverage? Why hasn't already?
>>>>
>>>> On Tue, Jun 9, 2015 at 10:37 AM, Andrew Janke <a.janke at gmail.com>
>>> wrote:
>>>>
>>>>> supports most options yes. Will be identical apart from very small
>>>> rounding
>>>>> errors.
>>>>>
>>>>> take note of the options for defining the number to average at once.
>>>>>
>>>>> a
>>>>> On 09/06/2015 6:25 PM, "Simon Eskildsen" <eskild at gmail.com> wrote:
>>>>>
>>>>>> Thanks. I'm trying this now. I assume same behaviour as mincaverage,
>>>>> right?
>>>>>>
>>>>>> On Tue, Jun 9, 2015 at 10:12 AM, Andrew Janke <a.janke at gmail.com>
>>>> wrote:
>>>>>>
>>>>>>> If it's memory, try mincbigaverage.  google knows where it is.
>>>>>>>
>>>>>>> a
>>>>>>> On 09/06/2015 5:15 PM, "Simon Eskildsen" <eskild at gmail.com>
>>> wrote:
>>>>>>>
>>>>>>>> Hi Robert,
>>>>>>>>
>>>>>>>> I'm averaging 38 deformation fields. Here's the info on one of
>>>> them:
>>>>>>>>
>>>>>>>> $ mincinfo 21/stx2_0004_20131113_091044_t1.mnc.21.i_grid_0.mnc
>>>>>>>> file: 21/stx2_0004_20131113_091044_t1.mnc.21.i_grid_0.mnc
>>>>>>>> image: signed__ float -10.166246414184570312 to
>>>> 11.006509780883789062
>>>>>>>> image dimensions: xspace yspace zspace vector_dimension
>>>>>>>>      dimension name         length         step        start
>>>>>>>>      --------------         ------         ----        -----
>>>>>>>>      xspace                    388          0.5        -96.5
>>>>>>>>      yspace                    460          0.5       -132.5
>>>>>>>>      zspace                    388          0.5        -78.5
>>>>>>>>      vector_dimension            3      unknown      unknown
>>>>>>>>
>>>>>>>> The files are quite big (~800 Mb each). Could this be a memory
>>>>> problem?
>>>>>>>>
>>>>>>>> I used mincaverage without any flags/options. Just a list of
>>> files
>>>> in
>>>>>>> and a
>>>>>>>> file out.
>>>>>>>>
>>>>>>>> Simon
>>>>>>>>
>>>>>>>> On Mon, Jun 8, 2015 at 3:46 PM, Robert D. Vincent <
>>>>>>>> robert.d.vincent at mcgill.ca> wrote:
>>>>>>>>
>>>>>>>>> Hi Simon,
>>>>>>>>>
>>>>>>>>> It looks like HDF5 might be running out of resource ids, which
>>>>> seems
>>>>>>>>> unlikely, but there could be a resource leak somewhere.
>>>>>>>>>
>>>>>>>>> Can you give me the details of the failing case (e.g. the
>>> files
>>>>>> you're
>>>>>>>>> using)?
>>>>>>>>>
>>>>>>>>>      -bert
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jun 8, 2015 at 9:33 AM, Simon Eskildsen <
>>>> eskild at gmail.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> Averaging a bunch of deformation fields using mincaverage, I
>>>> got
>>>>>> this
>>>>>>>>>> error:
>>>>>>>>>>
>>>>>>>>>> HDF5-DIAG: Error detected in HDF5 (1.8.14) thread 0:
>>>>>>>>>>    #000: /users/fristed/git/minc-toolkit/build/HDF5/src/H5A.c
>>>> line
>>>>>> 679
>>>>>>>> in
>>>>>>>>>> H5Aget_space(): can't get space ID of attribute
>>>>>>>>>>      major: Invalid arguments to routine
>>>>>>>>>>      minor: Can't get value
>>>>>>>>>>    #001:
>>> /users/fristed/git/minc-toolkit/build/HDF5/src/H5Aint.c
>>>>>> line
>>>>>>>> 719
>>>>>>>>> in
>>>>>>>>>> H5A_get_space(): unable to register dataspace atom
>>>>>>>>>>      major: Object atom
>>>>>>>>>>      minor: Unable to register new atom
>>>>>>>>>>    #002: /users/fristed/git/minc-toolkit/build/HDF5/src/H5I.c
>>>> line
>>>>>> 902
>>>>>>>> in
>>>>>>>>>> H5I_register(): no IDs available in type
>>>>>>>>>>      major: Object atom
>>>>>>>>>>      minor: Out of IDs for group
>>>>>>>>>> HDF5-DIAG: Error detected in HDF5 (1.8.14) thread 0:
>>>>>>>>>>    #000: /users/fristed/git/minc-toolkit/build/HDF5/src/H5S.c
>>>> line
>>>>>>> 1383
>>>>>>>> in
>>>>>>>>>> H5Screate_simple(): unable to register dataspace ID
>>>>>>>>>>      major: Object atom
>>>>>>>>>>      minor: Unable to register new atom
>>>>>>>>>>    #001: /users/fristed/git/minc-toolkit/build/HDF5/src/H5I.c
>>>> line
>>>>>> 902
>>>>>>>> in
>>>>>>>>>> H5I_register(): no IDs available in type
>>>>>>>>>>      major: Object atom
>>>>>>>>>>      minor: Out of IDs for group
>>>>>>>>>> (from miicv_get): Should not happen!
>>>>>>>>>>
>>>>>>>>>> Setting -debug flag does not provide further information.
>>>>>>>>>>
>>>>>>>>>> Ideas for debugging this are very welcome.

-- 
Best regards,

  Vladimir S. FONOV ~ vladimir.fonov <at> gmail.com