From zijdenbos at gmail.com  Thu Jun 25 16:26:02 2020
From: zijdenbos at gmail.com (Alex Zijdenbos)
Date: Thu, 25 Jun 2020 16:26:02 -0400
Subject: [MINC-users] Tempdir handling by some MINC tools
Message-ID: <CABEEk17vG-5pPXP89YRf=VH7WSidwajNN0rrX-AweNoSaZcunA@mail.gmail.com>

Hello all,

I recently figured out why I am now and again running into some annoying
mincdiff failures, where mincdiff does not report "Images are identical"
even though they actually are. This comes back to the way mincheader
creates and uses a temporary directory, here:

https://github.com/BIC-MNI/minc-tools/blob/master/progs/mincheader/mincheader#L68-L76


There are two problems with this bit of code:

   1. it will generate a tempdir solely based on the pid. The pid is of
   course unique at that moment, but the generated directory may not be - for
   example, in case previous invocations with a (past) identical pid were not
   cleaned up. So something like mincdiff (that calls mincheader) might fail,
   solely because of a tempdir name clash
   2. in the case of mincheader, the only use case for this appears to be
   in case of externally compressed .mnc.gz files. I am not sure who still
   uses those (I don't), but this means that working with internally
   compressed MINC2 files, each mincheader call will create a dir in /tmp that
   is subsequently not used for anything.

Similar bits of tempdir/file code exist in mincinfo, mincdiff, and likely
others.

I was wondering if anybody might have fixed this already. In perl this is
easy, using File::Temp that will guarantee unique temporary files - but
these tools are all shell scripts. If somebody has some .sh code lying
around that does this better, perhaps that can be slotted into these tools
pretty easily?

Thanks,

-- A

From gabriel.devenyi at mcgill.ca  Thu Jun 25 16:46:09 2020
From: gabriel.devenyi at mcgill.ca (Gabriel A. Devenyi)
Date: Thu, 25 Jun 2020 16:46:09 -0400
Subject: [MINC-users] Tempdir handling by some MINC tools
In-Reply-To: <4ea77642455444f4acdcb6327eaf41ea@YQXPR0101MB0838.CANPRD01.PROD.OUTLOOK.COM>
References: <4ea77642455444f4acdcb6327eaf41ea@YQXPR0101MB0838.CANPRD01.PROD.OUTLOOK.COM>
Message-ID: <CAFQOoOo7Ytg=m74n1DOJ-nqKmn7vX_GWBr1m6CiyoiVO60s7pg@mail.gmail.com>

We can fix this up with "mktemp -d" in shell scripts.
--------------------------------------------------------
Asking for technical help? Get help fast:
https://www.chiark.greenend.org.uk/~sgtatham/bugs.html
http://www.catb.org/~esr/faqs/smart-questions.html#beprecise
https://en.wikipedia.org/wiki/XY_problem
https://stackoverflow.com/help/minimal-reproducible-example


--
Gabriel A. Devenyi Ph.D.
Research Computing Associate
Computational Brain Anatomy Laboratory
Cerebral Imaging Center
Douglas Mental Health University Institute
Affiliate, Department of Psychiatry
McGill University
t: 514.761.6131x4781
e: gabriel.devenyi at mcgill.ca


On Thu, 25 Jun 2020 at 16:26, Alex Zijdenbos <zijdenbos at gmail.com> wrote:

> Hello all,
>
> I recently figured out why I am now and again running into some annoying
> mincdiff failures, where mincdiff does not report "Images are identical"
> even though they actually are. This comes back to the way mincheader
> creates and uses a temporary directory, here:
>
>
> https://github.com/BIC-MNI/minc-tools/blob/master/progs/mincheader/mincheader#L68-L76
>
>
> There are two problems with this bit of code:
>
>    1. it will generate a tempdir solely based on the pid. The pid is of
>    course unique at that moment, but the generated directory may not be -
> for
>    example, in case previous invocations with a (past) identical pid were
> not
>    cleaned up. So something like mincdiff (that calls mincheader) might
> fail,
>    solely because of a tempdir name clash
>    2. in the case of mincheader, the only use case for this appears to be
>    in case of externally compressed .mnc.gz files. I am not sure who still
>    uses those (I don't), but this means that working with internally
>    compressed MINC2 files, each mincheader call will create a dir in /tmp
> that
>    is subsequently not used for anything.
>
> Similar bits of tempdir/file code exist in mincinfo, mincdiff, and likely
> others.
>
> I was wondering if anybody might have fixed this already. In perl this is
> easy, using File::Temp that will guarantee unique temporary files - but
> these tools are all shell scripts. If somebody has some .sh code lying
> around that does this better, perhaps that can be slotted into these tools
> pretty easily?
>
> Thanks,
>
> -- A
> _______________________________________________
> MINC-users at bic.mni.mcgill.ca
> https://mailman.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>

From gabriel.devenyi at mcgill.ca  Thu Jun 25 16:53:06 2020
From: gabriel.devenyi at mcgill.ca (Gabriel A. Devenyi)
Date: Thu, 25 Jun 2020 16:53:06 -0400
Subject: [MINC-users] Tempdir handling by some MINC tools
In-Reply-To: <CAFQOoOo7Ytg=m74n1DOJ-nqKmn7vX_GWBr1m6CiyoiVO60s7pg@mail.gmail.com>
References: <4ea77642455444f4acdcb6327eaf41ea@YQXPR0101MB0838.CANPRD01.PROD.OUTLOOK.COM>
 <CAFQOoOo7Ytg=m74n1DOJ-nqKmn7vX_GWBr1m6CiyoiVO60s7pg@mail.gmail.com>
Message-ID: <CAFQOoOrp1ufyDCys-4jP8=sGTjXzVZ-Yp3YDNtT3pjEk-r6Cwg@mail.gmail.com>

Can you please open an issue listing all the scripts you caught this in and
I'll look into swapping in some more robust code. Maybe we can do this
before we push 1.9.18

Thanks!

--------------------------------------------------------
Asking for technical help? Get help fast:
https://www.chiark.greenend.org.uk/~sgtatham/bugs.html
http://www.catb.org/~esr/faqs/smart-questions.html#beprecise
https://en.wikipedia.org/wiki/XY_problem
https://stackoverflow.com/help/minimal-reproducible-example


--
Gabriel A. Devenyi Ph.D.
Research Computing Associate
Computational Brain Anatomy Laboratory
Cerebral Imaging Center
Douglas Mental Health University Institute
Affiliate, Department of Psychiatry
McGill University
t: 514.761.6131x4781
e: gabriel.devenyi at mcgill.ca


On Thu, 25 Jun 2020 at 16:46, Gabriel A. Devenyi <gabriel.devenyi at mcgill.ca>
wrote:

> We can fix this up with "mktemp -d" in shell scripts.
> --------------------------------------------------------
> Asking for technical help? Get help fast:
> https://www.chiark.greenend.org.uk/~sgtatham/bugs.html
> http://www.catb.org/~esr/faqs/smart-questions.html#beprecise
> https://en.wikipedia.org/wiki/XY_problem
> https://stackoverflow.com/help/minimal-reproducible-example
>
>
> --
> Gabriel A. Devenyi Ph.D.
> Research Computing Associate
> Computational Brain Anatomy Laboratory
> Cerebral Imaging Center
> Douglas Mental Health University Institute
> Affiliate, Department of Psychiatry
> McGill University
> t: 514.761.6131x4781
> e: gabriel.devenyi at mcgill.ca
>
>
> On Thu, 25 Jun 2020 at 16:26, Alex Zijdenbos <zijdenbos at gmail.com> wrote:
>
>> Hello all,
>>
>> I recently figured out why I am now and again running into some annoying
>> mincdiff failures, where mincdiff does not report "Images are identical"
>> even though they actually are. This comes back to the way mincheader
>> creates and uses a temporary directory, here:
>>
>>
>> https://github.com/BIC-MNI/minc-tools/blob/master/progs/mincheader/mincheader#L68-L76
>>
>>
>> There are two problems with this bit of code:
>>
>>    1. it will generate a tempdir solely based on the pid. The pid is of
>>    course unique at that moment, but the generated directory may not be -
>> for
>>    example, in case previous invocations with a (past) identical pid were
>> not
>>    cleaned up. So something like mincdiff (that calls mincheader) might
>> fail,
>>    solely because of a tempdir name clash
>>    2. in the case of mincheader, the only use case for this appears to be
>>    in case of externally compressed .mnc.gz files. I am not sure who still
>>    uses those (I don't), but this means that working with internally
>>    compressed MINC2 files, each mincheader call will create a dir in /tmp
>> that
>>    is subsequently not used for anything.
>>
>> Similar bits of tempdir/file code exist in mincinfo, mincdiff, and likely
>> others.
>>
>> I was wondering if anybody might have fixed this already. In perl this is
>> easy, using File::Temp that will guarantee unique temporary files - but
>> these tools are all shell scripts. If somebody has some .sh code lying
>> around that does this better, perhaps that can be slotted into these tools
>> pretty easily?
>>
>> Thanks,
>>
>> -- A
>> _______________________________________________
>> MINC-users at bic.mni.mcgill.ca
>> https://mailman.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>>
>

From zijdenbos at gmail.com  Fri Jun 26 10:16:50 2020
From: zijdenbos at gmail.com (Alex Zijdenbos)
Date: Fri, 26 Jun 2020 10:16:50 -0400
Subject: [MINC-users] Tempdir handling by some MINC tools
In-Reply-To: <CAFQOoOrp1ufyDCys-4jP8=sGTjXzVZ-Yp3YDNtT3pjEk-r6Cwg@mail.gmail.com>
References: <4ea77642455444f4acdcb6327eaf41ea@YQXPR0101MB0838.CANPRD01.PROD.OUTLOOK.COM>
 <CAFQOoOo7Ytg=m74n1DOJ-nqKmn7vX_GWBr1m6CiyoiVO60s7pg@mail.gmail.com>
 <CAFQOoOrp1ufyDCys-4jP8=sGTjXzVZ-Yp3YDNtT3pjEk-r6Cwg@mail.gmail.com>
Message-ID: <CABEEk15UiQr0G0k66ASsJNgOFgebSBV-t671WGVo2dvTaZ7jqg@mail.gmail.com>

Thanks, Gabriel - I filed  https://github.com/BIC-MNI/minc-tools/issues/105

On Thu, Jun 25, 2020 at 4:54 PM Gabriel A. Devenyi <
gabriel.devenyi at mcgill.ca> wrote:

> Can you please open an issue listing all the scripts you caught this in and
> I'll look into swapping in some more robust code. Maybe we can do this
> before we push 1.9.18
>
> Thanks!
>
> --------------------------------------------------------
> Asking for technical help? Get help fast:
> https://www.chiark.greenend.org.uk/~sgtatham/bugs.html
> http://www.catb.org/~esr/faqs/smart-questions.html#beprecise
> https://en.wikipedia.org/wiki/XY_problem
> https://stackoverflow.com/help/minimal-reproducible-example
>
>
> --
> Gabriel A. Devenyi Ph.D.
> Research Computing Associate
> Computational Brain Anatomy Laboratory
> Cerebral Imaging Center
> Douglas Mental Health University Institute
> Affiliate, Department of Psychiatry
> McGill University
> t: 514.761.6131x4781
> e: gabriel.devenyi at mcgill.ca
>
>
> On Thu, 25 Jun 2020 at 16:46, Gabriel A. Devenyi <
> gabriel.devenyi at mcgill.ca>
> wrote:
>
> > We can fix this up with "mktemp -d" in shell scripts.
> > --------------------------------------------------------
> > Asking for technical help? Get help fast:
> > https://www.chiark.greenend.org.uk/~sgtatham/bugs.html
> > http://www.catb.org/~esr/faqs/smart-questions.html#beprecise
> > https://en.wikipedia.org/wiki/XY_problem
> > https://stackoverflow.com/help/minimal-reproducible-example
> >
> >
> > --
> > Gabriel A. Devenyi Ph.D.
> > Research Computing Associate
> > Computational Brain Anatomy Laboratory
> > Cerebral Imaging Center
> > Douglas Mental Health University Institute
> > Affiliate, Department of Psychiatry
> > McGill University
> > t: 514.761.6131x4781
> > e: gabriel.devenyi at mcgill.ca
> >
> >
> > On Thu, 25 Jun 2020 at 16:26, Alex Zijdenbos <zijdenbos at gmail.com>
> wrote:
> >
> >> Hello all,
> >>
> >> I recently figured out why I am now and again running into some annoying
> >> mincdiff failures, where mincdiff does not report "Images are identical"
> >> even though they actually are. This comes back to the way mincheader
> >> creates and uses a temporary directory, here:
> >>
> >>
> >>
> https://github.com/BIC-MNI/minc-tools/blob/master/progs/mincheader/mincheader#L68-L76
> >>
> >>
> >> There are two problems with this bit of code:
> >>
> >>    1. it will generate a tempdir solely based on the pid. The pid is of
> >>    course unique at that moment, but the generated directory may not be
> -
> >> for
> >>    example, in case previous invocations with a (past) identical pid
> were
> >> not
> >>    cleaned up. So something like mincdiff (that calls mincheader) might
> >> fail,
> >>    solely because of a tempdir name clash
> >>    2. in the case of mincheader, the only use case for this appears to
> be
> >>    in case of externally compressed .mnc.gz files. I am not sure who
> still
> >>    uses those (I don't), but this means that working with internally
> >>    compressed MINC2 files, each mincheader call will create a dir in
> /tmp
> >> that
> >>    is subsequently not used for anything.
> >>
> >> Similar bits of tempdir/file code exist in mincinfo, mincdiff, and
> likely
> >> others.
> >>
> >> I was wondering if anybody might have fixed this already. In perl this
> is
> >> easy, using File::Temp that will guarantee unique temporary files - but
> >> these tools are all shell scripts. If somebody has some .sh code lying
> >> around that does this better, perhaps that can be slotted into these
> tools
> >> pretty easily?
> >>
> >> Thanks,
> >>
> >> -- A
> >> _______________________________________________
> >> MINC-users at bic.mni.mcgill.ca
> >> https://mailman.bic.mni.mcgill.ca/mailman/listinfo/minc-users
> >>
> >
> _______________________________________________
> MINC-users at bic.mni.mcgill.ca
> https://mailman.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>
>