[MINC-users] Announce - qbatch 1.0 - Execute shell command lines in parallel (serial farm) on SGE/PBS clusters

Wed May 11 20:30:57 EDT 2016

I've taken a proper look at this now. It currently is in no way a drop
in replacement but from what I can see it shouldn't be too hard. I'm
more than happy to do away with my old perl version. For one your
handling of dependencies is much nicer and self contained. There big
differences are:

1. qbatch(mine), was designed equivalent to nohup. ie: write a
command, test it, add qbatch.

   $ for i in *.mnc; do process_something.sh -args *.mnc; done

When happy you'd do this:

   $ for i in *.mnc; do qbatch --queue my.q --name $i --
process-something.sh -args *.mnc;
done

Whereas qbatch(python) expects commands to be in a file, this makes it
difficult to use things like globs with qbatch:

   $ qbatch --logfile blah.log -- do_something_args.pl *.mnc

I see a PR which is along these lines:

   https://github.com/pipitone/qbatch/issues/90

How hard would it be to support a single command after "--"? I note
that you can pipe things to qbatch via echo but this is a real pain to
use in things like perl and python scripts. OK in shell though.

2. I find debugging output in the logfiles to be invaluable when
debugging things. Stuff like the hostname and time of execution.

https://github.com/andrewjanke/qbatch/blob/master/qbatch#L151

and

https://github.com/andrewjanke/qbatch/blob/master/qbatch#L172

This means that I can quickly grep for "exit status" on a directory of logfiles.

3. I note slight differences in how we handle cwd, is there are a
reason for you cd?

   https://github.com/pipitone/qbatch/blob/master/bin/qbatch#L67

I found this problematic for NFS mounts if using relative paths, and
let gridengine sort out things itself.:

   https://github.com/andrewjanke/qbatch/blob/master/qbatch#L108

There are a number of smaller things (job name larger than 15
characters == FAIL in PBS). This causes issues when tracking
dependencies as your unique job name may be longer than this. You can
either bomb and error out to the user (choose a shorter job name!) or
you have to track an internal database of jobnumbers (capture the
return value of qsub) and match these to input job names.  Currently
in your implementation I think this will fail.

There are also issues around re-running jobs:

   https://github.com/andrewjanke/qbatch/blob/master/qbatch#L103

As this can also screw up checking dependencies. In your case I think
this is handled better as you query based upon name. I didn't see from
the code but can you specify a dependency as a job number?

ta

a

On 10 May 2016 at 21:23, Gabriel A. Devenyi <gdevenyi at gmail.com> wrote:
> Simon/Andrew, indeed this was written as an intended replacement for
> sge_batch and/or the original qbatch (we at CoBrALab also had an earlier
> internal qbatch which was not Andrew's version).
> Having said that, we didn't ensure that it was a one-to-one drop in
> replacement, so use may require some modification.
>
> Andrew:
> Yes, dependencies in PBS/SGE (and LSF, but I haven't tested LSF at all yet)
> are already implemented, just missing from the README, will add :) My
> antsRegistration-MaGeT pipeline makes extensive use of them already.
>
> For PBS we parse the XML joblist and do pattern matching on names:
> https://github.com/pipitone/qbatch/blob/master/bin/qbatch#L138-L171
> And for SGE we use jobname pattern matching built into SGE:
> https://github.com/pipitone/qbatch/blob/master/bin/qbatch#L381-L382
>
> As for the queue, I have an in progress branch to specify "other" batch
> options, handled by environment variable, see
> https://github.com/pipitone/qbatch/pull/80
> I just need to solve the append vs. replace issue I'm having with argparse.
> Thoughts here welcome :)
>
> For bugs/feature requests/modifications, please feel free to request! Our
> goal is to provide a maintained tool to support existing workflows and
> enable new ones.
>
> --
> Gabriel A. Devenyi B.Eng. Ph.D.
> Research Computing Associate
> Computational Brain Anatomy Laboratory
> Cerebral Imaging Center
> Douglas Mental Health University Institute
> Affiliate, Department of Psychiatry
> McGill University
> t: 514.761.6131x4781
> e: gdevenyi at gmail.com
>
> On Tue, May 10, 2016 at 9:07 AM, Andrew Janke <a.janke at gmail.com> wrote:
>>
>> Very tasty.
>>
>> Was it designed as a drop in replacement for qbatch?
>>
>>    https://github.com/andrewjanke/qbatch/blob/master/qbatch
>>
>> I note a number of similarities in the C/L arguments. Any plans for
>> handling dependencies? I see code in there but no arguments to make
>> use of it? In particular to get around the hoops of tracking
>> dependencies via job numbers (that my qbatch returns) in PBS and/or
>> via names in gridengine? ie:
>>
>>    $ for i in *.mnc; do qbatch --name STEP1-$i -- <blah>; done
>>    $ qbatch --name STEP2 --depends 'STEP1-*' -- <blah>
>>
>> I ask as I use qbatch internally in things like volgenmodel in order
>> to abstract away from the various qsub's. Currently I handle this via
>> some pretty horrendous perl code here:
>>
>>    https://github.com/andrewjanke/volgenmodel/blob/master/volgenmodel#L751
>>
>> It'd be nice to be able to do away with it via your version!
>>
>> I also find the notion of being able to define a queue via an ENV var
>> useful when you have a number of levels of scripts/perl/etc that don't
>> always pass arguments through.  ie:
>>
>>    https://github.com/andrewjanke/qbatch/blob/master/qbatch#L69
>>
>>
>> a
>>
>> On 10 May 2016 at 03:38, Gabriel A. Devenyi <gdevenyi at gmail.com> wrote:
>> > We (Gabriel A. Devenyi and Jon Pipitone) would like to announce the 1.0
>> > release <https://github.com/pipitone/qbatch/releases> of qbatch, a
>> > command-line tool for easily running a list of commands in parallel
>> > (serial
>> > farming) on a compute cluster. This tool takes the list of commands,
>> > divides them up into batches of arbitrary size, and then submits each
>> > batch
>> > as a separate job or as part of an array job. qbatch also gives you a
>> > consistent interface to use to submit commands on PBS and SGE clusters
>> > or
>> > locally, (support for others are planned/in testing, PRs welcome) while
>> > setting requirements for processors, walltime, memory and job
>> > dependencies.
>> > This tool can be used as a quick interface to spread work out on a
>> > cluster,
>> > or as the glue for for connecting a simple pipeline to a cluster (see
>> > https://github.com/CobraLab/antsRegistration-MAGeT for a sample
>> > implementation)
>> >
>> > The target audience of qbatch is two-fold: it is immediately available
>> > for
>> > users of PBS or SGE clusters to simplify their job construction, in
>> > addition, through the use of environment variables, cluster
>> > administrations
>> > can craft a default qbatch deployment which allows new cluster users to
>> > quickly submit jobs which honours the cluster’s policies.
>> >
>> > For more information, check out our github webpage here:
>> > http://github.com/pipitone/qbatch
>> > qbatch is also available in pypi via pip install qbatch
>> >
>> > —
>> > Gabriel A. Devenyi B.Eng. Ph.D.
>> > Research Computing Associate
>> > Computational Brain Anatomy Laboratory
>> > Cerebral Imaging Center
>> > Douglas Mental Health University Institute
>> > Affiliate, Department of Psychiatry
>> > McGill University
>> > t: 514.761.6131x4781
>> > e: gdevenyi at gmail.com
>> >
>> > _______________________________________________
>> > MINC-users at bic.mni.mcgill.ca
>> > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>> _______________________________________________
>> MINC-users at bic.mni.mcgill.ca
>> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>
>