[MINC-users] Announce - qbatch 1.0 - Execute shell command lines in parallel (serial farm) on SGE/PBS clusters

[Gabriel A. Devenyi]

Simon/Andrew, indeed this was written as an intended replacement for
sge_batch and/or the original qbatch (we at CoBrALab also had an earlier
internal qbatch which was not Andrew's version).
Having said that, we didn't ensure that it was a one-to-one drop in
replacement, so use may require some modification.

Andrew:
Yes, dependencies in PBS/SGE (and LSF, but I haven't tested LSF at all yet)
are already implemented, just missing from the README, will add :) My
antsRegistration-MaGeT pipeline makes extensive use of them already.

For PBS we parse the XML joblist and do pattern matching on names:
https://github.com/pipitone/qbatch/blob/master/bin/qbatch#L138-L171
And for SGE we use jobname pattern matching built into SGE:
https://github.com/pipitone/qbatch/blob/master/bin/qbatch#L381-L382

As for the queue, I have an in progress branch to specify "other" batch
options, handled by environment variable, see
https://github.com/pipitone/qbatch/pull/80
I just need to solve the append vs. replace issue I'm having with argparse.
Thoughts here welcome :)

For bugs/feature requests/modifications, please feel free to request! Our
goal is to provide a maintained tool to support existing workflows and
enable new ones.

--
Gabriel A. Devenyi B.Eng. Ph.D.
Research Computing Associate
Computational Brain Anatomy Laboratory
Cerebral Imaging Center
Douglas Mental Health University Institute
Affiliate, Department of Psychiatry
McGill University
t: 514.761.6131x4781
e: gdevenyi at gmail.com

On Tue, May 10, 2016 at 9:07 AM, Andrew Janke <a.janke at gmail.com> wrote:

> Very tasty.
>
> Was it designed as a drop in replacement for qbatch?
>
>    https://github.com/andrewjanke/qbatch/blob/master/qbatch
>
> I note a number of similarities in the C/L arguments. Any plans for
> handling dependencies? I see code in there but no arguments to make
> use of it? In particular to get around the hoops of tracking
> dependencies via job numbers (that my qbatch returns) in PBS and/or
> via names in gridengine? ie:
>
>    $ for i in *.mnc; do qbatch --name STEP1-$i -- <blah>; done
>    $ qbatch --name STEP2 --depends 'STEP1-*' -- <blah>
>
> I ask as I use qbatch internally in things like volgenmodel in order
> to abstract away from the various qsub's. Currently I handle this via
> some pretty horrendous perl code here:
>
>    https://github.com/andrewjanke/volgenmodel/blob/master/volgenmodel#L751
>
> It'd be nice to be able to do away with it via your version!
>
> I also find the notion of being able to define a queue via an ENV var
> useful when you have a number of levels of scripts/perl/etc that don't
> always pass arguments through.  ie:
>
>    https://github.com/andrewjanke/qbatch/blob/master/qbatch#L69
>
>
> a
>
> On 10 May 2016 at 03:38, Gabriel A. Devenyi <gdevenyi at gmail.com> wrote:
> > We (Gabriel A. Devenyi and Jon Pipitone) would like to announce the 1.0
> > release <https://github.com/pipitone/qbatch/releases> of qbatch, a
> > command-line tool for easily running a list of commands in parallel
> (serial
> > farming) on a compute cluster. This tool takes the list of commands,
> > divides them up into batches of arbitrary size, and then submits each
> batch
> > as a separate job or as part of an array job. qbatch also gives you a
> > consistent interface to use to submit commands on PBS and SGE clusters or
> > locally, (support for others are planned/in testing, PRs welcome) while
> > setting requirements for processors, walltime, memory and job
> dependencies.
> > This tool can be used as a quick interface to spread work out on a
> cluster,
> > or as the glue for for connecting a simple pipeline to a cluster (see
> > https://github.com/CobraLab/antsRegistration-MAGeT for a sample
> > implementation)
> >
> > The target audience of qbatch is two-fold: it is immediately available
> for
> > users of PBS or SGE clusters to simplify their job construction, in
> > addition, through the use of environment variables, cluster
> administrations
> > can craft a default qbatch deployment which allows new cluster users to
> > quickly submit jobs which honours the cluster’s policies.
> >
> > For more information, check out our github webpage here:
> > http://github.com/pipitone/qbatch
> > qbatch is also available in pypi via pip install qbatch
> >
> > —
> > Gabriel A. Devenyi B.Eng. Ph.D.
> > Research Computing Associate
> > Computational Brain Anatomy Laboratory
> > Cerebral Imaging Center
> > Douglas Mental Health University Institute
> > Affiliate, Department of Psychiatry
> > McGill University
> > t: 514.761.6131x4781
> > e: gdevenyi at gmail.com
> >
> > _______________________________________________
> > MINC-users at bic.mni.mcgill.ca
> > http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
> _______________________________________________
> MINC-users at bic.mni.mcgill.ca
> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users
>