[MINC-users] Announce - qbatch 1.0 - Execute shell command lines in parallel (serial farm) on SGE/PBS clusters

[Andrew Janke]

Very tasty.

Was it designed as a drop in replacement for qbatch?

   https://github.com/andrewjanke/qbatch/blob/master/qbatch

I note a number of similarities in the C/L arguments. Any plans for
handling dependencies? I see code in there but no arguments to make
use of it? In particular to get around the hoops of tracking
dependencies via job numbers (that my qbatch returns) in PBS and/or
via names in gridengine? ie:

   $ for i in *.mnc; do qbatch --name STEP1-$i -- <blah>; done
   $ qbatch --name STEP2 --depends 'STEP1-*' -- <blah>

I ask as I use qbatch internally in things like volgenmodel in order
to abstract away from the various qsub's. Currently I handle this via
some pretty horrendous perl code here:

   https://github.com/andrewjanke/volgenmodel/blob/master/volgenmodel#L751

It'd be nice to be able to do away with it via your version!

I also find the notion of being able to define a queue via an ENV var
useful when you have a number of levels of scripts/perl/etc that don't
always pass arguments through.  ie:

   https://github.com/andrewjanke/qbatch/blob/master/qbatch#L69


a

On 10 May 2016 at 03:38, Gabriel A. Devenyi <gdevenyi at gmail.com> wrote:
> We (Gabriel A. Devenyi and Jon Pipitone) would like to announce the 1.0
> release <https://github.com/pipitone/qbatch/releases> of qbatch, a
> command-line tool for easily running a list of commands in parallel (serial
> farming) on a compute cluster. This tool takes the list of commands,
> divides them up into batches of arbitrary size, and then submits each batch
> as a separate job or as part of an array job. qbatch also gives you a
> consistent interface to use to submit commands on PBS and SGE clusters or
> locally, (support for others are planned/in testing, PRs welcome) while
> setting requirements for processors, walltime, memory and job dependencies.
> This tool can be used as a quick interface to spread work out on a cluster,
> or as the glue for for connecting a simple pipeline to a cluster (see
> https://github.com/CobraLab/antsRegistration-MAGeT for a sample
> implementation)
>
> The target audience of qbatch is two-fold: it is immediately available for
> users of PBS or SGE clusters to simplify their job construction, in
> addition, through the use of environment variables, cluster administrations
> can craft a default qbatch deployment which allows new cluster users to
> quickly submit jobs which honours the cluster’s policies.
>
> For more information, check out our github webpage here:
> http://github.com/pipitone/qbatch
> qbatch is also available in pypi via pip install qbatch
>
> —
> Gabriel A. Devenyi B.Eng. Ph.D.
> Research Computing Associate
> Computational Brain Anatomy Laboratory
> Cerebral Imaging Center
> Douglas Mental Health University Institute
> Affiliate, Department of Psychiatry
> McGill University
> t: 514.761.6131x4781
> e: gdevenyi at gmail.com
>
> _______________________________________________
> MINC-users at bic.mni.mcgill.ca
> http://www.bic.mni.mcgill.ca/mailman/listinfo/minc-users