[MINC-users] Announce - qbatch 1.0 - Execute shell command lines in parallel (serial farm) on SGE/PBS clusters

[Gabriel A. Devenyi]

We (Gabriel A. Devenyi and Jon Pipitone) would like to announce the 1.0
release <https://github.com/pipitone/qbatch/releases> of qbatch, a
command-line tool for easily running a list of commands in parallel (serial
farming) on a compute cluster. This tool takes the list of commands,
divides them up into batches of arbitrary size, and then submits each batch
as a separate job or as part of an array job. qbatch also gives you a
consistent interface to use to submit commands on PBS and SGE clusters or
locally, (support for others are planned/in testing, PRs welcome) while
setting requirements for processors, walltime, memory and job dependencies.
This tool can be used as a quick interface to spread work out on a cluster,
or as the glue for for connecting a simple pipeline to a cluster (see
https://github.com/CobraLab/antsRegistration-MAGeT for a sample
implementation)

The target audience of qbatch is two-fold: it is immediately available for
users of PBS or SGE clusters to simplify their job construction, in
addition, through the use of environment variables, cluster administrations
can craft a default qbatch deployment which allows new cluster users to
quickly submit jobs which honours the cluster’s policies.

For more information, check out our github webpage here:
http://github.com/pipitone/qbatch
qbatch is also available in pypi via pip install qbatch

—
Gabriel A. Devenyi B.Eng. Ph.D.
Research Computing Associate
Computational Brain Anatomy Laboratory
Cerebral Imaging Center
Douglas Mental Health University Institute
Affiliate, Department of Psychiatry
McGill University
t: 514.761.6131x4781
e: gdevenyi at gmail.com
​