command-line tutorial

Divvy also provides a command-line interface that gives you the same power as the python API. You can use --help to get a list of the command-line options:

divvy --help
version: 0.4.0
usage: divvy [-h] [-V] [--silent] [--verbosity V] [--logdev]
             {write,list,init} ...

divvy - write compute job scripts that can be submitted to any computing
resource

positional arguments:
  {write,list,init}
    write            Write a job script
    list             List available compute packages
    init             Initialize a new divvy config file

optional arguments:
  -h, --help         show this help message and exit
  -V, --version      show program's version number and exit
  --silent           Silence logging. Overrides --verbosity.
  --verbosity V      Set logging level (1-5 or logging module level name)
  --logdev           Expand content of logging message format.

https://divvy.databio.org

The list command

Let's first use divvy list to show us our available computing packages:

divvy list
Divvy config: /home/nsheff/.local/lib/python3.5/site-packages/divvy/default_config/divvy_config.yaml
Available compute packages:

slurm
singularity
singularity_slurm
local
default
docker

The write command

Use divvy write to actually write a new script using a template. To do this, you'll need to provide 3 things: a template (which comes from your compute package), a settings file with variables, and an outfile.

The settings file

The settings argument is where you can pass an existing yaml file with key-value pairs. Here's a simple example:

cat settings.yaml

Now let's take a look at the template we are going to use by activating the slurm package

cat ../divvy/submit_templates/slurm_template.sub

We use divvy to populate that template with our list of variables above, like this:

divvy write -P slurm -S settings.yaml -O test.sub

Now we can take a look at what our sbumission scripts looks like.

cat test.sub

We populated several variables, like {LOGFILE} and {TIME}, from the settings.yaml file. However, the {CODE} and {JOBNAME} variables are still unpopulated, so this submission script is incomplete. To remedy this, we'll use divvy's command-line variable passing: any non-interpreted arguments passed to divvy are assumed to be variables to populate the template. These command-line variables are considered highest priority and so will override any values in the more distant locations. For example:

divvy write -P slurm -S settings.yaml -O test.sub --code run-this-cmd --jobname 12345 --time 6-0-0
cat test.sub

Now we have a complete script, which we can run with sbatch test.sub. Notice also that the time variable uses the one provided on the CLI rather than the one provided in the settings.yaml file, because the CLI has a higher priority.

Variables can come from these 3 sources, in order of increasing priority: 1) compute package (defined in the divvy configuration file and selected with the -P or --package argument); 2) settings.yaml file, passed with -S or --settings; 3) any additional variables passed on the command line.