Configuring containers with divvy
The divvy template framework is a natural way to run commands in a container, for example, using docker
or singularity
. All we need to do is 1) design a template that will run the job in the container, instead of natively; and 2) create a new compute package that will use that template.
A template for container runs
If you start up divvy without giving it a DIVCFG file, it will come with a few default compute packages that include templates for containers. You can also find these in the divcfg repository, which includes these scenarios:
- singularity on SLURM
- singularity on localhost
- docker on localhost
- others
If you need a different system, looking at those examples should get you started toward making your own. To take a quick example, using singularity on SLURM combines the basic SLURM script template with these lines to execute the run in container:
singularity instance.start {SINGULARITY_ARGS} {SINGULARITY_IMAGE} {JOBNAME}_image
srun singularity exec instance://{JOBNAME}_image {CODE}
singularity instance.stop {JOBNAME}_image
This particular template uses some variables provided by different sources: {JOBNAME}
, {CODE}
, {SINGULARITY_ARGS}
and {SINGULARITY_IMAGE}
. These arguments could be defined at different places. For example, the {SINGULARITY_IMAGE}
variable should point to a singularity image that could vary by pipeline, so it makes most sense to define this variable individually for each pipeline. So, any pipeline that provides a container should probably include a singularity_image
attribute providing a place to point to the appropriate container image.
Of course, you will also need to make sure that you have access to singularity
command from the compute nodes; on some clusters, you may need to add a module load singularity
(or some variation) to enable it.
The {SINGULARITY_ARGS}
variable comes just right after the instance.start
command, and can be used to pass any command-line arguments to singularity. We use these, for example, to bind host disk paths into the container. It is critical that you explicitly bind any file systems with data necessary for the pipeline so the running container can see those files. The singularity documentation explains this, and you can find other arguments detailed there. Because this setting describes something about the computing environment (rather than an individual pipeline or sample), it makes most sense to put it in the DIVCFG
file for a particular compute package. The next section includes examples of how to use singularity_args
.
If you're using looper, the {JOBNAME}
and {CODE}
variables will be provided automatically by looper.
Adding compute packages for container templates
To add a package for these templates to a DIVCFG
file, we just add a new section. There are a few examples in this repository. A singularity example we use at UVA looks like this:
singularity_slurm:
submission_template: templates/slurm_singularity_template.sub
submission_command: sbatch
singularity_args: --bind /sfs/lustre:/sfs/lustre,/nm/t1:/nm/t1
singularity_local:
submission_template: templates/localhost_singularity_template.sub
submission_command: sh
singularity_args: --bind /ext:/ext
These singularity compute packages look just like the typical ones, but just change the submission_template
to point to the new containerized templates described in the previous section, and then they add the singularity_args
variable, which is what will populate the {SINGULARITY_ARGS}
variable in the template. Here we've used these to bind (mount) particular file systems the container will need. You can use these to pass along any environment-specific settings to your singularity container.
With this setup, if you want to run a singularity container, just specify --compute singularity_slurm
or --compute singularity_local
and it will use the appropriate template.
For another example, take a look at the basic localhost_container.yaml
DIVCFG file, which describes a possible setup for running docker on a local computer:
compute:
default:
submission_template: templates/localhost_template.sub
submission_command: sh
singularity:
submission_template: templates/localhost_singularity_template.sub
submission_command: sh
singularity_args: --bind /ext:/ext
docker:
submission_template: templates/localhost_docker_template.sub
submission_command: sh
docker_args: |
--user=$(id -u) \
--env="DISPLAY" \
--volume ${HOME}:${HOME} \
--volume="/etc/group:/etc/group:ro" \
--volume="/etc/passwd:/etc/passwd:ro" \
--volume="/etc/shadow:/etc/shadow:ro" \
--volume="/etc/sudoers.d:/etc/sudoers.d:ro" \
--volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \
--workdir="`pwd`" \
Notice the --volume
arguments, which mount disk volumes from the host into the container. This should work out of the box for most docker users.