Containers WorkShop 2021
Various members of the CEMAC team attended the ARCHER 2 containers workshop in July 2021.
Here’s a quick start guide from that tutorial
Set Up
Installation of docker and sigularity are out lined here
Day 1 - Containers
The course outline is found here
Why use containers? - main point is reproducibility with software i.e.
time goes by and dependencies break
missing bits of documentation
systems no longer exist with software stack
cloud system no longer have access to
Docker command line
To not have to run as sudo look here
Once docker is installed
Start the docker engine
dockerd
check commands work
docker --version
docker container ls
docker container --help
docker container ls --help
docker build --help
hello world example
docker container ls
docker pull hello-world
docker container ls
docker run hello-world
Running a linux container
You don’t have to bother with pull command docker run will do this if the container isn’t found locally e.g.
docker run alpine
we can get the container to print its os info or print hello world
docker run alpine cat /etc/os-release
docker run alpine echo 'Hello World'
note the speed of this compared to a VM
we can also run this interactively
docker run -it alpine sh
you can run various linux commands now and and run exit to stop
Finding Containers on docker hub
Owned by docker team:
# docker pull CONTAINER:TAG
docker pull python:3.8
Docker hub community space (check the containers are legit before downloading)
# docker pull OWNER/CONTAINER:TAG
docker pull rocker/tidyverse:3.6.1
Tidying up
Check whats Running
# running
docker ps
# run recently and get list of recent exited containers
docker container ls --all
remove exited container
# single container
docker container rm <container ID>
# tidy all
docker container prune
Now we can remove images (if you’ve not done the steps above you may get an error)
# get image id
docker image ls
docker image rm <image id>
Creating your own container
Create Dockerfile
docker build -t <username>/docker_example Docker_example/
docker will build any Dockerfile it finds in the directory specified
now you can run:
docker image ls
docker run <username>/docker_example echo "Hello World"
Pushing to dockerhub
docker login
docker tag <username>docker_example <username>/docker_test_example:v1
docker push <username>/docker_test_example:v1
Starting points
keep simple software stack
use linux flavours like ubuntu or centos
mounting files
docker run -v <pathtodirhost>:/temp <username>/docker_example python3 <pathtodircontainer>/sum.py
sum.py takes input vars and sums them so you can add numbers to the end of the above script to alter the output
creating a more complex container
This Dockerfile will build the container with the sum function as the default fuction with some default input options
This Dockerfile will build the container with the sum function added to the path but you can also use other functions and commands easily
docker build -t <username>/complex_docker_example:v1 <pathtodirwithdockerfile>
This example COPY local files but you can also get files via wget or git
RUN git clone https://github.com/<username>/<repo>
RUN wget <url>
Archiving and reproducibility
This is mostly explained in the course documentation here
Singularity
Singularity is availble on archer2 and arc4 you might need to module load it in
on archer2
singularity --version
shows its working.
No need to start daemone like with docker. I a test dir
Pull, run and inspect:
singularity pull hello-world.sif shub://vsoch/hello-world
singularity run hello-world.sif
singularity inspect -r hello-world.sif
tidying up caches
rm hello-world.sif
singularity pull hello-world.sif shub://vsoch/hello-world
singularity cache list
singularity cache list -v
singularity cache clean
Run vs exc
exec allows us to run the non default command
singularity exec hello-world.sif /bin/echo Hello World!
singularity exec hello-world.sif /bin/date
interactive shell and files
enter the container with the shell command
singularity shell hello-world.sif
running whoami will display your username rather than root and will show the current directory, often bind mounting to home dir - works for your work directory as well
you won’t be able to access the other files on the system e.g. other users directory
you can create files on the host system with in the container
opening docker images with singularity
you can pull docker images and convert to sif files
singularity pull python-3.9.6.sif docker://python:3.9.6-slim-buster
Making SIF files
You need route access to create SIF files
There are three different options for accessing a suitable environment to undertake the material in this part of the course:
Run Singularity from within a Docker container - this will enable you to have the required privileges to build images
Install Singularity locally on a system where you have administrative access
Use Singularity on a system where it is already pre-installed and you have administrative (root) access
sigularity Installation on a local machine requires lots of dependencies so this tutorial uses a docker singularity contaier and aliases it as singularity
alias singularity='docker run --privileged -v ${PWD}:/home/singularity quay.io/singularity/singularity:v3.5.3-slim'
NB might need sudo in the alias on linux