Creating a VO-Box container

This guide describes how to create a networked Docker container for VO-Box use.

Update 10/01/20:

Please be aware that the instructions listed below are now outdated. A privileged CentOS 6 container could previously be set up the same way as a "traditional" VOBOX, as long as it was on a MACVLAN bridge. Due to the introduction of systemd in CentOS 7, the setup procedure of VOBOXes is now different for containers.

To simplify setup, the following Dockerfiles can instead be used to quickly deploy preconfigured container VOBOXes:

https://gitlab.cern.ch/mstoretv/dockervobox (HTCondor)

https://gitlab.cern.ch/mstoretv/dockervobox_arc (ARC/Generic)

 

Prerequisites:

 

Docker v. 1.12 or later (Tested on 17.03.1-CE a.k.a "1.14”).

CentOS 7.0 or later

CVMFS installed on the host

 

The commands listed below must be executed as root!

 

1. Set up networking

 

1.1 Create a new MACVLAN bridge named “docknet”:

 

$ docker network create -d macvlan --subnet=137.138.47.192/26 --gateway=137.138.47.193 --ipv6 --subnet 2001:1458:201:b50e::/64 --gateway 2001:1458:201:b50e::1 -o parent=eth0 docknet

 

Notes:

  • Subnet/gateway must be replaced with the settings applicable to your network, while parent must be the network interface. 
  • Why MACVLAN? Docker’s normal approach to bridging is simply a NAT translation scheme. MACVLAN bridges give the containers direct access to the network.
  • Containers in this bridge will be assigned their own MAC addresses, and appear as conventional computers on the local network
  • However, due to the way MACVLAN works, the host will not be able to ping the containers (and vice versa).

 

2. Creating the container

 

2.1 Download the latest CentOS image from the Docker Hub:

$ docker pull centos:latest

[For CentOS 6.9, replace “centos:latest” with “centos:centos6.9”] 

Warning: Be sure to change the default root password! This is simply "root" in the above image.

 

2.2 Create and launch a new CentOS container connected to the MACVLAN bridge:

$ docker run -it -v /cvmfs:/cvmfs -h myalienvo.cern.ch --name=myvocontainer --net=docknet --ip=137.138.47.251 --ip6=2001:1458:201:b50e::100:3e centos /bin/bash

 

Notes:

-v mounts the directory /cvmfs from the host at /cvmfs within the container

-h sets the hostname myalienvo.cern.ch

--name sets the container name (as seen from Docker) to myvocontainer

--net attaches the container to the previously created MACVLAN bridge docknet

--ip/ip6 sets the desired network IPs for the container.

 

Note that Docker gives containers limited access to the host kernel, which may prevent some tools from working properly. A possible workaround is to give the container additional privileges. See "additional notes" at the end of this page for more info.

As opposed to mounting CVMFS from the host, it is also possible to have it installed normally within the container. However, this will only work on a container with full access privileges. Be aware that unless otherwise configured, this will give each container its own CVMFS cache.

 

3. Using the container:

 

3.1 Exit the running container:

root@ myalienvo:~$ exit

[Note: Docker will stop the container once this bash session is closed.]

 

3.2 Start the container:

$ docker start myvocontainer

[This way, the container will remain active until you type “docker stop myvocontainer”.]

 

3.3 Start a bash session within the container:

$ docker exec -it myvocontainer /bin/bash

 

From this point onwards, you can proceed with configuring the VO-Box as you normally would on a VM, e.g. as a WLCG VOBOX. Once again, note that depending on the level of access given to the container, some tools may not function, and using "service start" may produce errors. For more information, see "additional notes" at the end of this page.

 

Useful commands:

 

List active containers:

$ docker ps

 

List all containers:

$ docker ps -a

 

List all images:

$ docker images

 

Save the container state as an image:

$ docker commit myvocontainer myvoimagename

[With “myvoimagename” being the desired image name]

 

Save the container as a tar file

$ docker export mycontainer > /home/myexportedcontainer.tar

[With “/home/myexportedcontainer.tar” being the directory and name for the exported container]

 

Docker commit vs export:

The "commit" command will save the container as an image in Docker's internal format. It will also preserve the container's history, layers and metadata, and also create a new layer on top. Beware that an image will rapidly grow in size if it is continuously being committed, as a result of all the stacked layers.

 

The "export" command will save the container to a tar file, that can be reimported and used to launch new containers on the same machine, or uploaded and/or transferred elsewhere. However, it does not store any history or metadata, and will delete all previous layers except the current. Exporting and reimporting an image can thus be used to flatten a container that has gained a large filesize, e.g as a result of having previously used "commit" several times.

 

Setting DNS:

 

When a container is restarted, Docker will overwrite its DNS configuration with a default one. To avoid having to reapply the DNS settings for every container restart, this can be changed either by adding --dns=YOUR_DNS_IP to the run command in 2.2, or by changing the default DNS used by Docker. This can be done by creating the file /etc/docker/daemon.json with the following contents:

{
    "dns": ["YOUR_DNS_IP1", "YOUR_DNS_IP2"]
}

Be aware that Docker must be restarted for the changes to take effect.

 

Automatically starting services at container launch:

 

Unlike a full OS in a virtual machine, Docker containers have no init system that can be used to automatically start services. One possible way to achieve this at container launch, is by using a Dockerfile.

In essence, a Dockerfile contains a set of commands and arguments that will be performed on a given image. This file can thus contain a full list of commands to execute at launch, or simply point to a bash script, like the following Dockerfile:

ROM mycontainerimage
ENTRYPOINT /etc/init.sh && /bin/bash

Here, "mycontainerimage" is the image to be used, and "init.sh" is a bash script to run at launch.

By running "docker build -t mynewcontainerimage ." Docker will in this case produce a new image "mynewcontainerimage" based on "mycontainerimage", where the contents of /etc/init.sh will be run at start. Note that the dockerfile must be named Dockerfile, and the previous build command executed in the same directory. The contents of Dockerfile will run independently of a container's access privileges, thus the /etc/init.sh file can contain commands such as "service start", despite having no additional privileges.

Alternatively, all commands can be given within the Dockerfile, without having to reference a separate bash script. However, this will require the image to be rebuilt every time changes are added to the Dockerfile.

 

Additional notes

 

For the sake of isolation, access to the kernel is restricted within the containers. To enable some access for networking tools, such as iptables, add --cap-add=NET_ADMIN when creating the container (step 2.2). More access can be given through --cap-add=SYS_ADMIN, and full access can be given with --privileged. Be aware that this will create pathways for potentially breaking container isolation.

 

The maximum container size can be limited by adding --storage-opt size=XXG to step 2.2 (replace "XX" with the desired storage capacity). However, note that Docker will most likely ask you to change its current storage driver to perform this. Be aware that doing this requires deleting all stored containers. Before continuing, back up any containers you want to keep using docker export. 

 

As mentioned earlier, the host will be unable to reach/ping the containers, and vice versa. A workaround is to create an additional ("normal") Docker bridge, and have the host / containers connect to it. 

 

CVMFS may respond with a “too many symbolic links” error when accessed from a new container or after a reboot. This is known autofs bug, and can be avoided by disabling autofs on the host and mounting your CVMFS directory manually. If you prefer to leave autofs enabled, you can remove the error by accessing a directory within CVMFS on the host (e.g /cvmfs/alice.cern.ch), and then restarting the container. Be aware that this error will likely return on the next reboot. 

 

HTCondor may throw an error “Failed DISCARD_SESSION_KEYRING_ON_STARTUP”. This can fixed by setting “DISCARD_SESSION_KEYRING_ON_STARTUP = false” in the Condor config file.

 

Example init.sh

-----------------------------------------------------------------------------
#! /bin/bash

d=`date '+%y%m%d-%H%M%S'`
log=/tmp/init-$d.log

> $log && exec > $log 2>&1 < /dev/null

f=/etc/sysconfig/iptables
[ -r $f ] && iptables-restore < $f

service dnsmasq start

#
# hack: work around PID file corruptions...
#

service rsyslog stop

for i in 1 2
do
    echo == rsyslog $i
    sleep 5
    service rsyslog status || service rsyslog start
    perl -ni -e 'print if $. < 2' /var/run/syslogd.pid
done

svcs='
    crond
    sshd
    gsisshd
    alice-box-proxyrenewal
    condor
    autofs
'

#
# random ones have been found absent...
# we try up to 3 times for now
#

for i in 1 2 3
do
    for svc in $svcs
    do
    echo == $svc $i
    sleep 1
    service $svc status || service $svc start
    done
done

service fetch-crl-boot start > /tmp/fetch-crl-$$.log 2>&1 < /dev/null &

exit 0
-----------------------------------------------------------------------------