Does Cloud Foundry work with containers? According to this video from the Cloud Foundry Foundation, containers are “part of the platform’s DNA.”
For those just getting started with Cloud Foundry and containers, we’ve put together an overview on different types of container implementations and how they might be used.
Until recently, Warden was the main container implementation used in Cloud Foundry. Docker is another option to easily and efficiently manage containers. That’s why a lot of effort has been put into enabling support for Docker in the Cloud Foundry Diego runtime.
Here, we briefly compare Warden and Docker, as well as explore the internals of Garden—the current container back end in Cloud Foundry.
What is Similar?
Warden and Docker containers have a number of similarities in their internal implementation; for instance:
- Both employ cgroups to isolate usage of resources and namespaces to separate applications running inside containers from each other and from host processes. (These two features are provided by the Linux kernel.)
- Both Warden and Docker use layers combined with a union file system, which organizes them into a single isolated root file system to be used inside a container.
What is Different?
There are also some differences. Warden is part of Cloud Foundry, so it doesn’t need to support many file systems. At the moment, it works with AUFS and OverlayFS. Docker covers more union file systems, including these two, as well as Btrfs, ZFS, VFS, and the devmapper framework.
The main difference between Docker and Warden is in the way container images are organized. Warden is designed to run applications that get all their dependencies from pieces of software called buildpacks. Warden containers usually have only two layers: a read-only layer with an OS root file system (for example, Ubuntu 14.04) and a nonpersistent, read/write layer for the application itself, all its dependencies, and temporary data.
Unlike Warden, Docker is built to run images, which are distributed through Docker Hub. An app must have an image created for it before it can work with Docker. Users can download publicly available images and make their own based on them.
Docker images consist of multiple layers, one for each RUN command. Layers are combined into a single file system, just as they are in Warden (see the diagram below). When a user creates an image based on someone else’s image, Docker reuses the layers. For example, if they have a Jenkins image based on JRE and want to make their own, with a Java application that also needs JRE, Docker will reuse the JRE image and all of its parents.
There are additional differences, which are outlined in the table below which depicts which resources can be isolated and which features are currently available in Warden and Docker containers.
Feature | Warden | Docker |
1. Resource isolation and control |
|
|
2. Dynamic resource management | Warden containers support this feature, but Cloud Foundry doesn’t use it. | Not supported. |
3. Image management | Only whole images can be reused to create new containers. | Layered—allows for reusing separate layers. |
4. Linking containers | No | Yes |
5. Exposing ports | Single port per container (Multiple ports will be available in Garden/Diego.) | Multiple ports per container |
What is Garden?
Garden is the current Cloud Foundry container back end, which became available in Diego—the current Cloud Foundry runtime. Garden is built around the same idea as Warden, but it has been refactored and re-implemented in Go. (Warden was originally written in Ruby.)
So, what’s the difference between Warden and Garden? First of all, Garden is modular. It supports multiple pluggable “back ends”—pieces of software responsible for creating containers. At the moment, three back ends are available: Linux, runC (a container specification from the Open Container Initiative), and Windows. Yes, you can run Windows (.NET) applications with Garden!
Another killer feature of the new Garden Linux back end is the possibility to run Docker images. Garden is able to fetch Docker images from Docker Hub, as well as from your own Docker repository. Generally, availability of Docker on Cloud Foundry is very good news for teams that already employ Docker images in their everyday activities and want to continue using them after they move to Cloud Foundry.
Some Cloud Foundry offerings already use Diego as the default app runtime, such as CenturyLink AppFog, HPE Helion Stackato, and Pivotal Cloud Foundry. According to the public tracker, Diego 1.0—which will finally replace DEA in open source Cloud Foundry—will be out in a matter of weeks.
Garden Internals
Let’s dig inside a Garden container to see how it works. First of all, I’ll create the simplest app—a static file index.html—and push it into Cloud Foundry.
$ cf app static
Showing health and status for app static in org altoros / space dev as admin...
OK
requested state: started
instances: 1/1
usage: 1G x 1 instances
urls: static.mycfdeployment.com
last uploaded: Tue Aug 9 13:25:09 UTC 2016
stack: cflinuxfs2
buildpack: staticfile 1.3.0
state since cpu memory disk details
#0 running 2016-08-09 04:25:48 PM 0.0% 3.5M of 1G 6.6M of 1G
$ cat index.html
hello
$ curl static.mycfdeployment.com
hello
Garden keeps its containers in a “depot” directory, which is usually located at /var/vcap/data/garden/depot in a typical Cloud Foundry deployment. Each subdirectory is a Garden container, and the directory name is a container ID.
root@afd496e3-339a-4ab6-87fd-87310fd8cc47:/var/vcap/data/garden/depot# ls -l
total 24
drwxr-xr-x 9 root root 4096 Jul 14 15:55 o9bkat0pmke
drwxr-xr-x 9 root root 4096 Jul 26 04:50 o9bkat0po8s
drwxr-xr-x 9 root root 4096 Jul 26 12:33 o9bkat0poai
drwxr-xr-x 9 root root 4096 Jul 27 09:24 o9bkat0poeg
drwxr-xr-x 9 root root 4096 Aug 9 11:58 o9bkat0pq98
drwxr-xr-x 9 root root 4096 Aug 9 13:25 o9bkat0pq9j
(It may be tricky to find “your” container without knowing its ID. I’ve found it by the directory’s creation time for my sample container).
Let’s see what’s inside:
root@afd496e3-339a-4ab6-87fd-87310fd8cc47:/var/vcap/data/garden/depot/o9bkat0pq9j# ls -l
total 64
drwxr-xr-x 2 root root 4096 Aug 9 13:25 bin
-rw-r--r-- 1 root root 15 Aug 9 13:25 bridge-name
-rwxr-xr-x 1 root root 1601 Aug 9 13:25 destroy.sh
drwxr-xr-x 2 root root 4096 Aug 9 13:25 etc
drwxr-xr-x 2 root root 4096 Aug 9 13:25 jobs
drwxr-xr-x 2 root root 4096 Aug 9 13:25 lib
-rwxr-xr-x 1 root root 1185 Aug 9 13:25 net.sh
-rwxr-xr-x 1 root root 1195 Aug 9 13:25 net_rate.sh
drwxr-xr-x 2 root root 4096 Aug 9 13:52 processes
-rw-r--r-- 1 root root 16 Aug 9 13:25 rootfs-provider
drwxr-xr-x 2 root root 4096 Aug 9 13:25 run
-rwxr-xr-x 1 root root 3820 Aug 9 13:25 setup.sh
-rwxr-xr-x 1 root root 484 Aug 9 13:25 start.sh
-rwxr-xr-x 1 root root 1195 Aug 9 13:25 stop.sh
drwxr-xr-x 2 root root 4096 Aug 9 13:25 tmp
-rw-r--r-- 1 root root 5 Aug 9 13:25 version
A Shell Inside Garden Containers
One of the most interesting parts here is a bin directory and the wsh utility, which allows you to run a shell inside a container.
root@afd496e3-339a-4ab6-87fd-87310fd8cc47:/var/vcap/data/garden/depot/o9bkat0pq9j# ./bin/wsh
Let’s check out the processes running in this container:
# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 8312 5672 ? S<l 13:25 0:00 initd -dropCapabilities=false -title="wshd: o9bkat0pq9j"
vcap 11 0.0 0.0 22528 4340 ? S< 13:25 0:00 nginx: master process /home/vcap/app/nginx/sbin/nginx -p /home/vcap/app/nginx -c /home/vcap/app/nginx/conf/nginx.conf
vcap 13 0.0 0.0 10360 7424 ? S<l 13:25 0:00 /tmp/lifecycle/diego-sshd -address=0.0.0.0:2222 -hostKey=-----BEGIN RSA PRIVATE KEY----- MIICXAIBAAKBgQCa2V4UlGUjE1x0pSr
vcap 24 0.0 0.0 48688 720 ? S< 13:25 0:00 cat
vcap 25 0.0 0.0 48688 828 ? S< 13:25 0:00 cat
vcap 30 0.0 0.0 22952 2336 ? S< 13:25 0:00 nginx: worker process
root 271 0.0 0.0 4448 688 pts/0 S<s 13:54 0:00 /bin/sh
root 292 0.0 0.0 15572 2056 pts/0 R<+ 13:55 0:00 ps aux
Here, we can see an nginx process, which serves our index.html file.
# curl localhost:8080
hello
Garden File System
Let’s view the container’s file system:
# ls -l /
total 84
lrwxrwxrwx 1 root root 14 Jan 21 2016 app -> /home/vcap/app
drwxr-xr-x 2 root root 4096 Jan 21 2016 bin
drwxr-xr-x 2 root root 4096 Apr 10 2014 boot
drwxr-xr-x 6 root root 4096 Aug 9 13:25 dev
drwxr-xr-x 78 root root 4096 Aug 9 13:25 etc
drwxr-xr-x 4 root root 4096 Aug 9 13:25 home
drwxr-xr-x 13 root root 4096 Jan 21 2016 lib
drwxr-xr-x 2 root root 4096 Jan 19 2016 lib64
drwx------ 2 root root 16384 Aug 9 13:25 lost+found
drwxr-xr-x 2 root root 4096 Jan 19 2016 media
drwxr-xr-x 2 root root 4096 Apr 10 2014 mnt
drwxr-xr-x 2 root root 4096 Jan 19 2016 opt
dr-xr-xr-x 216 root root 0 Aug 9 13:25 proc
drwx------ 2 root root 4096 Jan 21 2016 root
drwxr-xr-x 7 root root 4096 Jan 19 2016 run
drwxr-xr-x 2 root root 4096 Jan 21 2016 sbin
drwxr-xr-x 2 root root 4096 Jan 19 2016 srv
dr-xr-xr-x 13 root root 0 Aug 9 13:25 sys
drwxrwxrwt 4 root root 4096 Aug 9 13:25 tmp
drwxr-xr-x 10 root root 4096 Jan 21 2016 usr
drwxr-xr-x 12 root root 4096 Jan 21 2016 var
It looks just like a regular Linux file system, with an /app directory, which contains application bits:
# ls /app
Staticfile boot.sh index.html nginx public sources.yml
# cat /app/index.html
hello
Now, let’s leave a container shell:
# exit
root@afd496e3-339a-4ab6-87fd-87310fd8cc47:/var/vcap/data/garden/depot/o9bkat0pq9j#
Another interesting thing here is a container config located at etc/config.
root@afd496e3-339a-4ab6-87fd-87310fd8cc47:/var/vcap/data/garden/depot/o9bkat0pq9j# cat etc/config
id=o9bkat0pq9j
network_host_ip=10.254.0.21
network_host_iface=wo9bkat0pq9j-0
network_container_ip=10.254.0.22
network_container_iface=wo9bkat0pq9j-1
bridge_iface=wb-o9bkat0or7j0
network_cidr_suffix=30
container_iface_mtu=1500
network_cidr=10.254.0.20/30
root_uid=0
rootfs_path=/var/vcap/data/garden/aufs_graph/aufs/mnt/70d4c8fac017a4a3d7cdb4013f3662a379a7d56e21877206c8d2a77611853859
external_ip=192.168.111.77
You can see the rootfs_path parameter, which points to the path on a host file system. It is basically a container’s root file system:
root@afd496e3-339a-4ab6-87fd-87310fd8cc47:/var/vcap/data/garden/depot/o9bkat0pq9j# ls /var/vcap/data/garden/aufs_graph/aufs/mnt/70d4c8fac017a4a3d7cdb4013f3662a379a7d56e21877206c8d2a77611853859
app bin boot dev etc home lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var
To check it, let’s add a file from the host:
touch /var/vcap/data/garden/aufs_graph/aufs/mnt/70d4c8fac017a4a3d7cdb4013f3662a379a7d56e21877206c8d2a77611853859/created-from-host
and then load a container shell again to check whether the file is available:
root@afd496e3-339a-4ab6-87fd-87310fd8cc47:/var/vcap/data/garden/depot/o9bkat0pq9j# ./bin/wsh
# ls /
app bin boot created-from-host dev etc home lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var
Networking in Garden
Networks in containers are isolated from a host with the Linux network namespace. However, Garden provides a way to exchange traffic between a container and a host, as well as access container network services outside the host.
To exchange packets between the host and container, Garden creates a dedicated network in the /30 range for each container (which can hold only two IP addresses: one for the host and another for the container) with a pair of network interfaces. Dig into the etc/configfile inside the container directory to see the network settings:
root@afd496e3-339a-4ab6-87fd-87310fd8cc47:/var/vcap/data/garden/depot/o9bkat0pq9j# cat etc/config
network_host_ip=10.254.0.21
network_host_iface=wo9bkat0pq9j-0
network_container_ip=10.254.0.22
network_container_iface=wo9bkat0pq9j-1
bridge_iface=wb-o9bkat0or7j0
network_cidr_suffix=30
container_iface_mtu=1500
network_cidr=10.254.0.20/30
…
To make sure the network is working, let’s access nginx in the container from the host machine:
root@afd496e3-339a-4ab6-87fd-87310fd8cc47:/var/vcap/data/garden/depot/o9bkat0pq9j# curl 10.254.0.22:8080
hello
To provide network access to a container, Garden uses “network address translation” (NAT) based on a port. Garden randomly picks an unused port—60236 in my example—and then adds a rule to the routing table using iptables, which says, “forward everything that came on the port 60236 to 10.254.0.22:8080.”
The following diagram demonstrates the container network and the packet flow.
Garden appears very promising. Now, we’re just waiting for the Diego 1.0 release, which will bring even more power to container orchestration on Cloud Foundry.
About the Author
Maksim Zhylinski is a Cloud Foundry Engineer at Altoros. He is an expert in cloud computing, networking and Cloud Foundry BOSH, having worked on multiple BOSH CPIs, releases and service brokers. Maksim has 6+ years of experience in all things Ruby, JavaScript and Go, as well as extensive skills in server and client-side web app development. He is an active member of Ruby and Go communities and a frequent contributor to various open-source projects.