Multi-tenant Docker with OpenStack Heat

Published by Thomas Herve on29/07/2014| 4 Responses

While Heat exists mostly as an orchestration tool for OpenStack, it is also an interesting system for describing in templates interactions with APIs. There has been a resource to talk to the Docker API in Heat for a few months now [1], and we’ve seen some great examples of how to to use it. Most of them expect a Docker deployed on your Heat node, with all your users talking to it. With collegues, we thought about how you could use Nova servers with Docker installed and talk to the remote API [2]. This way we get a tenant-specific Docker instance on which we have full control.

While exploring those capabilities, I discovered that Docker introduced several exciting features that would make the template description much nicer. I wrote a patch to be able to use them [3], the following example relies on it, the good news being that it got recently merged in Heat master branch. You also need to enable the Docker resource in your heat deployment [4].

The main issue to solve when trying to deploy such a template is to make sure that your Docker service is ready before starting to create the containers in it. The new supported way to do this is to use software deployments resources. You need tools on your base p_w_picpath to talk to Heat, as used by TripleO. Building such an p_w_picpath is described in the heat-templates repository [5].

We’ll build the following example, creating our Docker server and 2 containers inside for having a WordPress deployment:

Heat and Docker

This is what an example template looks like:

heat_template_version: 2013-05-23 description: >  Heat Docker template using software deployments. parameters:  key_name:    type: string    description : Name of a KeyPair to enable SSH access to the instance    default: heat  instance_type:    type: string    description: Instance type for WordPress server    default: m1.small  p_w_picpath:    type: string    description: >      Name or ID of the p_w_picpath to use for the Docker server.  This needs to be      built with os-collect-config tools from a fedora base p_w_picpath. resources:  docker_sg:    type: OS::Neutron::SecurityGroup    properties:      description: Ping, SSH, Docker      rules:      - protocol: icmp      - protocol: tcp        port_range_min: 22        port_range_max: 22      - protocol: tcp        port_range_min: 80        port_range_max: 80      - protocol: tcp        port_range_min: 2345        port_range_max: 2345  docker_config:    type: OS::Heat::SoftwareConfig    properties:      group: script      config: |        #!/bin/bash -v        setenforce 0        yum -y install docker-io        cp /usr/lib/systemd/system/docker.service /etc/systemd/system/        sed -i -e '/ExecStart/ { s,fd://,tcp://0.0.0.0:2345, }' /etc/systemd/system/docker.service        systemctl start docker.service  docker_deployment:    type: OS::Heat::SoftwareDeployment    properties:      config: {get_resource: docker_config}      server: {get_resource: docker_host}  docker_host:    type: OS::Nova::Server    properties:      p_w_picpath: {get_param: p_w_picpath}      flavor: {get_param: instance_type}      key_name: {get_param: key_name}      security_groups:        - {get_resource: docker_sg}      user_data_format: SOFTWARE_CONFIG  database_password:    type: OS::Heat::RandomString  database:    type: DockerInc::Docker::Container    depends_on: [docker_deployment]    properties:      p_w_picpath: mysql      name: db      docker_endpoint:        str_replace:          template: http://host:2345/          params:            host: {get_attr: [docker_host, networks, private, 0]}      env:        - {str_replace: {template: MYSQL_ROOT_PASSWORD=password,                         params: {password: {get_attr: [database_password, value]}}}}  wordpress:    type: DockerInc::Docker::Container    depends_on: [database]    properties:      p_w_picpath: wordpress      links:        db: mysql      port_bindings:        80/tcp: [{"HostPort": "80"}]      docker_endpoint:        str_replace:          template: http://host:2345/          params:            host: {get_attr: [docker_host, networks, private, 0]} outputs:  url:    description: Public address of the web site    value:      str_replace:        template: http://host/wordpress        params:          host: {get_attr: [docker_host, networks, private, 0]}

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

heat_template_version: 2013-05-23

 

description: >

  Heat Docker template using software deployments.

 

parameters:

 

  key_name:

    type: string

    description : Name of a KeyPair to enable SSH access to the instance

    default: heat

 

  instance_type:

    type: string

    description: Instance type for WordPress server

    default: m1.small

 

  p_w_picpath:

    type: string

    description: >

      Name or ID of the p_w_picpath to use for the Docker server.  This needs to be

      built with os-collect-config tools from a fedora base p_w_picpath.

 

resources:

  docker_sg:

    type: OS::Neutron::SecurityGroup

    properties:

      description: Ping, SSH, Docker

      rules:

      - protocol: icmp

      - protocol: tcp

        port_range_min: 22

        port_range_max: 22

      - protocol: tcp

        port_range_min: 80

        port_range_max: 80

      - protocol: tcp

        port_range_min: 2345

        port_range_max: 2345

 

  docker_config:

    type: OS::Heat::SoftwareConfig

    properties:

      group: script

      config: |

        #!/bin/bash -v

        setenforce 0

        yum -y install docker-io

        cp /usr/lib/systemd/system/docker.service /etc/systemd/system/

        sed -i -e '/ExecStart/ { s,fd://,tcp://0.0.0.0:2345, }' /etc/systemd/system/docker.service

        systemctl start docker.service

 

  docker_deployment:

    type: OS::Heat::SoftwareDeployment

    properties:

      config: {get_resource: docker_config}

      server: {get_resource: docker_host}

 

  docker_host:

    type: OS::Nova::Server

    properties:

      p_w_picpath: {get_param: p_w_picpath}

      flavor: {get_param: instance_type}

      key_name: {get_param: key_name}

      security_groups:

        - {get_resource: docker_sg}

      user_data_format: SOFTWARE_CONFIG

 

  database_password:

    type: OS::Heat::RandomString

 

  database:

    type: DockerInc::Docker::Container

    depends_on: [docker_deployment]

    properties:

      p_w_picpath: mysql

      name: db

      docker_endpoint:

        str_replace:

          template: http://host:2345/

          params:

            host: {get_attr: [docker_host, networks, private, 0]}

      env:

        - {str_replace: {template: MYSQL_ROOT_PASSWORD=password,

                         params: {password: {get_attr: [database_password, value]}}}}

 

  wordpress:

    type: DockerInc::Docker::Container

    depends_on: [database]

    properties:

      p_w_picpath: wordpress

      links:

        db: mysql

      port_bindings:

        80/tcp: [{"HostPort": "80"}]

      docker_endpoint:

        str_replace:

          template: http://host:2345/

          params:

            host: {get_attr: [docker_host, networks, private, 0]}

 

outputs:

  url:

    description: Public address of the web site

    value:

      str_replace:

        template: http://host/wordpress

        params:

          host: {get_attr: [docker_host, networks, private, 0]}

You can deploy the template simply doing calling stack-create:

heat stack-create -f docker_sd.yaml -P p_w_picpath=fedora-software-config -P instance_type=m1.large docker_stack

The first part of the template is about exposing a Nova server with a Docker API endpoint listening for commands. We then create a database container and a WordPress container that we link to it. Using the port_bindings configuration, we expose the container on the Docker host. The URL output gives the private address where the service should be up once the stack is deployed.

Another alternative to software deployments is to simply use wait conditions. While less powerful, it can solve our problem easily; the corresponding template is shown at [6].

Being a first shot at solving that problem, there are several ways it can be improved. The main thing is that while you have a per-tenant Docker endpoint, it’s open without any authentication. It would be nice to use client certicates as shown at [7]. Then, for reproducibility, having an p_w_picpath with Docker installed and configured properly would simplify the template quite a bit.

We haven’t got the final word on how Docker and containers will integrate in OpenStack. There is a driver for Nova [8] actively developed, but it’s unclear (at least to me) if it fits Nova model or not, and it certainly doesn’t expose all features that Docker has to offer. I expect something like a container service to emerge, but in the mean time using Heat gives you some nice capabilities.

[1] http://docs.openstack.org/developer/heat/template_guide/contrib.html#dockerinc-resource

[2] https://docs.docker.com/reference/api/docker_remote_api/

[3] https://review.openstack.org/106120

[4] https://github.com/openstack/heat/blob/master/contrib/docker/docker/README.md

[5] https://github.com/openstack/heat-templates/blob/master/hot/software-config/elements/README.rst

[6] https://gist.github.com/therve/0e1148296c6c9b43cb55

[7] https://docs.docker.com/articles/https/

[8] https://github.com/stackforge/nova-docker