Heat Software Config

This article describes Software Configuration Hooks new in Icehouse.

All the following topics depend on https://github.com/openstack/heat-templates/tree/master/hot/software-config.

Pre-requisites

Images with cloud-init,  heat-config, os-collect-config, os-refresh-config, os-apply-config, heat-config-cfn-init, heat-config-puppet, heat-config-salt and heat-config-script  packages installed
Nova metadata server up and running
Heat CFN API service up and running

How hook works

1. When the instance boots up, it runs the cloud-init script. Cloud-init creates the directories under /var/lib/cloud:

root@web-server:/var/lib/cloud# ls
data  handlers  instance  instances  scripts  seed  sem

2. Download the data from the metadata server

root@web-server:/var/lib/cloud/instance# ll
-rw-r--r-- 1 root root    52 Aug  7 08:21 boot-finished
drwxr-xr-x 2 root root  4096 Aug  7 08:16 boothooks/
-rw------- 1 root root   128 Aug  7 08:16 cloud-config.txt
-rw-r--r-- 1 root root    29 Aug  7 08:16 datasource
drwxr-xr-x 2 root root  4096 Aug  7 08:16 handlers/
-r-------- 1 root root 10500 Aug  7 08:16 obj.pkl
drwxr-xr-x 2 root root  4096 Aug  7 08:16 scripts/
drwxr-xr-x 2 root root  4096 Aug  7 08:21 sem/
-rw------- 1 root root  3894 Aug  7 08:16 user-data.txt
-rw------- 1 root root  3954 Aug  7 08:16 user-data.txt.i
-rw------- 1 root root     4 Aug  7 08:16 vendor-data.txt
-rw------- 1 root root   345 Aug  7 08:16 vendor-data.txt.i

user-data.txt vs user-data.txt.i: Thecontent of user-data.txt is raw data from http://169.254.169.254/latest/user-data. It maybe contains gzip compressed user data. But user-data.txt.i contains the post-processed user data (uncompressed, with includes resolved).

3. Splits the multiple parts into the /var/lib/cloud/instance directory

3.1 Content-Type: text/part-handler & Content-Type: text/x-cfninitdata

Then content of this part-handers will be written to /var/lib/cloud/instance/handlers. The “part” handled here is subpart of user data. 

Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="part-handler.py"

#part-handler

def list_types():
    return(["text/x-cfninitdata"])

def handle_part(data, ctype, filename, payload):
    if ctype == "__begin__":
        try:
            os.makedirs('/var/lib/heat-cfntools', 0o700)
        except OSError as e:
            if e.errno != errno.EEXIST:
                raise
        return

    if ctype == "__end__":
        return

    with open('/var/log/part-handler.log', 'a') as log:
        timestamp = datetime.datetime.now()
        log.write('%s filename:%s, ctype:%s\n' % (timestamp, filename, ctype))

    if ctype == 'text/x-cfninitdata':
        with open('/var/lib/heat-cfntools/%s' % filename, 'w') as f:
            f.write(payload)

        # TODO(sdake) hopefully temporary until users move to heat-cfntools-1.3
        with open('/var/lib/cloud/data/%s' % filename, 'w') as f:
            f.write(payload)

For example, the following code process "text/x-cfninitdata" sections by writing them to /var/lib/heat-cfntools/ and /var/lib/cloud/data/. The payload is like this:

{"deployments": [], "os-collect-config": {"cfn": {"stack_name": "stackname", "metadata_url": "http://heat-server:8000/v1/", "path": "web_server.Metadata", "secret_access_key": "ba455502219c41a58757a62c5425d7ce", "access_key_id": "8776148cc65a4f95b3c4a0b390f87625"}}}

3.2  /var/lib/heat-cfntools

When part-handler executes , the directory looks like:

root@web-server:/var/lib/heat-cfntools# ls
cfn-init-data  cfn-watch-server

4. Execute user's script

4.1 The first loop of os-collect-config

os-collect-config is a daemon process. It collects data from defined configuration sources and runs a defined hook whenever the metadata has been changed.

root@web-server:~# ps aux  | grep os-collect-config
root      1229  2.5  1.2  99192 25816 ?        Ss   Aug07  33:39 /opt/stack/venvs/os-collect-config/bin/python /usr/local/bin/os-collect-config

Firstly, the default configuration of /etc/ os-collect-config.confjust contains default command options, like this:

[DEFAULT]
command = os-refresh-config

Afterone loop of os-collect-config execution, the directory of/var/lib/os-collect-config and /var/run/os-collect-config are like this:

root@web-server:/var/run/os-collect-config# ll
-rw-------  1 root root 111667 Aug  7 08:21 cfn.json
-rw-------  1 root root 111667 Aug  7 08:21 cfn.json.orig
-rw-------  1 root root   1061 Aug  7 08:21 ec2.json
-rw-------  1 root root   1061 Aug  7 08:20 ec2.json.last
-rw-------  1 root root   1061 Aug  7 08:19 ec2.json.orig
-rw-------  1 root root    300 Aug  7 08:21 heat_local.json
-rw-------  1 root root    300 Aug  7 08:20 heat_local.json.last
-rw-------  1 root root    300 Aug  7 08:19 heat_local.json.orig
-rw-------  1 root root    124 Aug  7 08:21 os_config_files.json

The content of /etc/heat_local.conf sources from/var/lib/heat-cfntools/cfn-init-data by heat_local::Collector. And ec2.json sources from metadata service by ec2::Collector.

But now, there are no cfn.json and cfn.json.orig because/etc/os-collect-config.conf is not configured fully yet.

Secondly, when os-refresh-config is called, the scripts of PHASES('pre-configure', 'configure',  'post-configure','migration') under /opt/stack/os-config-refresh will be executed. “os-apply-config”is among these scripts.

 

Thirdly, os-apply-config will create/etc/os-collect-collect.conf and /var/run/heat-config/heat-config by rendering templates predefined under /usr/libexec/os-apply-config/templates. But now/var/run/heat-config/heat-config is still empty.

root@web-server:~# cat /etc/os-collect-config.conf 
[DEFAULT]
command = os-refresh-config

[cfn]
metadata_url = http://heat-server:8000/v1/
stack_name = stack43
secret_access_key = ba455502219c41a58757a62c5425d7ce
access_key_id = 8776148cc65a4f95b3c4a0b390f87625
path = web_server.Metadata

4.2  The second loop of os-collect-config

Now, cfn.json and cfn.json.orig will be created by cfn::Collector in “os-collect-config” because now /etc/os-collect-config is configured correctly.

root@web-server:/var/run/os-collect-config# ll
-rw-------  1 root root 111667 Aug  7 08:21 cfn.json
-rw-------  1 root root 111667 Aug  7 08:21 cfn.json.orig

And /var/run/heat-config/heat-config will be filled with meaningful contents by “os-apply-config”.

root@web-server:/var/run/heat-config# ll
-rw-r--r--  1 root root 110761 Aug  8 09:09 heat-config

Now /opt/stack/os-config-refresh/configure.d/55-heat-config run, not only it will invoke proper hook under /var/lib/heat-config/hooks, but also it will signal heat with output.

Troubleshooting

/var/log/cloud-init.log
/var/log/cloud-init-output.log
/var/log/part-handler.log
/var/log/upstart/os-collect-config.log
/var/log/os-apply-config.log

References

https://help.ubuntu.com/community/CloudInit

http://cloudinit.readthedocs.org/en/latest/topics/examples.html

http://foss-boss.blogspot.com/2011/01/advanced-cloud-init-custom-handlers.html

https://github.com/openstack/heat-templates/tree/master/hot/software-config

Appendix

1. A Limit of Cloud-init

The first limit is osapi_max_request_body_size of nova-api service, and its default value is 114688. Define it in nova/api/sizelimit.py as following:

#default request size is 112k
max_request_body_size_opt = cfg.IntOpt('osapi_max_request_body_size',
                                       default=114688,
                                       help='The maximum body size '
                                           'per each osapi request(bytes)')

If your request body size is larger than osapi_max_request_body_size, it will complain like this “OverLimit: Over limit (HTTP 413)”.

The second limit is MAX_USERDATA_SIZE, and its default value is 65535. Define it in nova/compute/api.py:

MAX_USERDATA_SIZE = 65535

If your user-data size is larger than MAX_USERDATA_SIZE, it will complain directly like this “Instance User Data Too Large: User data too large. User data must be no larger than 65535 bytes once base64 encoded.”

2. One Template Sample

heat_template_version: 2013-05-23

description: Web Server Template

parameters:
  image:
    default: CentOS
    type: string
    description: Image use to boot a server
  
  flavor:
    default: m1.small
    type: string
    description: Flavor use to boot a server
    
  key_name:
    default: demo-key 
    type: string
    description: keypair name use to boot a server

resources:
  web_config:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config:
        get_file: scripts/web.sh

  web_deployment:
    type: OS::Heat::SoftwareDeployment
    properties:
      config:
        get_resource: web_config
      server:
        get_resource: web_server
        
  web_server:
    type: OS::Nova::Server
    properties:
      name: "Web Server"
      image: { get_param: image }
      flavor: { get_param: flavor } 
      key_name: { get_param: key_name } 
      user_data_format: SOFTWARE_CONFIG 

outputs:
  stdout:
    value:
      get_attr: [web_deployment, deploy_stdout]
  stderr:
    value:
      get_attr: [web_deployment, deploy_stderr]
  status_code:
    value:
      get_attr: [web_deployment, deploy_status_code]