Automation for the people: Deployment-automation patterns, Part 1

Patterns for one-click deployme

Software deployment is often treated as a necessary evil that can be cobbled together before going live. But you can and should apply software-engineering principles to deployment as well as to other parts of the development cycle. When manually applied, deployment is a repetitive and error-prone process. Just as you can automate builds to reduce errors and speed software development, you can automate deployment processes to reduce errors and expedite software delivery.

In an earlier Automation for the people installment, "Speed deployment with automation ," you learned a technique for remotely deploying software into multiple target environments. This two-part article takes the automated-deployment discussion to a higher level. Just as there are patterns for software development, there are patterns for deployment, and I've been cataloging them over the past few years. Here in Part 1, I'll introduce eight deployment patterns and provide examples for them:

• Managing all configuration files in a centralized Repository , which makes it possible to generate working software using a Scripted Deployment
  
• Scripted Deployment , which scripts all deployment actions so there's no need for human intervention when a deployment is executed
  
• Single Command , which reduces deployment complexities and ensures a Headless Execution of the deployment process
  
• Tokenize Configuration , which provides a repeatable way of injecting variable information into configuration files
  
• Externalized Configuration , which makes it simple to enter information only once that varies between target environments
  
• Template Verifier , which helps to ensure that all target environment properties are the same
  
• Headless Execution , which provides a secure way of accessing multiple machines in an automated process
  
• Unified Deployment , which promotes a single deployment script that can be run in many target environments

You'll learn about even more deployment patterns in Part 2.

About this series

As developers, we work to automate processes for end-users; yet, many of us overlook opportunities to automate our own development processes. To that end, Automation for the people is a series of articles dedicated to exploring the practical uses of automating software development processes and teaching you when and how to apply automation successfully.

Figure 1 illustrates the relationships among the deployment patterns covered in this article:

Figure 1. Deployment-automation patterns

I'll discuss each pattern in turn. As you read about them, you'll gain an understanding of the interrelationships illustrated in Figure 1.

Committing all files to a version-control repository

Name : Repository

Pattern : All files are committed to version-control repository — in the deployment context, all of the configuration files and tools.

Antipattern : Some teams put this information on a shared drive with controlled access. Others may keep the information only on their machines and then copy it to target environments.

As a general rule, I recommend that development teams check in all of their files necessary to create working software. Sometimes there are exceptions to this rule, but not often. In the context of deployment, some teams erroneously look at servers and server configuration as fixed assets — that they do not change. Although, at times, there are constraints on checking in large binaries, the configuration, database scripts, and all build and deployment scripts should be committed to the version-control repository. Using the Repository pattern greatly assists in the patterns I'll discuss next: Scripted Deployment and, particularly, Single Command.

---

Back to top

Scripting all deployment processes

Name : Scripted Deployment

Pattern : All deployment processes are written in a script.

Antipattern : Some may manually configure tasks such as installing and configuring a Web container. Others may use the GUI-based administration tool provided by the container to modify the container based on a specific environment. Although not likely, it could be simple to modify configuration manually the first time, but it doesn't scale when you perform multiple deployments to many target environments several times a week or more. Moreover, a GUI-based administration tool can be very helpful in deployment — the first time you use it. However, this approach also does not scale and is error-prone, because many people must perform these procedures again and again.

Listing 1 demonstrates the Scripted Deployment pattern by automating the process of (re)starting a Tomcat Web container through a scripted deployment. This process is written using the Apache Ant build-scripting language.

Listing 1. Example of starting a Tomcat Web container

<available file="@{tomcat.home}/server/@{tomcat.server.name}/bin" property="tomcat.bin.exists"/> <if> <isset property="tomcat.bin.exists"/> <then> <echo message="Starting tomcat instance at @{tomcat.home} with start_tomcat" /> <exec executable="@{tomcat.home}/server/@{tomcat.server.name}/bin/start_tomcat" osfamily="unix" /> </then> <else> <echo message="Starting tomcat instance at @{tomcat.home} with startup.sh" /> <exec osfamily="unix" executable="chmod" spawn="true"> <arg value="+x" /> <arg file="@{tomcat.home}/bin/startup.sh" /> <arg file="@{tomcat.home}/bin/shutdown.sh" /> </exec> <exec executable="sh" osfamily="unix" dir="@{tomcat.home}/bin" spawn="true"> <env key="NOPAUSE" value="true" /> <arg line="startup.sh" /> </exec> <exec osfamily="windows" executable="cmd" dir="@{tomcat.home}/bin" spawn="true" > <env key="NOPAUSE" value="true" /> <arg line="/c startup.sh" /> </exec> <sleep seconds="15" /> </else> </if>

 

By scripting this process, you eliminate the need to click through the GUI administration console that Tomcat, in this case, provides. What's more, because it's scripted, it can be run by a headless process as part of a comprehensive automated deployment.

---

Back to top

Getting deployment to run from a single command

Name : Single Command

Pattern : Deployers, or headless processes, can type a single command to generate working software for users.

Antipattern : Some deployment processes require people to enter multiple commands and procedures such as copying files, modifying configuration files, restarting a server, setting passwords, and other repetitive, error-prone actions. If they are lucky, they have access to step-by-step documented procedures for performing these actions. Regardless, requiring humans to perform deployment procedures introduces the risk of errors and causes time bottlenecks that delay the release of software into the multiple target environments.

Development does not equal production

The fact that software works in the development or QA environment does not mean it will work in production, because of the potential differences between the environments. This is why it's important to script all aspects of deployment.

When you write a deployment, your clients are often others on your team, organization, company — or even a machine. The more complex it is to run the deployment, the less likely someone or a headless process can successfully execute it without introducing errors. A simple single-command example for running a deployment is shown in Listing 2:

Listing 2. Single-command deployment using Ant

ant -Dproperties.file=$USERHOME/projects/petstore/properties/dev-install.properties / deploy:remote:install

 

The command in Listing 2 executes an Ant task called deploy:remote:install , passing along an environment-specific .properties file, to deploy the software remotely on other machines. This task performs actions such as securely copying files using the Secure Copy protocol (SCP); securely executing commands on remote machines via Secure Shell (SSH); installing, configuring, and restarting Web containers; and a host of other processes — without human intervention. Obviously, a user can enter the command, but because it's so terse, it can be easily run by a headless process, such as a Continuous Integration or build-management server.

---

Back to top

Injecting variable information into configuration files

Name : Tokenize Configuration

Pattern : Token values are entered into configuration files and then replaced during the Scripted Deployment based on Externalized Configuration properties checked into Repository.

Antipattern : Target-specific data is entered into configuration files in each environment.

Listing 3 is an XML file to manage the configuration between a Web container and a database server. In this file, I've placed tokens using the @ signs. During the automated deployment process, a script will replace these tokens with the actual values from the Externalized Configuration files.

Listing 3. Tokenized Web-container configuration file

<datasources> <local-tx-datasource> <jndi-name>@application.context.name@</jndi-name> <use-java-context>false</use-java-context> <connection-url>@database.url@</connection-url> <user-name>@database.user@</user-name> <password>@database.password@</password> <driver-class>@database.driver@</driver-class> </local-tx-datasource> </datasources>

 

Tokenizing what will become environment-specific values enables the Scripted Deployment to support multiple environments with a Unified Deployment.

---

Back to top

Extracting all environment-specific properties

Name : Externalize Configuration

Pattern : All variable values are externalized from the application configuration into build-time properties.

Antipattern : Some hardcode these values, manually, for each of the target environments, or they might use GUI tools to do the same.

Listing 4 shows some of the properties that are often contained in application-specific configuration files through the code base. By centralizing all variable values into a single .properties file, you can decouple data (the variable properties) from behavior (the deployment scripts). In other words, regardless of the target environment, the automated deployment process runs the same way every time.

Listing 4. Example properties that are external to application-specific files

authentication.type=db application.url=http://${tomcat.server.hostname}:${tomcat.server.port}/brewery-webapp database.type=mysql database.server=localhost database.port=3306 database.name=mydb database.user=myuser! database.password=mypa$$! database.url=jdbc:mysql://${database.server}:${database.port}/${database.name} tomcat.server.hostname=localhost tomcat.server.name=default tomcat.web.password=pa$$123! tomcat.cobraorb.port=12748

 

Often, the values in Listing 4 are scattered throughout source code, server configuration, XML, .properties, and other files. Moreover, I've found this data to be duplicated throughout a system, leading to complex deployment problems that are difficult to debug. By extracting this information from this many sources into a single .properties file, you'll reduce the myriad deployment problems that could occur.

---

Back to top

Save yourself with headless execution

Name : Headless Execution

Pattern : Securely interface with multiple machines without typing a command.

Antipattern : People manually access machines by logging into each of the machines as different users; then they copy files, configure values, and so on.

Despite its barbaric-sounding name, headless execution is quite an effective solution when you need to access other machines remotely through an automated process. By using a public key infrastructure (PKI), you can orchestrate commands that normally rely on a developer, build engineer, software configuration management (SCM), or operations person into an automated solution. In Figure 2, a private-key file is installed on the build machine and SSH. Specific values are defined in each of the target-specific .properties files. This typically includes the private-key file name and location, the SSH port number, and hostname. The target machines host the public SSH key file to complete the SSH handshake.

Figure 2. Using SSH keys to implement the Headless Execution pattern

If you use this approach, a Scripted Deployment can execute processes from the build environment to the target environment (or environments) without human intervention.

---

Back to top

Verifying properties are the same across environments

Name : Template Verifier

Pattern : Create a single template file that all target environment properties are based on.

Antipattern : Some use manual verification, trial and error (when deployment fails, check to see why), or keeping files "hidden" on a machine.

The problem is that you need to ensure that you've got the exact same attributes in every target environment. But how do you verify this in an automated environment? By using a single template .properties file that all of the target environment files verify against, you can ensure that all attributes are the same regardless of the environment you are running in. In Figure 3, the build script runs an Ant task that compares the attributes (not the values) between the template.properties and the target-specific .properties file (dev.properties, qa.properties, and so on). If it finds a difference, the deployment fails.

Figure 3. Implementation of Template Verifier pattern

Listing 5 shows an example template.properties file like the one illustrated in Figure 3. Notice that it includes only attributes , because the values are irrelevant.

Listing 5. A template file containing attributes, but no values

db.database= db.username= db.password= db.hostname= db.driver= db.port= db.url=

 

Listing 6 shows a snippet from a dev.properties (or qa.properties and so on) file as illustrated in Figure 3. Notice that it includes attributes and values. The values are specific to the target environment.

Listing 6. A target environment properties file based on template file

db.database=brewery db.username=root db.password=p@ssword db.hostname=dev1.domain.com db.driver=com.mysql.jdbc.Driver db.port=3306 db.url=jdbc:mysql://${db.hostname}:${db.port}/${db.database}

 

---

Back to top

Deploying once to multiple target environments

Name : Unified Deployment

Pattern : Create a single deployment script capable of running on different platforms and target environments.

Antipattern : Some may use a different deployment script for each target environment or even for a specific machine.

Although some deployment processes can run in certain environments, all processes should be capable of running in any target environment. For instance, the same Scripted Deployment is run in the development, test, stage, and production environments, but using a different Externalized Configuration file. The externalized configuration attributes are verified using the Template Verifier.

Figure 4 illustrates a single deployment script capable of deploying to multiple target environments:

Figure 4. Single deployment, multiple target environments

---

Back to top

But wait, there's more

Deployment is yet another aspect of software creation that lends itself well to automation. Automated deployments reap the benefits of a reliable, repeatable process: improved accuracy, speed, and control. In this article, I've covered eight patterns that can be effective for automating software deployments. Part 2 will cover several more patterns, including Remote Deployment, Disposable Containers, Deployment Test, and Environment Rollback.

 

Resources

Learn

• Software Configuration Management Patterns : (Stephen Bercuzk and Brad Appleton, Addison-Wesley Professional, 2003): The authors describe the Repository pattern and many other software configuration management patterns.
  
• Patterns of Deployment : HP Labs' SmartFrog wiki includes a catalog of many deployment patterns.
  
• Continuous Integration: Improving Software Quality and Reducing Risk (Paul Duvall, Steve Matyas, and Andrew Glover, Addison-Wesley Signature Series, 2007): Chapter 8, Continuous Deployment , demonstrates examples of incorporating deployment into an automated process.
  
• Pragmatic Project Automation: How to Build, Deploy, and Monitor Java Apps (Mike Clark, The Pragmatic Programmers, 2004): Enjoy pragmatic, automatic, unattended software production that's reliable and accurate every time.
  
• Release It!: Design and Deploy Production-Ready Software (Michael T. Nygard, The Pragmatic Programmers, 2007): If you're a developer and don't want to be on call at 3 a.m. for the rest of your working life, this book will help.
  
• "Speed deployment with automation " (Paul Duvall, IBM developerWorks, January 2008): Leverage automation to move software through different environments quickly.
  
• Browse the technology bookstore for books on these and other technical topics.
  
• developerWorks Java technology zone : Hundreds of articles about every aspect of Java programming.
  

Get products and technologies

Ant : Download Ant and start building software in a predictable and repeatable manner.