Implementing and Promoting Daily Builds

Introduction

<script language=JavaScript> </script>

Rational Unified Process (RUP), Extreme Programming (XP), and Agile methodologies trumpet the benefits of incremental development: software, like everything else, is easier to build in small steps than one big bang. Building an application daily is an iterative development best practice that helps expose bugs more quickly, raises clients' confidence and paces developers. But when daily build is put in place, a structured approach is required to promote the build and make it available to testers and end users. If not, the daily build quickly becomes unmanageable.

This paper describes a build promotion process that we in the Capgemini Technology Consulting group have successfully implemented in a number of large J2EE and .NET software developments. It shows how we have structured our daily build process and structured the build promotion from development to production servers. It also addresses the issues that arise when building daily, and how we managed them.

This paper is organized as follows. The daily build is first introduced, then a discussion of its positive effects on developers, testers and users is followed by a examination of the issues related to promoting and managing daily builds. Finally, the paper shows how difficult it can be to fix bugs and build databases when building daily.

The Daily Build

A build refers to the required steps to create a fully functional and executable version of a software application from its basic components (code, database, configuration files, etc.). A good build process must:

• Be automated and repeatable i.e. never require human intervention to complete its work. Automation is essential to reap the benefits of the daily build.
• Be portable to different computers and environments
• Report the results of its operation to humans that validate the results. Typically this takes the form of e-mails sent to the "build master". Other communication mediums such as Short Message Service (SMS) can also be leveraged.
• Perform multiple operations including but not restricted to:
  • Extracting/tagging most recent copy of the code from source control
  • Generate a change log file (useful for identifying who introduced the latest defect)
  • Generate version file identifying date of build and incorporate it in the executable
  • Building the application and its peripherals (documentation, etc.)
  • Packaging the application for deployment
  • Deploying and installing the application to target environments
  • Running regression tests
  • Reporting build results, including success/failure but also build metrics

Breaking the build is a capital offense. Developers who break the build must be identified and, at a minimum, fix the build. Authors have devised different incentives to keep the build running: have the developer who broke the build wake up at 5AM daily to smoke test the build, have them wear a funny hat, or have them contribute money to the office pool. Use one that works for your team but never tolerate broken builds.

The reference section provides additional references on the daily build process.

The Positive Impacts of Building Daily

Building your application daily has numerous positive side effects.

• Reduced integration work: developers are encouraged to commit their code as soon as it has been tested. This way their code does not get out of sync with the main development branch. Developers are also encouraged to get a fresh copy of the source code tree on a daily basis, or at least before starting each new feature.
• Quick spotting of defects: new commits in the code repository are included in the next daily build - which occurs at 5AM every morning - and then smoke tested. Defects are spotted before developers forget about the code changes they made. Furthermore, since the build process also generates a changelog file, it is relatively easy to identify which commit introduced the defect and who is to blame.
• Application quality improvement: since the whole software is tested on a daily basis, daily builds reduce the risk that new defects will go unnoticed. Defects are spotted and fixed on a daily basis. This prevents defects from accumulating in the software to the point where a significant effort would have to be expanded in order to fix them all.
• Progress visibility enhancements: since the software is built daily, it is easy for the project manager to see which features are there and which ones are not. Clients have visibility into the process and be reassured that milestones will be met.
• Team morale booster: seeing the software working provides a good morale boost to the troops. This benefit in itself is enough to justify the adoption of this best practice.
• High impact, low maintenance: the set-up of an automated daily build process takes time. Once automated, few minutes are needed for the configuration manager to read the e-mail sent by the build script and verify that the build is good. This process is very low maintenance and does not eat away valuable time from the team's busy schedule.

The Build Promotion Process

As we have seen, building daily goes beyond providing a sanity check on coding software. It brings new features or bug fixes that users and testers need to do their work. Once you start to build daily, requests will flow from all around to get the daily build into their server. Expect this behavior not only from testers but more generally from anyone using your application!

However, it is clear that you cannot push a build every day onto any server. Testers, end users, and support personnel need to have "predictable" behavior from their application, which the daily build cannot provide. At the same time, they would like to have the latest bug fixes and new features that the daily build process exposes. Having a daily build requires you to manage the releases very carefully.

The diagram below summarizes the build promotion process we have put in place.

Figure 1 - The build promotion process

Build promotion refers to the act of transferring a build from one server to another. Promoting a build means copying it to the target server; the build must not be rebuilt but rather simply copied, thus the need to identify and archive builds as they are created. We have defined roles for each server as well as rules to promote a build one server to the next. Table 1 - Build promotion rules -summarizes the build promotion process.

Daily Build Tips, Tricks, and Traps

Continuous builds
For large projects or projects where building takes a long time (hours), it is important to build continuously throughout the day. Typically this is done on a dedicated server that reports results and archives builds as it runs. This allows developers to quickly validate if their changes build properly without having to wait for the next daily build or consume their workstation CPU cycles for a long period of time.

Development Team Synchronization
When using the daily build process, it is essential to make sure the development team members are synchronized to avoid breaking the build. If developers are not synchronized, it is possible for one developer to check in code that builds properly on his workstation but still breaks the build if someone else checked in incompatible code.

To avoid this problem, you must define during the day certain periods:

• Safe check in: during that period anyone can check in code into the version control system. This is the default period during the day
• Safe check out: during this period, developers must update their code to match the version control. This period typically occurs just after the daily build has been smoke tested and shown not to be broken and lasts 1-2 hours. No developer is allowed to check in code during that period.
• No check in or check out: during this period no check in or check out is allowed. This period begins when the daily build occurs and lasts until the build is smoke tested and shown not to be broken.

Managing change: creating patches
We recommend that builds be promoted at regular intervals, thus continuing the rhythm put in place by the daily build. The build promotion process table suggests daily, weekly, monthly and quarterly promotions for daily, QA, staging, and production servers.

Sometimes however, the promotion process cannot be as straightforward as the table suggests. For instance, builds that have made it through the unit, smoke, and light integration tests do break during heavy functional testing. Sometimes the failure is major enough that it keeps testers from going any further until a new build fixes the problem. When that happens, the daily build can often fix the problem but it cannot be deployed because it introduces new features that are not stable enough for QA, staging, or production servers.

The solution is to patch the build in question (branching the code in the version control database from the point where the build in question was tagged), fix the problem, then promote the patch to the server where the issue exists. For example, testers find that a new account opening feature does not work on the QA server. That bug prevents them from exercising all test cases after account opening. The daily build already has that bug fixed but it cannot be promoted to QA server because other features are partially implemented. The account opening feature can be patched rapidly. A new, patched QA build is created and deployed, so as to provide testers with an updated build that lets them continue their tests while the more thorough fix finds its way in the normal build promotion process.

Building Databases
Building Java or .NET applications is relatively simple. Code is stored in a central repository and everything is built together at once. Simple and (relatively) easy to automate. Databases, however, are much more complex to build automatically because the build must not only create the schema but also create data. To make matters worse, often data must be migrated from a previous schema, and this is difficult, sometimes impossible, to automate.

Thus, the following must be done to properly promote databases during the build process:

• Re-create the database schema from scratch using SQL or other scripts extracted from the source code control system. This implies that all database schema changes are scripted, which is often not the case. But it can be done with a bit of rigor and constraints, especially in environments where database schema can only be modified by a few database administrators.
• Recreate the base data from scratch using SQL or other scripts. Again, this is seldom done but can be and should be the default best practice with all teams.
• Convert the data in the database from the previous schema. This is rarely ever done as it implies that scripted changes to the database schema also be accompanied with data conversion scripts.

As you can see, it is possible to automate the database creation and make it part of the automated daily build process. However, more often than not, you will find that databases are not built daily and are managed in an ad-hoc fashion.

However, Extreme Programming "Lessons Learned" [1] presents a great solution to DB build issues. It basically says that:

• Three databases are created: gold, silver, and bronze.
• Database schema changes are promoted from bronze, to silver, to gold then finally to production database (which we refer to as the "master database")
• Database data is migrated from production to gold, then to silver, and then bronze databases
• All changes to database schema must be scripted. The sum of all scripts, when applied in sequence, allows database migration to be possible
• The migration and promotion processes must be automated to ensure that they can be performed often, rapidly, and in a uniform manner.

They too, however, agree that managing promotion/migration scripts is a difficult yet essential tasks that must be followed closely by all involved.

Conclusion

Adopting a daily build process is essential to any software development project. It benefits all team members, from developers to end users. The build script must be customized for each project and fine tuned to map the application's structure and dependencies. Even when available and fully automated in an unbreakable, religious, daily routine, the build process raises issues with end users wishing they could have access as fast as possible to bug fixes and new features. A build release process must be put in place to ensure that appropriate levels of testing and predictability can be ensured for all users. Finally, we have seen how the daily build process creates needs for new developer's procedures, bug fixing shortcuts and non-trivial scripting and controls to ensure database are built along with the code they support.

References

[1] XP and Databases, Extreme Programming Lessons Learned, http://www.extremeprogramming.org/stories/testdb.html
[2] Rapid Development: Taming Wild Development Schedules, Steve McConnell, 1996, ISBN 1-55615-900-5, 650 pages
[3] Using Open Source .NET Tools for Sophisticated Builds, http://www.15seconds.com/issue/040621.htm

Special Thanks

Special thanks go to Pascal Forget, and Adnaan Sikandar for their review and comments of this paper.

About the Author

Farid Mheir is a Senior Technologist in the Capgemini Technology Consulting group. He has more than 20 years of experience in software development and engineering. He has significant experience in software application design and implementation, with a focus on large web-based solutions, leveraging Rational Unified Process and Unified Modeling Language to translate business requirements into technical specifications.

Mr. Mheir has integrated systems with existing applications and processes, recently using Web services to link remote Microsoft-Unix systems. His experience with the development and operations of 24/7 e-commerce applications has given him an understanding of performance and maintenance requirements. He is also experienced in the delivery of multiple applications via a distributed delivery mode, with teams based in North America and Europe. He can be reached at farid@mheir.com.