Change management and Drupal:AutoPilot

For those of you that read the enterprise article I posted recently, or attended the OSCMS Conference at Yahoo, you're probably familiar with the needs of corporate customers. Or at least the needs as I see it.

At the top of the list of almost every single customer we have is change management. Period.

Specifically, how to lower the cost of change management, how to reduce the cost of upgrading Drupal, and specifically how to estimate and understand the costs of doing Drupal-based change management .

Change is inevitable in all stages of a Drupal-based software project. Change management will help you direct and coordinate those changes, and provide best practices, so that chage itself can become a force that enhances-not hinders-your project.

In fact, the only constant in software development is change. So why is it so expensive in Drupal?

As this topic has been getting some attention as of late, we thought it was time to provide an update, and a vision about where we're going (hint: it's beyond site profiles , though that's extremely useful).

At WorkHabit, we have been working hard to address one of the most pressing concerns in the Drupal community. Drupal is currently somewhat expensive to upgrade and support, though not considerably more than many commercial software alternatives. But wouldn't it be nice if we could lower the cost complexity of Drupal change management?

Now, before you assume this is flame bait, let me tell you about some of the unique initiatives we've and working on in order to alleviate some of the problems. First let's outline some known quantities about Drupal:

• Drupal has a new release approximately every six months
• Drupal is made up of core, which is the code that it ships with, and contrib modules
• contrib modules are generally delivered from individual authors
• contrib modules are varying quality, and many have significant performance and code maintenance impact on the site
• traditionally, upgrades between major Drupal releases has been relatively seamless for core, at least compared to contrib modules
• contrib module upgrades have often been less seamless, and many times contrib module updates happen months after core has been released
• in addition, there can be a great deal of complexity migrating between contrib modules especially in cases where technology has transitioned from one module to another. A good specific example of this is when flexinode became cck, or at least when CCK became the preferred method

This is the current Drupal reality. To keep up at upgrades its required a great deal of time, and I don't expect that to change overnight. But what I do think in change is a reduction in the overall cost and complexity of upgrades, including upgrades between major versions and contrib modules.

So, how do you solve all that?

Well, let me first say that we're not out to build the universal business adapter (IBM does that, they advertised it), or a process tool (like SCM) - our goals are achievable and clear.

We're definitely not out to try to create a giant monolithic software product that "does everything" like so many become - quite the opposite. We need something simple that "just works" even if it requires a little setup.

That's because a successful change management system coordinates people, process, and technology in a logical, effective, flexible manner that makes sense. And to do that, people have to be able to use it.

And a large part of creating this tool is actually in creating the best practice - essentially providing guidelines about what works really well for creating Drupal-based websites.

This is going to drive down the cost of Drupal, but it's also going to speed up development, and make it easier for us all to do our jobs.

Change management gives you insite and control over the evolution of your Drupal software. The data you collect by implementing proper tools can reduce the risk of projects, make development more predictable, and help you learn from your mistakes. In the long run, managing change with a tool reduces the time to market considerably, improves code quality, and increases end-user satisfaction. We need a tool to manage, rather than just react to and watch, software change.

AutoPilot: Drupal Change Management

For several months we've been working with an internal project called AutoPilot .

Autopilot is named for Cruise Control , a fantastically useful application that's used to do continuous integration with Java based projects.

Autopilot has a number of goals, many of which are going to require the participation, support, and the willingness of the community to implement.

So what's the vision?

The first step is to implement social build management. This is a term that I coined, and I have no idea if it's unique or not, to describe the idea of being able to do changes to website in a multitier development, staging, live environment set up while people are contributing to your live environment.

One of the biggest problems has been the fact that as you work on stuff in development the copy of the database and website you have gets more and more out of date, to the point where becomes extremely difficult to reconcile with the live site. This is because people are actively working on the live site, and in the case of large communities there may be tens of thousands of posts between the time that you start a project and when you release it, especially with major engineering efforts.

This is impossibly difficult.

So the first goal of autopilot is to implement social build management. This is basically built on top of the idea that you can deliver a product that can keep track of, or make irrelevant the need to have, reconciliation between development, staging and production environments. Work on social build management has been going well. We announce the product as soon as we had a working for basic Drupal build management, but decided not to release it until we actually had social build management working, because the vast majority of sites would not actually be able to utilize the static build management system as it's really only useful for corporate websites, and that's not what's exciting about Drupal.

The second goal of autopilot is compliance. Delegating someone in your organization to run builds can provide a useful check off for QA, preventing the possibility of unauthorized pushes to live websites. In addition because of the fact that you can split who pushes to QA and who pushes to live, you're well on your way to Sarbanes-Oxley compliance (yep, Sarbox + Drupal = good).

The third goal of autopilot is test driven development and continuous integration. This would be so what astoundingly cool for the Drupal community. Effectively being able to bring Drupal into proper agile methodology compliance, and make it the preferred platform for test driven development and agile programming in PHP. This is not a small task, and will require the support of the Drupal community to accomplish, but it will extend Drupal's lead over the majority of systems available.

The fourth goal of autopilot is streaming Drupal delivery. This is about building this description-based Drupal, that enables customers to do updates on demand based on what's been committed to source control. This idea has been around in the community for some time, and we have used it ourselves with great effect on many projects. however, being able to combine test driven development and continuous integration with streaming Drupal is going to enable us to do DrupalCasting. There, I said it. Drupal + Casting = DrupalCasting, and it's the way man... it's so the way.

DrupalCasting is the ability to keep your site subscribed to updates, and simply be able to click a button inside of autopilot to sink to the latest version, test that versions against the unit tests that are written for your site including custom code, and get a result that lets you know that everything went okay for that you need to take a look. And remember, because everything is happening on a development server now, as opposed to just working in production, you have the opportunity to sit down and take a look without stressing out about site outage. This is an incredibly powerful capability, and will drastically drive down the cost of Drupal management.

Combined with installation profiles, DrupalCasting is going to have an impact on the way that Drupal gets delivered.

So as you can see autopilot has some fairly ambitious goals.

Our intention right now is simply to deliver the social build management. This is enough of a contribution to make a significant impact. And once released, we will be able to focus our time on improving the existing product based on user feedback. While I don't think it's possible to out innovate the Drupal community, I do think it would be possible to move too quickly on this platform and end up making some architecture decisions that would render render it less effective had it been architected otherwise. So the intention is to move through each one of these goals, basic build management, social build management, continuous integration, and then eventually test driven development with continuous integration, in a step-by-step fashion in a manner that "just works" for customers.

Right now we are continuing to do integration on the social build management component of autopilot. It's likely this integration work will take another few weeks. Many people asked me to commit to a solid release date, and I'd like to. But right now the entire project is funded by our ongoing consulting work, so are always looking for time and it's not always easy to find it. But we have now committed several months worth of work to the platform, and will be delivering another few months worth of work -- in terms of people hours -- over the next few months.

By the way, did we mention the entire system (so far) is built inside of Drupal? In fact, the way this works is that autopilot is able to work as a Drupal multisite installation, reaching out through the Apache Web server using an embedded certificate to other servers, so yes it actually does support multiserver setups. We're really excited to release his product, and consider it our first really major individual contribution to the Drupal community.

We'll be posting new articles on autopilot over the next two weeks as we implement and move towards a release date, including announcing a more firm release point for our open source code. We would strongly appreciate assistance with this work, and welcome earnest collaborators.

A solution to this problem that this project hopefully provides is far more important than anyone consulting company.

 

please download realted code from https://workhabit.svn.beanstalkapp.com/public/drupal/contrib/autopilot/trunk/