The State of Workflow by Tom Baeyens
The State of Workflow by Tom Baeyens
If database systems are like a respected, wise man telling a clear story, then workflow must be a bunch of spoiled brats shouting their own truth at each other. With this statement I would like to point out that workflow management systems are at the very initial phase of the technology hype curve. We are heading for some exciting times in this area. To illustrate this, we can make the comparison with relational database management systems (RDBMS). When talking about an RDBMS in a software development team most people will get the picture and shake their heads slightly up and down confirming they understand what you're saying. When using workflow terminology, the same crowds will shake their heads similarly but this time, every person will have a very different picture.
Figure 1: Workflow vs. RDBMS positioned in the hype-curve
One of the observations that form the basis of this statement is the overwhelming number of concepts used. None of the numerous specs and tools in this area is similar. Of course, they tend to overlap but it usually takes a dictionary and an extended concept mapper to compare any 2 of those.
A topic that certainly deserves to be included in an introduction about workflow is the relation between workflow and business process management (BPM). The term workflow is used to denote a series of related interactions between people and a computer system. Workflow is more often heard in developer communities. Sometimes, the meaning of workflow is being restricted to the different screens of a user interface. Business process management (BPM) has a broader scope. Where workflow is more often used in a technical context, BPM has a broader scope. It implies also the non-technical issues like analysis, organisational impact, from the viewpoint of a manager.
First I start with explaining what workflow management systems are. Then, the advantages of business process management are covered. Next is an explanation of why the workflow market looks confusing at first sight. The main conclusion will be that selecting a workflow management system is the hardest task companies have to face. To give you a solid basis for this task, the core section of this article defines the four layers of a process definition. It describes the common workflow & BPM concepts in a neutral language. At the end, a guided tour along the specs and tools is added.
What is a WFMS
A workflow management system (WFMS) is a software component that takes as input a formal description of business processes and maintains the state of processes executions, thereby delegating activities amongst people and applications.
To get started, lets define the basic terminology: a process definition and a process instance. A process definition is a formal description of a business process or procedure. A process instance is one execution of a process definition. So far so good. But when going one level deeper, we need to be very careful of the language we use. The way to express the steps within a process still hasn't found a common ground yet. That is the major difference between the specs and tools for workflow and BPM.
Why the term 'activity' should be banned...
Another important responsibility of a WFMS is maintaining context information for each process execution. A process context variable -or variable for short- is a variable associated with a process instance. E.g. the start-date of a holiday request, the key of a record in the database, a reference to a document in the document management system, ... Usually the process definition declares the process context variables. Then, at process instance creation time, the variables are instantiated. All mature WFMSs have customisable variable types.
One option is to use a WFMS as an enterprise application integration (EAI) platform. Currently, in most enterprise environments, the IT infrastructure has various heterogeneous applications and databases running in an intranet. Typically, these applications have a clear purpose when they are introduced in the organisation. Examples are customer management, document management, supply chain, order entry, billing, resource planning... Let's call them dedicated applications. Each of these dedicated applications contain knowledge about the business processes they have to support. The enterprise fits these automated processes of the software components into their overall non-automated business processes. Once such dedicated applications are installed and running, opportunities arise for new feature requirements that span over multiple applications. Enterprise application integration (EAI) is the discipline of implementing software requirements that involve multiple dedicated applications. Sometimes this is only data exchange that requires the implementation of a communication channel between two applications. Dedicated applications incorporate a number of business processes hard coded in the software. You could say that you buy a set of fixed, automated business processes. A WFMS on the other hand has got no prior knowledge of the domain. A WFMS takes a description of a business process as input and manages the executions of the process instances. That makes it much more flexible then dedicated applications but you have to do the effort to create a formal description of the business process. That is why WFMS and dedicated applications are complementary. A WFMS can be used to manage the overall process. As a rule of thumb, you could say use a dedicated application if it supports the processes you need. The first way to use a WFMS that was discussed here is to tie together all dedicated applications and create an EAI-platform with it.
A second option where a WFMS delivers high added value is for the development of workflow software that has a lot of people-related tasks. For this purpose, most WFMSs have a very convenient mechanism of creating forms for tasks. This target usage is especially productive in organisations focussing on ISO or CMM certification. Instead of documenting the procedures in a text format, a WFMS allows you to create automated support (e.g. by means of a web-application) of the procedures that are modelled as process definitions.
The third option is to embed a workflow engine inside another application. Remember from the first option that dedicated application incorporate a set of fixed business processes for a specific domain. The companies that develop the dedicated applications, can embed workflow engine software inside of their software. The workflow engine is in this case only used as an application component, hiding it from the application users. The main reason for embedding a workflow engine inside an application is for reuse (not reinventing wheels & hot water) and maintainability of the application software.
The case for workflowIntroducing workflow in an organisation delivers benefits on the software development level as well as the business level.
Easy developmentA workflow management system will ease the development and even more the maintenance of standard enterprise software.
- Reduced development risk - the business analyst will talk the same language as the developer (in terms of states and actions). That implies that the developer will not have to make a translation from user requirements to a software design.
- Centralized implementation - its the business processes that change so the biggest advantage of using a workflow system is that the implementation is not a fuzzy combination of software-pieces scattered over various systems.
- Rapid application development - your software is freed from the task of keeping track of the participants in a process leading to faster development and code that is better maintainable.
Business process management (BPM)
Before you can automate your business processes, the toughest but mostly rewarding task is analysing them and creating a formal description. e-workflow.org has a nice description about the business benefits gained by this analysis.
- Improved efficiency - automation of many business processes results in the elimination of many unnecessary steps
- Better process control - improved management of business processes achieved through standardizing working methods and the availability of audit trails
- Improved customer service - consistency in the processes leads to greater predictability in levels of response to customers
- Flexibility - software control over processes enables their re-design in line with changing business needs
- Business process improvement - focus on business processes leads to their streamlining and simplification
I think they even missed the most important reason to start using a WFMS: improved support for iterative development. If the business process part of a software system is not easy to change, organisations tend to spend a lot of effort in business process analysis and hope they'll get it right first time. Well sadly enough that is seldom the case as in any software development project. A WFMS makes it easy to deploy new versions of business processes. As a consequence the use of a WFMS enables more efficient and risk-reduced development because the business process software can be developed in an iterative way.
Missing linkI really believe a WFMS is the missing link in enterprise development. First of all I want to state that the default way of incorporating business process logic into enterprise software is scattered approach. This means that the logic of the business process is spread over various systems like the EJB-container, database-triggers, message broker and the likes. No need to mention that this leads to software that is not maintainable. As a consequence, these organisations will think of changing their business process software only as a last resort. They often prefer to change their processes to the software. This argument also applies to a larger extend ERP software packages. In a second attempt, suppose we are aware of this problem and really want to centralize all code related to one process. This is fine for one business process, but if you implement multiple processes, you see duplication of code that manages state and process variables. Then in a third approach we could extract the duplicated code and put it in a central library. Well, ... that is actually a WFMS so don't bother implementing it yourself and choose one from the list below.
A closer look
A WFMS takes process definitions as input. For the moment, think of a process definition as a UML activity diagram, a UML state diagram or a finite state machine. Examples are submitting an expense note, asking a holiday, requesting a quote, ... Then, the WFMS maintains the state and context of the executions of these process definitions. For this, it needs to be informed of state-changes. The state changes of the process executions can be logged for monitoring purposes.
Figure 2: Interfaces of a WFMS
- definition The definition interface of a WFMS allows process developers to deploy process definitions. Note that the actor 'process developer' is a placeholder for the combination of a business analyst and a software developer.
The 4 layers of a process definitionIn the next part, I try to answer the question "What is the content of a process definition?" This synthesis is based on principles and concepts that are used in the models of various specs & tools. It reflects the common ground between most of these models. The content of a process definition can be sliced into 4 different layers: state, context, programming logic and user interface.
The state layer
The state layer of a business process is all about expressing states and control flow. Control flow in standard programming languages is derived from Von Neuman architectures. There it defines the sequence of instructions that have to be executed. This control flow is determined by the order in which we write the instructions, if-statements, loop-statements, and so on. The control flow in a business process is basically the same. But instead of using an instruction as the basic element, the basic element in a business process is a state.
A state (also known as wait-state) in a process specifies a dependency upon an external actor. The meaning of a state is like "Now system X or person Y has to do something. Wait here until an external trigger of that actor signals the completion of the task." A state defines a dependency on a result provided by an external party. A typical example of a state is an approval step.
A state in a process definition also specifies which actor the execution depends on. The names given to these actors correspond with the notion of swimlanes in an activity diagram. With this information the WFMS can extract tasklists, which is the most common feature amongst WFMSs. As said before actors can be people or systems. For the states that require a human actor, the person has to be calculated at runtime by the WFMS. The dependency from a WFMS on the organisational data originates from this calculation. A very interesting article on this topic is 'Organizational Management in Workflow Applications' mentioned in the further reading section.
The control flow of a process definition is a set of states combined with the relations between the states. The logic between the states specifies which execution paths run concurrent and which are exclusive. Concurrent paths of execution are modelled with forks and joins, while exclusive paths of execution are modelled with decisions and merges. Note that in most models, every state has an implicit merge in front of it.
Often UML activity diagrams are used to model business processes. While it is an intuitive and widespread notation, a major problem poses in the diagram interpretation. In UML activity diagrams, no distinction is made between a state and an action (see 'Why the term 'activity' should be banned) They are both modelled as activities. The lack of this distinction (referred to as the lack of a state concept) is the most heard criticism by the academic community on UML activity diagrams. A second problem with UML activity diagrams was introduced in UML version 2.0. When multiple transitions arrive in an activity (read state), previous versions specified a default merge interpretation. The 2.0 version changed this to a default join, which requires synchronization. In my opinion, the graphical part of UML activity diagrams can still be used to model the state-layer of business processes on the condition that the semantics of these 2 constructs are changed as follows:
- in the diagram presentation of a business process, only model the state layer (states and control flow) and leave the actions out of the diagram. This means that every rectangle in the diagram should be a state instead of an activity
- if multiple transitions arrive in a state, define the default to be a default merge without synchronization
During process execution, a token is the pointer used by a WFMS to keep track of the current state of a process. This corresponds to the notion of program counter in Von Neuman architectures. When a token arrives in a state, the token is assigned to the external party for which the WFMS is waiting for. The external party can be a user, a group, or a computer system. Let's define an actor as a user or a system that acts upon the execution of a process. The assignment of an actor to a token is the only occasion where the WFMS needs to access the organisational data. This assignment allows the WFMS to extract task lists.
The context layer
A process context variable -or variable for short- is a variable associated with a process instance. Process variables allow a process developer to store data during the lifetime of the process instance. Some WFMS's have a fixed set of types, while in other systems, its possible to define your own custom types.
Note that variables can also be used to store references. A variable could reference e.g. a record in a database or a file on a network drive. When to use references typically depends on whether other applications use the referenced data.
Another interesting aspect related to process variables is how a WFMS turns data into information. Workflow is all about orchestrating the tasks and data between various heterogeneous systems across an organisation. For the human tasks of a business process, the WFMS collects the relevant data-items from the other involved systems like e.g. SAP, databases, CMR-system, document management system. In each human step of the business process, only the relevant data-items are shown that are collected and calculated from these heterogeneous systems. By presenting only relevant data-items, data that resides on the disparate systems is transformed and presented as information.
The programming logic layer
Recall that an action is a piece of programming logic to be executed by the WFMS upon a specified event that occurs during process execution. With programming logic we mean a piece of software in binary or in source format, in any programming or scripting language. The programming logic layer combines all these pieces of software with the information that specifies on which events these pieces of software need to be executed. Examples of programming logic for integration are sending an email, sending a message on a message broker, fetching data from an ERP-system and updating a database.
The user interface layerWhen an actor triggers the end of a state, usually that is the event where data is fed into the process variables. E.g. in the holiday example, when a boss signals to the WFMS that the state 'evaluation' is done, the boss provides the value 'approved' or 'disapproved' into the process. Some WFMSs allow to specify for each state what data can be fed into the process and how it should be stored into the process variables. From that information a user interface form can be generated to signal that requests information from a user. To see an example of a generic webapplication that allows actors to submit generated forms based upon a process definition, you can visit the jBpm online demo.
Executional processes versus a WFMS
A recent trend in the BPM community is the convergence of specifications about executional business processes. The approach promoted by XLANG, WSFL and BPML converged into BPEL. BPEL is based on interactions (message exchanges). BPEL is defined in the context of a service-oriented architecture. One of the prerequisites is that the services to be addressed are described in a WSDL declaration. BPEL then specifies an XML grammar that can be seen as a programming language to combine control flow with calls to the services defined in WSDL.
I see three main differences between the approach taken in executional business processes and state based workflow management systems:
- state based versus message oriented: state based WFMSs are centred around the notion of state (or activity). The workflow engine maintains the state and calculates the transitions from one state to the next. On the other hand, executional business processes as BPEL are centred around the definition of reactions upon incoming message. A set of those reactions, along with some other bells and whistles, can be seen as a business process. That explains why BPEL is somewhat complementary to state based WFMSs. A BPEL onMessage event handler, which is a reaction upon an incoming message, could be executed e.g. on transitions between states.
- process instance id versus message correlation: One of the complexities that are introduced with executional business processes is message correlation. Part of the process description must specify how the BPEL engine can detect the process instance identification from the incoming message. It has to be based on a data item in the message. On the other hand, state based WFMSs generate ids for each process instance that is created. The client can use this id in subsequent calls to the engine API.
- central workflow engine API versus an abstract service endpoint: state based WFMSs have a central API, meaning that the client talks to one central API to manage executions of all process instances. In executional business processes, each process is exposed as a service. This means a different access point for every process definition.
Academic communityThe academic community has the longest track record of involvement with workflow that dates back from the late seventies. The current trend in the research community considers petri nets as the mother of all process definition languages. For petri nets a wide variety of advanced analysis techniques have been described and last year I had the privilege of meeting professor Petri in person at the 2003 conference on Business Process Management. One of the best pieces of research that is made accessible and understandable for a large audience is workflow patterns. Workflow patterns compared a number of workflow management systems and expressed the common process modelling concepts in terms of petri nets.
Open source projects
Finally we are getting down to the links of real workflow management systems. This is where the difficult task starts of choosing one. Actually you should be happy: coping with making a choice is better then having no choice at all :-) One of the goals of this article was to explain the basic concepts of workflow so that you are better equipped to perform the selection process. But I realize that comparing workflow systems is one of the most challenging tasks at this moment for software architects.
- jBpm - jBpm is a flexible, extensible workflow management system written by the author of this article. Business processes, expressed in a simple and powerful language and packaged in process archives, serve as input for the jBpm runtime server. jBpm combines easy development of workflow-applications with excellent enterprise application integration (EAI) capabilities. jBpm includes a web-application and a scheduler. jBpm is a set of J2SE components that can also be deployed as a clustered J2EE application.
- OpenEbXML - The Open ebXML project is working to providing a ebXML framework that primarily supports the v2.0 set of ebXML specifications soon to be released by UN/CEFACT and OASIS.
- Werkflow - Werkflow is a flexible, extensible process- and state-based workflow engine. It aims to satisfy a myriad of possible workflow scenarios, from enterprise-scale business processes to small-scale user-interaction processes. Using a pluggable and layered architecture, workflows with varying semantics can easily be accomodated.
- OSWorkflow - What makes OSWorkflow different is that it is extremely flexible.
- wfmOpen - WfMOpen is a J2EE based implementation of a workflow facility (workflow engine) as proposed by the Workflow Management Coalition (WfMC) and the Object Management Group (OMG). Workflows are specified using WfMC's XML Process Definition Language (XPDL) with some extensions.
- OFBiz - The Open for Business Workflow Engine is based on the WfMC and OMG spec. OFBiz Workflow Engine uses XPDL as its process definition language.
- ObjectWeb Bonita - Bonita is a flexible cooperative workflow system, compliant to WfMC specifications. A comprehensive set of integrated graphical tools for performing different kind of actions such as process conception, definition, instanciation, control of processes, and interaction with the users and external applications. 100% browser-based environment with Web Services integration that uses SOAP and XML Data binding technologies in order to encapsulate existing workflow business methods and publish them as a J2EE-based web services. A Third Generation Worflow engine based in the activity anticipation model.
- Bigbross Bossa - The engine is very fast and lightweight, uses a very expressive Petri net notation to define workflows, does not requires a RDBMS and is very simple to use and to integrate with java applications. Actually, it was designed to be embedded.
- XFlow - XFlow runs within an EJB and servlet container.
- Taverna - The Taverna project aims to provide a language and software tools to facilitate easy use of workflow and distributed compute technology within the eScience community.
- Enhydra Shark - Shark is completely based on standards from WfMC and OMG using XPDL as its native workflow definition format. Storage of processes and activities is done using Enhydra DODS.
- PowerFolder - PowerFolder consists of a developer studio, administration environment, and a runtime engine.
- Breeze - Breeze is a lightweight, cross-platform component-based workflow engine prototype.
- Open Business Engine - Open Business Engine is an open source Java workflow engine which supports the Workflow Management Coalition's (WfMC) workflow specifications, including interface 1, also known as XPDL, interface 2/3 known as WAPI and interface 5 for auditing. OBE provides an environment for executing activities in a controlled, centralized environment. OBE supports both synchronous and asynchronous execution of workflows. The primary OBE implementation is based on J2EE.
- OpenWFE - OpenWFE is an open source java workflow engine. It features 3 components, easily scalable: an engine, a worklist and a web interface. Its workflow definition language is inspired of Scheme, a Lisp dialect, though it is expressed in XML.
- Freefluo - Freefluo is a workflow orchestration tool for web services. It can handle WSDL based web service invocation. It supports two XML workflow languages, one based on IBM's WSFL and another named XScufl. Freefluo is very flexible, at its core is a reusable orchestration framework that is not tied to any workflow language or execution architecture. Freefluo includes extension libraries that enable execution of workflows written in a subset of WSFL.
- ZBuilder - ZBuilder3 is a second generation of workflow development and management system which intends to be an open source product. It defines a set of standard JMX management interfaces for different workflow engines and their workflows.
- Twister - Twister's aim is to provide a new generation, easily integrable, B2B oriented workflow solution in Java, based on the latest specification efforts in this field. The process engine is based on the BPEL business process specifications and Web Services standards.
- Con:cern - con:cern is a workflow engine based on an extended case handling approach. A process is described as a set of activities with pre- and postconditions.
- Bea's WLI
- Fujitsu's i-Flow
- IBM's holosofx tool
- Oakgrove's reactor
- Oracle's integration platform
- SAP's NetWeaver
- Sonic's orchestration server
- WebMethod's process modeling
- A collection of links to tools for modelling business processes and workflows maintained by Bart-Jan Hommes at TU Delft, the Netherlands.
Michael zur Muehlen has made this fine overview presentation about all the workflow related specifications.
I agree with John Pyke and Wil van der Aalst that the standards defined for workflow are still a work in progress. Another argument that points in that direction is the huge amount of overlapping specifications out there.
In my opinion, the multitude of specifications and the limited adoption of each of those specifications are because of limited common knowledge about the primitives. You see: workflow is a subject from which you get easily distracted. The distraction occurs when workflow related concepts get mixed up with complementary technologies or concepts. To give a very concrete example: workflow is completely complementary to web-services. You could expose the interface to access a WFMS as a web-service, but its definitely a bad idea to presume that a WFMS *always* has to be accessed through a web-service interface. Some specifications make this presumption. Apart from web-services, other often occurring distractions include email, inter-business communication, web-applications & organisational structure.
The first standardisation effort in workflow was the Workflow Management Coalition (WfMC). The WfMC started in 1993. A very interesting publication of the WfMC is the reference model. It defines the interfaces between a workflow management system and other actors. Another piece of work is the XPDL specification. XPDL defines an XML schema for specifying the declarative part of a workflow. In my opinion, the reference model and XPDL are the best specifications
- JSR 207: Process Definition for Java - This is an initiative of the Java Community Process (JCP) that standardizes how to automate business processes on a J2EE server. The basic model of this JSR is to define a special type of ejb session bean that acts as the interface for a business process. The JSR wants to standardize a set of XML meta tags that should be specified as meta data (JSR175). The JSR207 compliant ejb-container then takes the session bean and the meta data as input and generates code that binds the methods together as specified in the meta data. This specification is still in an early phase, no publications have been made yet. The expert group was formed in March 2003.
- WfMC's XPDL - The Workflow Management Coalition (WfMC) is a consortium of about 300 that defines a set of related standards based on an interesting reference model. The reference model describes the relation between a WFMS and other actors in a use case fashion. XPDL is the specification of WfMC that specifies an XML grammar the control flow of business processes.
- ebXML's BPSS - ebXML is a set of related standards for business collaboration. Main focus is on communication between different companies. This can be seen as the successor of EDI. ebXML is a joint initiative of OASIS and UN/CEFACT. BPSS is the specification of ebXML that is closest to the concepts described in this article.
- BPMI's BPML & WSCI - (Intalio, Sun, SAP, ...). BPMI also has defined a specification (BPMN) that describes how the 'executional' business processes should be visually represented.
- BPEL - (Microsoft, BEA, IBM, SAP & Siebel) BPEL is the specification that results from a set of specifications based on message exchange: XLANG, WSFL, BPML. Also a proposal is being made that hooks this specification to Java: BPELJ. Instead of modelling the states of a process. This specification says how to create a reaction upon an incoming message. So this is not a specification for formal definition of business processes as laid out in this article. Simplified this can be seen as a programming language that is expressed in XML and that provides the control flow constructs for combining WSDL-services. As the name suggests, this specification is focussed (to be exact 'not limited to') to web services.
- OMG's Workflow management facility - Is based upon the WfMC specifications and defines how this gets translated into CORBA.
- UML - UML defines 9 types of diagrams for modelling and designing software systems. Each diagram type defines a visual representation and semantics for the things you can draw. One type of diagram, the activity diagram, has as a goal to specify how business processes should be visually represented. Note that with the 4 layers of a process definition, I tried to show that a process definition is more then just its visual representation. UML only specifies the visual part.
- RosettaNet - RosettaNet mainly defines a set of Partner Interface Processes (PIP). A PIP specifies a process between 2 trading partners, including the message formats.
- UBL - The Universal Business Language (UBL) defines a standard library of XML documents to be used for communication between different organisations. This can be seen as complementary to ebXML since ebXML only defines the infrastructure to set up inter-organisation processes. This standard is competing with a subset of the RosettaNet standard. The RosettaNet PIP's define a set of processes between trading partners, including the message format. The message formats of the RosettaNet PIP's are at the same level as UBL.
I have shown in this article that the workflow market is still young and wild but that there are already solid tools out there:
- Only now, mature integration platforms such as J2EE and .NET are available. Running a WFMS on such a platform really boosts their added value. That is why only recently the workflow systems have been rediscovered.
- In 'The case for workflow' we showed actually how the introduction of workflow management systems delivers return on investment, both at the technical and the business level.
- The positioning of workflow in the technology hype-curve shows that the concepts used in BPM and workflow management still need to settle.
- The section 'open source projects' and commercial vendors lists a variety of tools that are eager to bring you all the benefits of workflow and business process management.
The conclusion we can extract from all this is that
- Specifications are not yet mature, no standards have been adopted on a broad scale
- Comparing workflow tools is the hardest challenge today for companies that embrace the BPM ideas
- Despite the fact that standardisation still might take a while, good workflow management systems are already available. That is a real incentive for organisations to dive already into the tool-selection process.
I hope that I have stimulated your interest in workflow and gave you on the right background for making an effective comparison.
- Workflow Patterns - A website dedicated to the academic research on Workflow Patterns of professof Wil van der Aalst.
- ebpml.org - A complete and informative website on business process management and workflow.
- Business process management group - The Business Process Management Group (founded in 1992) is a global business club exchanging ideas and best practice in business process and change management.
- Enix - English consultancy process management consultancy firm with good content on the website
- ebizq.net - Commercial community that has booths, webinars and other interesting stuff put together in a nice website.
- An introduction to petri nets - The title says it all.
- Organizational Management in Workflow Applications - An interesting article that discusses the relation between business processes and the organisational data involved.
- Web services orchestration - an HP-paper that reviews the emerging technologies tools and standards. (Januari 2003)
- Business process standards for web services - An article that discusses the various aspects involved in workflow and BPM.