Xenomai - Implementing aRTOS emulation
framework on GNU/Linux
Copyright © 2002Philippe Gerum
Permission is granted to copy, distribute and/or modify thisdocument under the terms of the GNU Free Documentation License, Version 1.2 orany later version published by the Free Software Foundation; with no InvariantSections, no FrontCover Texts, and no Back-Cover Texts. A copy of the licenseis published on gnu.org: "GNU FreeDocumentation License" [http://www.gnu.org/licenses/fdl.html].
Generally speaking, the Xenomai technology first aims at helpingapplication designers relying on traditional RTOS to move as smoothly aspossible to a GNU/ Linux-based execution environment, without having to rewritetheir application entirely.
This paper discusses the motivations for proposing this framework,the general observations concerning the traditional RTOS directing thistechnology, and some in-depth details about its implementation.
The Xenomai project has been launched in August 2001. It has mergedin 2003 with theRTAI project[http://www.gna.org/projects/rtai/]to produce an industrialgrade real-time Free Software platform for GNU/Linuxcalled RTAI/fusion, on top of Xenomai's abstract RTOS core. Eventually, theRTAI/fusion effort became independent from RTAI in 2005 as thexenomai project [http://www.gna.org/projects/xenomai/].
Linux is a registered trademark ofLinus Torvalds. Other trademarks cited in this paper are the property of theirrespective owner.
Table of Contents
1. White paper.................................................................................................2
1.2. Porting traditional RTOS-basedapplications to GNU/Linux ........................ 3
1.3. A common emulation framework.......................................................... 7
1.4. Nucleus description........................................................................... 10
1 . White paper
1.1 . Introduction
A simpler migration path from traditionalRTOS to GNU/Linux can favour a wider acceptance of the latter as a real-timeembedded platform. Providing emulators to mimic the traditional RTOS APIs isone of the initiative the free software community can take to fill the gapbetween the very fragmented traditional RTOS world and the GNU/Linux world, inorder for the application designers relying on traditional RTOS to move assmoothly as possible to the GNU/
There is a lack of common softwareframework for developing these emulators, whereas the behavioural similaritiesbetween the traditional RTOS are obvious.
The Xenomai technology aims at fulfillingthis gap, by providing a consistent architecture neutral and generic emulationlayer taking advantage of these similarities. It is also committed to providean increasing set of traditional RTOS emulators built on top of this layer.
Xenomai relies on the common features andbehaviours found between many embedded traditional RTOS, especially from thethread scheduling and synchronization standpoints. These similarities areexploited to implement a nucleus exporting a set of generic services. Theseservices grouped in a high-level interface can be used in turn to implementemulation modules of real-time application programming interfaces, which mimicthe corresponding real-time kernel APIs.
A similar approach was used for the CarbonKernel [http://savannah.gnu.org/projects/carbonkernel/] project in the simulation field, in which RTOS simulation models are built on top of ageneric virtual RTOS based on event-driven techniques.
1.2 . Porting traditional RTOS-based applications toGNU/Linux（移植）
The idea of using GNU/Linux as an embeddedsystem with real-time capabilities is not novel. The reader can refer to JerryEpplin's article in the October 97 issue of Embedded Systems Programming for adiscussion about GNU/Linux potential in the embedded field .
Throughout this document, we will use theexpression source RTOS to indicatethe traditional real-time operating from which the application is to be ported,andtarget OS to indicate GNU/Linuxor any other free operating system to which the application could be ported.
1.2.1 . Limited high-level code modification（有限的高层代码修改）
Keeping the initial design andimplementation of a hard real-time application when attempting to port it toanother architecture is obviously of the greatest interest.Reliability andperformance may have been obtained after a long, complex and costly engineeringprocess，one does not want to compromise. Consequently, the best situation is to have the closest possibleequivalence between the source and destination RTOS programming interfaces, asfar as both the syntax and the semantics are concerned.
An illustration of this can be taken fromthe support of a priority inheritance protocol  by the mutual exclusion services. Theseservices allow concurrent threads to protect themselves from race conditionsthat could occur into critical sections of code. The purpose of this discussionis not to argue whether relying on priority inheritance for resolving priorityinversion problems is a major design flaw or a necessary safety belt for areal-time application, but only to emphasize that any cases, if this feature isused in the source RTOS, but not available from the target OS, the resourcemanagement strategy must be reevaluated for the application, since priorityinversion risks will exists.
1.2.2 . RTOS behavioural compatibility
During the past years, major embedded RTOS,such as VRTX, VxWorks, pSOS+ and a few others, have implemented a real-timekernel behaviour which has become a defacto standard, notably for threadscheduling, inter-thread synchronization, and asynchronous event management. Toillustrate this, let us talk about a specific concern in the interrupt servicemanagement.
A well-known behaviour of such RTOS is tolock the rescheduling procedure until the outer interrupt service routine (orISR) - called first upon possibly nested interrupts - has exited, after which aglobal rescheduling is finally stated. This way, an interrupt service routinecan always assume that no synchronous thread activity may run until it hasexited. Moreover, all changes impacting the scheduling order of threads, due toactions taken by any number of nested ISRs (e.g. signaling a synchronizationobject on which one or more threads are pending) are considered once andconjunctively, instead of disjunctively.
For instance, if a suspended thread isfirst resumed by an ISR, then forcibly suspended later by another part of thesame ISR, the outcome will be that the thread will not run, and remainsuspended after the ISR has exited. In the other hand, if the RTOS sees ISRs asnon-specific code that can be preempted by threads, the considered thread willbe given the opportunity to execute immediately after it is resumed, until itis suspended anew. Obviously, the respective resulting situations won't beidentical.
1.2.3 . Reevaluation of the real-time constraints
Making GNU/Linux a hard real-time system iscurrently achieved by using a co-kernel approach which takes control of thehardware interrupt management, and allows running real-time tasks seamlesslyaside of the hosting GNU/Linux system . The 'regular' Linux kernel iseventually seen as a low-priority, background of the small real-time executive.TheRTLinux [http://www.rtlinux.org]project is representative of this technical path. However, this approach has amajor drawback when it comes to port complex applications from a foreignsoftware platform: since the real-time tasks run outside the Linux kernelcontrol, the GNU/Linux programming model cannot be preserved when porting theseapplications. The result is an increased complexity in redesigning anddebugging the ported code.
In some cases, choosing a traditional RTOSto run an embedded application has been initially dictated by the memoryconstraints imposed by the target hardware, instead of actual realtimeconstraints imposed by the application itself. Since embedded devices tend toexhibit ever increasing memory and processing horsepower, it seems judicious toreevaluate the need for real-time guarantee when considering the porting effortto GNU/Linux on a new target hardware. This way, the best underlying softwarearchitecture can be selected. In this respect, the following, the followingcriteria need to be considered:
• Determinism and criticality.
What is theworst case interrupt and dispatch latencies needed ?
Does a misseddeadline lead to a catastrophic failure ?
• Programming model
What is theoverall application complexity, provided that the higher the complexity, thegreater the need for powerful debugging aid and monitoring tools.
• Is there a need need forlow-level hardware control ?
Is the real-timeactivity coupled to non real-time services, such as GUI or databases, requiringsophisticated communications with the non real-time world ?
1.2.4 . Some existing solutions
In order to get whether hard or softreal-time support, several GNU/Linux-based solutions exist . It is notthe purpose of this paper to present them all exhaustively. We will onlyconsider a two fold approach based on free software solutions which is likelyto be suited for many porting taskings, depending on the actual real-timeconstraints imposed by the application.
126.96.36.199 . Partial rewriting using a real-time GNU/Linuxextension
Real-timeenabling GNU/Linux using RTAI. Strictly speakingLinux/RTAI  is not a real-time operating system but rather a real-time Linuxkernel extension, which allows running real-time tasks seamlessly aside of thehosting GNU/Linux system. The RTAI cokernel shares hardware interrupts andsystem-originated events like traps and faults with the Linux kernel using theAdeos [http://www.adeos.org/]virtualization layer, which in turn ensures RTAI low interrupt latencies. Thenative RTAI API provides the applications a wealth of useful services,including counting semaphores, POSIX 1003.1-1996 facilities such as pthreads,mutexes and condition variables, also adding remote procedure call facility, mailboxes,and precision timers. Most services are symmetrically available from kernelmodule and user-space programs.
RTAI 2.x and 3.x provide a means to executehard real-time tasks in user-space context (x86 only), but still outside theLinux kernel control, which is best described as running 'user-space kernelmodules'. This feature, namely LXRT, is a major step toward a simpler migrationpath from traditional RTOS, since programming errors occuring within real-timetasks don't jeopardize the overall GNU/Linux system sanity, at the expense of afew microseconds more latency.
RTAI 2.X 和3.x版本提供一种用户空间内上下文的硬实时任务机制，但是仍然是脱离linux内核的控制，最贴切的描述为’用户空间内内核模块’。这种特点，美其名曰：LXRT，是在进行从传统RTOS移植过程中的主要步骤，且实时任务的代码编程错误并不影响linux系统的完整性，但需要消耗一部分微秒级的延迟时间。
Adhoc services emulation. A first approach consistsin emulating each real-time facility needed by the application using acombination of the RTAI services. An ad hoc wrapping interface has to bewritten to support the needed function calls. The benefit of the wrappingapproach lies in the limited modifications made to the original code. However,some RTAI behaviours may not be compliant with the source operating system's.For the very same reason, conflicts between the emulated and native RTAIservices may occur in some way.
Ad hoc 服务仿真。第一种方法由每一个实时设备根据应用所需的，同时利用RTAI服务来实现仿真。Adhoc封装接口必须支持所需的函数调用。封装的好处在于，在一个有限的范围内进行原始代码的修改。但是，一些RTAI的行为可能并不是与源操作系统匹配。基于许多原因，存在许多方式在仿真与原始RTAI服务的之间的冲突可能会发生。
Complete port to RTAI. A second approach consists in fully porting the application overthe native RTAI API. In such a case, RTAI facilities are globally substitutedfrom the facilities from the source RTOS. This solution brings improvedconsistency at the expense of a possible large-scale rewriting of theapplication, due to some fundamental behavioural differences that may existbetween the traditional RTOS and the native RTAI interface.
188.8.131.52 . Unconstrained user-space emulations
A few traditional RTOS emulators exists inthe free software world. There are generally designed on top of the GNU/LinuxPOSIX 1003.1-1996 layer, and allow to emulate the source RTOS API in auser-space execution context, under the control of the Linux kernel.
在自由软件世界，存在一小部分的传统的RTOS模拟器。他们被设计在运行在GNU/Linux的 POSIX层之上，允许用户接口的RTOS API，并且受linuxkernel的控制。
Once one of the most proeminent effort inthis area used to be the Legacy2linux project . This project, sponsored byMontavista Software, aimed at providing ["a series of Linux-residentemulators for various legacy RTOS kernels."] Just like Xenomai, [ theseemulators are designed to ease the task of porting legacy RTOS code to anembedded Linux environment".] Two emulators have been made available bythis project, respectively mimicking the APIs of WindRiver's pSOS+ and VxWorksreal-time operating systems. However, this project has stalled due to a lack ofmaintenance and contribution.
The benefits of this approach is mainly tokeep the development process in the GNU/ Linux user-space environment, insteadof moving to a rather 'hostile' kernel/supervisor mode context. This way, therich set of existing tools such as debuggers, code profilers, and monitorsusable in this context are immediatly available to the application developer. Moreover, the standard GNU/Linux programming model is preserved,allowing the application to use the full set of of facilities existing in theuser space (e.g. full POSIX support, including inter-process communication).Last but not least, programming errors occuring in this context don'tjeopardize the overall GNU/Linux system stability, unlike what can happen if abug is encountered on behalf of a hard real-time RTAI task which could causeserious damages to the running Linux kernel.
However, we can see at least three problemsin using these emulators, depending on the application constraints:
• First, the emulated API they provide is usually incomplete for aneasy port from the source RTOS. In other words, only a limited syntacticcompatibility is available.
• Second, the exact behaviour of the source RTOS is not reproduced forall the functional areas. In other words, the semantic compatibility might notbe guaranteed.
• These emulators don't share anycommon code base for implementing the fundamental real-time behaviours, even soboth pSOS+ and VxWorks share most of them.The resulting situation leads to redundant implementation efforts, without anybenefit one can see in code mutualization.
• And finally, even combined to the latest Linux 2.6 features likefine-grain kernel preemption and low latency efforts, these emulators cannotdeliver deterministic real-time performance.
1.3 . A common emulation framework
1.3.1 . Common traditional RTOS behaviours
In order to build a generic and versatileframework for emulating traditional RTOS, we chose to concentrate on a set ofcommon behaviours they all exhibit. A limited set of specific RTOS featureswhich are not so common, but would be more efficiently implemented into thenucleus than into the emulators, has also been retained. The basic behavioursselected cover four distinct fields:
184.108.40.206 . Multi-threading
Multi-threading provides the fundamentalmechanism for an application to control and react to multiple, discrete externalevents. The nucleus provides the basic multi-threading environment.
Threadstates. The nucleus has to maintain the currentstate of each thread in the system. A state transition from one state toanother may occur as the result of specific nucleus services called by the RTOSemulator. The fundamental thread states defined by the nucleus are:
• DORMANT and SUSPENDED states are cumulative, meaning that the newlycreated thread will still remain in a suspended state after being resumed fromthe DORMANT state.
• DORMANT 和 SUSPENDED状态是累计的，意味着新创建的线程在从DORMANT恢复后仍然保留在一个suspended的状态。
• PENDING and SUSPENDED states are cumulative too, meaning that athread can be forcibly suspended by another thread or service routine whilepending on a synchronization resource (e.g. semaphore, message queue). In sucha case, the resource is dispatched to it, but it remains suspended untilexplicitly resumed by the proper nucleus service.
• PENDING 和SUSPENDED也是一样，意味着，一个线程在悬停一个同步资源时，可以被另外一个线程强制suspend，或者另外一个服务程序suspend。在在这种情况下，资源被获得后，仍然保留suspend状态，直到合适的微内核服务将其唤醒。
• PENDING and DELAYED states may be combined to express a timed waiton a resource. In such a case, the time the thread can be blocked is bound to alimit enforced by a watchdog.
• PENDING 和DELAYED状态组合起来，可能是表达一个对资源的时间上的等待。在这种情况下，线程会阻塞，会被一个看门狗限定到一个有限的时间内。
Schedulingpolicies. By default, threads are scheduledaccording to a fixed priority value, using a preemptive algorithm. There isalso a support for round-robin scheduling among a group of threads having thesame priority, allowing them to run during a given time slice, in rotation.Moreover, each thread undergoing the round-robin scheduling is given anindividual time quantum.
Prioritymanagement. It is possible to use either anincreasing or decreasing thread priority ordering, depending on an initialconfiguration. In other words, numerically higher priority values could eitherrepresent higher or lower scheduling priorities depending on the configurationchosen. This feature is motivated by the existence of this two possibleordering among traditional RTOS. For instance, VxWorks, VRTX, ThreadX and ChorusO/S use a reversed priority management scheme, where the higher the value, thelower the priority. pSOS+ instead uses the opposite ordering, in which thehigher the value, the higher the priority.
Runningthread. At any given time, the highest prioritythread which has been ready to run for the longest time among the currently runnablethreads (i.e. not currently blocked by any delay or resource wait) is electedto run by the scheduler.
Preemption. When preempted by a higher priority thread, the running thread isput at the front of the ready thread queue waiting for the processor resource,provided it has not been suspended or blocked in any way. Thus it is expectedto regain the processor resource as soon as no other higher priority activity(i.e. a thread having a higher priority level, or an interrupt service routine)is eligible for running.
Manualround-robin. As a side-effect of attempting toresume an already runnable thread or the running thread itself, this thread ismoved at the end of its priority group in the ready thread queue. Thisoperation is functionally equivalent to a manual round-robin scheduling.
Even if they are not as widespread as thoseabove in traditional RTOS, the following features are also retained for thesake of efficiency in the implementation of some emulators:
Priorityinversion. In order to provide support forpreventing priority inversion when using inter-thread synchronization services,the priority inheritance protocol is supported.
Signaling. A support for sending signals to threads and running asynchronousservice routines to process them is available. The asynchronous service routineis run on behalf of the signaled thread context the next time it returns fromthe nucleus level of execution, as soon as one or more signals are pending.
Disjunctivewait. A thread is able to wait in a disjunctivemanner on multiple resources. The nucleus unblocks the thread when at least oneof the pending resources is available.
220.127.116.11 . Thread synchronization
Traditional RTOS provide a large spectrumof inter-thread communication facilities involving thread synchronization, suchas semaphores, message queues, event flags or mailboxes.Lookingat them closely, we can define the characteristics of a basic mechanism whichwill be usable in turn to build these facilities.
Pending mode. The thread synchronization facility provides a means for threads topend either by priority or FIFO ordering. Multiple threads should be able topend on a single resource.
Priorityinheritance protocol. In order to prevent priorityinversion problems, the thread synchronization facility implements a priorityinheritance protocol in conjunction with the thread scheduler. Theimplementation allows for supporting the priority ceiling protocol as a derivativeof the priority inheritance protocol.
Time-boundedwait. The thread synchronization facility providesa means to limit the time a thread waits for a given resource using a watchdog.
Forcibledeletion. It is legal to destroy a resource whilethreads are pending on it. This action resumes all waiters atomically.
18.104.22.168 . Interrupt management
Since the handling ofinterrupts is one of the least well defined areas in RTOS design,the nucleus focuses on providing a simple mechanism with sufficienthooks for specific implementations to be built onto according to the emulatedRTOS flavour.
Nesting. Interrupt management code is reentrant in order to supportinterrupt nesting safely.
Atomicity. Interrupts are associated with dedicated service routines calledISRs. In order for these routines not to be preempted by thread execution, therescheduling procedure is locked until the outer ISR has exited (i.e. in caseof nested interrupts).
Priority. ISRs are always considered as priority over thread execution.Interrupt prioritization is left to the underlying hardware.
22.214.171.124 . Time management
Traditional RTOS usually represent time inunits of ticks. These are clock-specific time units and are usually the periodof the hardware timer interrupt, or a multiple thereof. Since it needs tosupport both periodic and aperiodic time sources, the nucleus transparentlyswitches from periodic jiffies to time-stamp counter values depending on thecurrent timer operating mode.
Softwaretimer support. A watchdog facility is provided tomanage time-bound operations by the nucleus.
Absoluteand relative clock. The nucleus keeps a globalclock value which can be set by the RTOS emulator as being the system-definedepoch.
Some RTOS like pSOS+ also provide supportfor date-based timing, but conversion of ticks into conventional time and dateunits is an uncommon need that should be taken in charge by the RTOS emulatoritself.
1.3.2 . An architecture-neutral abstraction layer
After having selected the basic behaviours shared by traditionalRTOS, we have implemented them in a nucleus exporting a few service classes.These generic services will then serve as a founding layer for developing eachemulated RTOS API, according to their own flavour and semantics.
In order for this layer to be architectureneutral, the needed support for hardware control and real-time capabilitieswill be obtained from an underlying host software architecture, through arather simple standardized interface. Thus, porting the nucleus to a newrealtime architecture will solely consist in implementing this low-levelinterface for the target platform.
1.3.3 . Real-time capabilities
The host software architecture is expected to provide the primaryreal-time capabilities to the RTOS abstraction layer. Basically, the hostreal-time layer must handle at least the following tasks:
• On request start/stop dispatching the external interrupts to aspecialized handler ;
• Provide a means to mask and unmask interrupts ;
• Provide a means to create new threads of control in their simplestform ;
• Provide support for periodic and aperiodic interrupt sources used intimer management ;
• Provide support for allocating chunks of non-pageable memory.
1.3.4 . Benefits
Xenomai aims at helping applicationdesigners relying on traditional RTOS to move as smoothly as possible to aGNU/Linux-based execution environment, without having to rewrite theirapplications entirely. Aside of the advantages of using GNU/Linux as anembedded system, the benefits expected from the described approach is mainly areduced complexity in designing new RTOS emulations. The architecture-neutralabstraction layer provides the foundation for developing accurate emulations oftraditional RTOS API, saving the burden of repeatedly implementing theirfundamental real-time behaviours. Since the abstraction layer also favours codesharing and mutualization, we can expect the RTOS emulations to take advantageof them in terms of code stability and reliability.
1.4 . Nucleus description
RTOS emulations are software modules whichconnect to the nucleus through the pod abstraction. The pod is responsible forthe critical housekeeping chores, and the real-time scheduling of threads.
1.4.1 . Multi-threading support
The nucleus provides thread object (xnthread) and pod (xnpod)abstractions which exhibit the following characteristics:
• Threads are scheduled according to a 32bit integer priority value,using a preemptive algorithm. Priority ordering can be increasing or decreasingdepending on the pod configuration.
• A thread can be either waiting for initialization, forciblysuspended, pending on a resource, delayed for a count of ticks, ready-to-run orrunning.
• Timed wait for a resource can be bounded by a per-thread watchdog.
• The priority inheritance protocol is supported to prevent threadpriority inversion when it is detected by a synchronization object.
• A group of threads having the same base priority can undergo around-robin scheduling, each of them being given an individual time quantum.
• A support for sending signals to threads and running asynchronousservice routines (ASR) to process them is built-in.
• FPU support can be optionally enabled or disabled for any thread atcreation time.
• Each thread can enter a disjunctive wait on multiple resources.
1.4.2 . Basic synchronization support
The nucleus provides a synchronizationobject abstraction (xnsynch) aimed atimplementing the common behaviour of RTOS resources, which has the followingcharacteristics:
• Support for the priority inheritance protocol, in order to preventpriority inversion problems. The implementation is shared with the schedulercode.
• Support for time-bounded wait and forcible deletion with waitersawakening.
1.4.3 . Timer and clock management
The nucleus needs a time source to providethe time-related services to the upper interfaces. The timer hardware needs tobe configured so that a user-defined routine is called according to a givenfrequency. On architectures that provide a oneshot-programmable time source,the system timer can operate either in aperiodic or periodic mode. Using theaperiodic mode still allows to run periodic nucleus timers over it: theunderlying hardware will simply be reprogrammed after each tick by the timermanager using the appropriate interval value.
Each incoming clock tick is announced tothe timer manager which fires in turn the timeout handlers of elapsed timers.The scheduler itself uses per-thread watchdogs to wake up threads undergoing abounded time wait, while waiting for a resource availability or being delayed.
A special care has been taken to offerbounded worst-case time for starting, stopping and maintaining timers. Thetimer facility is based on the timer wheel algorithm described by Adam M.Costello and George Varghese, which is implemented in the NetBSD operatingsystem for instance.
1.4.4 . Basic memory allocation
Xenomai's nucleus provides dynamic memory allocation support withreal-time guarantee, based on McKusick's and Karels' proposal for a generalpurpose memory allocator. Any number of memory heaps can be maintaineddynamically by Xenomai, only limited by the actual amount of system memory.