Debugging in real-time embedded system(1)

For the recent months, I was always focus on debugging in the real-time system Vxworks, and off course wasted too much time on some bugs. These bugs were tricky but all turned out to be simple after finding the root cause. There are different ways of debugging for the different types of software.Obviously, some popular debugging technique cannot applied to the real-time system, Because real-time system is special and mostly have restrictive time. This series of articles just discuss some technique on debugging in the real-time system.I'll try to make the technique independent of test environment and the test hardware. Normally, the software should print the debugging information and some developers would like to have different priority level of trace. But how to print out the debug information in real-time system? Can we print the messages to the standard output? and how about write the message to a log file? My experience is that we must not use the the standard function in c language "printf". This function will largely reduce the performance of embedded real time system.This is because the printf function will use I/O by invoking interrupt, this may delay other interrupt in the system. Some developers would like to log the debug information by sending the messages to remote server via socket. This will help to reduce the times of invoking interrupt,but this still have impact on the system performance. The way we used in our current project is to log the important variable in the memory. First define a struct that store your message,it is note that there is no memory hole left for the struct, the struct should conform to your requirement. For example:

 
typedef struct
{
unsigned short info1;
unsigned short info2;
unsigned int info3;
}log_message;

Second step is that define an array for that struct, the size of array is decided case by case. If is is too large,it may takes too much memory, if it too short, it can't contain enough debug information. It may looks like that:

#define MAX_OF_ARRAY 5000
log_message debug_info[MAX_OF_ARRAY];

Actually,5000 is not enough for most projects,some projects may have thousands of messages, 5000 is two small. So we need use this array as ring buffer. Then we need an index to mark the recent position of the debug message in this array. That is:

unsigned int index_logmessage;

And it is suggested to define a global variable (logflag) to control this mechanism, i.e when to start to log the debug info and when to stop log the debug info. Because most real-time embedded system could set the variable by O&M system. So, in the application, the array will be used like this:

if(logflag ==1){
     if(index_logmessage < MAX_OF_ARRAY)
     {
      debug_info[index_logmessage].info1= message1;
      debug_info[index_logmessage].info2= message2;
      debug_info[index_logmessage].info3= message3;
      index_logmessage++;
      }
}

This technique is important for the real-time embedded system that can't use "printf" and the developer want to see what happened to the software every several milliseconds. We used a lot of times in our project in this way. Maybe there are other methods to print debug information that we don't know yet?