1. Introduction
1. Purposeof this document
LinuxTest Project (LTP) is an open source project with a goal to deliver test suitesto the open source community that validate the reliability, robustness, andstability of Linux. The LTP test suite is a collection of automated andsemi-automated tests for testing various aspects of the Linux operating system.The dominant programming language in LTP is ANSI-C (94%) followed by Bash (5%)and Perl (0.62%).
Thisdocument has several goals. First, it provides a cursory view of the contentsof LTP test suite. Second, it introduces a template for writing test-cases inANSI-C and Shell. Finally, it enumerates the steps that must to be followedinorder to add a new test cases to LTP. The document reflects the authors viewson code layout and coding practices.
2. Feedbackand corrections
In aneffort to keep this document current and useful, any form of feedback isgreatly appreciated. Feel free to point out any mistakes in this document orcontribute additional information. Please e-mail shubham@linux.vnet.ibm.com so that these changes can be made available in the nextversion.
2. RevisionHistory
| Version | Author | Date | Comments |
| 1.0 | Manoj Iyer | Jan 17 2003 | Initial Draft |
| 1.1 | Shubham Goyal | Apr 28 2012 | Updated Various URL's and other documentation |
3. BeforeYou Begin
Thissection will provide a cursory view of the contents of the LTP test suite, andalso provide instructions on downloading, compiling and executing the testsuite.
1. Downloadingthe LTP test suite
The stable version of the LTP test suite source,ltp-yyyymmdd.tgz is released on a monthly basis. The test suite may be used toverify stable and development release of the Linux kernel. The test suite isalso available in binary and source RPM format. The package ltp-yyyymmdd.tgzcan be downloaded from the LTPweb-site using a web browser or using the 'wget' command.
wget -chttp://prdownloads.sourceforge.net/ltp/ltp-full-yyyymmdd.tgz?download (Replaceyyyymmdd with appropriate values)
You can download the latest CVS snapshot of the LTP testsuite source via anonymous CVS access as follows.
cvs-d:pserver:anonymous@ltp.cvs.sourceforge.net:/cvsroot/ltp login
cvs -z3-d:pserver:anonymous@ltp.cvs.sourceforge.net:/cvsroot/ltp co ltp
Use the following to download without the CVS (directory)getting created in each subdirectory
cvs-d:pserver:anonymous@ltp.cvs.sourceforge.net:/cvsroot/ltp login
cvs -z3-d:pserver:anonymous@ltp.cvs.sourceforge.net:/cvsroot/ltp export -D "MONTHDAY, YEAR HH:MMam/pm" ltp
2. Contentsof the LTP test suite
Thetop level directory or LTP rootdirectory ./ltp-yyyymmdd contains SHELL scripts and various sourcedirectories. These scripts may be used to execute the test cases, the scriptsexecute the test cases using the test driver called pan.
Scripts
The test suite requires that certain users and groups(For example User:nobody Group:nobody.) exist on the system. This script willcreate these users and groups. This script is called from the top levelMakefile located in the LTP root directory.
This driver script runs a majority of tests in the LTP.It can also be easily modified to run a custom list of tests if required.
By default, this script will execute:
§ filesystemstress tests
§ disk I/Otests
§ memorymanagement stress tests
§ ipcstress
§ schedulertests
§ commandsfunctional varification tests
§ systemcall functional varification tests
The script is named runalltest.sh but it does not run allthe tests in the LTP.
This script executes the networking related tests. Beforeexecuting this script, two environment variables must be set RHOST (Remote HostName) and PASSWD (root password of the remote host).
This script executes tests related to floppy and CD-ROMdrives. These tests require that a floppy disk and CD-ROM media be present inthe respective drives.
Directories
This directory contains the source code for all the testcases. The test cases are located under directories named after the subsystemor service they are intended to test. For example, memory management tests arelocated under testcases/mem etc.
This directory contains command files that are used by pan to run the tests. Each file contains the name of the testcase "tag" and the "command" with arguments, if any, thatmust be executed to perform the test. For example, the file"syscalls" contains entries as follows.
| # Name of test Command access01 access01 alarm04 cp -f $LTPROOT/test-cases/bin/sig_rev $TMP;alarm04 chdir01A symlink01 -T chdir01; chdir01 |
This directory contains source code for the LTP-harnessAPIs (libltp.a). The libltp.a library gets linked with any test-case that usesthese API functions to log test results and status.
This directory contains header files that are included inany test-case that uses the LTP API-harness functions.
/opt/ltp/testcases/bin
This directory contains the test executables, the testcases get installed in this directory by typing "make install".
3. Compilingthe test suite
Tocompile the test suite change directory to ltp-yyyymmdd and type"./configure" followed by "make". To install the testsuite, type "make install" as root user. This will create symboliclinks of the executable to corresponding files in ltp-yyyymmdd/ test-cases/bindirectory. The test scripts (runltp for example) execute the binaries presentin this directory based on the tests listed in its command file.
Note that, "make install" can be run as non-root, but IDcheck.sh mustbe run separately before executing any test cases as root.
4. Executingthe test suite
Currentlythere are three scripts that can be used to execute test cases in the LTP testsuite, namely, runltp , networktests.sh and diskio.sh. The test cases and thetest driver "pan", require certain environment variables to be setand certain directories to be present in order function correctly. Thesescripts take care of such pre-requisites. The scripts invoke the test driver"pan" with certain parameters and pan executes the tests and logstest results. The parameters that are be passed to pan can be controlled byuser provided options to the scripts.
The most frequently used script, runltp , currently provides the followingoptions. If no options are specified while executing this script, the defaultbehaviour is as follows:
§ Run asingle iteration of all the default set of tests
§ Verboseoutput of of test-case status on screen
§ Doesnot produce any log file or test results by default
|
-a EMAIL_TO EMAIL all your Reports to this E-mail Address -c NUM_PROCS Run LTP under additional background CPU load [NUM_PROCS = no. of processes creating the CPU Load by spinning over sqrt() (Defaults to 1 when value)] -C FAILCMDFILE Command file with all failed test cases. -d TMPDIR Directory where temporary files will be created. -D NUM_PROCS,NUM_FILES,NUM_BYTES,CLEAN_FLAG Run LTP under additional background Load on Secondary Storage (Seperate by comma) [NUM_PROCS = no. of processes creating Storage Load by spinning over write()] [NUM_FILES = Write() to these many files (Defaults to 1 when value 0 or undefined)] [NUM_BYTES = write these many bytes (defaults to 1GB, when value 0 or undefined)] [CLEAN_FLAG = unlink file to which random data written, when value 1] -e Prints the date of the current LTP release -f CMDFILES Execute user defined list of testcases (separate with ',') -g HTMLFILE Create an additional HTML output format -h Help. Prints all available options. -i NUM_PROCS Run LTP under additional background Load on IO Bus [NUM_PROCS = no. of processes creating IO Bus Load by spinning over sync()] -l LOGFILE Log results of test in a logfile. -m NUM_PROCS,CHUNKS,BYTES,HANGUP_FLAG Run LTP under additional background Load on Main memory (Seperate by comma) [NUM_PROCS = no. of processes creating main Memory Load by spinning over malloc()] [CHUNKS = malloc these many chunks (default is 1 when value 0 or undefined)] [BYTES = malloc CHUNKS of BYTES bytes (default is 256MB when value 0 or undefined) ] [HANGUP_FLAG = hang in a sleep loop after memory allocated, when value 1] -N Run all the networking tests. -n Run LTP with network traffic in background. -o OUTPUTFILE Redirect test output to a file. -p Human readable format logfiles. -q Print less verbose output to screen. -r LTPROOT Fully qualified path where testsuite is installed. -s PATTERN Only run test cases which match PATTERN. -t DURATION Execute the testsuite for given duration. Examples: -t 60s = 60 seconds -t 45m = 45 minutes -t 24h = 24 hours -t 2d = 2 days -T REPETITION Execute the testsuite for REPETITION no. of times -v Print more verbose output to screen. -w CMDFILEADDR Uses wget to get the user's list of testcases. -x INSTANCES Run multiple instances of this testsuite.
|
Execuing default set of tests
If you wish to execute the default set of test case, you can use the followingcommands.
§ Changedirectory to /opt/ltp/
cd/opt/ltp/
§ Executerunltp script with the following options, or chose the options that are bestsuited for you from the list above. ./runltp -p -l result.yyyymmdd.log
Executing user defined set of tests
If you wish to run a select set of test cases, you have to do the following.
§ Create a command file. The command file contains the nameof the test case "tag" and the "test case" with arguments,if any, that must be executed to perform the test. For example.
| #Tag Test case |
§ Changeto the directory where LTP is installed i.e. /opt/ltp/
cd/opt/ltp/
§ Executetests in command file using runltp . You can execute runltp with the -f command file option to execute these tests.
./runltp-p -l result.02.log -f my_command_file
5. Contentsof output to screen and logfile
Outputto screen
The test cases, test driver, and scripts print various informational messagesto the screen. The test cases use the USCTEST functions to print information,the test driver "pan", prints tags that separate output from one testto another. Depending on the options that you may choose while running thescripts, the verbosity of these messages can be adjusted. For example.
§ Screen output of the command runltp -p -lresults.03.log -f my_command_file
| If some fields are empty or look unusual you may have an old version. Compare to the current minimal requirements in Documentation/Changes.
/etc/issue: Debian GNU/\s testing/unstable \n \l
Linux lazy 2.4.19 #1 Mon Aug 26 14:25:41 CDT 2002 i686 unknown
Gnu C 2.95.4 Gnu make 3.80 util-linux 2.11x mount 2.11x modutils 2.4.21 e2fsprogs 1.32 Linux C Library 2.3.1 Dynamic linker (ldd) 2.3.1 Procps 3.1.1 Net-tools 1.60 Kbd 66: Sh-utils 2.0.11 Modules Loaded soundcore uhci usbcore vfat fat free -m reports: total used free shared buffers cached Mem: 249 211 37 0 4 135 -/+ buffers/cache: 71 178 Swap: 258 49 208
/proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 8 model name : Pentium III (Coppermine) stepping : 10 cpu MHz : 863.881 cache size : 256 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse bogomips : 1723.59
t_start>>> tag=mtest01 stime=1043946397 cmdline=" mtest01" contacts="" analysis=exit initiation_status="ok" (((test_output))) stopped at 3076521984 bytes (((execution_status))) duration=0 termination_type=exited termination_id=1 corefile=no cutime=1 cstime=9 (((test_end)))
INFO: run mmstress -h for all options
test1: Test case tests the race condition between simultaneous read faults in the same address space. (((test_start))) tag=mmstress stime=1043946397 cmdline=" mmstress " contacts="" analysis=exit initiation_status="ok" (((test_output))) test2: Test case tests the race condition between simultaneous write faults in the same address space. test3: Test case tests the race condition between simultaneous COW faults in the same address space. test4: Test case tests the race condition between simultaneous READ faults in the same address space. The file maped is /dev/zero test5: Test case tests the race condition between simultaneous fork - exit faults in the same address space.
more messages here skipped.
(((execution_status))) duration=41 termination_type=exited termination_id=0 corefile=no cutime=605 cstime=3443 (((test_end))) (((test_start))) tag=fork01 stime=1043946438 cmdline=" fork01" contacts="" analysis=exit initiation_status="ok" (((test_output))) fork01 1 PASS : fork() returned 11825 fork01 2 PASS : child pid and fork() return agree: 11825 (((execution_status))) duration=0 termination_type=exited termination_id=0 corefile=no cutime=0 cstime=0 (((test_end))) (((test_start))) tag=chdir01A stime=1043946438 cmdline=" symlink01 -T chdir01" contacts="" analysis=exit initiation_status="ok" (((test_output))) incrementing stop chdir01 1 PASS : chdir(2) to object file location through symbolic link file is ok chdir01 2 PASS : chdir(2) to non-existent object file location through symbolic link file failed as expected chdir01 3 PASS : Nested symbolic link access condition caught. ELOOP is returned (((execution_status))) duration=0 termination_type=exited termination_id=0 corefile=no cutime=0 cstime=1 (((test_end))) pan reported FAIL
###############################################################
result log is in the /home/manjo/cvs_tree/ltp/results directory
###############################################################
|
§ Screen output of the command runltp -q -p -lresult.04.log -f my_command_file (Providing the -q option to runlt runltp will supress certain messages).
| If some fields are empty or look unusual you may have an old version. Compare to the current minimal requirements in Documentation/Changes.
/etc/issue: Debian GNU/\s testing/unstable \n \l
Linux lazy 2.4.19 #1 Mon Aug 26 14:25:41 CDT 2002 i686 unknown
Gnu C 2.95.4 Gnu make 3.80 util-linux 2.11x mount 2.11x modutils 2.4.21 e2fsprogs 1.32 Linux C Library 2.3.1 Dynamic linker (ldd) 2.3.1 Procps 3.1.1 Net-tools 1.60 Kbd 66: Sh-utils 2.0.11 Modules Loaded soundcore uhci usbcore vfat fat
free -m reports: total used free shared buffers cached Mem: 249 212 37 0 4 134 -/+ buffers/cache: 73 175 Swap: 258 49 208
more messages here skipped.
test1: Test case tests the race condition between simultaneous read faults in the same address space. test2: Test case tests the race condition between simultaneous write faults in the same address space. test3: Test case tests the race condition between simultaneous COW faults in the same address space. test4: Test case tests the race condition between simultaneous READ faults in the same address space. The file maped is /dev/zero test5: Test case tests the race condition between simultaneous fork - exit faults in the same address space. test6: Test case tests the race condition between simultaneous fork -exec - exit faults in the same address space. fork01 1 PASS : fork() returned 12156 fork01 2 PASS : child pid and fork() return agree: 12156 chdir01 1 PASS : chdir(2) to object file location through symbolic link file is ok chdir01 2 PASS : chdir(2) to non-existent object file location through symbolic link file failed as expected chdir01 3 PASS : Nested symbolic link access condition caught. ELOOP is returned pan reported FAIL
###############################################################
result log is in the /home/manjo/cvs_tree/ltp/results directory
###############################################################
|
Output to log file
The test driver pan uses the exit value of the test case to decide success orfailure of a test. If the test case exits with a non-zero value pan recordsthis as FAIL, if the test case exits with a value zero pan records this asPASS. The test driver "pan" generates log files, these logfilescontain results from executing the test case. You can provide the -l logfile_name option to runalltest.sh script in order to log results.If the use does not provide an absolute path, the file is created under theresults directory. Currently, the result log file is generated in two formats.
§ Human readable format
You can generate a logfile in this format by providing runltp with the -p option. This generates a log file in ASCII format, forexample.
| Test Start Time: Thu Jan 30 11:06:37 2003 ----------------------------------------- Testcase Result Exit Value -------- ------ ---------- mtest01 FAIL 1 mmstress PASS 0 fork01 PASS 0 chdir01A PASS 0 ----------------------------------------------- Total Tests: 4 Total Failures: 1 Kernel Version: 2.4.19 Machine Architecture: i686 Hostname: lazy |
§ Script readable format
This is the default format and may be used by scripts that are designed togenerate test result analysis reports. The file generated in ASCII format. Forexample.
| tag=mtest01 stime=1043947511 dur=0 exit=exited stat=1 core=no cu=0 cs=11 tag=mmstress stime=1043947511 dur=41 exit=exited stat=0 core=no cu=671 cs=3383 tag=fork01 stime=1043947552 dur=0 exit=exited stat=0 core=no cu=0 cs=1 tag=chdir01A stime=1043947552 dur=0 exit=exited stat=0 core=no cu=0 cs=0 |
4. Developingtests in ANSI-C using LTP test harness API's
At thetime of this writing, LTP has 156,775 physical source lines of code. The most dominantprogramming language is ANSI-C (94.43%) followed by shell (4.60%) and Perl(0.62%). This section provides an introduction to developing new test cases inANSI-C using LTP test harness APIs.
1. Basictemplate
Thissection provides a basic template that may be used to develop test-cases inANSI-C using LTP harness APIs. Please note that the following code will notcompile as is.
| /******************************************************************************/ /* */ /* Copyright (c) International Business Machines Corp., 2001 */ /* */ /* This program is free software; you can redistribute it and/or modify */ /* it under the terms of the GNU General Public License as published by */ /* the Free Software Foundation; either version 2 of the License, or */ /* (at your option) any later version. */ /* */ /* This program is distributed in the hope that it will be useful, */ /* but WITHOUT ANY WARRANTY; without even the implied warranty of */ /* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See */ /* the GNU General Public License for more details. */ /* */ /* You should have received a copy of the GNU General Public License */ /* along with this program; if not, write to the Free Software */ /* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ /* */ /******************************************************************************/
/******************************************************************************/ /* */ /* File: template.c */ /* */ /* Description: This is a template that can be used to develop new test-cases */ /* using the LTP-harness APIs. As an example a dummy_syscall() */ /* system call will be used as an example. */ /* */ /* Total Tests: 1 */ /* */ /* Test Name: dummy_syscall_test01 */ /* */ /* Test Assertion */ /* amp; Strategy: A brief description of the test Assertion and Strategy */ /* */ /* Author: Manoj Iyer manjo@austin.ibm.com */ /* */ /* History: Created - Jan 19 2003 - Manoj Iyer manjo@austin.ibm.com */ /* Added - Jan 20 2003 - Manoj Iyer, comments describing the */ /* the functions used by the test-case. */ /* */ /******************************************************************************/
/* Standard Include Files */ #include <stdio.h> #include <stdlib.h> #include <errno.h>
/* Harness Specific Include Files. */ #include "test.h" #include "usctest.h"
/* Local Defines */ #if !defined(TRUE) amp;amp; !defined(FALSE) #define TRUE 1 #define FALSE 0 #endif
/* Extern Global Variables */ extern int Tst_count; /* counter for tst_xxx routines. */ extern char *TESTDIR; /* temporary dir created by tst_tmpdir() */
/* Global Variables */ char *TCID = "dummy_syscall_test01"; /* test program identifier. */ int TST_TOTAL = 1; /* total number of tests in this file. */
/* Extern Global Functions */ /******************************************************************************/ /* */ /* Function: cleanup */ /* */ /* Description: Performs all one time clean up for this test on successful */ /* completion, premature exit or failure. Closes all temporary */ /* files, removes all temporary directories exits the test with */ /* appropriate return code by calling tst_exit() function. */ /* */ /* Input: None. */ /* */ /* Output: None. */ /* */ /* Return: On failure - Exits calling tst_exit(). Non '0' return code. */ /* On success - Exits calling tst_exit(). With '0' return code. */ /* */ /******************************************************************************/ extern void cleanup() { /* Close all open file descriptors. */
Insert code here. In case an unexpected failure occurs report it and exit setup(), the following lines of code will demonstrate this.
if (close(fd) == -1) { tst_resm(TWARN, "close(%s) Failed, errno=%d : %s", fname, errno, strerror(errno)); }
/* Remove all temporary directories used by this test. */
Insert real code here
/* kill child processes if any. */
Insert code here
/* Exit with appropriate return code. */
tst_exit(); }
/* Local Functions */ /******************************************************************************/ /* */ /* Function: setup */ /* */ /* Description: Performs all one time setup for this test. This function is */ /* typically used to capture signals, create temporary dirs */ /* and temporary files that may be used in the course of this */ /* test. */ /* */ /* Input: None. */ /* */ /* Output: None. */ /* */ /* Return: On failure - Exits by calling cleanup(). */ /* On success - returns 0. */ /* */ /******************************************************************************/ void setup() { /* Capture signals. */
Insert code here
/* Create temporary directories */
Insert code here
/* Create temporary files. */
Insert real code here. In case an unexpected failure occurs report it and exit setup(). Follow the code below for example.
if ((fd = open(fname, O_RDWR|O_CREAT, 0700)) == -1 ) { tst_brkm(TBROK, cleanup, "Unable to open %s for read/write. Error:%d, %s\n", fname, errno, strerror(errno)); }
return; }
/******************************************************************************/ /* */ /* Function: main */ /* */ /* Description: Entry point to this test-case. It parses all the command line */ /* inputs, calls the global setup and executes the test. It logs */ /* the test status and results appropriately using the LTP API's */ /* On successful completion or premature failure, cleanup() func */ /* is called and test exits with an appropriate return code. */ /* */ /* Input: Describe input arguments to this test-case */ /* -l - Number of iteration */ /* -v - Prints verbose output */ /* -V - Prints the version number */ /* */ /* Exit: On failure - Exits by calling cleanup(). */ /* On success - exits with 0 exit value. */ /* */ /******************************************************************************/ int main(int argc, /* number of command line parameters */ char **argv) /* pointer to the array of the command line parameters. */ {
/* parse options. */
LTP test harness provides a function called parse_opts() that may be used to parse standard options. For a list of standard option that are available refer on-line man page at the LTP web-site
/* perform global test setup, call setup() function. */ setup();
/* Print test Assertion using tst_resm() function with argument TINFO. */ tst_resm(TINFO, "Testing if dummy_syscall() function works\n");
/* Test Case Body. */
Insert real code goes here. In case of unexpected failure, or failure not directly related to the entity being tested report using tst_brk() or tst_brkm() functions.
if (something bad happened) { tst_brkm(TBROK, cleanup, "Unable to open test file %s\n", fname); }
/* print results and exit test-case */ if(results are favorable) tst_resm(TPASS, "dummy_syscall worked as designed\n"); else tst_resm(TFAIL, "dummy_syscall failed to work as designed\n");
tst_exit() } |
2. Understandingthe template
Copyrights and license statement
Each test-case starts with a copyright and licensestatement. All test-cases in LTP are covered under the GNU public license . Copyrights usually belong to the author or corporationthat developed the tests. Test case developed by IBM will have IBM copyrights,and test cases developed by SGI will have SGI copyrights and so on.
File header
The file header contains information contained in thetest case file under five main categories, these are as follows.
§ File:Name of the test case file. For example test01.c
§ Description:A brief description of the contents of this file, explains what this filecontains, namely, a test case to test functionality of certain system call etc.
§ TotalTests: Total number of test cases in this file.
§ TestName: Name of the test case
§ TestAssertion amp; Strategy: A brief description of the test Assertion andStrategy.
§ History:Date of creation or modification, Author and email address. Followed bycomments that describes the change.
For example: Created - Jan 19 2003 - Manoj Iyer manjo@austin.ibm.com
Standard include and header files
Theseinclude files must be based on recommendations made in the man pages ofstandard C library (libc) APIs that are used by the test case. Standard includefiles are followed by LTP harness specific header files and any other localheader file that the test might use.
Global variable definitions
Includeany global definitions, external global variables, and variables that areglobal to this test case. The variables Tst_count, TST_TOTAL and TCID are usedby the harness APIs for reporting status and test results. These are theminimum set of variables that must be defined in order to use the basicfunctions (tst_brk, tst_brkm, tst_res, tst_resm and tst_exit) to report teststatus.
Global function definitions
Thevariable definition is followed by global function definitions, it isrecommended that the developer define at least one function, namely, cleanup()and optionally a setup() as global extern function. The cleanup() function mustbe supplied to tst_brk() routine, this routine is used to report premature testfailure. The cleanup() function will call the tst_exit() function which willexit with appropriate exit values.
main()
Function definitions are followed by main(), the entrypoint to the test case. The activities expected from main() can be summarizedas follows:
§ parseinput options.
§ callsetup function or perform tasks that are a pre-requisite to perform tests.
§ printthe test assertion.
§ performthe test.
§ printtest results.
§ cleanup and exit with appropriate exit code.
TheLTP harness library provides certain macros and functions in an attempt tosimplify some of the above activities or help manage results and test statuseasily. For instance, The USCTEST macros and functions mightbe helpful for testing system calls. Below is an example of such a function.
The parse_opt() function, for instance, may be used to parse standard (orbuilt-in) options or user defined options to the test case. It supports a setof pre-defined standard options with the flexibility of adding more userdefined options.
| char * parse_opts(int argc, char *argv[], option_t option_array[], void (*user_help_func)());
where option_t is:
typedef struct { char *option; int *flag; char **arg; } option_t; |
User defined options may be specified in the option_array argument. A help() orusage() function may be specified in the user_help_func argument.
Although using these macros and functions are left to the developer'sdiscretion, the use of tst_resm/tst_res, tst_brk/tst_brkm,tst_brkloop/tst_brkloopm, and tst_exit functions is highly recommended. Fordetails on usage and syntax please refer man pages on(http://ltp.sourceforge.net) the LTP website.
The tst_brk/tst_brkm functions are typically used in a situation where a testcase breaks as a result of something other than the entity that is being testedfails. The tst_resm/tst_res functions may be used to print various messagedepending on the result type passed to it as argument. The result type may beone of the following.
TINFO
Print informational messages that report test executionstatus, test assertion , or other informational message. For example.
test01 0 INFO : Thistest will test the uname command.
TPASS
Print the test case result. The tst_resm/tst_res functionprints related messages to the screen indicating a test success. The test casewill exit with a zero exit value when tst_exit function is called.
test01 0 PASS : uname command works as expected.
TFAIL
Print the test case result. The tst_resm/tst_res functionprints related messages to the screen indicating a test failure. The test willexit with a non-zero exit value when tst_exit function is called.
test01 0 PASS : uname command exited with error code = 255
TWARN
Print warning messages. For example, when systemresources are low.
test01 0 PASS : System low on memory, test case might fail.
3. Recommendationon coding style
Thissection includes recommendations on code layout and style. They are optional,it reflects the authors personal coding style and practice and may beapplicable in general to any program written in ANSI-C.
§ Use afile header to describe the contents of a file and maintain a change history.CVS provides the option to maintain a change log while checking in code intothe CVS tree, but often times the developer need not be responsible formaintaining the code in CVS. The change log or history maintained in the filecan help both the developer and the maintainer of the code to track changesmade to the file.
§ Usemeaningful variable names. Avoid variable names like "i";"j", instead use variable names like "index","array_ndx" etc. Also initialize and comment the variablesappropriately. For example a for loop counter may be defined as follows:
int loop_cntr = 0; /* counter to keep track of number of iterations */
instead of defining the same as:
int j;
§ Commentcode appropriately. Code might be handled by seasoned programmers as well asnewbies. It is a good practice to add comments in the code where everappropriate for maintainance reasons. Comments can be stripped off from thecode using AWK or PERL scripts if required.
§ Use"/* --- */" style for comments, avoid using the "//" C++style comments.
§ UseANSI-C style function names instead of C++ style function names, for example
create_file()
instead of
CreateFile()
Declare functions local to the file as"static". Start the function names in a new line, and each input oroutput variable in separate lines with appropriate comments. For example:
| static int create_tmpfile(char *fname, /* name of temporary file created by test01 */ int flags, /* describes file as read, write only etc */ mode_t mode) /* file open modes, S_IRWXO etc. */ { } |
§ Use afunction prologue. The prologue should describe the function, its input andoutput variables if any, and exit or return values if any. Using a functionprologue is a good practice that makes code readable and maintainable. Refer tothe template for an example.
§ Checkwith man pages for #include <include file=""/> recommendationwhen using standard C library (libc) functions. Avoid using unnecessary headerfiles.
§ Checkfor return values from system calls. Report reason appropriately using thetst_brkm() API. The recommended style is as follows:
| if ((ret = open(filename, O_RDWR|O_CREAT, S_IRWXO|S_IRWXU|S_IRWXG)) == -1) { tst_brkm(TBROK, cleanup, "open(): Failed %s\n", strerror(errno)); } else{} |
§ Usetst_resm(TINFO, ...) to print informational messages as appropriate,alternately, a verbose option or a 'quiet' option may be provided to suppressor activate such messages.
§ Usespaces instead of tab characters. Typically four spaces may be used instead ofa tab character.
§ Compilecode using the "-Wall" option. Using this option, the compiler willgenerate more warning than when compiled without "-Wall". Treat allwarnings generated by the compiler as errors and try to minimize them.
4. More help
LTPtest suite has over a thousand test cases that can be studied in order tounderstand the use of the LTP APIs. For more information on syntax and usagefollow the "LTP Man Pages" link on the online man page located athttp://ltp.sourceforge.net under "Documentation" section. You canfind man pages for the LTP APIs, USCTEST macros and test driver"pan", in the ltp-yyyymmdd/doc/ directory.
5. Developingtests in Bash using the LTP command line harness interface
Limitednumber of test in the LTP test suite are written in Shell scripts (4.6%).Currently, the focus of the LTP test suite is the reliablity and stability ofthe Linux kernel. However, there is increasing interest in writing tests forother parts of the Linux operating system (for example deamons, services,etc.). It might be required to develop these tests in a language other thanANSI-C.(for example BASH script or AWK script). This section will provide atemplate for developing new test cases in SHELL using the command line harnessAPIs.
A minimum set of LTP harness APIs are available in the form of commands. Thesecommands help developers log test status and test results in a format similarto that done by tests in ANSI-C. The source code for these commands are locatedin the ltp-yyyymmdd/tools/apicmds/ directory. Once the commands are compiledand installed the test-cases reside under ltp-yyyymmdd/testcases/bin directory.
1. Basictemplate
Thissection will introduces a basic template that can be used to develop test casesin SHELL. Please note that the follwing code will fail to execute as is.
| ################################################################################ ## ## ## Copyright (c) International Business Machines Corp., 2001 ## ## ## ## This program is free software; you can redistribute it and#or modify ## ## it under the terms of the GNU General Public License as published by ## ## the Free Software Foundation; either version 2 of the License, or ## ## (at your option) any later version. ## ## ## ## This program is distributed in the hope that it will be useful, but ## ## WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY ## ## or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License ## ## for more details. ## ## ## ## You should have received a copy of the GNU General Public License ## ## along with this program; if not, write to the Free Software ## ## Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA ## ## ## ################################################################################ # # File : template.sh # # Description: This is a template that can be used to develop new test cases # using the LTP-harness command APIs. As an example a dummy_cmd # is used in this file. # # Author: Manoj Iyer, manjo@mail.utexas.edu # # History: Jan 19 2003 - Created - Manoj Iyer. # Jan 20 2003 - Added - Manoj Iyer, Test #1 # #! /bin/sh
# Function: setup # # Description: - Check if required commands exits # - Export global variables # - Check if required config files exits # - Create temporary files and directories # # Return - zero on success # - non zero on failure. return value from commands ($RC) setup() {
This variable is required by the LTP command line harness APIs to keep track of the number of testcases in this file. This variable is not local to this file.
export TST_TOTAL=1 # Total number of test cases in this file.
# Set up LTPTMP (temporary directory used by the tests).
Export PATH variable to point to the ltp-yyyymmdd/testcases/bin directory before running the tests manually, path is set up automatically when you use the runltp script. LTPTMP is user defined variables used only in this test case. These variables are local to this file. The TMP variable is exported by the runltp script.
LTPTMP=${TMP} # Temporary directory to create files, etc.
The TCID and TST_COUNT variables are required by the LTP command line harness APIs, these variables are not local to this program.
export TCID="setup" # Test case identifier export TST_COUNT=0 # Set up is initialized as test 0
# Initialize cleanup function to execute on program exit. # This function will be called before the test program exits. trap "cleanup" 0
# Initialize return code to zero.
RC is used to store the return code from commands that are used in this test. This variable is local to this file.
RC=0 # Exit values of system commands used
# Check if awk command exists which awk amp;>$LTPTMP/tst_template.out || RC=$? if [ $RC -ne 0 ] then tst_brkm TBROK NULL "INIT: Command awk not found" return $RC fi
# Check if temp_user exists. If not add one. RC=$(awk '/^temp_user/ {print 1}' /etc/passwd) if [ -z $RC ] then RC=0 tst_resm TINFO "INIT: Adding temporary user temp_user" useradd -m -s /bin/bash temp_user amp;>$LTPTMP/tst_template.out || RC=$? if [ $RC -ne 0 ] then tst_brk TBROK $LTPTMP/tst_template.out NULL \ "Test INIT: Failed adding user temp_user Reason:" return $RC fi fi return $RC }
# Function: cleanup # # Description - remove temporary files and directories. # # Return - zero on success # - non zero on failure. return value from commands ($RC) cleanup() { Clean up code goes here }
# Function: test01 # Description - Test if command dummy_cmd -f tst_file.in will print # contents of tst_file.in # - Create input file tst_file.in # - test01() { TCID="test01" # Identifier of this testcase. TST_COUNT=1 # Test case number. RC=0 # return code from commands.
In case of failure your testcase MUST exit with a non-zero value. This exit value will be used by pan, the test driver, to report FAIL. If the testcase exits with a value zero, pan reports PASS. This variable is local to this file.
# Print test assertion. tst_resm TINFO \ "Test #1: dummy_cmd -f tst_file.in will print contents of tst_file.in"
# Create input file tst_file.in cat amp;gt $LTPTMP/tst_file.in amp;ltamp;lt -EOF || RC=$? This is a test file EOF if [ $RC -ne 0 ] then tst_brkm TBROK NULL "Test #1: Unable to create input file" return $RC fi
# Perform test by calling command dummy_cmd with -f option # and tst_file.in as input.
Insert real code here. In case something wicked happenes use tst_brk or tst_brkm functions to report this for example, follow the lines of code below.
# Print test results and exit with appropriate value # Return code RC contains the value returned by dummy_cmd # zero on success non-zero on failure. if [ $RC -ne 0 ] then tst_res TFAIL $LTPTMP/tst_file.out \ "Test #1: dummy_cmd failed to print input file. Reason:" return $RC else tst_resm TPASS \ "Test #1: succeeded in printing the contents of input file." fi return $RC }
# Function: main # # Description: - Execute all tests, exit with test status. # # Exit: - zero on success # - non-zero on failure. # RC=0 # Return value from setup, and test functions.
setup || exit $RC
test01 || exit $RC |
2. Understandingthe template
Each test-case starts with a copyright and licensestatement. All test-cases in LTP are covered under the GNU public license . Copyrights usually belong to the author or corporationthat developed the tests. Test case developed by IBM will have IBM copyrights,and test cases developed by SGI will have SGI copyrights and so on.
The structure of the template is similar to tests written in ANSI-C. There arecertain important points to note while writing tests in SHELL using the LTPcommand-line APIs. These are summarized below.
§ exportTST_TOTAL="totalnumber of tests in this file". Thisvariable must be exported. It is used by the LTP command level APIs to displaystatus or results.
§ exportTCID="nameof the test case". Thisvariable must be exported. It is used by the LTP command-line APIs to reportthe name of the test while printing test status or results.
§ TST_COUNT="current test casenumber" If the test file has more than one test, each test mustset this to an appropriate value. For instance, third test (test #3) should setthis variable to 3, (TST_COUNT=3). The variable is used by the command levelAPIs for reporting status or results.
§ Thearguments to command APIs are in conformance with ANSI-C APIs. The FUNCargument to the commands is ignored, but a value must be provided forconformance with ANSI-C APIs (use NULL). The arguments to commands are asfollows:
§ export TCID=test name
§ export TST_TOTAL=Total Number of Tests
§ export TST_COUNT=Test case number
§
§ tst_brk TTYPE FNAME FUNC STRING
§ TTYPE -Test Result Type; one of TFAIL, TBROK, TCONF, and TRETR.
§ FNAME -Print contents of this file after the message
§ FUNC -Cleanup function (ignored), but MUST be provided. Use NULL.
§ STRING - Message explaining the test result
§
§ tst_brkm TTYPE FUNC STRING
§ TTYPE -Test Result Type; one of TFAIL, TBROK, TCONF, and TRETR.
§ FUNC -Cleanup function (ignored), but MUST be provided. Use NULL.
§ STRING - Message explaining the test result
§
§ tst_res TTYPE FNAME STRING
§ TTYPE -Test Result Type; one of TFAIL, TBROK, TCONF, and TRETR.
§ FNAME -Print contents of this file after the message
§ STRING - Message explaining the test result
§
§ tst_resm TTYPE STRING
§ TTYPE -Test Result Type; one of TFAIL, TBROK, TCONF, and TRETR.
§ STRING - Message explaining the test result
§
§ These messages are displayed if commands arecalled with no arguments.
§ TheUSCTEST functions and macros are not available in the form of commands. Thesefunctions and macros may be added at a later time. Check with the latestrelease of LTP test suite for recent updates.
3. Recommendationon coding style
This section makes certain recommendation on code layoutand style, but they are optional, it reflects the authors personal coding styleand practice and may be applicable in general to any program written in SHELL.
§ Use afile header to describe the contents of file and maintain a change history. Itmay be argued that CVS provides the option to maintain such a change log whilechecking in code to the CVS tree, but often times the developer need not beresponsible for maintaining the code in CVS. The change log or historymaintained in the file can help both developer and the maintainer of the codeto track changes made to the file.
§ Usemeaningful variable names, and capitalize all variables. Avoid variable like'i', 'j' etc instead use variable names like 'INDEX', 'ARRAY_NDX' etc.Initialize and comment the variables appropriately.
§ Commentcode appropriately. Code may by handled by seasoned programmers as well asnewbies, it is generally a good practice to add comments in the code where everappropriate for maintainability reasons. Comments can be stripped off from thecode using AWK or PERL scipts if required.
§ Usefunctions if possible, the template does not follow this rule for reasons ofconvenience.
§ Usetst_resm TINFO ... to print informational messages as appropriate, alternately,a verbose option or a 'quiet' option may be provided to suppress or activatesuch messages.
§ Use'spaces' instead of 'tab' characters. Typically four spaces may be used insteadof a tab character.
4. Morehelp
At thetime of this writing, only tst_brk, tst_brkm, tst_res and tst_resm functionsare available for use as commands. The syntax for using these commands conformswith the ANSI-C APIs. Man pages for these ANSI-C APIs may be helpful inunderstanding the arguments to these commands. Alternately executing thesecommands without any arguments will print usage. Testcases using these commandsare located under ltp-yyyymmdd/testcases/commands/ directory.
6. Addinga new test to LTP
Youcan add a new test to the LTP test suite can be done in two ways.
If the developers does not have developer/maintaineraccess to the the LTP CVS tree.
o Emailthe test case to the LTP mailing list and a developer or maintainer of LTP will add the testcase after reviewing the code.
If the test case develper has access to the LTP CVS tree.
o Getthe test case reviewed by peers. Test it throughly under different Linuxdistributions and kernel versions. This step is important in order to maintaina test suite that is of highest quality, and code free from bugs.
o Createa sub-directory under the appropriate directory. For example, when adding a newsystem call test case, create a new directory with an appropriate name underthe ltp-yyyymmmdd/testcases/kernel/syscalls/ directory.
o Addthe source code (written in ANSI-C) to this directory, use the genericMakefile, research existing test case for Makefile syntax. Generic template fortest-cases in ANSI-C may be as follows:
| CFLAGS+= -I../../../../include
LOADLIBES+= -L../../../../lib -lltp
SRCS=$(wildcard *.c) TARGETS=$(patsubst %.c,%,$(SRCS)) all: $(TARGETS)
install: @for i in $(TARGETS); do ln -f $$i ../../../bin/$$i ; done clean: rm -f $(TARGETS) |
o If thetest case is written in shell, then the Makefile might require only an"install:" section. Please make sure that there are empty tags for"all:" and "clean:". For example:
| all: install: @ln -f mail_tests.sh ../../bin/mail_tests.sh
clean: |
o Add adescription of the test cases to the 00_Descriptions.txt file located in theltp-yyyymmmdd/testcases/kernel/syscalls/ directory. The following template maybe used:
| test01 |
o Addthe new sub-directory to the top level Makefile, in this case add to Makefileunder ltp-yyyymmmdd/testcases/kernel/syscalls/ to the SUBDIR variable.
o Addthe test "tag" and "command" to execute to the appropriatecommand file under ltp-yyyymmdd/runtest/ directory, to the syscall file. Followthe syntax used in this file. This file will be used as input to the test suitedriver PAN, pan will use the contents of this file to execute the test case.For example, adding a hypothetical system call test01 to the command filesyscall:
| syslog11 syslog11 |
o Sendan email to the mailing list notifying the subscribers the addition of a a newtest. ltp-list@lists.sourceforge.net
Testcase might be included in the subsequent monthly release of the LTP test suite,and due credit will be given to the author of the test case.
OtherResources
Pleaserefer Linux Test Project website for updates, documentation, project details,email address, and to download the latest version of the test suite itself.
Please refer Linux Test Project Kernel CodeCoverage website for LTP's kernel code coverage tools.
The Technical papers section contains a wealth of information pertaining totesting Linux and also the LTP test suite.
Acknowledgements
Aspecial acknowledgement to Kristin Thomas for reviewing and editing thisdocument. I also thank everyone who provided me with valuable comments andsuggestions.
Disclaimer
IBM,the IBM logo, AIX are trademarks of the IBM Corporation.
Pentium is a trademark of Intel Corporation in the United States, othercountries, or both.
Linux is a registered trademark of Linus Torvalds.
All other trademarks and registered trademarks are the property of theirrespective owners.
This publication reflects the views of the author, and not the IBM Corporation.This publication may include typographical errors and technical inaccuraciesand may be changed or withdrawn at any time. The content is provided AS IS,without warranties of any kind, either express or implied, including theimplied warranties of merchantability and fitness for a particular purpose.
This publication may contain links to third party sites that are not under thecontrol of or maintained by IBM. Access to any such third party site is at theuser's own risk and IBM is not responsible for the accuracy or reliability ofany information, data, opinions, advice or statements made on these sites. IBMprovides these links merely as a convenience and the inclusion of such linksdoes not imply an endorsement.
The test results reported in this document represent results obtained fromrunning tests under specific conditions at IBM's laboratory facility in Austin,Texas. Users may achieve different results depending on their particularinstallations, configurations, workloads, or other factors. The information inthis document is provided solely for the information of the user. Theinformation is provided on an "AS IS" basis, without liability orwarranty. Users use such information at their own risk. Users are, thus,advised to evaluate the code referenced in this document as is appropriate fortheir specific installations.
Document Author:
Manoj Iyer
转载：http://ltp.sourceforge.net/documentation/how-to/ltp.php
1913
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
