A runtime test tool for applications

Thorough testing has been one of the principles of the Linux Standard Base Project since the beginning. Currently, tests suites exists to measure the runtime system, and to make a static analysis of an application. The one missing piece has been a test tool that can be used at runtime to evaluate the dynamic behavior of an application. Using a static analysis, an application may be proven to only call functions defined in the LSB, however the values of the parameter that are passed to those function may only be assessed at runtime. A different type of tool is needed to perform these tests. The dynamic application test tool being developed for the LSB is capable of performing these tests by inserting itself into the running application. Creating and maintaining this test tool has it's own unique set of challenges caused partly be the large number (several thousand) of interfaces involved. These challenges are met by automating the process of creating the tests. The LSB maintains a database containing representations of all of the interfaces included in the LSB. This database contains the interface names, the position and types of parameters to the interfaces, and definitions of the structures and constants which are used. This information is used to automatically generate testing stubs for each interface, which in turn use a library of type specific tests to validate the parameter values. In addition to the existing syntactic information in the database, some additional semantic information must be added to provide meta-type information that is needed to be able to evaluate the values being passed in as parameters. With this additional information, all but a handful of the interface test stubs can be generated automatically, and many of the type specific tests can also be created automatically. An initial prototype, which contains a small subset of the total tests has verified the mechanisms which will be used. Implementation of the remaining tests, and the process for automating their creation is occurring during the first half of 2004. This paper will explain the techniques used to create this test tool, and assess the results obtained against real applications. Lessons learned, unexpected application of this test, and how this methodology can be applied to APIs outside of the LSB will also be reviewed.

Stuart Anderson has been involved with Open Source since before the term was created. Stuart is a Software Engineer and has worked for NCR, AT&T and Metro Link, Inc. in the years since 1989 where he has done development work for a wide range of operating systems from small embedded devices to large multiprocessor servers. Stuart has been involved with the XFree86 Project since 1992 including a term on the XFree86 Core Team. Stuart has had previous standards experience with the X Consortium and the System V ABI in the mid '90s, and has been involved with the LSB since its begining. Additional details at: http://www.netsweng.com/~anderson/bio.html

Matt Elder just finished his second year at the University of South Carolina, where he studies Mathematics and Computer Science. His work with the LSB is his first open-source project.