An Empirical Study on Coverage-Based Software Testing and Software Fault Tolerance Techniques

Professor Michael R. Lyu

Computer Science & Engineering Department, The Chinese University of Hong Kong

Thu Feb 02 11:00:00 NZDT 2006 in Room 031, MSCS

Abstract

Software testing and software fault tolerance are two major techniques for developing reliable software systems, yet limited empirical data are available in the literature to evaluate their effectiveness. We conduct a major experiment to engage 34 programming teams to independently develop multiple software versions for an industry-scale critical flight application, and collect faults detected in these program versions. To evaluate software testing and software fault tolerance techniques, mutants are created by injecting real faults occurred in the development stage. The nature, manifestation, detection, and correlation of these faults are carefully investigated, and the relationship between code coverage and fault detection capability under different testing profiles is established.

Our experimental results show that coverage testing is generally an effective mean to detecting software faults, but the effectiveness of testing coverage depends on different testing profiles. It is a good estimator for fault detection of exceptional test cases, but a poor one for test cases in normal operations. A testing strategy can then be established to effectively incorporate testing coverage measures. We also perform modeling and analysis to investigate the reliability features in diverse software systems, and parameterize fault correlations to predict the reliability of various combinations of the program versions. The reliability modeling results for diverse software systems are evaluated and compared.

Biography

Michael R. Lyu received the B.S. (1981) in electrical engineering from National Taiwan University, the M.S. (1985) in computer engineering from University of California, Santa Barbara, and the Ph.D. (1988) in computer science from University of California, Los Angeles. He is a Professor in the Computer Science and Engineering Department of the Chinese University of Hong Kong. He worked at the Jet Propulsion Laboratory, Bellcore, and Bell Labs; and taught at the University of Iowa. His research interests include software reliability engineering, software fault tolerance, distributed systems, and multimedia applications. He has published over 220 papers in these areas. Professor Lyu initiated the International Symposium on Software Reliability Engineering (ISSRE), and was Program Chair for ISSRE'1996, Program Co-Chair for WWW10 and SRDS'2005, and General Chair for ISSRE'2001 and PRDC'2005. He is the editor-in-chief for two book volumes: Software Fault Tolerance (Wiley, 1995), and the Handbook of Software Reliability Engineering (IEEE and McGraw-Hill, 1996). He has been an Associate Editor of IEEE Transactions on Reliability, IEEE Transactions on Knowledge and Data Engineering, and Journal of Information Science and Engineering. Professor Lyu is an IEEE Fellow for his contributions to software reliability engineering and software fault tolerance.