Validating the Knowledge Base of a Therapy Planning System

Abstract:

Validation of expert system knowledge bases has proved to be difficult. This paper presents a description of a system called ScriptGen that generates test data for validating the knowledge base of the ONCOCIN cancer therapy planning system. Because of the size and complexity of the ONCOCIN knowledge base, we require tools for automated validation. ScriptGen, which applies techniques developed in testing both traditional software and expert systems, uses a parallel model of the ONCOCIN knowledge base and its own inference engine to generate test cases. We derived the limits of the system from a study that seeded errors into an existing knowledge base.

Key-Words:

Knowledge Base Validation - Program Validation - Knowledge Engineering - Expert Systems - Program Testing

• REFERENCES

• 1 Andriole SJ. ed Software Validation, Verification, Testing and Documentation. . Princeton NJ: Petrocelli Books; 1986
• 2 Howden WE. Introduction to software validation. In Miller E, Howden WE. ed Tutorial: Software Testing & Validation Techniques. . New York: IEEE Computer Society Press; 1978: 1-2.
• 3 Suwa M, Scott AC, Shortliffe EH. An approach to verifying completeness and consistency in a rule-based expert system. AI Magazine 1982; 2: 16-21.
• 4 Shortliffe EH, Scott AC, Bischoff MB, Campbell AB, van Melle W, Jacobs CD. ONCOCIN: an expert system for oncology protocol management. In: Proceedings of the 7th International Joint Conference on Artificial Intelligence. . Los Altos CA: Morgan Kaufman Publishers; 1981: 876-81.
• 5 Tu SW, Kahn MG, Musen MA, Ferguson JC, Shortliffe EH, Fagan LM. Episodic Monitoring of Time-Oriented Data for Heuristic Skeletal-Plan Refinement. Knowledge Systems Laboratory Memo no. KSL-87-70. Stanford CA: Stanford University; 1988
• 6 De Millo RA, McCracken WM, Martin RJ, Passafiume JF. Software Testing and Evaluation. . Menlo Park CA: Benjamin/ Cummings; 1987
• 7 Miller E. Introduction to software testing technology. In Miller E, Howden WE. ed Tutorial: Software Testing & Validation Techniques. . New York: IEEE Computer Society Press; 1978: 3-12.
• 8 Huang JC. An approach to program testing. ACM Comput Surveys 1975; 7: 113-28.
• 9 Miller EF, Bardens JA, Benson JP, Melton RA, Urban RJ, Wiscard WR. Structurally-based automatic program testing. In: EASCON 74 Record.. New York: IEEE Aerospace and Electronic Systems Group; 1974: 134-39.
• 10 Miller EF, Melton RA. Automated generation of testcase datasets. In: Proceedings International Conference on Reliable Software. . New York: IEEE Computer Society, IEEE Group on Reliability; 1975: 51-8.
• 11 Ramoorthy CV, Ho HF, Chen WT. On the automated generation of program test data. IEEE Trans Softw Eng. 1976. SE-2 293-300.
• 12 Lyons NR. An automatic data generating system for data base simulation and testing. Data Base 1977; 8: 10-3.
• 13 De Millo RA, Lipton RJ, Sayard FG. Hints on test data selection: help for the practicing programmer. Computer 1978; 11: 34-41.
• 14 Ostrand TJ, Balcer MJ. The category-partition method for specifying and generating functional tests. Comm ACM 1988; 31: 676-86.
• 15 Duran JW, Ntafos SC. An evaluation of random testing. IEEE Trans Softw Eng. 1984. SE-10 438-44.
• 16 Nguyen TA, Perkins WA, Laffey TJ, Pecora D. Knowledge base verification. AI Magazine 1987; 2: 69-75.
• 17 Stachowitz RA, Combs JB. Validation of expert systems. In: Proceedings Twentieth Annual Hawaii International Conference on System Sciences. . Hollywood CA: Western Periodicals; 1987: 686-95.
• 18 Davis R. Interactive transfer of expertise: acquisition of new inference rules. Artif Intell 1979; 12: 121-57.
• 19 Ginsberg A, Weiss S, Politakis P. SEEK2: A generalized approach to automatic knowledge base refinement. In: Proceedings Ninth International Joint Conference on Artificial Intelligence. . Los Altos CA: Morgan Kaufman Publishers; 1985: 367-74.
• 20 Perreault LE. Automatic Test Case Generation by Modeling Patient States and Physician Actions. Knowledge Systems Laboratory Memo no. KSL-S7-63. Stanford CA: Stanford University; 1987
• 21 Speedie S, Skarupa S, Oderada L, Blaschke TF, Leatherman E, Kondo J, Gray J, Olsen W. MENTOR: continuously monitoring drug therapy with an expert system. In Salamon R, Blum B, Jörgensen M. ed MEDINFO 86. Amsterdam: North-Holland Publ Comp; 1986: 237-39.
• 22 Mars NJI, Miller PL. Knowledge acquisition and verification tools for medical expert systems. Med Dec Making 1987; 7: 6-11.
• 23 Miller PL, Blumenfrucht SJ, Rose JR, Rothschild M, Swett HA, Weltin G, Mars NJI. HYDRA: A knowledge acquisition tool for expert systems that critique medical workup. Med Dec Making 1987; 7: 12-21.
• 24 Miller PL. Critiquing anesthetic management: the ATTENDING computer systern. Anesthesiol 1983; 58: 362-69.
• 25 Myers GJ. The Art of Software Testing. . New York: John Wiley & Sons; 1979
• 26 Nilsson NJ. Principles of Artificial Intelligence. . Palo Alto CA: Tioga Publishing; 1980
• 27 Musen MA, Rohn JA, Fagan LM, Short-Iiffe EH. Knowledge engineering for a clinical trial advice system: uncovering errors in protocol specification. In: Proceedings of AAMSI Congress 86. Washington DC: American Association for Medical Systems and Informatics; 1986: 24-7.
• 28 Mills HD. On the Statistical Validation of Computer Programs. IBM Report no. FSC72-6015. Gaithersburg MD: IBM Federal Systems Division; 1972
• 29 Musen MA, Fagan LM, Shortliffe EH. Graphical specification of procedural knowledge for an expert system. In: 1986 Computer Society Workshop on Visual Languages. . Washington DC: IEEE Computer Society; 1986: 167-78.
• 30 Hamlet R. Special section on software testing. Comm ACM 1988; 31: 662-7.