Download PDF
Methods Inf Med 1989; 28(01): 36-50
DOI: 10.1055/s-0038-1635547
Original Article
Schattauer GmbH

Validating the Knowledge Base of a Therapy Planning System

M. A. Shwe
1   Section on Medical Informatics, Department of Medicine, Stanford University School of Medicine, USA
,
S. W. Tu
1   Section on Medical Informatics, Department of Medicine, Stanford University School of Medicine, USA
,
L. M. Fagan
1   Section on Medical Informatics, Department of Medicine, Stanford University School of Medicine, USA
› Author Affiliations
Further Information

Publication History

Publication Date:
20 February 2018 (online)

    Permissions and Reprints

Abstract:

Validation of expert system knowledge bases has proved to be difficult. This paper presents a description of a system called ScriptGen that generates test data for validating the knowledge base of the ONCOCIN cancer therapy planning system. Because of the size and complexity of the ONCOCIN knowledge base, we require tools for automated validation. ScriptGen, which applies techniques developed in testing both traditional software and expert systems, uses a parallel model of the ONCOCIN knowledge base and its own inference engine to generate test cases. We derived the limits of the system from a study that seeded errors into an existing knowledge base.

Key-Words:

Knowledge Base Validation - Program Validation - Knowledge Engineering - Expert Systems - Program Testing
 

  • REFERENCES

  • 1 Andriole SJ. ed Software Validation, Verification, Testing and Documentation. . Princeton NJ: Petrocelli Books; 1986
    Google Scholar
  • 2 Howden WE. Introduction to software validation. In Miller E, Howden WE. ed Tutorial: Software Testing & Validation Techniques. . New York: IEEE Computer Society Press; 1978: 1-2.
    Google Scholar
  • 3 Suwa M, Scott AC, Shortliffe EH. An approach to verifying completeness and consistency in a rule-based expert system. AI Magazine 1982; 2: 16-21.
    PubMedGoogle Scholar
  • 4 Shortliffe EH, Scott AC, Bischoff MB, Campbell AB, van Melle W, Jacobs CD. ONCOCIN: an expert system for oncology protocol management. In: Proceedings of the 7th International Joint Conference on Artificial Intelligence. . Los Altos CA: Morgan Kaufman Publishers; 1981: 876-81.
    Google Scholar
  • 5 Tu SW, Kahn MG, Musen MA, Ferguson JC, Shortliffe EH, Fagan LM. Episodic Monitoring of Time-Oriented Data for Heuristic Skeletal-Plan Refinement. Knowledge Systems Laboratory Memo no. KSL-87-70. Stanford CA: Stanford University; 1988
    Google Scholar
  • 6 De Millo RA, McCracken WM, Martin RJ, Passafiume JF. Software Testing and Evaluation. . Menlo Park CA: Benjamin/ Cummings; 1987
    Google Scholar
  • 7 Miller E. Introduction to software testing technology. In Miller E, Howden WE. ed Tutorial: Software Testing & Validation Techniques. . New York: IEEE Computer Society Press; 1978: 3-12.
    Google Scholar
  • 8 Huang JC. An approach to program testing. ACM Comput Surveys 1975; 7: 113-28.
    CrossrefPubMedGoogle Scholar
  • 9 Miller EF, Bardens JA, Benson JP, Melton RA, Urban RJ, Wiscard WR. Structurally-based automatic program testing. In: EASCON 74 Record.. New York: IEEE Aerospace and Electronic Systems Group; 1974: 134-39.
    Google Scholar
  • 10 Miller EF, Melton RA. Automated generation of testcase datasets. In: Proceedings International Conference on Reliable Software. . New York: IEEE Computer Society, IEEE Group on Reliability; 1975: 51-8.
    Google Scholar
  • 11 Ramoorthy CV, Ho HF, Chen WT. On the automated generation of program test data. IEEE Trans Softw Eng. 1976. SE-2 293-300.
    Google Scholar
  • 12 Lyons NR. An automatic data generating system for data base simulation and testing. Data Base 1977; 8: 10-3.
    CrossrefPubMedGoogle Scholar
  • 13 De Millo RA, Lipton RJ, Sayard FG. Hints on test data selection: help for the practicing programmer. Computer 1978; 11: 34-41.
    PubMedGoogle Scholar
  • 14 Ostrand TJ, Balcer MJ. The category-partition method for specifying and generating functional tests. Comm ACM 1988; 31: 676-86.
    CrossrefPubMedGoogle Scholar
  • 15 Duran JW, Ntafos SC. An evaluation of random testing. IEEE Trans Softw Eng. 1984. SE-10 438-44.
    Google Scholar
  • 16 Nguyen TA, Perkins WA, Laffey TJ, Pecora D. Knowledge base verification. AI Magazine 1987; 2: 69-75.
    PubMedGoogle Scholar
  • 17 Stachowitz RA, Combs JB. Validation of expert systems. In: Proceedings Twentieth Annual Hawaii International Conference on System Sciences. . Hollywood CA: Western Periodicals; 1987: 686-95.
    Google Scholar
  • 18 Davis R. Interactive transfer of expertise: acquisition of new inference rules. Artif Intell 1979; 12: 121-57.
    CrossrefPubMedGoogle Scholar
  • 19 Ginsberg A, Weiss S, Politakis P. SEEK2: A generalized approach to automatic knowledge base refinement. In: Proceedings Ninth International Joint Conference on Artificial Intelligence. . Los Altos CA: Morgan Kaufman Publishers; 1985: 367-74.
    Google Scholar
  • 20 Perreault LE. Automatic Test Case Generation by Modeling Patient States and Physician Actions. Knowledge Systems Laboratory Memo no. KSL-S7-63. Stanford CA: Stanford University; 1987
    Google Scholar
  • 21 Speedie S, Skarupa S, Oderada L, Blaschke TF, Leatherman E, Kondo J, Gray J, Olsen W. MENTOR: continuously monitoring drug therapy with an expert system. In Salamon R, Blum B, Jörgensen M. ed MEDINFO 86. Amsterdam: North-Holland Publ Comp; 1986: 237-39.
    Google Scholar
  • 22 Mars NJI, Miller PL. Knowledge acquisition and verification tools for medical expert systems. Med Dec Making 1987; 7: 6-11.
    CrossrefPubMedGoogle Scholar
  • 23 Miller PL, Blumenfrucht SJ, Rose JR, Rothschild M, Swett HA, Weltin G, Mars NJI. HYDRA: A knowledge acquisition tool for expert systems that critique medical workup. Med Dec Making 1987; 7: 12-21.
    CrossrefPubMedGoogle Scholar
  • 24 Miller PL. Critiquing anesthetic management: the ATTENDING computer systern. Anesthesiol 1983; 58: 362-69.
    CrossrefPubMedGoogle Scholar
  • 25 Myers GJ. The Art of Software Testing. . New York: John Wiley & Sons; 1979
    Google Scholar
  • 26 Nilsson NJ. Principles of Artificial Intelligence. . Palo Alto CA: Tioga Publishing; 1980
    Google Scholar
  • 27 Musen MA, Rohn JA, Fagan LM, Short-Iiffe EH. Knowledge engineering for a clinical trial advice system: uncovering errors in protocol specification. In: Proceedings of AAMSI Congress 86. Washington DC: American Association for Medical Systems and Informatics; 1986: 24-7.
    Google Scholar
  • 28 Mills HD. On the Statistical Validation of Computer Programs. IBM Report no. FSC72-6015. Gaithersburg MD: IBM Federal Systems Division; 1972
    Google Scholar
  • 29 Musen MA, Fagan LM, Shortliffe EH. Graphical specification of procedural knowledge for an expert system. In: 1986 Computer Society Workshop on Visual Languages. . Washington DC: IEEE Computer Society; 1986: 167-78.
    Google Scholar
  • 30 Hamlet R. Special section on software testing. Comm ACM 1988; 31: 662-7.
    CrossrefPubMedGoogle Scholar