Background: A test protocol is created when individual tests are combined. Protocol performance
can be calculated prior to clinical use; however, the necessary information is seldom
available. Thus, protocols are frequently used with limited information as to performance.
The next best strategy is to base protocol design on available information combined
with a thorough understanding of the factors that determine protocol performance.
Unfortunately, there is limited information as to these factors and how they interact.
Purpose: The objective of this article and the next article in this issue is to examine in
detail the three factors that determine protocol performance: (1) protocol criterion,
(2) test correlation, (3) test performance. This article examines protocol criterion
and test correlation. The next article examines the impact of individual test performance
and summarizes the results of this series. The ultimate goal is to provide guidance
on the formulation of a protocol using available information and an understanding
of the impact of these three factors on performance.
Research Design: A mathematical model is used to calculate protocol performance for different protocol
criteria and test correlations while assuming that all individual tests in the protocol
have the same performance. The advantages and disadvantages of the different criteria
are evaluated for different test correlations.
Results: A loose criterion will produce the highest protocol hit and false alarm rates; however, the
false alarm rate may be unacceptably high. A strict criterion will produce the smallest protocol hit and false alarm rates; however,
the hit rate may be unacceptably low. Adding tests to a protocol increases the probability
that the protocol false alarm rate will be too high with a loose criterion and that
the protocol hit rate will be too low with a strict criterion. The intermediate criterion, about which little has been known, provides advantages not available with
the other two criteria. This criterion is much more likely to produce acceptable protocol
hit and false alarm rates. It also has the potential to simultaneously produce a protocol
hit rate higher, and a false alarm rate lower, than the individual tests. The intermediate
criteria produce better protocol performance than the loose and strict criteria for
protocols with the same number of tests. For all criteria, best protocol performance
is obtained when the tests are uncorrelated and decreases as test correlation increases.
When there is some test correlation, adding tests to the protocol can decrease protocol
performance for a loose or strict criterion. The ability of a protocol to manipulate
hit and false alarm rates, or improve performance relative to that of the individual
tests, is reduced with increasing test correlation.
Conclusions: The three criteria, loose, strict, and intermediate, have definite advantages and
disadvantages over a large range of test correlations. Some of the advantages and
disadvantages of the loose and strict criteria are impacted by test correlation. The
advantages of the intermediate criteria are relatively independent of test correlation.
When three or more tests are used in a protocol, consideration should be given to
using an intermediate criterion, particularly if there is some test correlation. Greater
test correlation diminishes the advantages of adding tests to a protocol, particularly
with a loose or strict criterion. At higher test correlations, fewer tests in the
protocol may be appropriate.
Key Words
Audiology - false alarm rate - hearing - hit rate - intermediate criterion - loose
criterion - strict criterion - test correlation - test protocol
