Semin Hear 2023; 44(02): 140-154
DOI: 10.1055/s-0043-1767667
Review Article

Test–Retest Reliability and the Role of Task Instructions when Measuring Listening Effort Using a Verbal Response Time Paradigm

Samantha J. Gustafson
1   Department of Communication Sciences and Disorders, University of Utah, Salt Lake City, Utah
Crystal Ortiz
1   Department of Communication Sciences and Disorders, University of Utah, Salt Lake City, Utah
Loren Nelson
1   Department of Communication Sciences and Disorders, University of Utah, Salt Lake City, Utah
› Author Affiliations
Funding Work reported here was funded by a Research Grant in Hearing and Balance from the American Academy of Audiology Foundation (to S.J.G.). This research was also supported by the Office of Undergraduate Research through the Undergraduate Research Opportunity Program (to C.O.) at the University of Utah and by the National Center for Advancing Translational Sciences of the National Institutes of Health (UL1TR002538).


Listening amidst competing noise taxes one's limited cognitive resources, leading to increased listening effort. As such, there is interest in incorporating a reliable test of listening effort into the clinical test battery. One clinically promising method for measuring listening effort is verbal response time (VRT) because it can be obtained using already-established clinical tasks. In order for widespread implementation of the VRT paradigm, a better understanding of the psychometric properties is needed. The purpose of this work was to improve the understanding of the reliability and sensitivity of the VRT listening task. Using within-subject study designs, we completed a pilot study to evaluate the test–retest reliability (Study 1) and the effects of task instructions and listening condition (Study 2). Preliminary results show that the VRT paradigm enjoys good to excellent test–retest reliability and that neither task instructions nor listening condition meaningfully influence VRT once measurement error is accounted for. Future studies should account for measurement error when considering statistically significant versus meaningful effects of experimental parameters when using listening effort tasks.

Publication History

Article published online:
04 April 2023

© 2023. Thieme. All rights reserved.

Thieme Medical Publishers, Inc.
333 Seventh Avenue, 18th Floor, New York, NY 10001, USA

  • References

  • 1 Mattys SL, Davis MH, Bradlow AR, Scott SK. Speech recognition in adverse conditions: a review. Lang Cogn Process 2012; 27 (7–8): 953-978
  • 2 Pals C, Sarampalis A, van Rijn H, Başkent D. Validation of a simple response-time measure of listening effort. J Acoust Soc Am 2015; 138 (03) EL187-EL192
  • 3 Peelle JE. Listening effort: how the cognitive consequences of acoustic challenge are reflected in brain and behavior. Ear Hear 2018; 39 (02) 204-214
  • 4 Pichora-Fuller MK, Kramer SE, Eckert MA. et al. Hearing impairment and cognitive energy: the framework for understanding effortful listening (FUEL). Ear Hear 2016; 37 (Suppl. 01) 5S-27S
  • 5 Alhanbali S, Dawes P, Lloyd S, Munro KJ. Hearing handicap and speech recognition correlate with self-reported listening effort and fatigue. Ear Hear 2018; 39 (03) 470-474
  • 6 Hughes SE, Hutchings HA, Rapport FL, McMahon CM, Boisvert I. Social connectedness and perceived listening effort in adult cochlear implant users: a grounded theory to establish content validity for a new patient-reported outcome measure. Ear Hear 2018; 39 (05) 922-934
  • 7 Rosemann S, Thiel CM. Neuroanatomical changes associated with age-related hearing loss and listening effort. Brain Struct Funct 2020; 225 (09) 2689-2700
  • 8 Hétu R, Jones L, Getty L. The impact of acquired hearing impairment on intimate relationships: implications for rehabilitation. Audiology 1993; 32 (06) 363-381
  • 9 Kramer SE, Kapteyn TS, Houtgast T. Occupational performance: comparing normally-hearing and hearing-impaired employees using the Amsterdam Checklist for Hearing and Work. Int J Audiol 2006; 45 (09) 503-512
  • 10 Hornsby BWY. The effects of hearing aid use on listening effort and mental fatigue associated with sustained speech processing demands. Ear Hear 2013; 34 (05) 523-534
  • 11 Hick CB, Tharpe AM. Listening effort and fatigue in school-age children with and without hearing loss. J Speech Lang Hear Res 2002; 45 (03) 573-584
  • 12 Johnson J, Xu J, Cox R, Pendergraft P. A comparison of two methods for measuring listening effort as part of an audiologic test battery. Am J Audiol 2015; 24 (03) 419-431
  • 13 Meister H, Rählmann S, Lemke U, Besser J. Verbal response times as a potential indicator of cognitive load during conventional speech audiometry with matrix sentences. Trends Hear 2018; 22: 2331216518793255
  • 14 Wu YH, Stangl E, Zhang X, Perkins J, Eilers E. Psychometric functions of dual-task paradigms for measuring listening effort. Ear Hear 2016; 37 (06) 660-670
  • 15 McGarrigle R, Munro KJ, Dawes P. et al. Listening effort and fatigue: what exactly are we measuring? A British Society of Audiology Cognition in Hearing Special Interest Group ‘white paper’. Int J Audiol 2014; 53 (07) 433-440
  • 16 Alhanbali S, Dawes P, Millman RE, Munro KJ. Measures of listening effort are multidimensional. Ear Hear 2019; 40 (05) 1084-1097
  • 17 Ohlenforst B, Zekveld AA, Jansma EP. et al. Effects of hearing impairment and hearing aid amplification on listening effort: a systematic review. Ear Hear 2017; 38 (03) 267-281
  • 18 Strand JF, Brown VA, Merchant MB, Brown HE, Smith J. Measuring listening effort: convergent validity, sensitivity, and links with cognitive and personality measures. J Speech Lang Hear Res 2018; 61 (06) 1463-1486
  • 19 Piquado T, Benichov JI, Brownell H, Wingfield A. The hidden effect of hearing acuity on speech recall, and compensatory effects of self-paced listening. Int J Audiol 2012; 51 (08) 576-583
  • 20 Rönnberg J, Lunner T, Zekveld A. et al. The Ease of Language Understanding (ELU) model: theoretical, empirical, and clinical advances. Front Syst Neurosci 2013; 7: 31
  • 21 Oates PA, Kurtzberg D, Stapells DR. Effects of sensorineural hearing loss on cortical event-related potential and behavioral measures of speech-sound processing. Ear Hear 2002; 23 (05) 399-415
  • 22 Korczak PA, Kurtzberg D, Stapells DR. Effects of sensorineural hearing loss and personal hearing AIDS on cortical event-related potential and behavioral measures of speech-sound processing. Ear Hear 2005; 26 (02) 165-185
  • 23 Gillis M, Decruy L, Vanthornhout J, Francart T. Hearing loss is associated with delayed neural responses to continuous speech. Eur J Neurosci 2022; 55 (06) 1671-1690
  • 24 Nada NM, Kolkaila EA, Gabr TA, El-Mahallawi TH. Speech auditory brainstem response audiometry in adults with sensorineural hearing loss. Egyptian J Ear, Nose, Throat Allied Sci 2016; 17 (02) 87-94
  • 25 White BE, Langdon C. The cortical organization of listening effort: new insight from functional near-infrared spectroscopy. Neuroimage 2021; 240: 118324
  • 26 Gatehouse S, Gordon J. Response times to speech stimuli as measures of benefit from amplification. Br J Audiol 1990; 24 (01) 63-68
  • 27 Brennan MA, Lewis D, McCreery R, Kopun J, Alexander JM. Listening effort and speech recognition with frequency compression amplification for children and adults with hearing loss. J Am Acad Audiol 2017; 28 (09) 823-837
  • 28 Gustafson S, McCreery R, Hoover B, Kopun JG, Stelmachowicz P. Listening effort and perceived clarity for normal-hearing children with the use of digital noise reduction. Ear Hear 2014; 35 (02) 183-194
  • 29 Holube I. Speech intelligibility in fluctuating maskers. Proc Int Symp Audit Audiol Res 2011; 3: 57-64
  • 30 Lewis D, Schmid K, O'Leary S, Spalding J, Heinrichs-Graham E, High R. Effects of noise on speech recognition and listening effort in children with normal hearing and children with mild bilateral or unilateral hearing loss. J Speech Lang Hear Res 2016; 59 (05) 1218-1232
  • 31 McGarrigle R, Gustafson SJ, Hornsby BWY, Bess FH. Behavioral measures of listening effort in school-age children: examining the effects of signal-to-noise ratio, hearing loss, and amplification. Ear Hear 2019; 40 (02) 381-392
  • 32 Oosthuizen I, Picou EM, Pottas L, Myburgh HC, Swanepoel W. Listening effort in native and nonnative English-speaking children using low linguistic single- and dual-task paradigms. J Speech Lang Hear Res 2020; 63 (06) 1979-1989
  • 33 Oosthuizen I, Picou EM, Pottas L, Myburgh HC, Swanepoel W. Listening effort in school-age children with normal hearing compared to children with limited useable hearing unilaterally. Am J Audiol 2021; 30 (02) 309-324
  • 34 Visentin C, Valzolgher C, Pellegatti M, Potente P, Pavani F, Prodi N. A comparison of simultaneously-obtained measures of listening effort: pupil dilation, verbal response time and self-rating. Int J Audiol 2022; 61 (07) 561-573
  • 35 Houben R, van Doorn-Bierman M, Dreschler WA. Using response time to speech as a measure for listening effort. Int J Audiol 2013; 52 (11) 753-761
  • 36 Gustafson SJ, Nelson L, Silcox JW. Effect of auditory distractors on speech recognition and listening effort. Ear and Hearing. Accessed March 21, 2023
  • 37 Gagné JP, Besser J, Lemke U. Behavioral assessment of listening effort using a dual-task paradigm: a review. Trends Hear 2017; 21: 2331216516687287
  • 38 McCreery RW, Stelmachowicz PG. The effects of limited bandwidth and noise on verbal processing time and word recall in normal-hearing children. Ear Hear 2013; 34 (05) 585-591
  • 39 Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med 2016; 15 (02) 155-163
  • 40 Le Prell CG. Current issues in clinical and translational research in the hearing sciences, audiology, and otolaryngology. In: Translational Research in Audiology, Neurotology, and the Hearing Sciences. Springer; 2016: 19-50
  • 41 Batterham AM, George KP. Reliability in evidence-based clinical practice: a primer for allied health professionals. Phys Ther Sport 2000; 1 (02) 54-62
  • 42 Crocker L, Algina J. Introduction to Classical and Modern Test Theory. ERIC; 1986
  • 43 Weir JP. Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. J Strength Cond Res 2005; 19 (01) 231-240
  • 44 Bruton A, Conway JH, Holgate ST. Reliability: what is it, and how is it measured?. Physiotherapy 2000; 86 (02) 94-99
  • 45 Krueger M, Schulte M, Brand T, Holube I. Development of an adaptive scaling method for subjective listening effort. J Acoust Soc Am 2017; 141 (06) 4680-4693
  • 46 Picou EM, Ricketts TA. The relationship between speech recognition, behavioural listening effort, and subjective ratings. Int J Audiol 2018; 57 (06) 457-467
  • 47 Reinten I, De Ronde-Brons I, Houben R, Dreschler W. Measuring the influence of noise reduction on listening effort in hearing-impaired listeners using response times to an arithmetic task in noise. Trends Hear 2021;25:23312165211014437
  • 48 Devocht EMJ, Janssen AML, Chalupper J, Stokroos RJ, George ELJ. The benefits of bimodal aiding on extended dimensions of speech perception: intelligibility, listening effort, and sound quality. Trends Hear 2017; 21: 2331216517727900
  • 49 Picou EM, Ricketts TA. The effect of changing the secondary task in dual-task paradigms for measuring listening effort. Ear Hear 2014; 35 (06) 611-622
  • 50 Neagu MB, Dau T, Hyvärinen P, Bækgaard P, Lunner T, Wendt D. Investigating pupillometry as a reliable measure of individual's listening effort. Proceedings of the International Symposium on Auditory and Audiological Research 2019; 7: 365-372
  • 51 Jones M, Dobson A, O'Brian S. A graphical method for assessing agreement with the mean between multiple observers using continuous measures. Int J Epidemiol 2011; 40 (05) 1308-1313
  • 52 Degeest S, Corthals P, Keppler H. A Dutch version of a dual-task paradigm for measuring listening effort: a pilot study regarding its short-term test-retest reliability. B-ENT 2021; 17 (03) 135-144
  • 53 Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Int J Nurs Stud 2010; 47 (08) 931-936
  • 54 Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 1 (8476): 307-310
  • 55 Bland JM, Altman DG. Comparing methods of measurement: why plotting difference against standard method is misleading. Lancet 1995; 346 (8982): 1085-1087
  • 56 Beninato M, Portney LG. Applying concepts of responsiveness to patient management in neurologic physical therapy. J Neurol Phys Ther 2011; 35 (02) 75-81
  • 57 Portney LG, Watkins MP. Foundations of Clinical Research: Applications to Practice. Vol 892. Pearson/Prentice Hall Upper Saddle River, NJ; 2009
  • 58 Beckerman H, Roebroeck ME, Lankhorst GJ, Becher JG, Bezemer PD, Verbeek ALM. Smallest real difference, a link between reproducibility and responsiveness. Qual Life Res 2001; 10 (07) 571-578
  • 59 Donoghue D, Stokes EK. Physiotherapy Research and Older People (PROP) group. How much change is true change? The minimum detectable change of the Berg Balance Scale in elderly people. J Rehabil Med 2009; 41 (05) 343-346
  • 60 Choi S, Lotto A, Lewis D, Hoover B, Stelmachowicz P. Attentional modulation of word recognition by children in a dual-task paradigm. J Speech Lang Hear Res 2008; 51 (04) 1042-1054
  • 61 Woodworth RS. Accuracy of voluntary movement. Psychological Review 1899; 3: 1-106
  • 62 Heitz RP. The speed-accuracy tradeoff: history, physiology, methodology, and behavior. Front Neurosci 2014; 8: 150
  • 63 Adler LA, Kessler RC, Spencer T. Adult ADHD Self-Report Scale-v1. 1 (ASRS-v1. 1) Symptom Checklist. New York, NY: World Health Organization; 2003
  • 64 Bonino AY, Malley AR. Measuring open-set, word recognition in school-aged children: corpus of monosyllabic target words and speech maskers. J Acoust Soc Am 2019; 146 (04) EL393-EL398
  • 65 Nagaraj NK, Magimairaj BM, Schwartz S. Auditory distraction in school-age children relative to individual differences in working memory capacity. Atten Percept Psychophys 2020; 82 (07) 3581-3593
  • 66 Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess 1994; 6 (04) 284
  • 67 Hopkins WG. Measures of reliability in sports medicine and science. Sports Med 2000; 30 (01) 1-15
  • 68 Ohlenforst B, Zekveld AA, Lunner T. et al. Impact of stimulus-related factors and hearing impairment on listening effort as indicated by pupil dilation. Hear Res 2017; 351: 68-79
  • 69 Oosthuizen I, Picou EM, Pottas L, Myburgh HC, Swanepoel W. Listening effort in school-aged children with limited useable hearing unilaterally: examining the effects of a personal, digital remote microphone system and a contralateral routing of signal system. Trends Hear 2021; 25: 2331216520984700
  • 70 Watson PF, Petrie A. Method agreement analysis: a review of correct methodology. Theriogenology 2010; 73 (09) 1167-1179