Download PDF
CC BY-NC-ND 4.0 · Avicenna J Med 2022; 12(03): 138-147
DOI: 10.1055/s-0042-1755332
Original Article

Construction and Writing Flaws of the Multiple-Choice Questions in the Published Test Banks of Obstetrics and Gynecology: Adoption, Caution, or Mitigation?

Magdy H. Balaha 
1   Department of Obstetrics and Gynecology, College of Medicine, King Faisal University, Kingdom of Saudi Arabia
2   Department of Obstetrics and Gynecology, Faculty of Medicine, Tanta University, Tanta, Egypt
,
Mona T. El-Ibiary
2   Department of Obstetrics and Gynecology, Faculty of Medicine, Tanta University, Tanta, Egypt
,
Ayman A. El-Dorf
2   Department of Obstetrics and Gynecology, Faculty of Medicine, Tanta University, Tanta, Egypt
,
Shereef L. El-Shewaikh
2   Department of Obstetrics and Gynecology, Faculty of Medicine, Tanta University, Tanta, Egypt
,
Hossam M. Balaha 
3   Department of Computer Engineering and Systems, Faculty of Engineering, Mansoura University, Mansoura, Egypt
› Author Affiliations Funding None.
› Further Information
   Permissions and Reprints

Abstract

Background The item-writing flaws (IWFs) in multiple-choice questions (MCQs) can affect test validity. The purpose of this study was to explore the IWFs in the published resources, estimate their frequency and pattern, rank, and compare the current study resources, and propose a possible impact for teachers and test writers.

Methods This cross-sectional study was conducted from September 2017 to December 2020. MCQs from the published MCQ books in Obstetrics and Gynecology was the target resources. They were stratified into four clusters (study-book related, review books, self-assessment books, and online-shared test banks). The sample size was estimated and 2,300 out of 11,195 eligible MCQs were randomly selected. The MCQs (items) were judged on a 20-element compiled checklist that is organized under three sections as follows: (1) structural flaws (seven elements), (2) test-wiseness flaws (five elements), and (3) irrelevant difficulty flaws (eight elements). Rating was done dichotomously, 0 = violating and 1 = not violating. Item flaws were recorded and analyzed using the Excel spreadsheets and IBM SPSS.

Results Twenty three percent of the items (n = 537) were free from any violations, whereas 30% (n = 690) contained one violation, and 47% (n = 1073) contained more than one violation. The most commonly reported IWFs were “Options are Not in Order (61%).” The best questions with the least flaws (75th percentiles) were obtained from the self-assessment books followed by study-related MCQ books. The average scores of good-quality items in the cluster of self-assessment books were significantly higher than other book clusters.

Conclusion There were variable presentations and percentages of item violations. Lower quality questions were observed in review-related MCQ books and the online-shared test banks. Using questions from these resources needs a caution or avoidance strategy. Relative higher quality questions were reported for the self-assessment followed by the study-related MCQ books. An adoption strategy may be applied with mitigation if needed.

Keywords

multiple-choice questions writing flaws - multiple-choice questions violations - test bank quality

Authors' Contributions

M.H.B. contributed with idea, concept, design, acquisition of data, studying of collected data, data analysis, interpretation of data, drafting the article, revising the article, and shared final approval.

M.T.El.-I. took part in design, studying of collected data, interpretation of data, revising the article, and also shared final approval.

A.A.El.-D. took part in conceptualization, design, studying of collected data, interpretation of data, revising the article, and shared final approval.

S.L.El.-S. took part in conceptualization, design, studying of collected data, interpretation of data, drafting the article, revising the article, and shared final approval.

H.M.B. contributed with conceptualization, design, acquisition of data, data analysis, drafting the article, revising the article, and shared final approval.


Supplementary Material



Publication History

Article published online:
31 August 2022

© 2022. Syrian American Medical Society. This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/)

Thieme Medical and Scientific Publishers Pvt. Ltd.
A-12, 2nd Floor, Sector 2, Noida-201301 UP, India

 

  • References

  • 1 Farley JK. The multiple-choice test: writing the questions. Nurse Educ 1989; 14 (06) 10-12 , 39
    CrossrefPubMedGoogle Scholar
  • 2 Vyas R, Supe A. Multiple choice questions: a literature review on the optimal number of options. Natl Med J India 2008; 21 (03) 130-133
    PubMedGoogle Scholar
  • 3 Cynthia B, Whitney DR. The effect of selected poor item-writing practices on test difficulty, reliability and validity. J Educ Meas 1972; 9 (03) 225-233
    CrossrefGoogle Scholar
  • 4 Downing SM. Construct-irrelevant variance and flawed test questions: Do multiple-choice item-writing principles make any difference?. Acad Med 2002; 77 (10(Suppl 10): S103-S104
    CrossrefPubMedGoogle Scholar
  • 5 Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ 2003; 37 (09) 830-837
    CrossrefPubMedGoogle Scholar
  • 6 Idika DO, Shogbesan YO, Bamikole OI. Effect of test item compromise and test item practice on validity of economics achievement test scores among secondary school students in cross river state, Nigeria. African Journal of Theory and Practice of Educational Assessment 2016; 3 (06) 33-47
    Google Scholar
  • 7 Breakall J, Randles C, Tasker R. Development and use of a multiple-choice item writing flaws evaluation instrument in the context of general chemistry. Chem Educ Res Pract 2019; 20 (02) 369-382
    CrossrefGoogle Scholar
  • 8 Determining the quality of assessment items in collaborations: aspects to discuss to reach agreement.. Developed by the Australian Medical Assessment Collaboration. Accessed July 19, 2022 at: https://www.acer.org/files/quality-determination-of-assessment-items-amac-resource.pdf
  • 9 Tenore A, Mathysen DGP, Mills P, Westwood M, Rouffet J-B. Papalois V ea. UEMS-CESMA Guidelines. Accessed July 19, 2022 at: https://www.uems.eu/news-and-events/news/news-more/uems-cesma-guidelines
    Google Scholar
  • 10 Davis MH, Ponnamperuma G, McAleer SDR. The Joint Committee on Intercollegiate Examinations. Accessed July 19, 2022 at: https://www.jcie.org.uk/Content/content.aspx
    Google Scholar
  • 11 Paniagua M, Swygert K. Constructing Written Test Questions for the Basic and Clinical Sciences. 4th ed.. Philadelphia, PA: National Board of Medical Examiners (NBME); 2016
    Google Scholar
  • 12 Balaha M, El-Baramawi M, El-Hawary E. Three option multiple choice questions had the least non-functioning distractors: analysis of 1855 MCQs in first year competency based medical program at tanta faculty of medicine, Egypt. International Journal of Scientific and Engineering Research (IJSER) 2019; 10 (02) 1432-1438
    Google Scholar
  • 13 Tarrant M, Knierim A, Hayes SK, Ware J. The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments. Nurse Educ Today 2006; 26 (08) 662-671
    CrossrefPubMedGoogle Scholar
  • 14 Downing SM. The effects of violating standard item writing principles on tests and students: the consequences of using flawed test items on achievement examinations in medical education. Adv Health Sci Educ Theory Pract 2005; 10 (02) 133-143
    CrossrefPubMedGoogle Scholar
  • 15 Ellsworth RA, Dunnell P, Duell OK. Multiple-choice test items: what are textbook authors telling teachers?. J Educ Res 1990; 83 (05) 289-293
    CrossrefGoogle Scholar
  • 16 Hansen JD, Dexter L. Quality multiple-choice test questions: item-writing guidelines and an analysis of auditing testbanks. J Educ Bus 1997; 73 (02) 94-97
    CrossrefGoogle Scholar
  • 17 Masters JC, Hulsmeyer BS, Pike ME, Leichty K, Miller MT, Verst AL. Assessment of multiple-choice questions in selected test banks accompanying text books used in nursing education. J Nurs Educ 2001; 40 (01) 25-32
    CrossrefPubMedGoogle Scholar
  • 18 Bosher S. Barriers to creating a more culturally diverse nursing profession. Linguistic bias in multiple-choice nursing exams. Nurs Educ Perspect 2003; 24 (01) 25-34
    PubMedGoogle Scholar
  • 19 Naeem N, van der Vleuten C, Alfaris EA. Faculty development on item writing substantially improves item quality. Adv Health Sci Educ Theory Pract 2012; 17 (03) 369-376
    CrossrefPubMedGoogle Scholar
  • 20 Nedeau-Cayo R, Laughlin D, Rus L, Hall J. Assessment of item-writing flaws in multiple-choice questions. J Nurses Prof Dev 2013; 29 (02) 52-57 , quiz E1–E2
    CrossrefPubMedGoogle Scholar
  • 21 Medical Sciences Board of Examiners. IBMG. Rules and Regulations for the BSc degree programme. Appendix 3–5. Faculty of Medical Sciences, University of Groningen, the Netherlands; 2011: 25-32
    Google Scholar
  • 22 Yockey RD. SPSS demystified: a simple guide and reference. London, United Kingdom: Routledge; 2018
    Google Scholar
  • 23 Vargha A, Delaney HD. The Kruskal-Wallis test and stochastic homogeneity. J Educ Behav Stat 1998; 23 (02) 170-192
    CrossrefGoogle Scholar
  • 24 Tarrant M, Ware J. Impact of item-writing flaws in multiple-choice questions on student achievement in high-stakes nursing assessments. Med Educ 2008; 42 (02) 198-206
    CrossrefPubMedGoogle Scholar
  • 25 Pham H, Besanko J, Devitt P. Examining the impact of specific types of item-writing flaws on student performance and psychometric properties of the multiple choice question. MedEdPublish 2018; 7 (04) 4
    Google Scholar
  • 26 Garrison MJ, Hansen JD, Knoepfle TW. An analysis of multiple-choice questions from business law testbanks and from the CPA examination. J Leg Stud Educ 1997; 15 (01) 91-105
    CrossrefGoogle Scholar
  • 27 Bailey CD, Karcher JN, Clevenger B. A comparison of the quality of multiple-choice questions from CPA exams and textbook test banks. Accounting Educators' Journal. 1998; 10 (02) 12-30
    Google Scholar
  • 28 Moncada SM, Harmon M. Test item quality: an assessment of accounting test banks. Journal of Accounting & Finance Research. 2004; 12 (04) 28-39
    Google Scholar
  • 29 Ibbett NL, Wheldon BJ. The incidence of clueing in multiple choice testbank questions in accounting: some evidence from Australia. e-Journal of Business Education and Scholarship of Teaching 2016; 10 (01) 20-35
    Google Scholar
  • 30 Royal KD, Hedgpeth M-W, Posner LP. A simple methodology for discerning item construction flaws in health professions examinations. Health Prof Educ 2019; 5 (01) 82-89
    CrossrefGoogle Scholar
  • 31 O'Neill LD, Mortensen S, Nørgaard C, Holm AL, Friis UG. Screening for technical flaws in multiple-choice items: a generalizability study. Dansk Universitetspaedagogisk Tidsskrift. 2019; 14 (26) 51-65
    CrossrefGoogle Scholar
  • 32 Richman H, Hrezo MJ. The trouble with test banks. Perspectives In Learning. 2017; 16 (01) 3-8
    Google Scholar
  • 33 Mulready-Shick J, Edward J, Sitthisongkram S. Developing local evidence about faculty written exam questions: Asian ESL nursing student perceptions about linguistic modification. Nurs Educ Perspect 2020; 41 (02) 109-111
    CrossrefPubMedGoogle Scholar