Improving the Quality of MCQs by Enhancing Cognitive Level and using Psychometric Analysis: Improving the quality of MCQs by enhancing cognitive level

Muhammad Faisal  Rahim; Shireen  Qassim Bham; Shahjabeen Khan; Tahir  Ansari; Masood  Ahmed

doi:10.54393/pjhs.v4i04.700

Authors

Muhammad Faisal Rahim Department of Medical Education, Fazaia Ruth Pfau Medical College, AIR University, Karachi, Pakistan
Shireen Qassim Bham Department of Paediatrics, Fazaia Ruth Pfau Medical College, AIR University, Karachi, Pakistan
Shahjabeen Khan Department of Physiology, Fazaia Ruth Pfau Medical College, AIR University, Karachi, Pakistan
Tahir Ansari Department of Medical Education, Fazaia Ruth Pfau Medical College, AIR University, Karachi, Pakistan
Masood Ahmed Department of Anatomy, Fazaia Ruth Pfau Medical College, AIR University, Karachi, Pakistan

DOI:

https://doi.org/10.54393/pjhs.v4i04.700

Keywords:

Item Analysis, Cognition, Discrimination Index, Difficulty Index, Distractor

Abstract

Faculty development programs are an essential part of any reputable institution. Faculty training through various conferences and workshops will help develop competencies for high quality assessment. Objective: To evaluate the effectiveness of ongoing faculty training initiatives aimed at improving the standard of MCQ development. Methods: Faculty members of FRPMC were engaged in this observational, quasi-experimental study. Convenient sampling was done. Three different questionnaires were provided to the participants. The participants were tasked with accurately identifying flaws, cognitive levels in the pre- and post-test questionnaire items, as well as post hoc analysis to discard or modify the Items. Results: Items with multiple flaws were assessed with mean value in pre-test score was 1.349 compared to post-test score 3.442 which were statistically significant (p-value <0.05). The pre- and post-test questionnaire to correctly identify the cognitive levels showed pre-test 77.5% compared to 87.5% participants identified correct cognitive level in post-test. In post hoc analysis, participants were able to identify 5 questions out of 7 which need to be discarded and 3 questions out of 5 which needed improvement according to the key given to them. Conclusions: Our study revealed that workshop helped faculty identify item flaws with significant improvement in pre- and post-test scores. The cognitive capabilities of faculty were improved in all three levels of cognition, with greatest improvements shown in higher complexity questions (C3). There was a noticeable improvement in knowledge and the participant's capacity to identify and correct errors.

References

Lockyer J, Carraccio C, Chan MK, Hart D, Smee S, Touchie C, et al. Core principles of assessment in competency-based medical education. Medical Teacher. 2017 Jun; 39(6): 609-16. doi: 10.1080/0142159X.2017.1315082.

Schuwirth LW and van der Vleuten CP. A history of assessment in medical education. Advances in Health Sciences Education. 2020 Dec; 25(5): 1045-56. doi: 10.1007/s10459- 020-10003-0.

Farooqui F, Saeed N, Aaraj S, Sami MA, Amir M. A Comparison Between Written Assessment Methods: Multiple-choice and Short Answer Questions in End-of-clerkship Examinations for Final Year Medical Students. Cureus. 2018 Dec; 10(12): e3773. doi: 10.7759/cureus.3773.

Holzinger A, Lettner S, Steiner-Hofbauer V, Capan Melser M. How to assess? Perceptions and preferences of undergraduate medical students concerning traditional assessment methods. BMC Medical Education. 2020 Sep; 20(1): 312. doi: 10.1186/s12909-020-02239-6.

Adnan S, Sarfaraz S, Nisar MK, Jouhar R. Faculty perceptions on one-best MCQ development. The Clinical Teacher. 2023 Feb; 20(1): e13529. doi: 10.1111/tct.13529.

Balaha MH, El-Ibiary MT, El-Dorf AA, El-Shewaikh SL, Balaha HM. Construction and Writing Flaws of the Multiple-Choice Questions in the Published Test Banks of Obstetrics and Gynecology: Adoption, Caution, or Mitigation? Avicenna Journal of Medical Sciences. 2022 Aug; 12(3): 138-47. doi: 10.1055/s-0042-1755332.

Arooj M, Mukhtar K, Khan RA, Azhar T. Assessing the educational impact of cognitive level of MCQ and SEQ on learning approaches of dental students. Pakistan Journal of Medical Sciences. 2021 Mar; 37(2): 445-9. doi: 10.12669/pjms.37.2.3475.

Pickford JC and Newcomb LH. Relationship of cognitive level of instruction to students' cognitive level of achievement. NACTA Journal. 1989 Jun; 33(2): 56-9.

Zaidi NB, Hwang C, Scott S, Stallard S, Purkiss J, Hortsch M. Climbing Bloom's taxonomy pyramid: Lessons from a graduate histology course. Anatomical Sciences Education. 2017 Sep; 10(5): 456-464. doi: 10.1002/ase.1685.

Stringer JK, Santen SA, Lee E, Rawls M, Bailey J, Richards A, et al. Examining Bloom's Taxonomy in Multiple Choice Questions: Students' Approach to Questions. Medical Science Education. 2021 May; 31(4): 1311-7. doi: 10.1007/s40670-021-01305-y.

Khan MU and Aljarallah BM. Evaluation of Modified Essay Questions (MEQ) and Multiple Choice Questions (MCQ) as a tool for Assessing the Cognitive Skills of Undergraduate Medical Students. International Journal of Health Sciences. 2011 Jan; 5(1): 39-43.

Grainger R, Dai W, Osborne E, Kenwright D. Medical students create multiple-choice questions for learning in pathology education: a pilot study. BMC Medical Education. 2018 Aug; 18(1): 201. doi: 10.1186/s12909-018-1312-1.

Kowash M, Hussein I, Al Halabi M. Evaluating the Quality of Multiple Choice Question in Paediatric Dentistry Postgraduate Examinations. Sultan Qaboos University Medical Journal. 2019 May; 19(2): e135-e141. doi: 10.18295/squmj.2019.19.02.009.

Islam ZU and Usmani A. Psychometric analysis of Anatomy MCQs in Modular examination. Pakistan Journal of Medical Sciences. 2017 Sep; 33(5): 1138-43. doi: 10.12669/pjms.335.12382.

Sajjad M, Iltaf S, Khan RA. Nonfunctional distractor analysis: An indicator for quality of Multiple choice questions. Pakistan Journal of Medical Sciences. 2020 Jul; 36(5): 982-6. doi: 10.12669/pjms.36.5.2439.

Ali SH, Carr PA, Ruit KG. Validity and Reliability of Scores Obtained on Multiple-Choice Questions: Why Functioning Distractors Matter. Journal of the Scholarship of Teaching and Learning. 2016 Feb; 16(1): 1-4. doi: 10.14434/josotl.v16i1.19106.

Abdulghani HM, Ahmad F, Irshad M, Khalil MS, Al-Shaikh GK, Syed S, et al. Faculty development programs improve the quality of Multiple Choice Questions items' writing. Scientific Reports. 2015 Apr; 5(1): 1-7. doi: 10.1038/srep09556.

Nemec EC and Welch B. The impact of a faculty development seminar on the quality of multiple-choice questions. Currents in Pharmacy Teaching and Learning. 2016 Mar; 8(2): 160-3. doi: 10.1016/j.cptl.2015.12.008.

AlFaris E, Naeem N, Irfan F, Qureshi R, Saad H, Al Sadhan RE, et al. A one‐day dental faculty workshop in writing multiple‐choice questions: an impact evaluation. Journal of Dental Education. 2015 Nov; 79(11): 1305-13. doi: 10.1002/j.0022-0337.2015.79.11.tb06026.x.

Dellinges MA and Curtis DA. Will a short training session improve multiple‐choice item‐writing quality by dental school faculty? A pilot study. Journal of Dental Education. 2017 Aug; 81(8): 948-55. doi: 10.21815/JDE.017.047.

Rush BR, Rankin DC, White BJ. The impact of item-writing flaws and item complexity on examination item difficulty and discrimination value. BMC Medical Education. 2016 Dec; 16(1): 1-10. doi: 10.1186/s12909-016-0773-3.

Rauf AR and Sultana S. Effect of faculty training on quality of Multiple Choice Questions. Rawal Medical Journal. 2021 Dec; 46(2): 430.

Hingorjo MR and Jaleel F. Analysis of one-best MCQs: the difficulty index, discrimination index and distractor efficiency. Journal of Pakistan Medical Association. 2012 Feb; 62(2): 142-7.

Abdulghani HM, Irshad M, Haque S, Ahmad T, Sattar K, Khalil MS. Effectiveness of longitudinal faculty development programs on MCQs items writing skills: A follow-up study. PloS One. 2017 Oct; 12(10): e0185895. doi: 10.1371/journal.pone.0185895.

Tenzin K, Dorji T, Tenzin T. Construction of multiple choice questions before and after an educational intervention. Journal of Nepal Medical Association. 2017 Jan; 56(205): 112-6. doi: 10.31729/jnma.2976.

Gupta P, Meena P, Khan AM, Malhotra RK, Singh T. Effect of faculty training on quality of multiple-choice questions. International Journal of Applied and Basic Medical Research. 2020 Jul; 10(3): 210. doi: 10.4103/ijabmr.IJABMR_30_20.