Actionable companion · Assessment & reasoning

From AI evidence to responsible practice: Assessing the Utility of AI Versus Human-Created MCQs in Pediatric Medical Education

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not findings or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paper Search paper agent CC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Higher education. Health-professions education findings do not establish effectiveness in K–12 classrooms.

Original authors’ abstract, reproduced under the linked source license:

Background Multiple-choice questions (MCQs) remain central to assessment in medical education, but their development is resource intensive. Generative artificial intelligence (AI) offers a potential solution by automating MCQ creation. However, little is known about the psychometric quality of AI-generated MCQs compared with human-authored items, particularly in pediatric education. Objective This study aimed to directly compare the quality of AI- and human-generated MCQs in pediatrics using item analysis grounded in classical test theory. Methods A formative exam comprising both AI (Microsoft Copilot) and human-generated pediatric MCQs was administered to 4th-year medical students. Item analysis was performed to calculate difficulty indices, discrimination indices, item-total correlations, and distractor functioning. Reliability was assessed using KR-20. Descriptive and inferential statistics, including paired t-tests, compared performance between AI and human items. Results Human-authored questions outperformed AI-generated questions across all quality indicators. AI questions showed lower discrimination (mean 0.19 vs. 0.29) and a higher proportion outside the acceptable difficulty range (56% vs. 32%). Distractor analysis also favored human questions, with fewer nonfunctioning distractors and more items containing fully functional distractors. While some AI items met ideal psychometric thresholds, overall consistency was lower. Conclusion Generative AI in its current form cannot yet match human expertise in producing consistently high-quality MCQs for pediatrics. However, AI shows potential as a supplementary tool, particularly within hybrid human–AI workflows that combine efficiency with expert oversight. These findings highlight both the opportunities and limitations of AI in medical education assessment and underscore the importance of balancing reliability, validity, acceptability, and cost-effectiveness when integrating AI into assessment design.

What kind of evidence is this?

Supporting AI learning science · Design requires full-text appraisal

Stage: Higher education. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

The key strength of this study is the use of a cohort of medical students as test-takers, which allows the analysis of the generated questions within a realistic context which reflects the way the MCQs are used most in medical education. The use of a test taking cohort is also provided the ability to quantitatively compare the AI and human groups through item and distractor analysis. Additionally, a large sample size of 157 participants allows for sufficient power for the item analysis. Previous work assessing AI MCQ questions in a medical education context have assessed question quality using expert panels or student perception.14,15 Another strength of this study compared to previous work on generative AI MCQs is that respondents were not asked to rank the quality of the questions and were blinded as to the origin of the question. 14 This addresses the issue that respondents may be inclined toward perceiving questions as higher or lower quality based on the belief that they are AI generated. 41

Limitations and uncertainty · source passage

The key strength of this study is the use of a cohort of medical students as test-takers, which allows the analysis of the generated questions within a realistic context which reflects the way the MCQs are used most in medical education. The use of a test taking cohort is also provided the ability to quantitatively compare the AI and human groups through item and distractor analysis. Additionally, a large sample size of 157 participants allows for sufficient power for the item analysis. Previous work assessing AI MCQ questions in a medical education context have assessed question quality using expert panels or student perception.14,15 Another strength of this study compared to previous work on generative AI MCQs is that respondents were not asked to rank the quality of the questions and were blinded as to the origin of the question. 14 This addresses the issue that respondents may be inclined toward perceiving questions as higher or lower quality based on the belief that they are AI generated. 41

Design the system before the pilot

Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.

Use common rubrics, blind scoring where feasible and a fresh unaided reasoning task. Document permissible assistance and avoid treating AI-detector scores as proof of misconduct.

A 90-day district pilot

Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
Days 16–30 · Design. Choose one disciplinary task. Require a first attempt, a documented AI consultation, a critique of the output and an independent explanation. Align the comparison group and scoring rubric with the same objective. Name an accountable owner and set the baseline, success threshold and budget before implementation.
Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
Days 61–90 · Decide. Blind-score reasoning and accuracy on a new task without AI. Distinguish model examination scores, student perceptions and actual learner gains; only the last directly tests educational benefit. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

James Knight; Richard G. McGee; Bunmi S. Malau-Aduli; for the ACHIEVE network. Assessing the Utility of AI Versus Human-Created MCQs in Pediatric Medical Education. Journal of Medical Education and Curricular Development (2026-01-01). https://doi.org/10.1177/23821205261427885

Companion prepared 2026-11-01. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.