Actionable companion · Assessment and evidence of learning

From evidence to district action: Comparison of Large Language Model Performance on the United Kingdom Neurology Specialty Certificate Examination

AI-assisted editorial implementation guide. The district actions are proposed applications, not findings or recommendations attributed to the original authors. No additional peer review is asserted.

Evidence and context

Health-professions education context: test transfer to K–12 rather than assuming it.

Original authors’ abstract, reproduced under the linked source license:

Background Large language models (LLMs) are increasingly used in medical education; however, a gap remains in the literature comparing the performance of different LLMs on neurology-specific higher-specialty training assessments. This study aimed to compare the performance of three leading LLMs, including ChatGPT (OpenAI, San Francisco, CA, USA), Claude (Anthropic, San Francisco, CA, USA), and Gemini (Google LLC, Mountain View, CA, USA), on UK Neurology Specialty Certificate Examination (SCE) sample questions. Materials and methods This comparative cross-sectional performance evaluation used 83 official UK Neurology SCE single-best-answer (SBA) questions obtained from the Membership of the Royal Colleges of Physicians of the UK (MRCP (UK)). Explicit permission was obtained from MRCP (UK) for research and publication. Evaluated models included ChatGPT (version 5.5, thinking mode), Claude (Opus 4.6), and Gemini (Pro, thinking mode). Each LLM was presented with an identically formatted question set and identical prompts, instructing it to select the SBA and provide a brief explanation limited to 200 words. Each question was entered into a new chat session, so responses were generated independently of previous items. Official answers were used as the reference standard. Responses were coded as correct or incorrect, with binary answer accuracy as the primary outcome. Mixed-effects logistic regression compared the probability of a correct answer between LLMs, with question ID as a random intercept to account for question-level variation. Pairwise comparisons were adjusted using Holm correction to account for multiple testing. Confirmatory paired analyses were performed using Cochran’s Q test and exact McNemar tests. Results ChatGPT answered 78 of 83 questions correctly, corresponding to an accuracy of 94.0% (95% Wilson CI 86.7-97.4%). Gemini also answered 78 of 83 questions correctly, with an accuracy of 94.0% (95% Wilson CI 86.7-97.4%), while Claude answered 66 of 83 questions correctly, corresponding to a lower accuracy of 79.5% (95% Wilson CI 69.6-86.8%). The mixed-effects model demonstrated a significant association between LLM identity and answer accuracy. Confirmatory paired analyses supported these findings. Cochran’s Q test indicated a significant overall difference in accuracy among models (Q = 16.94, df = 2, p = 0.00021). Exact McNemar tests confirmed that ChatGPT and Gemini both significantly outperformed Claude, with no significant difference between them. Conclusions ChatGPT and Gemini demonstrated identical accuracy on official UK Neurology SCE sample questions, while Claude achieved notably lower accuracy on this dataset. Together, these findings suggest LLMs may perform well on selected specialist neurology training assessment questions and favor their use as a supplementary learning tool. However, these findings are based on a relatively small sample of questions and, in isolation, do not translate into real-world clinical competence or reasoning ability. Further studies should evaluate the quality of LLM clinical reasoning through predefined mark schemes and specialty-specific expert review.

A 90-day district pilot

1. Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
2. Days 16–30 · Design. Redesign one low-stakes assessment: require a first attempt, an AI-assisted revision, a source-check log and a short oral explanation. Use the same rubric for the comparison task. Name an accountable owner and set the baseline, success threshold and budget before implementation.
3. Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
4. Days 61–90 · Decide. Blind-score reasoning, factual accuracy and unaided transfer on a fresh task. Record suspected misconduct separately from demonstrated evidence. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Josiah Cho; Arvindh Sekaran. Comparison of Large Language Model Performance on the United Kingdom Neurology Specialty Certificate Examination. Cureus (2026-07-10). https://doi.org/10.7759/cureus.112413

Companion prepared 2026-10-30. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.