Agentic education · Learning science

Actionable companion · Feedback & scaffolding

From AI evidence to responsible practice: Development and pilot evaluation of a large language model-based clinical simulation system for medical education in gastroenterology

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not findings or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paperSearch paper agentCC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Age / stage not established. Health-professions education findings do not establish effectiveness in K–12 classrooms.

Original authors’ abstract, reproduced under the linked source license:

Background Medical education relies on experience to develop clinical reasoning skills, yet patient access is often limited. Large language models (LLMs) offer an accessible alternative for simulating patient encounters. We aimed to develop and evaluate an LLM-based clinical simulation system for gastroenterology that simulates entire clinical encounters in natural language, from history-taking through to treatment and follow-up planning. Methods The study was conducted in three phases. First, a system comprising an LLM-based clinical case generator, an LLM-based patient simulator, and a web chat interface was developed and iteratively refined. Second, two gastroenterology fellows performed 20 clinical case simulations each as initial testers, providing feedback that guided further system adjustments. Third, six independent gastroenterology fellows, blinded to the development process, evaluated the final system by completing the same 20 clinical simulations. All evaluators rated the system using a 5-point Likert scale across four domains: case presentation accuracy, adaptability to user interactions, realism, and educational value. Results A total of 160 evaluations were collected across 20 cases and 8 raters. At final evaluation, all domains scored highly: educational value (mean 4.57 ± 0.63), accuracy (4.42 ± 0.63), adaptability (4.33 ± 0.75), and realism (4.32 ± 0.81). Scores differed significantly across domains (p < 0.01), with educational value rated highest. Evaluation panellists rated all domains significantly higher than initial testers (all p < 0.01) at an exploratory comparison. Conclusions The LLM-based clinical simulator demonstrated high accuracy, realism, adaptability, and educational value for gastroenterology case simulation in this pilot evaluation, and may represent an accessible and cost-effective tool for clinical reasoning training. Supplementary Information The online version contains supplementary material available at https://doi.org/10.1186/s12909-026-09709-3.

What kind of evidence is this?

Supporting AI learning science · Model benchmark / tool evaluation

Stage: Age / stage not established. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

The study was conducted in three phases: In the first phase the clinical simulator was developed and iteratively refined. Then in a second phase, initial testing and evaluation of the system was performed by two testers (GAM, CC). Based on feedback from the initial testers, adjustments were made to the system to improve the quality of the simulation system. Finally, in the third phase a final pilot evaluation panel composed of six gastroenterology fellows (AA, CS, EB, GM, GM, PM), who were blinded to and not involved in the development and initial evaluation phases, independently evaluated the final version of the system.

Limitations and uncertainty · source passage

Medical education requires interaction with patients to acquire clinical reasoning skills, but access to real patients can often be limited [1]. Simulation-based methods can provide additional clinical reasoning practice in a safe, controlled environment, and have historically employed actors trained to portray medical conditions consistently, known as standardized patients [1, 2]. While standardised patients have previously been shown to be cost-effective [1], they also have several limitations including training requirements, cost, and issues with scheduling and limited availability.

Design the system before the pilot

Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.

  • Give feedback on the learner’s reasoning, then fade hints. Record whether the learner can complete the next task without that scaffold.
  • Use common rubrics, blind scoring where feasible and a fresh unaided reasoning task. Document permissible assistance and avoid treating AI-detector scores as proof of misconduct.
  • Co-design accommodations with learners and specialists. Check access and error burden across groups without inferring subgroup effectiveness from small or unreported samples.

A 90-day district pilot

  1. Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
  2. Days 16–30 · Design. Have a subject specialist reproduce a small set of the paper’s tool-evaluation tasks using the documented model and prompt conditions. Keep this as a tool-reliability exercise until a separate learner study is designed. Require learners to attempt equivalent tasks independently before any AI comparison. Name an accountable owner and set the baseline, success threshold and budget before implementation.
  3. Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
  4. Days 61–90 · Decide. Report tool accuracy and error severity separately from learner outcomes. Measure learner understanding and transfer with unaided parallel tasks; model examination scores are not evidence of improved student learning. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Stiliano Maimaris; Gianmarco Tedino; Giusi Aurora Memoli; Carlotta Crisciotti; Alessandra Albrizio; Camilla Sammartino; Erica Bartolotta; Giulio Massetti; Giulia Mantica; Paolo Minerba; Federico Biagi; Annalisa Schiepatti. Development and pilot evaluation of a large language model-based clinical simulation system for medical education in gastroenterology. BMC Medical Education (2026-06-25). https://doi.org/10.1186/s12909-026-09709-3

Companion prepared 2026-11-01. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.