Agentic education · Learning science

Actionable companion · Feedback & scaffolding

Learning with agency: Ambient AI Scribes to Create Educational Feedback Notes for Medical Students: Randomized Trial

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not claims or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paperSearch paper agentCC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Higher education. Health-professions learning must be distinguished from K–12 instruction.

Original authors’ abstract, reproduced under the linked source license:

Abstract Background High-quality observation and feedback contribute to the development of clinical competence and professional growth in medical education. Faculty often struggle to translate verbal observations into written feedback because of documentation burden and competing demands. Ambient artificial intelligence (AI) scribes, already adopted in clinical practice, may address this challenge by capturing verbal exchanges and generating structured notes. Objective The purpose of this study was to examine the use of ambient AI scribes to generate educational feedback notes during a formative medical interviewing workshop for first-year medical students in March and April 2025. Methods Thirteen instructors were randomized to control (human-only) or intervention (AI scribe–assisted) workflows to complete narrative feedback forms. The intervention group used an AI scribe to generate transcripts of student-instructor encounters, which were then summarized into feedback notes using a large language model and edited by the instructors before submission. All narratives were scored using the Evaluation of Feedback Captured Tool (EFeCT). Factual accuracy of a subsample of unedited AI feedback summaries was reviewed against the source transcripts. Task load and usability were measured using NASA Task Load Index and System Usability Scale, respectively. Results Instructors submitted feedback on 92.2% (94/102) of the students. EFeCT scores on the scale from 0 to 5 were higher for human-edited AI narratives (median 3.00, IQR 2.00-4.00) and unedited AI summaries (median 3.00, IQR 3.00-4.00) than for human-only narratives (median 2.00, IQR 1.75-3.00; P<.001). Human-only narratives were shorter than AI-assisted outputs (P<.001). Review of 117 AI-generated feedback elements showed a 6.8% (n=8) mischaracterization and 1.7% (n=2) hallucination rate, with most errors corrected during editing. Task load was high, and usability was marginal in both the control and intervention groups, with no significant differences (P=.31 and P=.40, respectively). Conclusions An ambient AI scribe–assisted workflow improved the quality of written narrative feedback with no observed increase in instructor effort compared to human-only documentation. Although occasional inaccuracies required review, this innovation has the potential to transform feedback documentation.

What kind of evidence is this?

Supporting AI learning science · Randomized study reported

Stage: Higher education. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

This study took place in spring 2025 as part of a formative medical interviewing workshop for all first-year medical students at Yale School of Medicine. During the workshop, each medical student spent 20 minutes conducting a complete history with an SP while being observed by an instructor and 3 peers. Students and instructors were permitted to call “time-out” for interspersed guidance and feedback [6]. Upon completion of the interview, 10 minutes were allotted for self-, peer, and instructor feedback. SPs provided no feedback. The exercise was repeated until all 4 students in the room had interviewed separate SPs portraying different scenarios and received individual verbal feedback. After the workshop, instructors completed a postsession feedback narrative (Multimedia Appendix 1) for each student within a learning management system (Medtrics Lab LLC). The workshop was cycled over 4 afternoons, with each student participating once. Instructors devoted 3 and a half hours per workshop, including presession faculty development. This was the seventh iteration of the workshop in the first-year clinical skills curriculum, with each workshop structured similarly, offering students opportunities for repeat practice as their foundational knowledge grows during medical school.

Limitations and uncertainty · source passage

This study has several limitations. It was conducted at a single institution with a small number of highly trained educators, limiting generalizability. Participants were diligent in reviewing AI outputs, which may not reflect the behaviors of a broader faculty population in usual teaching environments. However, in real-world practice, we can reasonably expect human-only narratives to be less diligently submitted, potentially further increasing the observed differences in quality between AI-assisted and human-only narratives.

Design the system before the pilot

Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.

  • Give feedback on the learner’s reasoning, then fade hints. Record whether the learner can complete the next task without that scaffold.

A 90-day district pilot

  1. Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
  2. Days 16–30 · Design. Choose one disciplinary task. Require a first attempt, a documented AI consultation, a critique of the output and an independent explanation. Align the comparison group and scoring rubric with the same objective. Name an accountable owner and set the baseline, success threshold and budget before implementation.
  3. Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
  4. Days 61–90 · Decide. Blind-score reasoning and accuracy on a new task without AI. Distinguish model examination scores, student perceptions and actual learner gains; only the last directly tests educational benefit. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Jaideep S Talwalkar; David Chartash; Lisa Zhang; Michael Makutonin; Conrad W Safranek; Anne Elizabeth Sidamon-Eristoff; Lee H Schwamm; Donald S Wright. Ambient AI Scribes to Create Educational Feedback Notes for Medical Students: Randomized Trial. JMIR Medical Education (2026-05-28). https://doi.org/10.2196/89996

Companion prepared 2026-10-30. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.