Agentic education · Learning science

Actionable companion · Learning design & outcomes

Learning with agency: Chatgpt vs traditional pedagogy: a comparative study in urological learning

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not claims or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paperSearch paper agentCC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Higher education. Verify the participant population and task before transferring the findings.

Original authors’ abstract, reproduced under the linked source license:

Abstract Purpose Technological evolution is radically changing medical learning models. We evaluated the learning outcomes of urological concepts using ChatGPT, traditional lecture and combined approach. Methods We conducted a randomized triple-blind study on 121 medical students with no previous formal curriculum in urology. Students were randomly divided into three study classes with different learning methods: ChatGPT, Lecture and ChatGPT + Lecture. The “adrenal glands” were randomly extracted as the subject of the lessons. Students were evaluated using a thirty-question test. Results The evaluation test median score was higher for students who underwent ChatGPT + Lecture compared with those who had only ChatGPT (10 vs. 12, p = 0.007). Such differences remained statistically significant also in multivariable models adjusting according to year of course, gender and previous ChatGPT experience (estimate: 2.6, p-value = 0.002). For most of the questions (about 70%), the proportion of students correctly answering was higher in the ChatGPT + Lecture learning groups than in the other groups. Conclusion ChatGPT loses its potential if used without a previous background. The limits of scientific reliability persist and a teacher-guided method is still essential. ChatGPT + traditional lecture gives more effective results than the single traditional lecture also allowing a better use of the chatbot. Supplementary Information The online version contains supplementary material available at 10.1007/s00345-025-05654-w. Keywords: ChatGPT, Learning methods, Medical education, Pedagogy, Urology

What kind of evidence is this?

Conversational & pedagogical agents · Evidence synthesis

Stage: Higher education. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

We conducted a prospective, single-center, randomized, triple-blind study on a total of 121 students enrolled in the 3rd and 4th year of the Medicine and Surgery course at the University “G. d’Annunzio” with no previous formal curriculum in urology.

Limitations and uncertainty · source passage

Many studies tested ChatGPT performance using university exam and medical licensing test questions, to analyze the accuracy of the answers [11–13]. Kung et al. [12] found that ChatGPT answered United States Medical Licensing Exam (USMLE) questions with good accuracy (> 50%) and high agreement. Gilson et al. [16] also tested ChatGPT using questions from the USMLE exam. It achieved a passing score, providing logical and informative responses. Conversely, Fijačko et al. [17], showed that ChatGPT failed the Basic Life Support (BLS) and Advanced Cardiovascular Life Support (ACLS) exams, despite its scientifically relevant answers. In Eysenbach’s study [18], the potential and limits of ChatGPT are empirically and broadly tested through conversation with the tool. In several points the user must direct and correct the chatbot with suggestions about the topics, the correct way to conduct an anamnesis, or provide information to a patient within the simulations carried out. Clear limitations are still those expressed in the interview by ChatGPT itself: the lack of direct generation of diagrams and animations, the absence of critical analysis skills, the lack of access to all scientific articles, the need for professional medical supervision for critical content analysis.

Design the system before the pilot

Define whether the agent asks questions, gives hints, explains or assesses. Test hallucinated feedback, excessive praise, misleading confidence and responses when the learner is confused. A conversational interface alone does not establish autonomous agency.

  • Specify a measurable learning objective, a comparator and a task without tool assistance before adopting a system.

A 90-day district pilot

  1. Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
  2. Days 16–30 · Design. Choose one disciplinary task. Require a first attempt, a documented AI consultation, a critique of the output and an independent explanation. Align the comparison group and scoring rubric with the same objective. Name an accountable owner and set the baseline, success threshold and budget before implementation.
  3. Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
  4. Days 61–90 · Decide. Blind-score reasoning and accuracy on a new task without AI. Distinguish model examination scores, student perceptions and actual learner gains; only the last directly tests educational benefit. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Alessio Digiacomo; Angelo Orsini; Rossella Cicchetti; Ludovica Spadano; Sara De Santis; Laura Di Sessa; Miriana Vitale; Marta Di Nicola; Flavia Tamborino; Martina Basconi; Riccardo De Archangelis; Gaetano Salzano; Guglielmo Dello Stritto; Peppino Lannutti; Luigi Schips; Michele Marchioni. Chatgpt vs traditional pedagogy: a comparative study in urological learning. World Journal of Urology (2025-05-08). https://doi.org/10.1007/s00345-025-05654-w

Companion prepared 2026-10-30. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.