Agentic education · Learning science

Actionable companion · Assessment & reasoning

Learning with agency: Let's chat(GPT): Implementation of a ChatGPT-generated social determinants of health activity

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not claims or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paperSearch paper agentCC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Higher education, Adult & professional learning. Verify the participant population and task before transferring the findings.

Original authors’ abstract, reproduced under the linked source license:

Abstract Background Artificial intelligence (AI)- powered chatbots have provided some notable benefits for learners. Educators are beginning to explore their possible utility and find ways to leverage AI in their classrooms. Objective This study aimed to evaluate the implementation of ChatGPT-generated social determinants of health (SDOH) activity in a team-based pharmacy education course. Methods Instructors asked the software to generate a set of learning objectives, an in-class activity, assessment strategies, and summative assessments for a student's conceptual understanding of SDOH. During a required first-year, team-based pharmacy course at the University of Tennessee Health Science Center, participants (n = 95) completed a ChatGPT-generated, in-class activity on SDOH within groups. The students' views on the quality of the activity were evaluated using five Likert-scale questions. Four of the questions assessed the applicability and usefulness of the assignment, with rankings on a scale of 1–4 (1 = strongly disagree; 4 = strongly agree). The fifth question evaluated the quality of the activity compared to activities generated by the instructor, using a scale of 1–5 (1 = far worse; 5 = far better). Results For applicability and usefulness,”94.7 % (n = 90) of students agreed that “This in-class exercise was valuable to my professional development as a pharmacist;” 96.8 % (n = 92) agreed with “It is necessary for pharmacists to understand SDOH;” 94.7 % (n = 90) agreement with the statement, “The quality of this in-class activity was on-par with other in-class activities in the course;” and 90.5 % (n = 86) agreed with “This in-class exercise was just as impactful to my professional development as other in-class activities.” The majority of students (63.2 %; n = 60) selected either “somewhat better” (40 %; n = 38) or far better (23.2 %; n = 22) for, “Regarding quality, I feel the in-class activity was ____ than other in-class activities.” Conclusions Most students reported that the ChatGPT-generated activity on social determinants of health was useful, applicable, and somewhat or far better than instructor activities. However, AI can generate incorrect information and potentially hinder student learning of conceptual frameworks; thus, instructors should review all output carefully. Keywords: Artificial intelligence, ChatGPT, Social determinants of health, Pharmacy education, Team-based learning

What kind of evidence is this?

Conversational & pedagogical agents · Design requires full-text appraisal

Stage: Higher education; Adult & professional learning. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

Appendix B includes the final version of the ChatGPT-generated in-class activity. Participants (n = 95) completed the ChatGPT-designed SDOH activity within their pre-defined groups (groups assigned by the instructor) of 5–6 students, for a total of 16 groups. Students were not aware that this was a ChatGPT-generated activity.

Limitations and uncertainty · source passage

When using ChatGPT to create classroom content, it is crucial to remember that the premise of ChatGPT is to generate material. As noted above, when prompted for “Validated tools to assess student understanding of social determinants of health,” ChatGPT generated several false SDOH assessment scales with erroneous references. Because no validated tools exist to assess SDOH, the program generated a response, but it created invalid, non-existent content to meet the needs of the prompt entered. ChatGPT's capacity to generate non-existent information, often called “hallucinations,” poses a significant challenge. These hallucinations arise from algorithmic limitations and training data constraints, leading the chatbot to produce seemingly valid but ultimately inaccurate information.22,23 An alternative term, “confabulations,” coined by Hatem et al., more accurately describes this phenomenon as a fabrication within the data.24 Thus, cautious use and fact-checking are essential when relying on chatbots for information.1 This highlights the necessity of a critical mindset for reviewing any AI-generated materials. To the general population, these programs can offer valuable insight and information; however, for educators responsible for preparing students with accurate and up-to-date resources, the strict reliance on ChatGPT-generated materials is not recommended. Furthermore, educators should assess all items' accuracy, utility, and applicability to a student's development and learning goals for the course.

Design the system before the pilot

Define whether the agent asks questions, gives hints, explains or assesses. Test hallucinated feedback, excessive praise, misleading confidence and responses when the learner is confused. A conversational interface alone does not establish autonomous agency.

  • Use common rubrics, blind scoring where feasible and a fresh unaided reasoning task. Document permissible assistance and avoid treating AI-detector scores as proof of misconduct.

A 90-day district pilot

  1. Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
  2. Days 16–30 · Design. Choose one disciplinary task. Require a first attempt, a documented AI consultation, a critique of the output and an independent explanation. Align the comparison group and scoring rubric with the same objective. Name an accountable owner and set the baseline, success threshold and budget before implementation.
  3. Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
  4. Days 61–90 · Decide. Blind-score reasoning and accuracy on a new task without AI. Distinguish model examination scores, student perceptions and actual learner gains; only the last directly tests educational benefit. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Karl R Kodweis; Theodore J Cory; Elizabeth A Hall; Christa M George; Katherine L March. Let's chat(GPT): Implementation of a ChatGPT-generated social determinants of health activity. Exploratory Research in Clinical and Social Pharmacy (2025-05-06). https://doi.org/10.1016/j.rcsop.2024.100553

Companion prepared 2026-10-30. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.