Actionable companion · Feedback & scaffolding
From AI evidence to responsible practice: Development and content validity of an AI-assisted simulation scenario for orthotic fabrication in occupational therapy education
AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not findings or recommendations attributed to the source authors. No independent peer review of this guide is asserted.
Evidence and context
Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Age / stage not established. Verify the participants, intervention and comparator before applying this study in a new setting.
Original authors’ abstract, reproduced under the linked source license:
Abstract Background Developing high-fidelity simulation scenarios is a time-consuming process, and validated resources for complex psychomotor skills such as orthotic fabrication are scarce. This study aimed to examine the content validity index (CVI) of a simulation scenario for anticlaw orthosis fabrication developed with the support of artificial intelligence (AI). Methods A mixed-methods methodological study was conducted in an academic education program in Türkiye. Six subject matter experts with professional experience in hand rehabilitation, orthotics, and simulation-based education participated. A simulation scenario package was structured by the researchers, while the specific scenario flow was generated using a seven-step process supported by the generative AI tool ChatGPT. Expert reviewers evaluated five components, specifically objectives, background, prebriefing, scenario flow, and debriefing, using a 4-point Likert scale and open-ended questions. The item-level content validity index (I-CVI) and scale level content validity index (S-CVI) were calculated. Results The simulation scenario package was evaluated across five core components. The scenario achieved an I-CVI and S-CVI of 1.00. This indicated that all expert reviewers rated the scenario components as content valid by assigning a score of 3 or 4. Qualitative feedback highlighted the need for refinements in clinical fidelity, specific material choices, and safety warnings regarding heat and sharp tools. Conclusions The study provided educators with a comprehensive, ready-to-implement simulation scenario package, including validated learning objectives, prebriefing instructions, the full scenario script developed via AI-assisted methodology, and debriefing guides, that could be adopted to enhance orthotic training in rehabilitation curricula. Supplementary Information The online version contains supplementary material available at https://doi.org/10.1186/s12909-026-10043-x. Keywords: Orthotic devices, Active learning, Large language models, Competency, Simulation training
What kind of evidence is this?
Supporting AI learning science · Qualitative / mixed methods
Stage: Age / stage not established. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.
Study methods · source passage
The present study employed a mixed methods design to examine the content validity of a simulation scenario developed with the support of generative AI. The validity of the scenario was assessed quantitatively through the calculation of the content validity index (CVI), while expert reviewers’ open-ended responses were utilized descriptively to establish a recommended refinement and implementation blueprint for future educational usage, rather than modifying the raw AI-generated scenario flow examined in this study. The reporting of this study adheres to the guidelines for health care simulation research as extensions to the CONSORT and STROBE statements [21].
Limitations and uncertainty · source passage
Several limitations should be considered when interpreting the findings of this study. First, the content validity assessment was based on expert reviewer judgment, and the simulation scenario had not yet been pilot tested with occupational therapy students at the time of the study. Therefore, the direct effects of the scenario on learners’ psychomotor performance, learning outcomes, and educational experiences were not examined and remained beyond the scope of this research. This approach was intentionally selected to prioritize review by expert reviewers and content validation prior to learner exposure. Second, the study methodology focused on orthosis fabrication as a specific psychomotor skill domain within occupational therapy education. Although elements related to communication and patient interaction were incorporated into the scenario design, the primary emphasis was placed on technical and procedural competencies. As a result, the findings may have limited applicability to simulation scenarios primarily targeting communication skills, learner affect, or complex psychosocial processes. Finally, the generative AI model used to support scenario development was subject to ongoing updates and version changes, meaning that the findings reflected the performance and characteristics of the model at the time of data collection. The reproducibility of AI assisted scenario drafts across different model versions was not examined. Future studies may therefore explore learner outcomes, perceived fidelity, and user experience associated with AI assisted simulation scenarios, as well as the consistency of AI supported content generation across evolving models and diverse educational contexts.
Design the system before the pilot
Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.
- Give feedback on the learner’s reasoning, then fade hints. Record whether the learner can complete the next task without that scaffold.
A 90-day district pilot
- Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
- Days 16–30 · Design. Treat the paper as background evidence until its population is verified in the full text. Do not assign an age-specific intervention or extend adult results to children. Name an accountable owner and set the baseline, success threshold and budget before implementation.
- Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
- Days 61–90 · Decide. Extract the participant ages or education stage, recruitment, task and comparator before selecting a local pilot. If these cannot be established, retain the paper for design discussion rather than age-band efficacy claims. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.
The board decision brief
Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.
Attribution and reuse
Duygu Mine Alataş; Çiğdem Öksüz; Nilgün Bek; Özlem Ülkü Bulut; Kamile Akarsu; Mevlüt Karadağ; Aynur Ayşe Karaduman. Development and content validity of an AI-assisted simulation scenario for orthotic fabrication in occupational therapy education. BMC Medical Education (2026-07-29). https://doi.org/10.1186/s12909-026-10043-x
Companion prepared 2026-11-01. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.