Agentic education · Learning science

Actionable companion · Assessment & reasoning

Learning with agency: Evaluating a Large Language Model’s Ability to Synthesize a Health Science Master’s Thesis: Case Study

AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not claims or recommendations attributed to the source authors. No independent peer review of this guide is asserted.

Read original paperSearch paper agentCC BY 4.0

Evidence and context

Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): Higher education. Verify the participant population and task before transferring the findings.

Original authors’ abstract, reproduced under the linked source license:

Abstract Background Large language models (LLMs) can aid students in mastering a new topic fast, but for the educational institutions responsible for assessing and grading the academic level of students, it can be difficult to discern whether a text has originated from a student’s own cognition or has been synthesized by an LLM. Universities have traditionally relied on a submitted written thesis as proof of higher-level learning, on which to grant grades and diplomas. But what happens when LLMs are able to mimic the academic writing of subject matter experts? This is now a real dilemma. The ubiquitous availability of LLMs challenges trust in the master’s thesis as evidence of subject matter comprehension and academic competencies. Objective In this study, we aimed to assess the quality of rapid machine-generated papers against the standards of the health science master’s program we are currently affiliated with. Methods In an exploratory case study, we used ChatGPT (OpenAI) to generate 2 research papers as conceivable student submissions for master’s thesis graduation from a health science master’s program. One paper simulated a qualitative health science research project and another simulated a quantitative health science research project. Results Using a stepwise approach, we prompted ChatGPT to (1) synthesize 2 credible datasets, and (2) generate 2 papers, that—in our judgment—would have been able to pass as credible medium-quality graduation research papers at the health science master’s program the authors are currently affiliated with. It took 2.5 hours of iterative dialogue with ChatGPT to develop the qualitative paper and 3.5 hours to develop the quantitative paper. Making the synthetic datasets that served as a starting point for our ChatGPT-driven paper development took 1.5 and 16 hours for the qualitative and quantitative datasets, respectively. This included learning and prompt optimization, and for the quantitative dataset, it included the time it took to create tables, estimate relevant bivariate correlation coefficients, and prepare these coefficients to be read by ChatGPT. Conclusions Our demonstration highlights the ease with which an LLM can synthesize research data, conduct scientific analyses, and produce credible research papers required for graduation from a master’s program. A clear and well-written master’s thesis, citing subject matter authorities and true to the expectations for academic writing, can no longer be regarded as solid proof of either extensive study or subject matter mastery. To uphold the integrity of academic standards and the value of university diplomas, we recommend that master’s programs prioritize oral examinations and school exams. This shift is now crucial to ensure a fair and rigorous assessment of higher-order learning and abilities at the master’s level.

What kind of evidence is this?

Supporting AI learning science · Qualitative / mixed methods

Stage: Higher education. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.

Study methods · source passage

In an exploratory case study, we asked ChatGPT (ChatGPT-4o and ChatGPT-o1) to synthesize two datasets: one qualitative and one quantitative, inspired by existing real-world datasets the authors had previously analyzed. The generation of synthetic datasets and subsequent data analysis took place in October and November of 2024.

Limitations and uncertainty · source passage

One limitation of our ChatGPT-4o–generated master’s paper assessment is that the finished products were neither put through a proper peer assessment nor submitted to the university examination system as sham student projects. As master’s program lecturers, supervisors, course leaders, and master’s thesis examiners, we believe we are capable of assessing whether the fake papers would have been able to obtain a pass grade. A master’s thesis in our system contains more than just the scientific paper that we have used as an example in our case study. A short introduction usually precedes the paper. However, we believe that the process of developing scientific papers is relevant to illustrate how easy it is to produce scientific work at a high level using LLMs. We only made 2 example papers. More papers using a diverse set of analyses could perhaps have revealed relevant nuances between analytical methods. Given the highly competitive field of AI tools, we could have tested more than 1 LLM. Further prompt optimization could have reduced the number of false references and prompted the use of the COREQ (Consolidated Criteria for Reporting Qualitative Research) checklist [21] or a similar tool, which could have further enhanced the quality of the qualitative paper.

Design the system before the pilot

Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.

  • Use common rubrics, blind scoring where feasible and a fresh unaided reasoning task. Document permissible assistance and avoid treating AI-detector scores as proof of misconduct.

A 90-day district pilot

  1. Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
  2. Days 16–30 · Design. Choose one disciplinary task. Require a first attempt, a documented AI consultation, a critique of the output and an independent explanation. Align the comparison group and scoring rubric with the same objective. Name an accountable owner and set the baseline, success threshold and budget before implementation.
  3. Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
  4. Days 61–90 · Decide. Blind-score reasoning and accuracy on a new task without AI. Distinguish model examination scores, student perceptions and actual learner gains; only the last directly tests educational benefit. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.

The board decision brief

Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.

Attribution and reuse

Pål Joranger; Sara Rivenes Lafontan; Asgeir Brevik. Evaluating a Large Language Model’s Ability to Synthesize a Health Science Master’s Thesis: Case Study. JMIR Formative Research (2025-07-03). https://doi.org/10.2196/73248

Companion prepared 2026-10-30. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.