Actionable companion · Assessment & reasoning
Learning with agency: Investigating generative AI models and detection techniques: impacts of tokenization and dataset size on identification of AI-generated text
AI-assisted editorial implementation guide, separate from the original research. Proposed actions are not claims or recommendations attributed to the source authors. No independent peer review of this guide is asserted.
Evidence and context
Title and abstract screening; stage labels describe reported settings, not verified numerical ages. Design labels require full-text appraisal. Reported stage(s): K–12 · mixed / unspecified. Verify the participant population and task before transferring the findings.
Original authors’ abstract, reproduced under the linked source license:
Generative AI models, including ChatGPT, Gemini, and Claude, are increasingly significant in enhancing K–12 education, offering support across various disciplines. These models provide sample answers for humanities prompts, solve mathematical equations, and brainstorm novel ideas. Despite their educational value, ethical concerns have emerged regarding their potential to mislead students into copying answers directly from AI when completing assignments, assessments, or research papers. Current detectors, such as GPT-Zero, struggle to identify modified AI-generated texts and show reduced reliability for English as a Second Language learners. This study investigates detection of academic cheating by use of generative AI in high-stakes writing assessments. Classical machine learning models, including logistic regression, XGBoost, and support vector machine, are used to distinguish between AI-generated and student-written essays. Additionally, large language models including BERT, RoBERTa, and Electra are examined and compared to traditional machine learning models. The analysis focuses on prompt 1 from the ASAP Kaggle competition. To evaluate the effectiveness of various detection methods and generative AI models, we include ChatGPT, Claude, and Gemini in their base, pro, and latest versions. Furthermore, we examine the impact of paraphrasing tools such as GPT-Humanizer and QuillBot and introduce a new method of using synonym information to detect humanized AI texts. Additionally, the relationship between dataset size and model performance is explored to inform data collection in future research.
What kind of evidence is this?
Supporting AI learning science · Design requires full-text appraisal
Stage: K–12 · mixed / unspecified. Screening labels describe reported context and study-design language. They are not a risk-of-bias rating or proof of effectiveness for an age band.
Study methods · source passage
This study's human-written text and prompts are acquired from the Kaggle Automated Student Assessment Prize (ASAP) dataset sponsored by the Hewlett Foundation. We use prompt 1 as the main prompt to train and evaluate models for detection of machine-generated essays. This prompt is designed to guide high school students in writing essays on various topics. It requires students to critically think about a specific subject or problem and express their viewpoints and arguments in a short essay. ASAP prompts can cover a wide range of topics, including social issues, science, literature, history, and more, depending on the specific competition or task. Prompt 1 asks students to share their opinions about technology's effects to a local newspaper. Although not part of this research's scope, the essays are evaluated and scored by human raters from 1-6 and each text is being scored twice. The total combined score from 1-12 is record as the final score for each essay. The dataset of human-written essays used to train our models comprises 1,500 essays (Hamner et al., 2012). Research on machine-generated texts has examined various state-of-the-art GenAI models including Chat-GPT, Claude, and Gemini/Bard. This study uses prompt 1 as a question prompt with an additional requirement to generate high-scoring text based on the scoring guideline provided for 50% of the dataset, and with a specific score prompt equally distributed for the other 50%. The dataset size for each of the model versions is 500. For Chat-GPT 4o, 500 additional essays are collected to investigate the relationship between dataset size and the detectors' performance. The model versions collected in this study include Chat-GPT 3.5, 4.0, and 4o, Claude 3 and 3.5, and Gemini base and Advanced. We thus include most of the popular GenAI that are available to the public, with both base and advanced versions of each. To test our new tokenization technique, 200 essays produced by Chat-GPT 4o are paraphrased using the QuillBot paraphraser (premium mode) or Chat-GPT Humanizer. These 200 essays form a secondary dataset used to evaluate how well our new tokenization technique counters the effect of word-changing on detection of AI-generated essays. For the various stages of the experiment, the dataset is split into training, testing, and validation samples in the ratio 0.8 : 0.1 : 0.1.
Limitations and uncertainty · source passage
Generative AI models, including ChatGPT, Gemini, and Claude, are increasingly significant in enhancing K–12 education, offering support across various disciplines. These models provide sample answers for humanities prompts, solve mathematical equations, and brainstorm novel ideas. Despite their educational value, ethical concerns have emerged regarding their potential to mislead students into copying answers directly from AI when completing assignments, assessments, or research papers. Current detectors, such as GPT-Zero, struggle to identify modified AI-generated texts and show reduced reliability for English as a Second Language learners. This study investigates detection of academic cheating by use of generative AI in high-stakes writing assessments. Classical machine learning models, including logistic regression, XGBoost, and support vector machine, are used to distinguish between AI-generated and student-written essays. Additionally, large language models including BERT, RoBERTa, and Electra are examined and compared to traditional machine learning models. The analysis focuses on prompt 1 from the ASAP Kaggle competition. To evaluate the effectiveness of various detection methods and generative AI models, we include ChatGPT, Claude, and Gemini in their base, pro, and latest versions. Furthermore, we examine the impact of paraphrasing tools such as GPT-Humanizer and QuillBot and introduce a new method of using synonym information to detect humanized AI texts. Additionally, the relationship between dataset size and model performance is explored to inform data collection in future research.
Design the system before the pilot
Use this study to inform a learning mechanism or implementation question. It is not direct evidence that autonomous agents improve learning. Keep the proposed agent behavior separate from what the source actually tested.
- Use common rubrics, blind scoring where feasible and a fresh unaided reasoning task. Document permissible assistance and avoid treating AI-detector scores as proof of misconduct.
A 90-day district pilot
- Days 1–15 · Appraise. Have the curriculum lead and a teacher read the methods and limitations. Record the study design, sample, comparator, outcomes and evidence gaps. Define a local question that the study can help inform.
- Days 16–30 · Design. Verify the grades and participant ages in each included study before selecting a classroom pilot. Use teacher-led hints and preserve a non-AI route; a K–12 label does not establish the same benefit across every school year. Name an accountable owner and set the baseline, success threshold and budget before implementation.
- Days 31–60 · Test. Start with a small voluntary cohort, preserve an accessible non-AI route and require teacher review. Use synthetic or de-identified examples; approve data handling and tool terms through district processes.
- Days 61–90 · Decide. Compare independent learning, delayed retention and transfer within the locally selected grade band. Report age-stage uncertainty rather than pooling young children and adolescents as one population. Compare with baseline, document uncertainty and stop or revise if learning worsens, access is unequal or critical errors persist.
The board decision brief
Report the learning problem, evidence quality, participating population, total cost including review time, measured benefit, errors, access differences and remaining uncertainties. Make a dated recommendation to stop, revise or expand. A positive perception survey alone does not establish learning improvement.
Attribution and reuse
Haowei Hua; Co-Jiayu Yao. Investigating generative AI models and detection techniques: impacts of tokenization and dataset size on identification of AI-generated text. Frontiers in Artificial Intelligence (2024-11-19). https://doi.org/10.3389/frai.2024.1469197
Companion prepared 2026-10-30. Original authorship and DOI are preserved. The abstract is an attributed source extract; the implementation plan is new AI-assisted editorial material, offered under the same CC BY 4.0 license. This guide has not undergone independent peer review.