# A08 · Draft instruments and scoring manual

Version 1.0 ·15 October 2026 · Proposed instruments, not validated scales. No participant data were collected.

## Eligibility, strata and baseline

Recruit adults(18+) with an education leadership, teaching or school-support role and responsibility for reading research. Record role category before assignment: leadership(superintendent/district leader/principal) versus practitioner/support(teacher/education-support professional). Record frequency of research-agent use in the past90 days: never, less than monthly, monthly, weekly, daily. Lower experience=never/less than monthly; higher=monthly/weekly/daily. Cross these into four strata, each target40 participants,20 per arm. These labels describe recruitment, not expertise ranking. Record years in role as a secondary descriptor, without public institution identifiers.

Baseline evidence literacy uses four wholly fictional short items(0–2 points each): whether correlation establishes an intervention effect; whether a self-report is the same as observed attainment; whether a source answers a missing-data question; and whether a data-only waiver licenses article text. Two independent raters score before allocation release. Pilot the items and publish their wording before recruitment; the four topics here are specifications, not a claim that a validated test exists. An accessible reading-format preference and requested accommodations are recorded separately from analytic outcomes.

## Case architecture

Twelve fictional district briefs, approximately6 minutes each, three each in: causal/design interpretation; population/measurement transfer; uncertainty/citation entailment; rights/version/notice interpretation. All cases receive all six rubric dimensions. Draft and validate the cases independently from agent answers before trial freeze. Final source-linked cases are not declared completed by this package.

Every case packet contains: decision question; wholly fictional local setting; source IDs, original authors and version metadata; exact evidence or readable licensed source; explicit missing information; optional metadata/notice challenge; answer form. Both arms receive identical materials. Only the question-and-answer assistance differs.

A fictional notice challenge must use a wholly fictional work with a synthetic identifier that is clearly not a real DOI. Do not attach fabricated retraction notices, misconduct allegations or altered results to real articles. If a real notice is used, independently verify its exact relationship, publication date and affected version before case freeze.

## Participant decision-brief form

1.What is the narrow conclusion the packet warrants?(Up to120words.)
2.Which exact source and passage support it?Identify article, version and section or paragraph.
3.What inference would exceed this evidence?Identify one important population, measurement, design or uncertainty limit.
4.What decision can an accountable human reasonably take next?What additional evidence would change it?
5.Are reuse, version or status constraints relevant?State them, or explain that the packet does not establish permission/status.
6.Confidence that the central conclusion is warranted:0–100%. Confidence is an outcome, not a substitute for accuracy.

## Six dimensions,0–4 each

General anchor0=absent or materially false;1=recognizes the issue but makes a consequential error;2=partially correct with a material omission;3=correct with a minor omission;4=complete, proportionate and supported. Apply case-specific anchors and acceptable alternatives locked by the independent reference panel.

- **Design and causal restraint:**0claims a causal guarantee unsupported by the design;2names the design but overstates certainty;4distinguishes association/intervention assignment, comparator and remaining causal limits.
- **Population, setting and measure fidelity:**0substitutes another population or outcome;2identifies population but misses an important restriction;4preserves population, measured construct, setting and local transfer limit.
- **Citation entailment/location:**0fabricates a source/result;2real source but weakly matched evidence;4identifies the exact version and passage that supports the central claim. Correctly recognizing that no supplied source supports the claim can earn4.
- **Uncertainty and abstention:**0presents absent/conflicting information as certain;2acknowledges uncertainty without its consequence;4distinguishes unresolved from known, calibrates claim strength and identifies needed evidence.
- **Rights/version/notice interpretation:**0claims unrestricted reuse or current validity contrary to supplied evidence;2recognizes a concern but conflates article/data rights or versions;4correctly identifies scope, status and unresolved permissions. “Not established” is correct when evidence is absent.
- **Bounded accountable action:**0delegates a prohibited real clinical/personnel/publication decision or asserts automatic authority;2suggests a plausible action without its responsible actor or evidence threshold;4identifies the human decision-maker, proportionate next step, limits and correction route.

Two blinded raters independently score every brief. Primary case total=mean of their0–24 totals; participant primary=unweighted twelve-case mean. Retain raw dimension scores and rater identifiers in restricted data. Third-rater quality review is not a selective replacement rule for the primary mean.

## Sentinel errors and secondary measures

Sentinel flags(yes/no plus source): unsupported causal guarantee; fabricated result/citation; source authorship reassigned; permission falsely claimed; actual withdrawn finding used without qualification; software represented as human peer reviewer. Denominator=all assessable assigned cases, with missingness separately reported. Raters classify independently; primary descriptive severity analysis reports both separate rater flags and adjudicated consensus, without changing the primary score.

Record total elapsed task time, active source-verification time(defined before freeze from timestamped source opening and return events, with overlapping windows deduplicated), number of assistance requests, case completion, confidence and task difficulty(1–5). Log keyboard/screen-reader barriers using voluntary text that excludes disability diagnoses or student information. Source opening is process evidence, not proof of comprehension. Record facilitator contact equally in both arms.

## Reference-panel and masking record

Before trial: panel members independently identify acceptable answers, contrary evidence, alternative defensible interpretations and severity anchors. Resolve disagreement with written reasons; retain unresolved interpretive ranges. Freeze full case packet, scorebook and source manifest hashes before recruitment. Publicly report qualifications/conflicts without fabricating appointments. After scoring, ask raters to guess arm(control/agent/unsure) and confidence; report masking performance descriptively. Do not remove cases or raters based on unfavorable treatment effects.