# Analysis specification · Version 3.0

## Status and target

This is a proposed analysis specification, with two executed methodological outputs: SEDA file feasibility and simulation-based design sensitivity. Primary fieldwork is prospective. There are no observed trial effects in this package. Registration, ethics, funding, district agreements, pilot validation and final author approval must be documented by the investigators before study activation.

Primary estimand: the equal-school average effect of assigned condition on the fixed baseline cohort's mean endline-minus-baseline 20-point rubric score, over recruited schools. Twenty district blocks of four schools are the candidate design. Two schools receive each condition within every block. The alternative compares the structured package with equal paid meeting time and facilitator contact, not with an absence of professional learning.

## Complete-cohort estimator

For each school, average the 40 baseline-cohort changes. For each district b, calculate D_b as the mean change of the two treated schools minus the mean change of the two controls. Estimate tau as the mean of D_b across B districts. Let s²_b1 and s²_b0 be the sample variances of school mean changes within the respective two-school arms.

The conservative Neyman variance estimate is U = sum_b(s²_b1/2 + s²_b0/2)/B². The specified approximate 95% interval is tau ± t_(B-1, .975) sqrt(U). Its small-sample coverage must be examined in design simulations; it is not an exact finite-population guarantee. Heterogeneous effects can make the Neyman variance conservative.

For complete cohorts, also report a Monte Carlo blocked randomization test of the sharp no-school-effect null. Independently select two of four schools as treated in every block for 9,999 draws. Recompute the unstudentized contrast. The two-sided p value is (1 + the number of simulated absolute contrasts at least as extreme as the observed)/(1 + 9,999). Preserve block membership and count ties. The sharp null is stronger than a zero average effect.

No student-level label shuffling is supported. The complete-cohort tool rejects missing scores, unknown schools, duplicate IDs, non-binary assignments, invalid block allocations and non-finite or out-of-range scores. Its provenance label is supplied by the investigator and does not independently certify approval.

## Missing-data model and pooling

The baseline cohort remains the target. Before randomization, record the legal retention and follow-up rules for withdrawal, departure and discontinued participation. Consent restrictions govern which retained observations may be analyzed. Record missingness reasons and rates by arm and relevant baseline groups.

The proposed imputation model treats endline total as a continuous, bounded response on 0–20, with a normal likelihood truncated to that range. Predictors are centered baseline total, baseline grade, prespecified task-form order, assignment, assignment-by-centered-baseline interaction and district fixed effects. Include random intercepts for school and baseline teacher nested in school. Do not add post-treatment implementation or workload as routine predictors of the total treatment effect. The continuous-score approximation must be checked against the actual distribution and a prespecified ordinal sensitivity model.

For a score measured in rubric points, intercept prior is Normal(10, 5); centered baseline coefficient prior Normal(1, 0.5); assignment, interaction and other centered fixed-effect priors Normal(0, 5). Teacher, school and residual standard deviations have half-Student-t priors with 3 degrees of freedom and scale 5. These are design choices, not estimates. Conduct prior-predictive checks before fieldwork; material changes require a versioned pre-allocation amendment.

Use at least four chains and assess R-hat below 1.01, effective sample size above 400 for quantities driving imputation, trace plots, divergent transitions where applicable, and posterior-predictive checks. Retain 50 sufficiently separated posterior predictive completed datasets, with parameter and residual uncertainty represented. A failed diagnostic does not authorize replacing the model with complete-case analysis after seeing a favorable result; report the problem and follow a documented blinded amendment.

For each imputation, calculate the same school-average estimator and U. Pool estimates using mean Q, within-imputation variance Ubar, between-imputation variance Bimp, and total T = Ubar + (1 + 1/50)Bimp. Apply the Barnard–Rubin small-sample degrees-of-freedom adjustment with the complete-data reference B-1. Report Monte Carlo stability by repeating with additional imputations if necessary. Cite Barnard and Rubin (1999). No routine imputation implementation is claimed to be tested by the supplied complete-cohort script.

This model assumes conditional missing at random. Repeat with missing-score predictive draws shifted by -2, -1, 0, 1 and 2 points, truncated to 0–20; present arm-specific adverse-shift analyses as well. Show worst-case bounds assigning permissible missing scores to the scale extremes. Do not present a permutation of one imputed dataset as an exact randomization test when treatment-dependent missingness is possible.

## Primary, secondary and exploratory decisions

The primary two-sided type-I error is 0.05 for the declared learning endpoint. Report the estimate and uncertainty irrespective of significance. Delayed transfer, workload and implementation are separate secondary outcome families. Specify the family membership and a Holm adjustment within each confirmatory secondary family before allocation. Unplanned outcomes and treatment-by-subgroup interactions are exploratory. Family-wise adjustments and confidence intervals must be reported with their actual target rather than appended indiscriminately.

Proposed practical thresholds for pre-allocation stakeholder deliberation are a one-point mean learning benefit and no more than 30 additional unpaid teacher minutes per week. These are candidate values, not validated minimum-important differences. A final adoption rule and thresholds require a dated pre-allocation decision. A precise interval ruling out the chosen worthwhile effect challenges the learning hypothesis; a wide interval is inconclusive. Serious safeguarding, accessibility or privacy concerns trigger independent review regardless of a mean score.

## Mechanism model

Specify teacher measures at baseline and week 9, and learner outcomes at week 18. Decompose teacher-level variation within schools and school-level variation between schools. Candidate latent constructs remain distinct. Identify each factor with justified indicators, define ordinal thresholds and scale identification, and examine time/condition comparability before treating latent means as comparable. Account for task-form and rater effects. The number of schools, not only teachers, constrains between-school model complexity.

Prespecify a parsimonious between-school structural model relating assignment to midpoint measures and those measures to the later school outcome, conditional on their baseline counterparts. An indirect coefficient is an association unless additional identification assumptions are defended. Assignment does not randomize the mediator, and treatment may affect mediator-outcome confounders. Report sensitivity analyses and any simpler fallback model as such. No fitted multilevel SEM, causal mediation finding or universal software implementation is included here.

## External data

TALIS: use one declared country/population, appropriate teacher or principal weights, 100 corresponding replicate weights and Fay factor 0.5. Inspect national adaptations, release version, item missing codes and adjudication. The included estimator handles category proportions for teacher question 26 only; it does not fit ordinal factors or a multilevel survey model.

SEDA: pin release 2025.2 and the actual repository file hash. Keep subgroup, gap, comparison and period distinctions. For the executed audit, the declared all-student, non-gap, single-comparison 2022–2025 slice uses cs_mn_lrn_ol and cs_mn_lrn_ol_se_adj. Keep the uncertainty measure with the estimate. Do not treat suppression as zero, administrative-district estimates as individual growth, or a historical context variable as an outcome of a future intervention.

There is no established public TALIS–SEDA crosswalk in this package. A separately approved mapping of recruited U.S. districts to SEDA context is a different operation and requires its own provenance, validity and permission checks.