AI evaluations produce scores. The harder question is what those scores allow us to claim. My work examines their reliability, what they measure, and whether they predict performance in a different setting. I draw on psychometricsThe statistics of human testing, which asks whether a test is reliable, comparable, and measuring what it claims., statistical modelling and economics, with a growing interest in evaluations of interactive and tool-using systems.
under reviewQ2 · measure
Do economically framed benchmarks measure a distinct capability? A structural test and a predictive test of the same frontier leaderboard give different answers.
421 model configurationsHash-pinned frontier model configurations across a frozen battery of twelve benchmarks, four of them economically framed. Of these configurations, 103 carry all three sparsely scored economic benchmarks and 96 carry all twelve.
no distinct economic factorUnder the pre-specified dimensionality rule, the economic benchmarks form no distinct factor. Under linear learners, a multi-factor representation of the other benchmarks still predicts held-out economic scores better than a mean-score index, and the advantage reverses for tree learners.
preprint, arXiv v2 · under review, ICLR 2027 · pre-specified hypotheses, analysis plan deposited retrospectively
paper · code & data · analysis plan
acceptedQ4 · decide
Builds a quality-adjusted price index for AI inference on an IRT-estimated capability scale, and audits that scale before relying on it.
several times faster than matched-modelQuality-adjusted inference prices fell several times faster than matched-model methods record. The figures follow with the revised preprint in November.
accepted, NeurIPS 2026 EconML workshop · journal version in preparation for the Review of Income and Wealth · pre-registered validity audit
paper · dataset · Hugging Face · pre-registered audit
acceptedQ4 · decide
Argues from the lifecycle of MMLU that a benchmark score should travel with its repair history, meaning which errors were found, which were repaired, and which version a reported model was evaluated on.
position paper · accepted, NeurIPS 2026 workshop on AI for Meta-Science · full version in preparation for the ICML 2027 Position Paper Track
SocArXiv preprint
under reviewQ4 · decide
A versioned, hash-pinned corpus of the safety frameworks published by twelve frontier AI developers, and a measure of how much of each revision the developer’s own account discloses.
67% silent revisionShare of material changes not identifiable from the developer’s own account of the revision, under the strict criterion, with a 95% interval of 62% to 72%. Under the lenient criterion the share is 53%.
710 commitmentsCommitment instances traced across twelve consecutive version pairs of the developers’ safety frameworks.
preprint, arXiv 2609.08789 · under review, ACM FAccT 2027
paper · corpus and code · GitHub
preprintQ1 · trust
Examines when reliability statistics answer the wrong question in LLM-judge pipelines, with an empirical item bank and simulation studies.
3 statistics misindexedKR-20, the dependability index and Livingston-Lewis classification accuracy, each answering a question the judge pipeline does not ask.
preprint · full version in preparation for NeurIPS 2027 Evaluations and Datasets
paper · code & data
workshop paper
Studies how an AI agent’s institutional role and access to information affect group decisions, in an agent-based simulation.
1.1M simulation runsRuns of the agent-based model across authority structures and information conditions. A simulation, not a human-participant experiment.
Tian, Zhang, Zhu and colleagues · Workshop on Human-Agent Collaboration, CHI 2026
paper
Earlier and dormant work · adversarial CAPTCHAs that humans pass and AI agents fail, with UCL Computer Science and Holistic AI · a study of cognitive load, XAI and trust in human-AI teams with UCL Computer Science, study design complete
