Why education reform fails, and why generative AI has not changed that
Public systems optimize for legitimacy over efficiency. A public school system has to certify millions of students in a way that is legible to strangers: an employer or university reading a diploma, a parent, a politician deciding whether the system works. For the certificate to be fair, the result has to mean the same thing in every school, so it needs a standardized metric at scale.
That standard had to survive the constraints of its time, which meant authored by humans, teachable by the average teacher, scored by non-experts, under budget constraints. What survives that compression is largely recall and procedure, with the judgment stripped out. Too often, mathematics and physics become formula-plugging and history becomes fact recall. What arrives in the classroom is not a smaller version of the discipline. It is something else, wearing the same name.
Under pressure, proxies are optimized for and stop measuring what they were meant to. Scores on high-stakes tests rise for years while low-stakes tests of the same material rise far less, or not at all (Campbell 1976; Koretz 2008, 2017). Nobody is behaving badly; every actor is locally rational: the teacher has to spend their scarce hours drilling what the test will see; the student who refuses to care is declining to invest in a parody of a discipline, and the system grades the refusal as a defect in the student.
But the metric cannot be removed, because the system fundamentally requires it. Abolishing it has been tried, from progressive education onward, and has repeatedly failed politically, for good reason: the metric protects the student from the arbitrariness of whoever happens to grade them, and the stranger who has to trust the diploma.
Accountability reforms went the other way: they raised the stakes on the proxy and got the results you would expect from gaming the metric: narrowed curricula, unconfirmed gains and in some places fraud.
Technological innovation in delivery, and now generative AI, a truly revolutionary innovation, has not changed this structure. That is surprising until you look at where it was pointed. Generation and explanation were not the main bottleneck. The bottleneck is the measure, and everything adaptive that depends on it.
Many building with AI models in education have seen this problem, and there have been roughly two responses.
The first keeps the objects the constraints produced, the proxies and the curriculum built around them, and places an AI model on top: more effective, personalized, engaging delivery of the same paradigm.
The second sees that the standard hollows the discipline and answers by redefining what should be learned: critical thinking, reasoning games, a curriculum of the builder's own design. It is the more serious response, with the deeper flaw. The target has moved from the discipline to the builder. Nobody outside the company can check it, so nobody can find out it is wrong, and a target that cannot lose cannot be iterated against. A builder's account of what reasoning is ends up competing with physics, history and mathematics, with no way to be corrected.
Neither approach produces a verifier.
One lineage understood the problem and rebuilt the instrument: the Cognitive Tutors project at Carnegie Mellon measured the components of mathematical reasoning step by step and showed real gains in large trials (Anderson et al. 1995). Its cost was authoring: each hour of instruction took 200 to 300 hours of expert development (Aleven et al. 2006), and it remained difficult in subjects outside of mathematics.
It is not that we lack data: learning research has an abundance of learner data, and plenty of talented researchers. The problem is what that data is made of. Most of that data is item responses on proxy tasks, often collected while a system was steering the learner toward them, so a million of them is a precise picture of the proxy under the very pressure that bends it: sample size buys reliability, but the defect is validity.
For three years the Hubble telescope's images were blurred by a mirror ground to the wrong shape. Processing could sharpen bright objects but could not recover the faint ones the telescope was built to study; the fix was correcting its optics, not more data.
So the metric is required, but the metric hollows the underlying discipline, and a better metric failed on authoring cost. Taken together, these have looked like a law for a century: a standard cheap enough to run at scale has to be a proxy, a score that can rise while the ability it stands for does not.
But look again at what the requirement says. It demands a standard, comparable and cheap at scale. It never demanded that the standard measure a proxy. The proxy was only the cheapest thing that met it.
The inversion
A measure under pressure is optimized for, so good metrics take their input from the activity itself, so that the only way to improve them is to do the thing better.
Elo is an excellent metric for chess ability: it is estimated from game results, and short of cheating nothing moves it except winning games, so optimizing for it is playing better.
Chess puzzles can also be rated, but a chess puzzle tells you there is a winning move, without the player noticing it themselves or setting up the position within their strategy. Puzzle rating is only a loose guide to playing strength.
Many high school test items are like chess puzzles. The physics question gives you exactly the numbers you need, so the only question is which formula uses them all. The history question asks you to identify a source's bias, so you know there is one. Each such item has already done the part the discipline is about: noticing what matters, inside a larger piece of work the student is directing themselves, the way a winning chess move only arises inside a player's own plan. What is left to the student is the step, cut out of the work it belonged to.
The question Norem asks is whether a measure of reasoning can take the reasoning itself as its input, so that, like Elo, nothing moves it but the underlying discipline it measures.
The instrument
History is not as deterministic as a chess board, but it has its own game: a written argument that survives review by historians. Writing that argument is doing real history. Norem's first prototype instrument has the learner answer a historical problem by constructing an argument from primary sources. An essay assignment is doing real history too; the difference lies in how the performance is assessed.
Norem's measurement is built to move only by doing the historical reasoning. Not by recall, not by writing well, not by writing what the grader wants to read. It does this by scaling expert judgment rather than replacing it. Every written history book is a reviewed expert doing history, and an AI model can draw out of the book what the historian did with each document and build the lesson from that, so the historian's judgment is the material the learner works in. There is no separate assessment layer: the structure that generates the lesson is the structure the learner's moves are read against. What remains is validating that process at a few points, rather than validating every item, and that is an open bet Norem seeks to verify.
The referent is never one historian's reading. A topic is built from more than one, so what the learner is read against is the space historians occupy, including where they disagree with each other. A learner who argues against one of them is not marked wrong for it; they are read against the others.
The instrument does not claim an ontology of history, or that the moves it measures are a decomposition of history. It claims only that they are the discipline's, and that learning them is worthwhile. Nor are the moves ours. What historians do when they reason from evidence has been described extensively: sourcing, contextualization and corroboration (Wineburg 1991, 2001); the second-order concepts of evidence, cause, change and accounts (Lee and Ashby 2000); the components of historical reasoning (Van Drie and Van Boxtel 2008); the Big Six (Seixas and Morton 2013). Our register is not a rival account of historical thinking. It is the cut of that literature that follows from the constraints our architecture imposes. Those frameworks were built to describe historical thinking and to teach it; ours has primarily to be recorded by an instrument. That forces a finer grain: acts with a completion condition rather than competences with a description. Where that cut and the frameworks disagree, they are the established thing and ours is the one under test.
Measuring these moves has been attempted too. Stanford's short assessments of historical thinking isolate a single move in a short item, validated by watching students think aloud (Breakstone, Smith and Wineburg 2013; Smith, Breakstone and Wineburg 2019). That is the right standard, and it is expensive: expert-built and expert-validated. Norem's bet is that the same moves can be measured inside the learner's own argument, with the expert cost paid an abstraction level up, rather than once per item as Stanford paid it or by the hour of instruction as Carnegie Mellon did. The rest is generation, and generation is nearly free. That is the cost half of the claim, the half the Cognitive Tutors could not make.
The demonstration shows a learner working one problem on the current prototype.
What this unlocks
If the measurement holds, at scale and across disciplines, then the requirement for a comparable standard, cheap at scale, is met by something that is not a proxy, and much of what the proxy blocked becomes possible.
The pressure reverses. The teacher drilling what the test will see is drilling the discipline. The student optimizing the score is doing history.
The student gets real history, physics and mathematics, and engagement no longer has to be bolted on (badges, streaks, a tutor in their favourite singer's voice) to make up for the content.
Nobody's incentives change; the thing they point at does.
Instruction becomes adaptive outside of mathematics. A tutor can only adapt to a signal, and the signal has been a proxy, so adaptive instruction has meant adaptive drilling of the proxy. With a valid measure, every tutor and learning environment can be pointed at the discipline.
Education becomes an experimental science. Learner data measures the discipline rather than the proxy, at a much finer grain of the discipline than the test. Whether a method improves reasoning stops being a decades-long argument over proxy scores and becomes a trial that can be run in weeks. The field has had abundant proxy data and few fine-grained valid measures; this is meant to be the corrected lens.
Norem's mission is to build the instruments those trials run on and to run the loop: cohort, reading, redesign, next instrument, next cohort. The first instrument is not a proposal for the final one. It is built for first contact, and we expect it to be badly miscalibrated; its job is to make contact well enough to show where. Each cohort shows that, and the next instrument is built from it. The company's bet is that the understanding and data this accumulates beat a frontier model judging prose at measuring reasoning.
And if we let ourselves dream a little: in ten years, history might mean reconstructing the past from its sources and watching it assemble as you argue. Physics might mean being dropped on a planet where things fall wrong and designing the experiments to figure out why. Or the subjects might dissolve into one another: a city to build, with its bridges, budgets, epidemics and elections, and every discipline arriving as the problem it solves. None of this is new as a dream. It has always stalled on the same question: whether anyone learned anything in there. That is the question we are building the answer to.
What we have to prove
Let's start from the end: what would have to be true for Norem to be what it claims. Then work backwards; each step is a thing that must be proven before the rung above can be tested.
The destination. A learner steered by Norem's measurement ends up a better historical thinker, judged by historians, in the actual discipline, not in a format.
For that to be true, steering by the measure has to beat the alternatives: a frontier model reading prose holistically and giving feedback. If steering by our structural measurement does not improve independent outcomes more than steering by that model judge, by a margin set in advance, there is no reason for the decomposition to exist.
For steering to beat anything, there has to be something to steer on: the measurable decomposition we impose upon history has to be real in a few ways. The decomposed structure has to be real history. The decomposed moves need to fail in distinct, separable ways, so they can be acted on; for that they need reasonably clear boundaries from other moves. And the moves need to be repeatable rather than local, so that steering on them achieves something.
For any of that to be readable, each single measurement has to be warranted: our instruments have to accurately capture the intended reasoning from the student, defensible against a historian reading the same trace, on a metric that isn't gameable like a proxy.
For the referent to mean anything: the referent has to be the discipline's. The right answer is the historian's reading, not ours, and the kinds of mistake are the ones real students make. And beneath that the machine has to work: the lesson sets up the move without giving it away, starves no commit of the material it needs, and records every commit.
Beneath all of it, one presupposition: the moves we chose have to be enough of the discipline. You could pass every step above with a thin slice of history, and produce a brilliant source-reading trainer that teaches nothing about argument.
We plan to verify these through a series of cohorts. Lay the chain out, ask of each link the cheapest point at which it could be false, and order by cost. That ordering is the first phase. Everything that can fail with no learner (fidelity to the page, a working machine) is checked before anyone is enrolled. Everything a small cohort can falsify is batched into that first phase. Everything that needs multiple cohorts, larger samples or multiple topics follows in subsequent phases.
Small cohorts can refute but cannot confirm. The tests are written as falsification cards in our public ledger, each with the observation that would fail it, published before any data exists.