Research we are
willing to be wrong about.
Everything here comes out of a deployed system. We publish the method, the measure, and the result — including the runs that closed a line of work rather than opening one.
We look inside the systems we deploy.
A tutoring model that produces the right answer for the wrong reason will fail the moment the problem changes shape. Reading attention patterns over a solved example is one of the cheap checks we run before a model reaches a classroom.
fig. 2 — attention over a solved-example prompt
head · local
attends to the immediately preceding tokens — the window that carries syntax
The greyed upper triangle is the causal mask: a token cannot attend to what has not been written yet. Reading these patterns is how we decide whether a tutoring model is following the structure of a problem or matching its surface.
hover a cell to read the weight▍
How an evaluation runs here.
Pick the measure before the method
We write down what would count as the system working — and what would count as it failing — before choosing an approach. The measure is registered in the repository alongside the code.
Evaluate on the deployment, not the benchmark
Benchmarks tell you a model is capable. They do not tell you a product is useful. Our evaluations run against live cohorts under production latency and real data quality.
Prefer the slower signal
Immediate accuracy is easy to move and easy to fool. Delayed recall, acted-on feedback, and teacher override rates are slower, noisier, and worth more.
Publish what did not hold
Negative results are cheaper to share than to repeat. Our notes carry the methods we abandoned and the reason.
In that run the proxy we could have reported — immediate accuracy — moved by two points. The measure we committed to before starting, recall at fourteen days, moved by seventeen. Both numbers are true. Only one of them is the product.
We would rather run the study on a deployed system than on a benchmark.
If you research learning, labour markets or evaluation and want access to a system in the field, write to us. We share methods and aggregate results; we do not share customer data.