Same test. Every day.

100 questions, graded by code, compared only with the model's own first week. Here's everything we do and don't measure.

100questions per test

1dbetween tests

7dbaseline window

3latest runs per verdict

The questions

Five kinds, 20 of each, ordered from easy to hard. Every question has one correct answer, checked by code rather than by another AI.

ReasoningKeep track of a list through dozens of steps.

Logic puzzlesWork out who always tells the truth and who always lies, from what they say.

Reading codeWork out what a short program prints, without running it.

Following instructionsWrite a few lines that obey several exact rules at once.

Long documentsFollow a chain of references through a directory of up to 1,200 people.

The baseline

A model's baseline is every run in its first 7 days of tracking. Later results are compared with that week, never with other models.

We can't test the past. If a model launched before we started tracking it, anything that changed before then won't show up. Every model page says when its baseline began.

The verdict

We compare the last 3 runs with the baseline and work out the likely range of the difference. A model is only called nerfed when that whole range is below zero. A dip inside normal variation is no change.

Scores show that a model changed, not why.

Nerfed

No change

What stays fixed

Questions go to each company's public API, directly or through OpenRouter. Through OpenRouter, each model is pinned to the company that makes it, so a question can't be quietly routed to someone else. Every setting is spelled out: the model, how much it may think, and the answer length limit. If a company changes its defaults, our results don't move. A refusal counts as a wrong answer.

Tokens and cost

Because the questions never change, we can see when the same work gets more expensive: the model spends more tokens, or the company counts the same text as more tokens. Each model page tracks both, plus the list-price cost of the 100 questions.

We can't see chat subscription limits, like how fast the percentage left on a plan goes down.

Not covered

We test what developers get through the API, not chat apps like ChatGPT or Claude.ai. Those apps add their own instructions and may route questions differently.

Speed

The 100 questions run through a batch service, which is cheaper and can take hours, so these runs can't show how fast a model answers. We don't time live answers yet.