Same test. Every day.

100 questions, graded by code, compared only with the model's own first week, with your votes right beside the score. Here's everything we do and don't measure.

100questions per test

1dbetween tests

7dbaseline window

3latest runs per verdict

The questions

Five kinds, 20 of each, ordered from easy to hard. Every question has one correct answer, checked by code rather than by another AI.

ReasoningKeep track of a list through dozens of steps.

Logic puzzlesWork out who always tells the truth and who always lies, from what they say.

Reading codeWork out what a short program prints, without running it.

Following instructionsWrite a few lines that obey several exact rules at once.

Long documentsFollow chains of managers and count people across a directory of up to 1,200.

The baseline

A model's baseline is every run in its first 7 days of tracking. Later results are compared with that week, never with other models.

We can't test the past. If a model launched before we started tracking it, anything that changed before then won't show up. Every model page says when its baseline began.

The verdict

We compare the last 3 runs with the baseline and work out the likely range of the difference. A model is only called nerfed when that whole range is below zero. A dip inside normal variation is no change.

Scores show that a model changed, not why.

Nerfed

No change

Your vote

You use these models every day, so you often notice a change before any test does: shorter answers, more mistakes, a sudden habit of refusing. Every model card has two buttons for that, Nerfed and Feels fine.

Vote once a day on each model. Changed your mind? Tap the other button, or tap yours again to take it back. The card shows what everyone said over the past week, right next to the test score.

When the tests and the voters agree, that's a strong sign. When they disagree, it's worth a closer look. Your votes never move the score or the verdict. They sit beside them, so you can compare.

You do decide the order on the home page, though: it's 70% your votes and 30% the test score. A model nobody has voted on yet is placed by its test score.

What stays fixed

Questions go to each company's public API, directly or through OpenRouter. Through OpenRouter, each model is pinned to the company that makes it, so a question can't be quietly routed to someone else. Every setting is spelled out: the model, how much it may think, and the answer length limit. If a company changes its defaults, our results don't move. A refusal counts as a wrong answer.

Tokens and cost

Because the questions never change, we can see when the same work gets more expensive: the model spends more tokens, or the company counts the same text as more tokens. Each model page tracks both, plus the list-price cost of the 100 questions.

We can't see chat subscription limits, like how fast the percentage left on a plan goes down.

Not covered

We test what developers get through the API, not chat apps like ChatGPT or Claude.ai. Those apps add their own instructions and may route questions differently.

Speed

The 100 questions run through a batch service, which is cheaper and can take hours, so these runs can't show how fast a model answers. We don't time live answers yet.