Stanford: many claims about AI flaws are already out of date
Stanford researchers tested 50 widely cited claims about language models and found 44 percent have already expired, according to The Decoder.
The facts
- Stanford researchers Owen Queen and James Zou tested 50 widely cited claims about large language models against 20 models, according to The Decoder.
- 44 percent of the claims have expired and 34 percent have weakened significantly. Only 16 percent still hold and 6 percent have grown stronger.
- Among the 22 findings with an estimated expiry date, the average lifespan was 15.7 months. Claims about what models can't do expired fastest.
- Behaviour findings lasted longer, including models favouring their own answers, sycophancy and a left-leaning political bias. Results also varied widely between models, even within the same release window.
Case notes · Danny's take
Why it matters
Most people repeat AI weaknesses as if they were laws of nature. This study says many are snapshots. A flaw that was true in 2023 can be gone in 2026. The lasting problems are the softer ones: bias, flattery and models grading their own work. They have no single right answer for developers to train against. Not the headline failure. The habit underneath it. If you judge a model by what you read last year, you are working from old evidence. And the reverse is also true: one model fabricating 69 percent of its references shows the fix is not universal.
For traders
If you trade AI narratives, check the date and model version behind any claim about what AI can't do. Old criticism can be priced in on stale evidence, and so can old praise.
For builders
Re-test the limits you assumed when designing your product on each new model, and log the model, version and date. Don't assume a fix carries across providers, because the study found results vary even among models released in the same window.
Not financial advice. Do your own investigation.
This story was summarised with AI from The Decoder's reporting and checked against our editorial policy. Danny's take reflects his views. Disclaimer.
Danny Walter
Founder of Moon Republic, Volt Terminal and Moon Republic Academy. Author of The Market Is a Crime Scene and After the Applause.