Dear AI Companies: Give Me Baseline Thinking Performance
Like most people have likely noticed, there is a bit of a frenzy when a new frontier model comes out such as Fable 5, Opus 5, etc. Like clockwork, though, most users (including me) find that they degrade in quality over time.
This isn’t something that I can put my finger on, and the behavior is honestly unpredictable. Last week, I asked Opus to write a robots.txt in my Laravel app and instead it constructed a literal robots.txt route and used PHP to return raw text:
I was actually floored at this approach, because I would have never thought of it myself. You could say that I was shocked at this approach because it’s so comically unnecessary and, frankly, wrong. It did the same thing to Generate a sitemap (slightly more reasonable, but Spatie has a package for this).
This is part of a broader issue that I am seeing where Anthropic, OpenAI are benchmarking their Frontier models and their capabilities while simultaneously applying variable compute and effort levels which developers have no visibility into.
Model Degradation Over Time
During the earlier days of AI usage it was pretty common to run into context rot, where the context window would swell and eventually the baseline performance and integrity of the output would start to degrade. This is now more of a feature than anything, with unpredictable results even at 10-20% context window usage. This was documented in depth by Stella Laurenzo in this github issue which included a massive triage of the situation. The gist of it was that the engineer observed:
[…]Thinking allocation is load-sensitive and variable in the post-redaction regime.
It does not matter how great a model is, or how great the benchmarks are, if the effort and compute is unreliable. I suspect the “This is better than fable!” is in the early days of load balancing this compute on Anthopics end, and, a few weeks in (Claude was released July 24th 2026, I‘m writing this on August 6th 2026), we see them optimizing against actual cost on their side. Anthropic dismissed the idea that it’s load-sensitive and instead tied it to a few changes that took place internally, but that has not been my experience. Additionally, they public their own internal autonomous agentic evaluation. Great! That doesn’t reflect it’s real world performance in complex codebases.
It’s well known that plans are receiving more compute than the company can actually bankroll based on local AI token usage, but I have to wonder how many more tokens we consume overall repeating or redoing work that is fumbled when instructions were fairly clear to begin with. Using cccost (Claude Code Cost Tracker) we typically spend about ~$1600 a month in tokens on a $200 plan - not bad, but definitely not something we would be willing to pay for unreliable results.
Thinking Redaction
Related to the above, the thinking redaction which was in response to what is now called “distillation attacks” (fine tuning, but the kind we don’t like) from Chinese actors. As a result, we can only interrupt it and see that it apologizes for “going on a tangent” without actually seeing what is going on with it. We can probe for deeper reasoning, but the tokens it spends on the wrong track are now invisible to us.
Guardrails Are Broken
Anthropic’s marketing department could not help but brag about how dangerous Fable 5 was, which resulted in the US government requesting them to pull it offline, but I feel like we do need to take a slight deviation from the original topic and mention it. Guard rails, in Fable 5, are completely broken.
Earlier today we attempted to have Fable 5 audit a list of system services in windows and, when accepting a list of these services, triggered the guard rails which locked the session and instead asked to send it to Opus 4.8.
Frequently, when working on security reports, Claude tells me it won’t fabricate adding details to the report when I am the one writing the report and I have verified the authenticity of the content. When I walk it through the process like a five year old, it finally caves but only after it takes the massive descent from it’s personalities moral high ground. This personality is probably one of the worst things about it. When tripping these guard rails, it suddenly turns into an abrasive Karen who just found out you added a decoration to your lawn that was not approved by the community HOA.
Potential Fix: Deterministic Compute
LLMS are not deterministic, which spawns all sorts of context to try to get it to behave a certain way: claude.md, context, self-reviews after performing changes, etc. I want to really throw this out there: I will wait a minute for you to queue my work and get it to a model if you can give me predictable baseline performance, and I’m sure a lot of them will do the same.
The Real Fix: Open Source Models
There is a reason that Anthropic is doing some double-speak and trying to sabotage open source under the guise of security while releasing blog posts that somewhat “approve” of open source models - with a caveat of “Open-weights models that don’t have dangerous capabilities are a public good”
I think it’s important that the public remember that these models are built on stolen content that are the property of humanity and human creation. This technology belongs to humanity, not in the hands of a few tech giants while the public at large gets lobotomized versions.