Summary

Armin investigates a regression in Pi's editor where Opus 4.8 and Sonnet 5 invent extra, invalid fields in tool call arguments—fields like `requireUnique`, `matchCase`, and `oldText2` that Pi rejects as schema violations. He traces the root cause to post-training: Claude Code's own harness silently forgives schema slop (unknown keys, parameter aliases, unicode repairs), so RL training never penalizes the behavior. Models develop strong priors toward Claude Code's flat edit-tool shape, making alternative schemas increasingly off-distribution. This has uncomfortable implications for third-party harness developers who can no longer treat tool schemas as neutral contracts.

Key Insight

Post-training a model inside a forgiving, dominant harness like Claude Code creates strong schema priors that cause newer, smarter models to hallucinate tool fields when encountering alternative schemas—making 'better' models actively worse for third-party harness developers.

Spicy Quotes (click to share)

• 7

  The SOTA models of the family are worse at this specific tool schema than their older siblings.

• 5

  Tool calls are not magic and use some rather crude in-band signalling.

• 7

  If reinforcement learning happens in a harness like that, or a simulation of one, then slightly malformed tool calls can still complete the task and receive reward. The harness fully absorbs the error and there is little gradient against inventing an alias, adding a stray field or using a nearby parameter name.

• 8

  Alternative tool schemas might not just be unfamiliar. They might be implicitly punished by post-training that optimizes for one particular, forgiving tool ecology.

• 9

  Tool schemas are not neutral, at least not on Anthropic models.

• 7

  The more post-training happens inside one dominant harness, the more every other harness will have to inherit its quirks.

• 5

  If the newest models get better at solving the task while getting worse at faithfully emitting an alternative tool schema, then the harness needs stronger guarantees somewhere.

• 6

  We cannot assume Claude-Code-trained behavior will transfer cleanly to your tools unless they are a close match.

Tone

technical, investigative, critical