Rendered at 22:36:22 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
HarHarVeryFunny 1 days ago [-]
This seems a strange thing to test given that Claude Code is optimized for Anthropic models.
A test of how different models perform in more of a model-agnostic harness like OpenCode would be much more interesting, perhaps paired with a control experiment of how those same models performed on the same task when using their respective native harnesses.
rpdillon 1 days ago [-]
Yeah, it's an interesting test because he gave Kimi credit for running inside the harness he liked, which I presume is Claude Code. But he also said it was noted for its harness sensitivity, and then said it didn't do as well as Opus.
A lot of folks don't seem to understand that it's the harness and its behavior paired with a particular model that produces the result, and you have to focus on both aspects. He barely acknowledges the importance of the harness in his write-up, beyond the two notes above.
HarHarVeryFunny 1 days ago [-]
There's a very interesting benchmark comparison below of Opus 4.7 run under three different harnesses : OpenCode, Cursor and Claude Code where it's not very close at all and Opus's native harness, Claude Code, performs worst of all three.
The pass@1 scores are 50/45/40 for OpenCode/Cursor/Claude Code respectively.
A test of how different models perform in more of a model-agnostic harness like OpenCode would be much more interesting, perhaps paired with a control experiment of how those same models performed on the same task when using their respective native harnesses.
A lot of folks don't seem to understand that it's the harness and its behavior paired with a particular model that produces the result, and you have to focus on both aspects. He barely acknowledges the importance of the harness in his write-up, beyond the two notes above.
The pass@1 scores are 50/45/40 for OpenCode/Cursor/Claude Code respectively.
https://artificialanalysis.ai/agents/coding-agents#harness-c...
I've seen other benchmarks where Pi also outperforms Claude Code both in model performance and in much reduced token usage.