Paul Gauthier@paulgauthierGemini 2.5 Pro sets SOTA on the aider polyglot leaderboard with a score of 73%. This is well ahead of thinking/reasoning models. A huge jump from prior Gemini models. The first Gemini model to effectively use efficient diff-like editing formats. t.coOpens with an observation
Paul Gauthier@paulgauthierThe new Qwen 2.5 Coder models did very well on aider's code editing benchmark. The 32B Instruct model scored in between GPT-4o and 3.5 Haiku. 84% 3.5 Sonnet 75% 3.5 Haiku 74% Qwen2.5 Coder 32B 71% GPT-4o 69% Qwen2.5 Coder 14B 58% Qwen2.5 Coder 7B t.coOpens with an observation
Paul Gauthier@paulgauthierNew Claude 3.5 Haiku comes in 4th on the aider code editing leaderboard with 75%. This is just behind the old 3.5 Sonnet 06/20. 84% Sonnet 10/22 80% o1-preview 77% Sonnet 06/20 75% Claude 3.5 Haiku 47% Claude 3 Haiku t.coOpens with an observation