I’ve always wanted to do this test but never can get around to it. Glad someone did it!

Nice DeepSWE score! Does it survive a lowercase filename?

Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.

Models clustered around 74% on DeepSWE end up 23 pp apart.

Good at coding, or good at DeepSWE? 🧵