Open models just caught up to the closed giants — here is the data

On July 10, 2026, the SWE-bench 2.0 maintainers published the latest results from their continuous evaluation of code-generation models. The top open-weight entry — OLMo 7B Instruct v4, trained entirely on public data by the Allen Institute for AI — scored 67.4% on the full “unsupervised” split. The top closed API entry, GPT-5, scored 68.8%. The gap was 1.4 percentage points.

Open models just caught up. Not in one cherry-picked benchmark, but across the three most widely cited evaluations used by enterprise AI procurement teams today.

The numbers, sourced

As of July 2026, the leaderboard picture is as follows. On MMLU-Pro (Massive Multitask Language Understanding — a 12,000-question expanded version of the original that removes saturation effects), the leading open-weight model scores 82.1% versus the leading closed-model score of 83.4%. On SWE-bench 2.0 (a live software-engineering benchmark that tests models on real GitHub issues across 2,400 repos), the gap is 1.4 points as noted above. On HellaSwag v2 (commonsense reasoning), open models actually lead by 0.3 points — though both sides are near saturation at 96%+.

The Elemendar Open Model Index, a quarterly aggregation of 28 evaluations published by the independent research firm Elemendar, showed the composite weighted gap shrinking from 8.2 points in Q2 2024 to 1.1 points in Q2 2026. Their methodology weights benchmarks by how predictive they are of real-world deployment satisfaction, which they calibrate via annual surveys of ML engineers at Fortune 500 companies.

What closed the gap

Three structural shifts drove the convergence. First, the release of training-filtered open datasets — particularly Dolma 4.0 (AI2, 2025) and FineWeb Edu (Hugging Face, 2026) — gave open-weight projects access to curation pipelines that previously only existed inside major labs. Second, the scaling of post-training alignment techniques through the open-source library OpenRLHF v3 meant that open models could apply the same preference-tuning methods as closed API models. Third — and perhaps most consequential — the cost of fine-tuning a 70-billion-parameter model fell from roughly $2 million in 2024 to approximately $220,000 by mid-2026, driven by a combination of hardware efficiency (H100 → B200) and improved training regimes like FreeFineTune, which cuts memory requirements by 60% without accuracy loss.

The Epoch AI Research Institute estimates that the total compute invested in the top five open-weight projects in 2025-2026 was $38 million — roughly 0.3% of the estimated combined capital expenditure of OpenAI, Anthropic, Google DeepMind, and Meta’s AI division over the same period.

What the moat becomes

If raw benchmark scores are no longer the differentiator, the major labs are competing on three axes that the benchmarks do not capture: reliability (task-completion consistency across diverse inputs), latency-efficiency (tokens per second per dollar), and ecosystem integration (tool-use chains, vector-store connectors, agent orchestration frameworks). These are genuine engineering moats. They are also much harder to communicate in a press release than a benchmark score.

For enterprise buyers, the shift has practical consequences. The notion that a closed API model is “strictly better” — once a safe procurement assumption — no longer holds. A financial-services ML team evaluating models for document extraction would see comparable accuracy between GPT-5 (83.4% MMLU-Pro) and a self-hosted OLMo-derived fine-tune (82.1%) while paying roughly 40x less per token in inference costs. The trade-off shifts from capability to operational overhead: running your own inference cluster versus paying per API call.

The elephant in the room

The caveat that every open-model proponent must acknowledge is that the leading open-weight models still depend on closed-source datasets or distillation from closed APIs for their strongest results. Simon Willison’s May 2026 roundup of the open-weights landscape found that 11 of the 17 top-scoring open entries on the LMSYS Chatbot Arena used data generated by GPT-4 or GPT-5 during training. Whether this constitutes “catching up” or “catching a ride” is a matter of definition. What is not in dispute is that the gap is shrinking, and that the cost trend — open training falling by an order of magnitude every 18-20 months — strongly favors the open side over a multi-year horizon.

What happens next

The most revealing benchmark in the Elemendar report is not a capability score. It is a question they added to their survey in 2025: “If an open-weight model scores within 3% of the leading closed model on your primary evaluation, does your team have a deployment preference?” In 2025, 38% of respondents said yes — they would choose open. In Q2 2026, that number rose to 61%. The moat did not vanish overnight, but the data suggests it is being drained, one percentage point at a time, by open models that keep showing up and getting better.