In January 2024, the best closed model beat the best open-weight model by 8.04% on the Chatbot Arena leaderboard. By February 2025 the gap was 1.70%, according to Stanford’s 2025 AI Index. Between November 2022 and October 2024, the cost of running a model as good as GPT-3.5 fell more than 280-fold.
Those two numbers changed how I think about where the value in AI ends up.
The shovels are free now
There’s an old line about gold rushes: the reliable money is in selling picks and shovels. For a while the frontier model was the shovel, and only a few labs could make one.
Pretraining a frontier base model is still a job for a handful of companies. But making a model good at a specific thing, by continuing its training on your domain and then reinforcing the behavior you care about, is now within reach of a small team. Open-weight families like Qwen, Gemma, Phi and DeepSeek give you a strong starting point, and some of them run on a laptop. I run a few of them on mine.
Model creation is diffusing. When the shovels are free, nobody gets rich selling shovels.
So where is the moat?
If you can’t win on weights, you win on the things weights can’t give you. I see three.
Data nobody else has. A base model knows what’s on the internet. It doesn’t know your customers’ failures, your domain’s edge cases, or the questions people bring to your product when something breaks. That data only comes from being deployed, and it compounds: every failure you see is training data your competitors don’t have.
Evals that define “better.” Whoever writes the eval decides what the model gets better at. On Copilot, the LLM-as-judge eval we built, calibrated against domain experts, became the bar every change had to clear, and session success rose 7 points. It turned “it feels better” into a number the team could argue about. A good eval is a product spec with teeth, and it’s specific to your users in a way a public benchmark can’t be.
The loop between them. Data without evals is a pile, and evals without data are a scoreboard. The advantage is the loop: deploy, watch where the model fails, turn those failures into training data and new eval cases, retrain, and ship again. Each turn is cheaper than the last, because the eval tells you exactly where to spend.
What this means if you’re building
The question I hear most is “which model should we use?” It’s the least durable decision you’ll make. The rankings change every few months, and the gap between the best open and closed models is now small enough that your switching costs matter more than the model.
Better questions:
- What does our product see that nobody else’s does, and are we capturing it?
- Do we have an eval that predicts what users will feel, and does it gate releases?
- How many days does it take us to turn a failure into a fix?
Teams that answer those well will be fine on any model. Teams that don’t will keep switching models and wondering why nothing improves.
Picks and shovels made money in the gold rush because they were scarce. The scarce thing now is knowing where to dig.