Why the Wait, But Why meme will yet again be misread

When Mythos drops we're going to see another round of METR results and with it another round of this meme. Regardless of what the results show, this is the wrong way to think about it.

The first problem is mathematical. Implicit in the meme is that we're about to experience take-off, but what it misses is early on an exponential and a logistic look nearly identical. You genuinely can't tell them apart until you start hitting diminishing returns. It's just as possible we're going to start approaching a ceiling, but the meme just assumes the answer.

The second and bigger problem is that the meme treats intelligence as a single dimension. One Y-axis, one line. In reality, model capability is deeply jagged. There are areas where models already far exceed human capability and areas where they fall well short. And as these models improve, that jaggedness is getting more pronounced, not less.

When you see METR results or any major benchmark, you're mostly seeing the jagged peaks, not the troughs. The headline number looks like rapid linear progress on a single axis. The reality underneath is much more uneven, and the unevenness is where all the interesting questions live.

Instead of asking "how close have we gotten to human intelligence" as a single dimension, to me the more interesting versions of the same question are:

1. in what areas is it performing like the top 1% of humans?
2. in what areas is it approaching or exceeding the most capable humans?
3. in what areas does it still fall short of the average person?

We're seeing a lot of bucket 1. Coding is the classic example — less than 1% of humans can program, so in a sense the models are already at the 99th percentile of all humans. Give them the SAT or GRE and they'll typically score between the 90th and 99th percentile. This extends across fields with high verifiability like science, math, and engineering. There's a reason verifiability is the key and if you're curious I've got a whole separate thread on that.

But bucket 1 is very different from bucket 2. Writing a basic for loop makes you better at coding better than 99% of all humans, but you would still be the worst developer on the planet. Getting to bucket 2 means going from 99th percentile to 99.99th. While the first two 9s are hard, the next two 9s are an order of magnitude harder. We might be getting a glimpse of bucket 2 with Mythos's ability to discover vulnerabilities in computer systems. The question is which other domains will follow.

If this movie feels familiar, it's because we've seen it play out before with autonomous vehicles. Ten years ago AVs felt like they were around the corner. Turns out there's a massive gap between getting to the 90th percentile of human driving performance, the 99th percentile, and becoming good enough to fully remove humans from the loop. Closing those last few percentage points took a decade.

And then there's the areas where the models are still genuinely weak. Not just "not superhuman" but worse than the average person. Navigating ambiguity, taste, judgment in novel situations, knowing when they're wrong. The meme erases this bucket entirely. It has to because the only performance we can track must inherently be verifiable. The models will do best in verifiable areas and those are the only areas we can test. It becomes tautological.

Finally, the meme also conflates is the gap between AGI and ASI. The implicit assumption is once we make computers as smart as humans, making them an order of magnitude smarter is right around the corner. But if humans are the metaphor here, there's a major fallacy in the argument. If you're teaching another person, it's far easier to teach them up to your skill level than it is to train them past it. I suspect models will be no different.

Up until the boundary of human capability, the corpus is rich. Beyond it they're fumbling around in the dark just like we are. Once models hit the limits of what humans can do, extending past that probably starts to look more like human progress. Still quite fast compared to historical standards, but quite slow compared to what we've gotten used to over the last few years.

All of this doesn't change how transformative these tools already are. AI getting as good as humans in even a couple of dimensions is already a seismic change. We've already reached a point where programming as we used to know it is dead. We're probably at the point of needing to completely reimagine vulnerability and system security. The change will be dramatic even if the path to summoning god is a whole lot longer than this meme might imply.