Almost nothing optimizes against what it actually wants. It optimizes against a proxy, a stand-in it can measure, because the real goal is too expensive to score or impossible to write down. The proxy tracks the goal across the range where somebody checked that it does. Optimization pushes past that range, to where the cheap wins are, and there proxy and goal come apart. The distance between them is the gap.
Below are twenty problems people are pointing AI at, each drawn as the same gap. Pick one to see how far it reaches past what has been checked, what keeps its gap open, and where the same gap has shown up before.
Pick a problem to see what keeps its gap open and where the same gap has shown up before. Pick a barrier or an earlier case to go the other way. Hover for a short explanation.
Every version of every problem, placed by how hard its remaining gap is and how much of that gap no engineering removes. The selected problem's versions are joined in order, from deployed today to the ideal.