Fortunately, this is the kind of flaw that doesn’t irritate much because the overestimation leaves us with spare time. Probably it’s also the reason why labs have not yet fixed this problem. So what, right? Companies have always encouraged software engineers to overestimate their budgeted time.
But think of it this way: if you’re weighing your options and the time estimates are completely overblown, you’re consistently steered towards the quick fix instead of the higher ‘effort’ solution. It encourages tech debt.
Why do models do this?
Likely the pretrain datasets are at fault. Countless stackoverflow answers, forum posts, github issues and pull requests, where developers anecdotally reference how long a task will take, eventually form a pattern. Just like models like to swoon over Japan because the internet is swarmed with posts romanticizing its culture, our internet is full of developers giving task estimates from a pre-LLM era.
How can we fix it?
Pretrain corpuses lead to many types of LLM behaviors that we prefer to train away. One approach is to use reinforcement learning to reward the model to correct its behavior. Another approach is to use heuristics to adjust the given estimate to something far more likely. Either way, the first step is to measure.
How much do models overestimate?
The first step to solving the problem is to reliably measure the behavior. And luckily, doing this is relatively simple. Take a list of verifiable tasks, ask the LLM to look at the task (prompt + environment files) and ask it to generate a time estimate. Then ask it to perform the task and measure how long it takes.
There are confounding factors of course that should be minimized. For example: is internet speed constant? Is the task being performed on the same system each time? Is inference speed constant? And so on...
Nevertheless, the measurements can be made.
We generated 200 simple verifiable tasks and cooked them into a benchmark called ETABench, then ran GPT 5.4 (because we’re not made of money) against all 200 tasks to measure results.
The results
Laid out side by side, we see that GPT 5.4 estimated 1,334 minutes when in reality the tasks only took 180 minutes in total.
Not a single task was underestimated, or estimated correctly. And on average there was an 8× discrepancy between prediction and reality.
When laid out task by task, we clearly see that the estimations live almost an order of magnitude higher than the measurements.
We also measured different discrepancy multiples between different task categories. Where frontend tasks seemed to have the least discrepancy (albeit still 6×) and debugging and performance tasks the highest.
Next steps
All avid LLM agent users have noticed this phenomenon before. But as much as the model capabilities have improved over the last year, this problem remains. Measuring it was simple. And once you can measure it you can hillclimb the solution.
ColophonBy @chkn_little · Researched by GPT 5.6 sol (high) and Fable · Measurement tasks generated by GPT 5.6 sol · Written by @chkn_little · Edited by Kimi 3