OpenAI engineers told colleagues earlier this month that they had found a way to cut inference costs by more than half on the models it was applied to, according to The Information. The first place it showed up: logged-out ChatGPT, the traffic from people who never bothered making an account, which the company was reportedly able to serve with just a couple hundred Nvidia GPUs.
That is the whole confirmed story, and it is thinner than the headlines racing around it suggest.
What we actually know
One person familiar with the internal discussion talked to a reporter. No blog post, no press release, no chart, no CEO quote on stage. OpenAI has said nothing publicly, and it has not explained the technique. The reporting notes it's unclear how many GPUs the guest tier needed before, which makes the "more than half" figure impossible to check from the outside.
Guest ChatGPT is also the easiest possible target. Logged-out users get a stripped-down version with limited features, the lowest-value traffic OpenAI serves. Squeezing it is not the same as squeezing the paid product or the reasoning models, and whether the gains carry over is an open question nobody at the company has answered.
So what did they do?
Guessing, mostly. The usual suspects are quantization, KV caching, smarter batching, and routing simple queries to cheaper models. Coverage from heise runs through the same list and adds a useful caveat: these tricks carry real risk. Aggressive quantization can degrade output quality, and bad routing can hand a hard question to a model too small to answer it. Some users have already blamed recent drops in ChatGPT quality on exactly this kind of thing, though that's speculation stacked on speculation.
The number that matters more
The GPU count is the flashy detail. The margin figure is the real one. The Information puts OpenAI's API gross margin at 39% at the end of Q1, up from 33% a year earlier, with a stated target of 52% by year-end. Hitting that reportedly means averaging around 56% for the rest of the year, which is a steep climb. A software-only halving of inference cost, if it generalizes, is one of the few levers that gets you there without new hardware.
Which is also why the timing is convenient. OpenAI is heading toward a period where investors will poke hard at whether the business works at scale, and a vague-but-positive cost story leaked through an unnamed source lands at a helpful moment. Compare it to how the company handled Jalapeño, its Broadcom-built inference chip, which got a name, a partner, and a specific 50% efficiency claim. This one got a whisper.
Anthropic and Google are chasing the same server-level efficiency, and none of the three have published benchmarks or code. Treat the whole thing as reported, not settled. The real test is whether the same optimization reaches paid tiers and the API. Until OpenAI says something on the record, that's the number to wait for.
