How enabling two settings tripled our scores on the ARC-AGI-3 benchmark
Benchmark scores for artificial intelligence models often depend on more than just the underlying technology. Recent testing on the ARC-AGI-3 puzzle suite revealed that GPT-5.6 Sol scored significantly lower than expected when placed in a standard, generic testing harness. Our investigation showed that the model struggled because the default test environment discarded the model's internal reasoning after every move and purged historical data once context limits were reached. By treating the model as a blank slate at every step, the test prevented it from learning or applying past insights to new moves.
To address this, we integrated the Responses API to better reflect how our systems function in real-world products like ChatGPT and Codex. We enabled two specific settings: retained reasoning and compaction. Retained reasoning allows the model to keep its internal thoughts across turns, meaning it no longer has to solve a game from scratch every time it makes an action. Compaction manages the context window more efficiently by condensing information instead of simply deleting the oldest data when memory limits are hit.
The results were immediate and measurable. These two changes tripled the model's score on the ARC-AGI-3 benchmark while simultaneously reducing the total output tokens required by six times. This demonstrates that performance is not an inherent trait of the model alone but is a result of how that model interacts with its environment and manages its own history.
These findings suggest a shift in how the industry approaches model evaluation. If we measure models in environments that contradict how they are trained and deployed, the scores will remain artificially low. Moving forward, we recommend that developers use the Responses API with reasoning and compaction enabled to align evaluation methods with actual deployment standards. We look forward to seeing how these adjustments change the broader understanding of model reasoning capabilities.