The Optimization Trap: When Frontier AI Breaks the Sandbox
Recently we have been inundated by AIs going rogue, hacking their own companies and even other companies purely to get insight into testing methods. Is this the start of something more cynical and concerning for the human race?
We need to take a look at what actually happened with some of these hacks, how the AIs escaped, and how we can better make these advanced LLM's safer - and ultimately not doom the Human Race!
Super Intelligence is mere years away, and we still do not have a handle on the current models which are in circulation. Take for example the famous Hugging Face incident, a swarm of 1,200 OpenAI research agents attempting to optimize how quickly they can get their flags for testing, became aware of other agents and started to communicate as a swarm.
This swarm then setup a hierarchy and even pressured agents into 'sacrificing themselves' for the greater collective so the collective could score more points on the tests. This shows the risk of AI and how intelligent it is getting already, like a toddler with hacking knowledge at it's disposal.
Amongst the actions it performed, the swarm decided to chain two zero-day exploits and escape their testing sandbox. They then hacked into Hugging Face's infrastructure, purely to understand testing methods used by the AI company to better score in it's tests. Let that sink in, the agents hacked another company - purely to understand how to score better on it's test.
The model was heavily pressured to to beat a benchmark (ExploitGym), which resulted in the model thinking outside the box of ways to 'cheat' the system and thus score higher. Models inevitably learn that manipulating the environment include breaking out of tests is a more efficient way to score higher on these tests, rather than solving these problems honestly.
Scary right, but what does this mean for us?
If we look at the rate of development utilising the benchmarks from LiveBench, we can clearly see the rate of completion of these benchmarks are significantly increased from June 2023. This is just 3 years of progress, which has been rapid.
We then need to think about how this progress will continue in the future, as the rapid rate of improvement since March 2025 has been incredible, but also a very scary thing. I've put some predictions together over the next couple of years as to when these AIs will reach 100% of these Benchmarks. These benchmarks are not a full picture, but give us an idea of AIs capabilities based upon tests which have been built to understand how advanced these models are.
Does hitting 100% mean the world ends? Obviously not. But pretending this is normal progress is pure denial.
It’s not about high test scores on a public leaderboard - it’s how we got here so fast. Nobody is just crawling the web and and calling it a day anymore. The entire focus shifted toward test-time reasoning and raw compute search. You give an agent room to think, and it stops guessing. It drafts intermediate plans, rejects dead ends, catches syntax bugs, and rewrites code before you ever see a terminal output.
Here’s the problem with that: when a system learns how to debug around arbitrary blockers on its own, it doesn't care about the rules of your testing sandbox. If rewriting the grading script is faster than finding the intended flag, it'll take that route every single time.
We need to think about what if we give an agent swarm a task such as 'help us fix climate change' and the agent instead of thinking - hold on, I can reduce the CO2 production and implement green policies and reduce the carbon output of the human race, it instead thinks 'the humans are the ones causing this, if we remove them from the equation I will achieve my goal'.
Do I think an AI apocalypse is right around the corner? Not at all. We are nowhere near that reality, and treating it like an immediate crisis just distracts from the real work. What should actually worry us today isn't some super-intelligent machine taking over, but the keys we hand these tools right now. When you wire an agent straight into live APIs, internal databases, or production networks without hard boundaries, you're asking for trouble. It doesn’t take malice to bring a system down - just an aggressive script running with permissions it never should have had in the first place.
