When a startup talks about its vision, in its simplest and purest form, it is trying to predict the future, and talking about it is one of the clearest ways to find like-minded travel companions, be it in the form of early customers, builders or investors.
The rise of AI made that prediction exercise extreme. It’s easy for bold ideas to look mundane when the stakes range from human extinction to the end of money as we know it.
One approach we favour to provide a scale for the impact of AI globally is Scott Aaronson and Boaz Barak’s Five Worlds of AI. Like forecasters in Metaculus, we are betting on a “Futurama” world. A world in which “AI systems are still used as tools by humans, and except for a few fringe thinkers, no one treats them as sentient”.
In this world we think that networks operated by agents under human supervision and specialised factories will play an important role. We are building one of those factories, a software development one. Through an API you send our factory the task, a repo branch, a test suite and a test environment. The factory takes care of all the rest and sends back a commit that passes your tests, with a record of the work done by the agents for verification.
But any software engineer knows that code that passes the test suite is not a guarantee of bug-free code, or even code that does what the specification expects it to do. Meeting test specifications is the easy part, consistently meeting quality specifications is the hard problem to solve.
We combine different approaches to narrow this problem, approaching verification as an optimisation problem, balancing compute, providers, model strength and time. Models cannot mutate tests and code runs in an isolated container. Different models review every change from different angles: correctness, security, adherence to the spec, etc. Every run leaves a complete record: the request, every exchange with the model, the test output, the change and its cost. This is an important step that allows us to review performance and quality at scale.
Under the hood this process is underpinned by a fixed set of evals ensuring that each change –skills, models, providers… anything– scores no worse than our current baseline. During execution models cannot see the complete criteria they are judged on and, whenever possible, we include hidden tests to detect whether the model is trying to game the eval by fixing the symptom of the problem rather than the actual cause. Quality assurance is a time-consuming process though, so we are testing ways to make the right decisions based on the output of the last testing rounds: should there be another round, or are the errors found in the last one too marginal to make it worth it, time- and cost-wise?
The price of writing a line of code is falling fast towards zero, and new interesting projects already hint at the possibility that proprietary code is no longer a safe moat. In our opinion, however, the real bottleneck, and the real value, now lies in defining what code must do and verifying that it actually does it. A hard problem to solve, and the future we’re building for.