Can software factories actually work?
Software factories, explained
Agents write code faster than any human can review.
To keep from becoming the bottleneck, engineers everywhere have been building software factories: automated systems and guardrails that let agents ship and test on their own.
But the hype came to a sudden halt when HumanLayer CEO Dex Horthy went viral with a three-part series on Why Software Factories Fail last month. He argues that no amount of harness engineering can ever make them work at scale.
So, will we always be stuck reading the code, or can software factories actually work?
What is a software factory?
The idea is simple. With the right pipeline in place, you can automate most (if not all) of software development, much like an industrial assembly line.
It involves several loops scaffolded with tools like version control, CI/CD, and monitoring that make it easy for anyone – human or agent – to ship code in a standardized, repeatable process.
Current software factories that incorporate agents look something like this:
Humans decide what to build and put it into a ticketing system.
Agents pull tickets, build and test PRs that satisfy the requirements, then submit them for CI and review.
If the PR is approved, the code is deployed to production via CD.
If it’s rejected, it goes back to an agent for further iteration until it gets approved.
Once the code is in production, users engage with it and emit data, feedback, and signals that inform what gets built next.
Since it’s just a metaphor, there’s no actual criteria for what counts as a software factory. It’s easier to think about it as a spectrum based on how much of the code is produced by agents, and how closely humans still read the code:
Ramp, Cursor, and Uber land near the middle today. They’ve posted about using autonomous background agents without explicitly calling them software factories. PostHog is just a little further than that with agents writing about 70% of our PRs, but humans still skim at least 80% of those.
The top right corner is where it gets controversial with “lights-off” software factories, or “dark” software factories. In these, agents write all of the code while humans read none. This requires:
A high degree of AI adoption in a team. Just having access to coding agents doesn’t mean anything if no one is using them.
A system where agent output gets validated to the point of mergeability, such as with peer reviews, local testing, or browser clickthroughs; writing unit tests or reasoning about their own code is not enough.
Guardrails that assume failure, such as sandboxes, permissions, and deterministic checks. Humans only act as decision makers on high risk PRs.
Only a few companies like StrongDM have publicly written about going fully lights-off, and many developers are skeptical about the approach.
Why is everyone talking about them now?
As humans shifted from writing code to orchestrating agents that write code for them, code review has become the new bottleneck.
At the same time, models keep getting better, so people are building loops, systems, and tools where agents review and/or merge PRs instead of humans – a.k.a., software factories.
But the discussion really heated up when Dex Horthy went viral with his three-part piece on Why Software Factories Fail a few weeks ago.
He argues that no amount of harness engineering makes lights-off software factories viable because the problem lies in the models.
He specifically highlights how coding agents are rewarded for one-off code correctness during RL training:
“How the model got to a correct answer doesn’t matter. If the tests pass, we win, but there is no penalty for eroding codebase maintainability.”
This happens because there simply aren’t many benchmarks that can detect quality and maintainability in the first place. The paper behind one recently published benchmark, SlopCodeBench, describes the challenge well:
“Every design decision in software engineering is a compromise with unknown future requirements.”
In other words, whether a design is good or not depends on what an agent will get asked to build next. That rubric is unique to every codebase and impossible to know in advance.
So, until there are better benchmarks in place, Dex’s recommendation is to either keep reading the code, or front-load human effort in the planning phase.
What software factories are missing
The problem with most software factories today is that they enforce a clean separation between “deciding what to build” and “building it.”
This is evident in every blog about software factories we’ve seen so far. The pipeline always begins with a step where a human does all the planning and decision-making first, stuffs the requirements in a ticket, then hands it off to an agent with zero prior context on the problem.
Dex’s argument follows this pattern, too. In part 3, he says a better oracle for code quality would be whether the output “passes all verifiers for an incrementally-divulged spec.” Again, it’s assumed that the agent that writes the code must receive a spec written by a human.
In the real world, that’s not how good product engineering works.
Every engineer knows that to design a system well, you need both the technical constraints as well as a deep understanding of the problem. This is why product engineers need to talk to users, measure product-market fit, and validate product ideas. It’s also why our engineers decide what to build, not PMs.
The same applies to coding agents. An agent makes better architectural choices if they have access to relevant information, such as how your product is currently being used, your ideal customer profile, what users are complaining about, or what else is going on in your company:
Looking at signals from prod is especially valuable for enabling code maintainability since most software changes happen in response to actual usage. At PostHog, for example, 63% of changed lines are in files that already exist, and fix is our largest commit type at 40%.
And if you think about the last time you addressed an incident in prod or a piece of user feedback, you probably relied more on logs and traces to figure out a fix instead of trusting someone else’s secondhand account of the problem.
These are the kinds of problems we’re working on at PostHog to make self-driving products a reality: giving agents the same context that human engineers would use to make better design decisions.
Once you have that, turning off the lights might not be so crazy after all.
Words by Jina Yoon, who still can’t believe we got software factories before GTA 6.
📕 Long-term quality reads
Are agents really killing UI? – Cleo Lant
Own the Outer Loop – Addy Osmani
The Session You Cannot Take With You – EARENDIL
Stateless MCP has recaptured my interest – Simon Willison
We spend $45,000 on doing more weird every month – Charles Cook