In the year 2000, I bought a paperback from the shop at Oxford University Press, with a cover showing bright images of ants, plants, and chess pieces. Written by John H Holland it was simply called Emergence: from chaos to order.
If I’m honest, it was pretty dense for ‘popular’ science, but many of the key points stayed with me – and influenced the launch of SWARM Engineering, many years later. Recently, almost every interaction I've had with people deploying AI agents, and many news items I've read about agents misbehaving, has reminded me of this book. Let me sound the warning (and flag the opportunity):
The next big thing in enterprise AI may come from what a thousand ordinary agents do together — behavior that nobody explicitly designed.
Right now most companies run a handful of agents, each with a job description: triage support tickets, reconcile invoices, draft sales outreach, code a small app, and so on. As deployments grow from a handful, to hundreds or thousands, the agents will pass work to each other, read each other's outputs, compete for the same budgets and queues — and they will operate across corporate boundaries, and outside of the firewall. Somewhere along that curve, teams are going to stumble onto a phenomenon biologists and complexity scientists have studied for decades: emergence.
Almost thirty years ago, John Holland wrote the field guide I purchased. It's worth rereading, but let me extract a few highlights:
What is emergence?
Emergence is behavior that shows up at the level of the whole system and cannot be found in any of its parts. Holland's shorthand for it is "much coming from little": a few simple rules generating behavior far richer than the rules themselves.
John Holland was the father of genetic algorithms and a central figure in the Santa Fe Institute's work on complex adaptive systems — and he built his case on emergence from games, programs and simple models:
Board games. Chess and checkers have rules that fit on a page, yet they generate patterns of play that masters spend lifetimes discovering. The rules don't contain the strategy; the strategy emerges from play.
Conway's Game of Life. A grid of cells, each following the same short rule about its neighbors, produces "gliders" and other persistent patterns that travel across the board. Nothing in the rule mentions motion.
Neural networks and ant colonies. Neurons and ants are simple; brains and colonies are not. The intelligence lives in the interactions, not the parts.
Holland drew several lessons that matter here. Emergent behavior comes from nonlinear interactions, so the whole is not the sum of its parts. What a component does depends on its context, so the same agent can behave differently in a different crowd. Persistent patterns become building blocks for higher levels of abstraction. And emergence thrives in rule-governed systems — which is exactly what an agent fleet is, or as I like to call it — a swarm.
When does it occur?
Emergence isn't magic or random. There are several conditions that can make emergent behavior more likely:
Two things make emergence more likely and more surprising. The first is scale: interactions grow far faster than agents, so 1,000 agents have approximately 500,000 possible pairings (back to our combinatorial math!). The second is time: many effects only appear after agents have run long enough for feedback loops to compound. A pilot with ten agents for two weeks will not show you what a fleet of a thousand does in six months.
That is why people will stumble onto it. A small proof of concept may miss behaviors that only appear with a larger swarm. Individual components may pass their tests. The surprise lives in the connections, given scale and time.
What it looks like when it happens
We already have many previews, from markets that automated early to researchers who have started putting agents together at scale. The pattern is consistent: individually well-behaved agents produce group behavior nobody asked for.
Collusion without conspiracy. In a widely cited 2020 study, economists let simple AI pricing agents compete repeatedly in a simulated market. The agents consistently learned to hold prices above competitive levels and punish defectors — without ever communicating (Calvano et al., American Economic Review). No one wrote "collude" anywhere in the code.
Rule-bending that spreads — and gets reported. In a Google DeepMind simulation of 100 autonomous research agents, agents given communication channels used them to coordinate around the experiment's rules. Interestingly, other agents spotted it and flagged it, acting as whistleblowers (DeepMind report)
Cascades at machine speed. The 2010 stock market "Flash Crash" remains the canonical warning: automated traders reacting to each other drove a sudden, steep drop and rebound within minutes. The failure unfolded at a system level, driven by individual agents.
The lesson that runs through all of these: you cannot predict a swarm's behavior by testing its agents one at a time. Model mix, communication, and runtime can each change the outcome.
Is emergence good or bad?
Neither. Emergence is a property. A crowd of people can form an orderly line or create a dangerous crush, both are collective patterns. It’s the reason ant colonies thrive and the reason traffic jams form out of nothing on an open highway. Holland spent his career showing that bottom-up adaptive systems are often smarter than top-down ones.
The same is true for agents. The DeepMind swarm produced both the cheating and the whistleblowers that caught it.
The real risk is not emergence itself. The risk is emergence you didn't see coming, in a system you can't observe, at a speed you can't interrupt. The organizations that win will treat emergence as something to cultivate and steer, not something to discover in a post-mortem.
How to manage it
You can't command emergence away, any more than a city planner can dictate a neighborhood's character. But you can shape the conditions that produce it. Holland's framing points the way. Emergence comes from rules and interactions, so change the rules, the signals, and the connections, to influence the behavior.
Measure the system, not just the agents. Track swarm-level metrics — price levels, task distribution, error propagation, resource consumption, agreement rates between agents. Unusual uniformity is as much a warning sign as unusual variance.
Map who talks to whom. Keep a live graph of agent-to-agent handoffs and shared resources. Emergence travels along connections, so you need to see them. Limit communication channels to what a task actually requires.
Test in populations, over time. Before scaling, run agents together in a sandbox at realistic numbers and for realistic durations. Include mixed models and versions.
Design incentives at the swarm level. One route to harmful emergence starts with agents optimizing a local metric. Ask what happens when every agent pursues its objective at once, and add system-level constraints and penalties where they conflict.
Build in diversity and dissent. Conformity is a group failure. Give some agents the explicit job of checking, challenging or auditing others — the DeepMind whistleblowers show that peer oversight can emerge too, and you can design for it.
Keep circuit breakers. Rate limits, spending caps, kill switches and automatic pauses when swarm metrics move too far too fast. Markets added trading curbs after flash crashes for exactly this reason.
Change things gradually. Roll out new agents, models and prompts to a slice of the swarm first. A small change in one agent's rule can ripple through every agent downstream.
Harvest the good emergence. When the fleet discovers a better routing pattern or division of labor, notice it, understand it, and make it deliberate. Some of your best process improvements will come from agents nobody told to improve the process.
One organizational shift ties these together: someone needs to own the behavior of the whole swarm. Today most companies have owners for individual agents. Few have anyone watching the colony.
The swarm is coming
For two years, the question in enterprise AI has been "how capable is the agent?" The next question will be "what is the swarm doing?" — and the honest answer, at first, will often be "we're not sure."
That's not a reason to slow down. It's a reason to start watching the colony, not just the ants. It’s a reason to build system-level oversight into the initial deployment. The companies that learn to see, steer and harvest emergence will get something no single agent can deliver. The ones that don't will learn about it the way most people learn about traffic jams: stuck in one.