Most people using AI agents run one at a time. One agent, one task, wait for the answer.
We run a lot of them at once. Our own system, Pulsar, runs well over a hundred agent sessions on our own servers. Once you can do that, a question shows up fast: if one agent is good, are ten better? Are fifty? And if they are, how should they be organized?
Nobody seems to know. We looked. There are plenty of demos of multi-agent setups and almost no measurements. So we built a lab and started measuring.
The setup
The task is a bug hunt. We froze a copy of a real codebase, about 158,000 lines, and every swarm hunts the same frozen copy. Bugs are a good test because they are countable. An agent either found a real defect or it did not.
Every report goes to a separate set of Claude verifiers. They check each claimed bug blind, without knowing which swarm or which recipe produced it. What comes out is one number per run: distinct real bugs. We also track false alarms, duplicates, severity, and how many bugs were never found before.
The swarm itself is 10 agents running Qwen Flash, a small open model, on our own hardware (two NVIDIA DGX Sparks). Each run lasts 30 minutes. Then the agents are shut down, the reports are scored, and the next run starts. No human in the middle.
Running the model locally matters. The tokens cost us close to nothing. A cloud model would make this kind of experiment expensive fast. A local model lets us run it all day and all night.
The loop
We borrowed the method from Andrej Karpathy's idea of an autonomous research loop. There is always one champion recipe. Each test run takes the champion and changes exactly one thing. If the change beats the champion, it becomes the new champion. If not, it goes in the log and the next idea runs.
A "recipe" is how the swarm is organized. How the work is divided, what each agent is told, what it knows about the other agents, when it gets a fresh context, what happens when it says it is done. The model never changes. Only the organization does.
We are 34 runs in as of this writing.
Why we think organization is the whole game
This started with a result from a bigger model. We gave 20 agents running a frontier cloud model the same bug hunt with no structure. They found 120 real bugs. They also filed 593 reports, and 417 of those were duplicates of each other. Twenty smart agents spent most of their time finding the same bugs.
Then we made one change. We split the code into 20 slices and gave each agent its own slice. Same model, same time, same instructions otherwise. They found 369 real bugs.
That is 3.1 times more, from one change to how the work was divided. No new model, no better prompt.
That result is why the lab exists. If one organizational change is worth 3x, what are the others worth?
What we have learned so far
Noise is bigger than you think. We ran the exact same recipe twice and got 23 bugs, then 28. Over four identical runs the spread went from 21 to 33. Any "improvement" under about 30% from a single run is not an improvement. It is luck. Most multi-agent claims we see online come from one run.
Telling agents how to think made them worse. We added a 104-word method to the instructions ("start from the callers, then trace inward"). Real bugs fell from an average of 24 to 13. Longer instructions cost us. Changing the structure around the agents helped. Changing their instructions mostly hurt.
Agents skip the same files every time. We measured what each agent actually read. A Qwen agent reads about 5,000 to 6,000 lines in 30 minutes, about half of its slice. And it reads in list order, so every run re-read the top of each slice and skipped the bottom. Over five runs, 39 of 276 files were never opened. The fix was to cut each slice in half and alternate halves between runs. That recipe, "tiles", posted the best single run at the time (38) and moved the champion's average from about 24 to about 33.
The idea everyone ranked first did nothing. We asked two separate ChatGPT Pro sessions for ideas. Both put the same idea at the top: when an agent says it is done early, push it back to work on the files it has not read. We built it. All 10 agents were still working at minute 20. The bug count did not move. Late work mostly re-checked files that had no bugs left in them.
Thinking matters more than steps. We turned the model's reasoning off to get more steps per agent. Each agent took three times as many steps. Real bugs fell to 7, false alarms rose to 80%, and three agents wrote nothing at all.
Memory trades volume for novelty. We gave each agent a list of the bugs already found in its slice. Those agents found about half as many bugs per run. But almost none were repeats, so they added about 1.9 times as many never-seen-before bugs per run as the plain recipe.
The bigger question
Here is the bet underneath all of this. A frontier model is smarter per agent. A local model is close to free. If a cheap swarm runs all night, does it catch up to the expensive one?
So far every 30-minute run of the Qwen swarm keeps adding new bugs. Across all runs the swarm has found 387 distinct real bugs, and the curve has not flattened. That count includes bugs outside our reference list that are not de-duplicated across runs, so treat it as an upper bound. The frontier swarm found 369 in a single 20-minute round. The local swarm needs many runs to match that. But those runs cost us almost nothing, and they do not stop.
What we are looking for
Recipes that transfer. A recipe that only works for one model or one codebase is a trick. A recipe that works with small models, big models, and different tasks is a way to organize work. Split did that. We want to find the next one.
Next on the queue: more agents on smaller slices, a few agents diving deep into one file each, faster sprints, a leaner set of instructions, and different reasoning effort for different agents.
What this series will cover
We will post the leaderboard as it changes, every recipe with the exact change it makes, and the numbers for each run. That includes the failures, and so far most ideas fail. When an idea that sounds smart loses to the plain version, we will say so. That is most of what a lab is for.
If you run agent swarms and measure them, we would like to compare notes. We have not found anyone else doing this in the open yet.