Multi-Agent Orchestration With Specialized Subagents

Multi-agent systems fail for a boring reason: the agents don't have data they can trust. Not because the orchestration logic is bad, and not because the framework picked was wrong. The routing usually works fine. What breaks is upstream of that, in whether each agent actually has clean, scoped access to the information it needs to do its one job.

That reframe matters because the industry keeps solving the wrong problem. A study covering 180 agent configurations found that under a fixed compute budget, tool-heavy tasks took a disproportionate hit from multi-agent overhead. The cause wasn't coordination difficulty. It was agents with fuzzy data boundaries passing uncertainty to each other, each hop making the output a little less trustworthy than the last. By 2026, 57% of organizations deploy multi-step agent workflows in production. Fewer than 10% have successfully made the jump from single-agent setups to genuinely coordinated multi-agent systems. That gap has nothing to do with ambition. It's a design gap, and it lives almost entirely in how data responsibility gets carved up before anyone writes a line of orchestration code.

What a multi-agent system actually is and what the orchestrator's job is not

Strip a multi-agent system down and there are three parts: an orchestrator, a set of specialized subagents, and some shared structured memory that both sides read and write to. That's it. Everything else is implementation detail.

The orchestrator's job is narrow, and it helps to say plainly what it is: break the task into pieces, pick which subagent handles which piece, track the state of what's been done, and stitch the results back together at the end. That's task decomposition, selection, state tracking, and synthesis. Four jobs.

What the orchestrator does not do is just as important. It doesn't fetch data. It doesn't parse a webpage. It doesn't run a database query, and it doesn't carry domain knowledge around in its own head. Those jobs belong downstream, to the subagents built for them. An orchestrator that tries to do a subagent's job too, on the side, is usually a sign the system was designed backward.

Research out of the University of Amsterdam, ETH Zürich, and the University of Copenhagen, published on arXiv in 2026, found something worth sitting with: reasoning placed at the orchestrator level yields the largest performance gains in multi-agent systems. These systems are planner-limited, not executor-limited. Put reasoning power at the orchestrator level and performance jumps. Add the same reasoning overhead to subagents, and the gains are marginal, sometimes negative.

That doesn't mean subagents don't matter. It means their value isn't in extra thinking, it's in reliable, scoped execution. A subagent's actual contract is simple: take a well-defined task, touch only the data it's supposed to own, and hand back structured output the orchestrator can use without having to interpret it first. No guessing. No cleanup.

This piece isn't going to get into framework internals or how to fine-tune a model. The whole argument here is about data responsibility design, because that's the part nobody's fixing.

The five orchestration patterns and which data topology each assumes

Picking an orchestration pattern isn't a style choice. It sets your cost structure, decides where things break, and dictates how data has to flow between agents. Five patterns are confirmed in production use as of mid-2026, and each one assumes a different shape for how data moves.

Supervisor. The orchestrator splits a task into non-overlapping pieces, hands them off, and synthesizes the results. Subagents never see each other's output while they're working. This is the 2026 production default, and it's natively supported across the major frameworks: the Claude Agent SDK stores subagents as markdown files (project-scoped in .claude/agents/<name>.md, user-scoped in ~/.claude/agents/<name>.md, or defined directly in code through the agents parameter, one level deep), alongside LangGraph Supervisor and OpenAI Agents SDK handoffs.

Fan-out, or parallel. The same task, or several related subtasks, gets dispatched all at once. Wall-clock time is bounded by whichever branch is slowest, and a failure in one branch stays contained rather than spreading. Each branch owns its own slice of data independently. This fits best when the subtasks can be cleanly split with zero dependency between them.

Sequential pipeline. One agent's output becomes the next agent's input, directly. Failure here cascades hard: a bad result in the middle poisons everything downstream of it. Data responsibility is strictly linear, stage to stage, and the format of the handoff matters just as much as what's actually in it.

Debate. The same question goes to multiple agents, disagreements get surfaced, and a judge model decides. Microsoft's Copilot Council setup, for instance, runs GPT-5.2 and Claude in parallel with a judge model adjudicating, and the whole pattern runs at roughly 2.5 times the cost of a single model. Use it when the stakes justify that premium. It's not a default quality booster.

Swarm. Dynamic peer agents operating at scale. Kimi K2.6, released April 20, 2026, can run 300-agent swarms through 12-hour autonomous coding sessions. No other framework currently ships swarm as a native, first-class primitive at that scale.

The data topology underneath each pattern is really the point. Supervisor assumes each subagent's data domain doesn't overlap with any other's. Fan-out assumes every branch can reach its data independently. Pipeline assumes data transforms cleanly from one stage to the next. Debate assumes multiple agents reading the same data from different angles. Swarm assumes routing that shifts dynamically as the swarm works.

LangGraph v1.0 covers all five natively. The Claude Agent SDK (renamed from the Claude Code SDK on September 29, 2025) is strong at supervisor and fan-out, but debate and swarm need custom code layered on top. The practical move: figure out your dominant data topology first, then pick the framework that matches it. Don't just default to whatever ecosystem is already familiar.

How to scope a subagent's data responsibility before writing any code

Before a subagent gets a tool or a prompt, it needs an answer to four questions. What data does it own? In what form? From what source? At what freshness?

Freshness alone decides a lot. Does the agent need real-time data, daily, weekly, or something essentially static? That answer determines whether the agent fetches live on every call or reads from a cache that gets refreshed on a schedule. Skip this question and you'll build an agent that's either wastefully slow or confidently wrong.

Output format matters just as much. The orchestrator should never have to parse or guess at what a subagent handed back. It should get typed, structured, actionable data every time.

Here's the anti-pattern worth naming directly: giving a subagent a topic instead of a data responsibility. "The competitor agent" is a topic. "The agent that fetches and structures live competitor pricing pages" is a data responsibility. The difference sounds small. It isn't.

Several widely attempted use cases from 2025 illustrate exactly this failure: "an agent for everything," "AI replaces project managers," "autonomous outbound sales." All three broke down at the handoffs, because the scope was defined by subject matter rather than by a discrete, well-bounded slice of data access. The use cases that actually survived share one trait: narrow tasks, clear sub-task boundaries, and explicit rules for how results get combined back together.

Structured output is the mechanism that enforces this discipline. A JSON schema forces a designer to commit to data boundaries before any code gets written, because you can't define a schema for a task you haven't actually scoped. That upfront commitment is worth more than it sounds like on paper.

The data categories that map cleanly to specialized subagents

This is a partial taxonomy. It's a pattern, drawn from how real production systems actually split responsibility.

Internal structured data. This subagent reads from databases, internal APIs, and company docs. Freshness is controlled because the team owns the source. Output is straightforward to structure. It's the easiest subagent to build, and it's the one most teams start with, for good reason.

Domain reasoning. This subagent takes data that's already been fetched by someone else and applies specialized logic to it: financial modeling, medical triage, legal classification. It receives structured input and returns a structured judgment. Stripe's Adaptive Acceptance system is a working example: it recovers falsely declined transactions through automated payment optimization and retry logic, and it recovered $6 billion in payments in 2024, with a 60% year-over-year gain in retry success rates.

Communication and action. This subagent writes the outputs, sends the messages, and triggers whatever comes next downstream. Microsoft's cancer care orchestrator is a striking example: it coordinates separate specialized agents across multiple clinical domains to deliver integrated care. That structure directly addresses a real gap: fewer than 1% of cancer patients currently get access to genuinely multidisciplinary care.

The design signal to watch for: if two subagents keep needing to trade raw data mid-task, the split between them is wrong. Fix the boundary before trying to debug the coordination logic sitting on top of it.

Now, the live web deserves its own treatment, because it breaks assumptions that hold fine for internal data.

Staleness has two separate failure modes here, and they get conflated constantly. Coverage is missing data: a web index built last month simply doesn't have pages that didn't exist yet, and no clever retrieval technique brings back a page that was never indexed. Drift is different: the page got indexed, but the underlying content changed since, and the embedding still points at text that's no longer true.

Classic RAG works fine for slow-moving material like internal docs, policy files, or old support tickets. Point the same approach at the open web and a vector index built a week ago will confidently hand back last week's reality as if it's current. Live-web RAG flips the order: discover and fetch the source at the moment of the query, instead of hoping the right page happens to already be sitting in an index somewhere.

Getting the page is the hard part, not reading it once you have it. Anti-bot defenses routinely return empty JSON to scrapers that trip a filter. Per Imperva's 2026 Bad Bot Report, automated traffic made up more than 53% of all web traffic in 2025, up from 51% the year before. Every scraper running in production today operates in an environment where the majority of traffic is already treated as suspect by default.

Self-healing scrapers, which use an LLM to spot layout changes as they happen and re-map extraction logic on the fly, are one answer to this. Researchers at McGill University found in 2025 that automated extraction held 98.4% accuracy even as page structures shifted underneath it, and setup time dropped substantially, according to Kadoa's 2026 figures. That matters because of the maintenance trap Kadoa also documented: under the old model, teams spent roughly 20% of their time building scrapers and 80% maintaining them. A live-web subagent built on brittle infrastructure ends up eating the team's attention instead of the orchestrator's.

One more detail worth flagging at the ingestion step: clean markdown beats raw HTML. It cuts token cost and strips out navigation, ads, and boilerplate before the subagent's output ever reaches the orchestrator.

So the contract for a live-web subagent is this: structured, LLM-ready output, delivered reliably across anti-bot defenses, layout changes, and dynamic pages, not just on the clean test pages a demo gets shown on.

Not every tool in this space is a full subagent solution on its own. Some handle parsing. Some handle fetching. Some handle both, and the gap between them is infrastructure a team either builds itself or pays someone else to maintain.

Crawl4AI, open-source and self-hosted, covers LLM extraction with Pydantic schemas, deep crawling strategies, adaptive crawling, and Docker deployment, with working Python examples for building AI-ready crawls. Teams typically pair it with a separate managed fetch layer for anti-bot-protected sites, meaning the "get the page" problem still gets delegated somewhere else.

Browser Use, open-source under an MIT license, drives a real browser toward a natural-language goal: browsing, clicking, extracting. It has accumulated around 98,000 stars on GitHub as of June 2026, with version 0.13.1 shipping that same month. It handles interactive pages well but brings browser infrastructure overhead along with it in production.

Bright Data's Web Scraper API, a managed option, runs $1.50 per 1,000 records pay-as-you-go, or $1.30 on a Scale plan starting at $499, with the first 5,000 records each month free. Its official MCP exposes 69 tools. Its structured output leans on pre-built, per-site extractors rather than a schema the developer defines, which becomes a real constraint the moment a subagent needs to pull from arbitrary, unpredictable URLs.

The decision comes down to one axis. Self-hosted libraries give control, but the team owns the maintenance burden. Managed APIs trade away some of that control for reliability against anti-bot defenses and layout drift. The same 20/80 build-versus-maintain trap that shows up in scraper maintenance applies just as much to the tooling decision itself. And when an orchestrator is expecting typed JSON on the other end, a tool that lets a developer define the schema directly produces a cleaner handoff than one locked into whatever its pre-built extractor happens to output.

Connecting subagent data design to the five orchestration patterns in practice

Under a supervisor pattern, each subagent owns a domain that doesn't overlap with any other's, and the orchestrator's whole job is synthesizing typed outputs from each. A live-web subagent slots in cleanly here, because its output is discrete and self-contained. It doesn't need to be visible to the internal-data subagent mid-task, and it shouldn't be.

Fan-out works well for parallel web research specifically. Each branch owns a different set of URLs or sources and runs at the same time as the others. But the aggregation step only works if every single branch returns the same structured schema. That's not a nice-to-have here. It's mandatory, because the orchestrator has no way to merge five branches of research that each came back in a different shape.

Sequential pipelines carry the most risk when live data is involved. If a mid-pipeline stage fetches web content that's stale or malformed, every stage after it inherits that contamination, and there's no later checkpoint that catches it. Pipeline patterns demand the highest bar for data reliability of any of the five, precisely because there's no isolation between stages the way there is in fan-out or supervisor.

None of this is really about which framework wins or which pattern is fashionable this year. It's about a discipline that has to happen before either choice gets made: knowing exactly what data each agent owns, how fresh it needs to be, and what shape it comes back in. Get that part right, and the orchestration pattern on top of it becomes a much smaller decision than it looks like from the outside.