Crawl Scope and Depth Control for Research Agents

Crawl scope and depth are decisions, the same kind a human analyst makes when deciding which sources to trust and how far down a rabbit hole to follow a lead. Get them wrong and a research agent either starves on too little context or drowns in pages that have nothing to do with the question at hand. Most engineers treat scope as a config detail to set once and forget. That's backwards: scope is the single decision that makes an agent's answer either trustworthy or just confident-sounding noise. Automated traffic passed 53% of all web traffic in 2025, up from 51% the year before, so these calls about domain boundaries and depth limits now get made constantly, by machines, at a scale where one bad default compounds fast.

Domain boundaries, depth limits, and page caps as constraints on what an agent can know

Three separate controls decide what a crawl can see, and each does a different job. Mixing them up is the most common scope mistake in agent design, and it's an easy one to make because all three feel like the same kind of setting.

Domain scope is the coarsest lever. It decides whether the crawler stays on the seed domain, follows subdomains, or wanders off into external links. Get this one wrong and every downstream decision inherits the mistake, because there's no filter after this that catches an entire wrong domain.

Depth limit is about hops, not domains. It controls how many links away from the starting URL the crawler will follow. A depth of 1 means the homepage and whatever it links to directly. A depth of 4 means the agent can land somewhere only loosely connected to where it started, even while staying inside the same domain the whole time. The deeper an agent goes, the worse the topical drift gets, and nobody notices until the output reads strangely off-topic.

Page cap is the budget guardrail. Regardless of depth or domain, it's the hard ceiling on how many pages get fetched. Crawl4AI's BFSDeepCrawlStrategy exposes this directly through parameters like max_depth, include_external, and max_pages (which defaults to infinite, and that default has burned more than one engineer).

Changing one of these settings makes the other two behave differently, so nothing here should be tuned in isolation. A deep crawl with a tight domain scope is usually safe, since everything stays contained even many hops in. A shallow crawl with external links turned on can still pull in garbage, because even one hop outside the seed domain opens the door to unrelated territory. The include_external flag is the sharpest domain-boundary decision an engineer makes, and it defaults to off for good reason: flipping it on should be a deliberate call tied to the task, never something inherited by copying a config from another project.

BFS, DFS, and best-first traversal

Scope sets the boundaries. Traversal order decides what actually gets pulled from inside them, and picking the wrong strategy wastes the scope work already done.

BFS, breadth-first search, visits every link at the current depth before moving to the next. It tends to favor high-authority, high-visibility pages, since important content usually sits close to the homepage in most site architectures. In a live news-site benchmark, BFS indexed 949 files in 886.94 seconds at depth 4, real coverage at real time cost. That's real coverage, but it costs real time. BFS earns its keep when an agent needs to map the shape of a domain before committing to going deep anywhere in particular.

DFS, depth-first search, does the opposite: it follows one path all the way down before backtracking to try another. In the same benchmark, DFS returned 470 files in 233.02 seconds, roughly a third of BFS's time, because it wasn't trying to cover as much ground. DFS fits a documentation tree, a threaded forum, a single product's changelog history. Running it on the wrong site, though, leaves an agent with a long chain of pages that all say roughly the same thing, while an entire adjacent branch holding the actual answer goes untouched.

Best-first should be the default for agentic pipelines. Instead of visiting links in a fixed order, it scores each candidate URL before fetching it and only follows the ones that clear a threshold. Crawl4AI implements this through BestFirstCrawlingStrategy, with a url_scorer parameter and an optional score_threshold as a secondary filter. A keyword match, semantic similarity, a URL pattern, or an LLM call judging relevance directly can serve as the scoring signal, and this signal is what makes this work. Skip the real signal and best-first buys nothing over plain BFS; it's just BFS with extra steps.

Adaptive stopping pushes further still. Crawl4AI's AdaptiveCrawler.digest() function halts the crawl on its own once it decides enough relevant content has been gathered. That's a real shift in behavior: the agent stops when it has what it needs instead of crawling until a page cap or depth limit forces the issue, and it quits spending budget on pages that add nothing new.

The token and noise cost of miscalibrated scope

Scope mistakes don't announce themselves, and that's what makes them dangerous. An under-scoped crawl quietly misses the two pages, three hops deep, that actually answered the question. The agent produces an answer anyway. That answer comes out wrong or incomplete, and nothing downstream flags it, because nothing downstream was built to check.

Over-scoped crawls fail in the opposite direction, and the cost lands mostly in tokens. Every extra page fetched drags nav bars, footers, cookie banners, and related-post carousels along with whatever real content sits on the page. Crawl4AI's fit_markdown feature strips a lot of that boilerplate before it reaches the LLM, and that helps, but it's a downstream fix. It cleans up a mess after the mess already got made, and does nothing to stop the agent from fetching the wrong pages to begin with.

The damage compounds hardest at the handoff point. A research agent hands its retrieved chunks to a synthesis agent, and that handoff carries no built-in error correction. If the source material was off-topic, the synthesis agent has no way to know. It synthesizes whatever it received, confidently, because confidence never required accuracy.

Staleness is a related but separate failure. A perfectly scoped crawl, hitting exactly the right domain and depth, against an index that hasn't been refreshed in months, still produces wrong answers. Coverage and freshness are two different problems, and fixing one does nothing for the other.

Scope boundaries for different research agent tasks

Different research tasks call for different scope defaults. Treating them all the same causes a pipeline to go quietly wrong in ways that are hard to trace back to the setting that caused it.

Domain survey work, competitor analysis or company research, wants a single domain, BFS traversal, and moderate depth, somewhere around 2 to 3 hops. Page cap should scale to how much content the site actually has: a five-page marketing site and a documentation portal running into the thousands have no business sharing a cap.

Documentation ingestion for RAG pipelines usually wants a single domain or subdomain, BFS or best-first depending on how large the docs are, and often deeper crawls than a survey task needs. URL path filtering keeps changelogs and legal pages, the stuff that adds nothing to a knowledge base, from riding along into the crawl next to the documentation most doc sites mix them with.

Focused topic research across multiple sources is the one case where external links get turned on, and even then only carefully: best-first traversal with a real relevance scorer, and a tight page cap so the agent doesn't wander off into the weeds.

Live-query grounding, where an agent answers a user's question in real time, wants shallow depth and a tight page cap almost by necessity. There's no time budget for a deep crawl mid-conversation. Freshness TTL decides whether to re-fetch or just reuse what's cached.

URL pattern filtering, by regex or path prefix, cuts across all of these use cases. It lets an engineer treat different sections of the same domain as separate scopes, so a docs subfolder gets deep coverage while a blog subfolder gets skipped.

Confidence produces this heuristic: high confidence about where the relevant content lives means a narrower, deeper crawl. High confidence about where the relevant content lives means a narrower, deeper crawl. Low confidence means a broad BFS sweep first, then a second, targeted pass once the terrain is clearer. If a task is answerable from top-nav pages, depth 1 or 2 is plenty, and going any deeper should answer a specific need rather than serve as a default nobody bothered to question.

Structured extraction as a scope enforcement mechanism inside retrieved pages

Scope doesn't stop mattering once the crawler stops moving. A retrieved page still carries nav menus, footers, ad units, comment threads, and related-articles carousels sitting right next to the content an agent actually wants.

Schema-based extraction handles this at the page level, and it beats selector-based scraping on durability alone. Define the fields once, and extraction finds them by meaning rather than HTML position. It survives a layout redesign that would break a CSS selector overnight. Research on a system called AXE (Adaptive X-Path Extractor) showed a 0.6B parameter model hitting an F1 score of 88.1% on extraction tasks when paired with DOM pruning that cut input tokens by 97.9%. The small model did well because the input it got was already scoped down to what mattered. Efficient extraction doesn't need a bigger model, it needs less garbage fed into the one it already has; the extraction task's F1 score of 88.1% with DOM pruning cutting input tokens by 97.9% shows this directly.

Two practical paths exist here, and they solve different problems rather than competing to solve the same one. CSS and XPath selectors work well for predictable, repeated HTML structure: fast, deterministic, zero LLM cost. LLM-based extraction, through a JSON schema or a plain-language prompt, fits pages that vary in layout or get redesigned without warning. Choosing between them comes down to one question: how stable is this page structure actually going to stay?

Scope decisions in multi-agent and orchestrated research pipelines

Multi-agent pipelines split a research task across specialized agents: one discovers sources, one fetches and extracts, one synthesizes. Every handoff between them is a scope boundary, whether anyone designed it that way or not.

Three orchestration ecosystems dominate going into 2026, and they coordinate agents in different ways. LangGraph offers stateful graphs with retries and resumable checkpoints. AutoGen 0.4's AgentChat is built around coordinated multi-agent setups. CrewAI structures things around defined agent roles and handoffs.

Whichever framework is in play, scope has to travel as a shared contract between agents, not something each one figures out for itself on the fly. If the discovery agent settles on a domain and depth constraint, that constraint has to ride along in the message passed to the fetch agent, not get silently re-derived, or dropped, at the next step. Lose that contract between steps and a tightly scoped discovery phase turns into an unscoped, runaway fetch phase two steps later, without anyone changing a setting on purpose.

Try a structured API first. Fall back to page-level scraping only if no API route exists, and reach for cached chunks when the freshness TTL still allows it. Scope decisions have to apply at every tier an agent tries, not just the first one.

Freshness as the temporal dimension of crawl scope

Domain and depth define where a crawl looks. The last-crawl date defines when it looked. Both have to be right, because a spatially perfect crawl built on stale data still produces answers that sound confident and are quietly wrong.

Staleness hides by design. An agent's index can sit three months out of date without anyone knowing it, and the agent just answers using what it has, in the same confident tone it would use if the data had been fetched five minutes ago.

Two distinct staleness problems live here, and they need different fixes. A coverage gap happens when pages published after the last crawl simply don't exist in the index yet: no retrieval strategy pulls in something that was never fetched. Drift is a different animal. The indexed pages are still there, but the live content underneath them has changed, while the stored embeddings still point at the old text. Re-crawling on a tighter schedule narrows that window. It never closes it completely.

The real fix is a per-topic freshness TTL, not one blanket setting applied across the board. News sites, pricing pages, and regulatory content change constantly and need short TTLs, sometimes measured in hours. Stable documentation can go far longer between re-crawls without losing meaningful accuracy.

Choosing a crawling API that exposes the scope controls research agents need

A crawling API built for research agents needs to expose depth limit, page cap, a domain-scope toggle, URL pattern filtering, and clean output (Markdown or schema-driven JSON) that doesn't need a second cleanup pass before it's usable. Pricing needs to be predictable per page, and failed or blocked fetches shouldn't raise the bill. Paying for pages that never returned content is a bad deal on its face, and any vendor billing that way is optimizing for their revenue, not the quality of anyone's research.

Crawl4AI, open-source under Apache 2.0, is a concrete example of a tool that puts most of these controls directly in an engineer's hands rather than hiding them behind a black-box "crawl this site" button. Its BFSDeepCrawlStrategy, BestFirstCrawlingStrategy, and adaptive crawling strategy all leave scope and depth decisions where they belong: with the person who actually understands the task. Whatever API a given pipeline ends up using, the test stays the same. Can scope be set with precision, adjusted per task, reasoned about like the research decision it actually is? Or is it a fixed setting some vendor picked months ago, buried where nobody thinks to look? Build on the tools that pass that test, and treat the ones that fail it as a liability; that liability appears in production.