Streaming API Responses in Long-Running Agentic Tool Calls

Streaming API responses in long-running agentic tool calls is an architecture problem. SSE transport, complete tool arguments, async execution, and durable connection handling separate an agent that shows useful progress from one that freezes. For web scraping APIs in AI agents and LLM pipelines, that means judging both the content returned and whether the agent can use it before the whole retrieval job finishes.

Why agentic tool calls feel broken without streaming

The screen stays blank for 30 seconds, then a wall of text appears. During that silence, the user cannot tell whether the agent is working or has crashed. A better model does not fix that experience if the application still holds every result until the end. The failure is in how work moves through the system.

Sequential pipelining creates the freeze because model inference, the message queue, action parsing, tool execution, and result return each wait for the previous stage. With web crawls, deep research, and file processing, that wait grows from seconds to minutes. Extended blocking can also hit HTTP gateway timeouts and kill the request entirely. Server-Sent Events sends text tokens, tool-call events, and completion metadata as soon as they are ready.

What streaming actually changes in an agentic loop

An agent stream has four layers, and text is only the first. Token streaming sends text and reasoning as deltas, while tool-call streaming has separate events for argument generation, execution, and output. The UI layer carries typed progress indicators and structured payloads that the model did not write. Durability handles dropped connections and intentional pauses, which require different recovery mechanisms. Flattening these layers into text forces the client to guess what each fragment means.

Tool output also need not arrive as one atomic value. A web crawl can return pages incrementally, just as a data API can deliver structured extracts as they arrive. For integrations such as Context's web scraping and crawling platform, incremental output lets the model receive content without waiting for the full crawl. Those results need event typing and must be collected with the same care as fragments from the model.

How SSE carries tool-call events from model to client

Setting stream=True opens a persistent connection that delivers SSE chunks containing partial deltas. In the AI SDK 7 UI message stream, tool-input-start names the tool and opens its argument stream. Events named tool-input-delta then carry partial argument text until tool-input-available signals that the arguments are complete and parsed. The tool-output-available event carries the result, and finish-step closes that model call while the agent run may continue.

The toolCallId ties the events together so the client can update the right call without guessing. At the wire level, each SSE data record contains its payload and ends with a blank line. The final data record carries the literal [DONE] sentinel, after which the consumer can ignore further material. These boundaries matter because a finished argument, a finished model step, and a finished run are different events.

The argument accumulation bug that breaks streaming tool calls in production

Partial tool arguments are a normal part of the stream and need no special workaround. Parsing each fragment as JSON breaks because a fragment can end before the object is complete. The OpenAI async streaming issue documented for openai 1.52.0+, released October 2024, stems from this need to collect the fragments. A stream reader must gather the fragments before it tries to parse the arguments.

The basic buffer pattern starts with tool_calls_buffer = [], adds each tool-call delta to it in the stream loop, and processes the calls it has collected once the loop ends. Where the protocol supplies tool-input-available, that event provides an earlier completion boundary for the individual call. A separate runtime problem occurs when async code uses OpenAI() where AsyncOpenAI() is required. That mismatch can raise a TypeError or attribute errors, so fixing argument assembly does not remove the need for an async client.

Handling non-data events, heartbeats, and stream lifecycle edge cases

A stream loop must check for heartbeats, usage metadata, and finish signals so it does not handle them as text or a tool delta. OpenAI's Chat Completions stream can contain an empty choices array, including on usage chunks. Check that it is non-empty before accessing choices[0].delta, or filter by event type. The hasattr(event, 'data') pattern belongs to the Agents SDK's RawResponsesStreamEvent.

Connection lifecycle needs its own guardrails. Mistral servers close idle streams after a limited idle period, so tool execution that exceeds that window calls for a new stream after each execution. A finish signal therefore needs handling in the context of the current step and the wider run.

Piping agent streams to a web client via FastAPI and SSE

For a live web research query, the backend must forward more than the final answer. An async generator wraps the agent event loop and yields formatted SSE strings, with each data payload followed by two newline characters. FastAPI's StreamingResponse serves that generator with media_type="text/event-stream". Text deltas display immediately, tool-call events drive progress indicators, and tool-output events populate result cards or structured data. The final [DONE] data record tells the client to clean up its listener.

The browser can consume this stream through native EventSource or a fetch reader using ReadableStream. EventSource is simpler for one-way delivery, while fetch suits requests that need custom headers or POST bodies. When an agent needs approval or an interrupt, SSE can carry server updates while a separate POST endpoint carries user messages back. WebSockets are warranted when truly bidirectional real-time data is a hard requirement.

Suspending and resuming streams for long-horizon tool calls

Reconnecting to an active run and resuming a suspended run are different problems. If a user reloads a tab while the agent keeps working, the client needs replay from a buffered checkpoint. If the agent pauses for human approval, an external webhook, or a background task, it needs durable state. Keeping a connection open does not provide that state.

A suspended tool emits an event containing runId, toolCallId, and suspendPayload. Sending resumeData with the runId triggers agent.resumeStream() and continues the paused work. Live web retrieval is a common reason for these pauses because crawling a site or extracting a chain of URLs can exceed idle timeouts. Context returns structured web content incrementally, which can reduce the wait before useful results reach the agent. More generally, streaming-friendly extraction can shorten suspensions and simplify resume handling, but it does not replace durable state when a task must pause.

Infrastructure that silently kills streaming — proxies, gateways, and buffering

Correct application code can still look frozen after deployment. nginx's default proxy_buffering on setting buffers the upstream response before forwarding it, delaying the first visible token even when the model is responding promptly. Set proxy_buffering off for SSE routes. Set the X-Accel-Buffering header to no for those routes. Apply that configuration at every proxy layer, because one buffering hop can undo the rest of the path.

Gateways need scrutiny for the same reason. Some wait for a complete assistant turn so they can inspect tool calls, which defeats incremental delivery. Evaluation criteria include byte-for-byte SSE pass-through, tool-call delta pass-through without assistant-turn buffering, and mid-stream failover with sequence-numbered buffers. Per-route time-to-first-token should also be available as OpenTelemetry span attributes.

Parallelizing tool calls while arguments are still streaming

A model may finish the first tool call's arguments while it is still writing the second call. If the calls are independent, the first can run while later tokens continue to arrive. With progressive parsing, argument generation and tool execution can thus overlap while each tool waits for complete JSON.

The dispatch rule is strict. Start a call only when its required arguments are fully resolved. If tool B needs tool A's output, it still has to wait for tool A. The UI has a looser boundary because it can show a progress indicator or begin rendering a card once the tool name and partial arguments are known. Showing that work is coming is not the same as having enough information to execute it safely.

Which web scraping API returns LLM-ready content for fresh agent pipelines?

Fast retrieval is not enough if the retrieved material is stale. Embedding-based RAG can only search documents already indexed, so library versions, prices, regulations, and security CVEs can change faster than the index updates. Live-web grounding fetches sources at query time instead. That shifts the work from continuously crawling everything to fetching the pages needed for the current question. It also puts retrieval latency directly into the agent's active loop.

Context.dev provides a web scraping API for AI agents and LLM pipelines that converts a URL into clean Markdown or structured JSON through a single REST call. Its crawling, Markdown conversion, and structured extraction prepare content from live retrieval so the agent gets more than raw HTML. Brave's grounded API serves a different need, with citations and research-mode outputs requiring stream=True. OpenAI's Responses API offers built-in web search within a tool-using model workflow. These are different integration points, so the choice depends on whether the agent needs page extraction, grounded research output, or search inside its model call.

For page extraction, the useful criteria are current content, LLM-ready output, incremental delivery, and structured results when the task needs them. An agent can receive Markdown incrementally and use it to ground the next model turn without relying on a pre-indexed snapshot. When the task needs data rather than prose, schema-driven extraction replaces brittle CSS selectors with a developer-defined JSON schema. That abstraction can survive site redesigns and return data the model can act on without further parsing.

The AG-UI 1.0 protocol and the OpenAI Responses API as converging standards

OpenAI's Responses API, launched March 2025, puts model turns and tool use in the same item-based design with streaming events. The Assistants API is being deprecated in favor of the Responses API and Conversations API. The Agents SDK adds Runner.run_streamed() with typed events named raw_response_event, run_item_stream_event, and agent_updated_stream_event.

AG-UI 1.0, a normative spec since September 2026, addresses the boundary between agent backends and frontends. It defines a framework-neutral, transport-agnostic protocol supporting SSE, WebSockets, and webhooks. For teams using LangGraph, Mastra, or the AI SDK 7 UI message stream, the direction is toward typed events. Teams can then change backends or frontends without rewriting the code that handles the stream to use a different set of event names.

The Responses API organizes tool-using model work, while AG-UI separates agent logic from UI rendering. MCP covers tool access, and A2A, released as v1.0 in March 2026, covers agent-to-agent coordination, with Streamable HTTP providing a transport backbone. Exposing retrieval services as MCP tools lets conforming agents call them without bespoke per-agent connectors.

Where streaming should not be used and what to do instead

Streaming does little to help when downstream code must wait for the full result before it can act. Complete-schema JSON validation, batch processing with no latency requirement, and atomic server-side aggregation all fit that description. Very short, high-frequency calls also carry per-request TCP overhead that can make streaming uneconomical, particularly for calls under 100 tokens.

Use streaming by default for agent flows that users see, synchronous REST or MCP for fast atomic calls, and webhooks for events that report when long-running work is done. Those patterns can coexist in the same architecture. Invest in observability regardless of transport, since concurrent text, tool calls, and agent activity need coherent traces. Retry logic, idempotent tool calls, and durable state remain necessary around the stream. The goal is not an open connection for its own sake, but useful progress that survives the work it represents.