Agent Concurrency and Scheduling

Getting a single Agent running isn't hard these days. Any LangGraph or OpenAI Agents SDK demo, run locally once, with tools working, looks like it's "ready to ship."

But between demo and service lies a mountain: concurrency.

A single Agent is a long task—often tens of seconds, tens of steps, holding a model connection, holding tools, holding the in-memory KV cache and session state. The moment 100 users arrive at once, the model API's RPM/TPM caps out, the database connection pool fills up, the queue jams, and a paying user's request sits behind a free user writing junk prompts... these are problems a demo never encounters but that will cost you in production every day.

This article explains clearly: how an Agent service actually bears traffic.

1. First see why Agent concurrency is hard

Ordinary web requests are stateless, millisecond-level, one-shot. Agents are the exact opposite:

• Long lifecycle: a single conversation may run 30 seconds to 5 minutes, with multiple rounds of model calls + multiple tool calls in between.
• Stateful: the Agent holds conversation memory, tool handles, an intermediate plan; it cannot be "use-and-discard" like HTTP.
• Resource-heavy: it holds a concurrent channel to the model API, it holds tool connections (database, browser, sandbox), it holds GPU/KV cache (when self-hosting models).
• Non-interruptible intermediate state: a tool call killed halfway may leave dirty data and half-completed side effects.

So "concurrency" in an Agent service is essentially competing for resources across multiple dimensions at the same time: model API concurrency quota, tool-side concurrency quota, process/memory. Any one dimension maxes out and the service collapses.

A brutal fact: what you bought is 1000 RPM on the model API, not 1000 concurrent Agents. One Agent in one task may hit the model API 8 times, so your real concurrent Agent count is roughly RPM / average call rounds / user peak multiplier. Once you do this math, all subsequent design has a target.

2. Concurrency models: per-request Agent vs Agent pool

Two mainstream models, each with its own pitfalls.

Model A: create an Agent per request (Per-request Agent)

Each request does new Agent(), destroys it when done. Simplest, least error-prone, naturally isolated state.

async def handle_turn(session_id: str, user_msg: str):
    agent = Agent(
        llm=build_llm(),
        tools=build_tools(),
        memory=await load_memory(session_id),
    )
    try:
        return await agent.run(user_msg)
    finally:
        await agent.close()  # release tool connections, flush memory

Applicable to: short sessions, light tools, no heavy warm-up cost scenarios (customer service, Q&A, single-turn RAG). The downside is having to rebuild tool connections and reload system prompts each time, with high cold-start cost.

Model B: pre-warmed Agent pool (Agent Pool)

Warm up a batch of long-lived Agents in advance; when a request arrives, borrow one from the pool and return it when done. Suited to scenarios with heavy tools (browser, sandbox, long database connections) and long system prompts (tens of thousands of tokens to rebuild once).

class AgentPool {
  private idle: AsyncQueue<Agent> = new AsyncQueue();
  private inUse = new Set<Agent>();

  constructor(size: number) {
    for (let i = 0; i < size; i++) {
      this.idle.push(warmAgent()); // warm up: load tools, establish long connections
    }
  }

  async acquire(userId: string, timeout = 30_000): Promise<Agent> {
    const agent = await this.idle.pop(timeout); // full = queue / reject
    if (!agent) throw new PoolExhaustedError();
    this.inUse.add(agent);
    return agent.bindContext(userId); // key: bind user context each time
  }

  release(agent: Agent) {
    agent.resetContext(); // key: state must be cleared after use
    this.inUse.delete(agent);
    this.idle.push(agent);
  }
}

The biggest pitfall of the pooled model is state bleed. One Agent just finished answering "my order" for one user; the next user takes over, and if resetContext isn't clean, the next user sees the previous user's order. This is the single, and most fatal, bug source for Agent pools. My advice: whenever team size allows, start with the Per-request model; switch to pooling only when a cold-start bottleneck actually appears (visible in monitoring), and pooling must be paired with comprehensive reset tests.

How do you size the pool? Take min(model API concurrency quota, tool concurrency quota, memory budget) then apply a 0.8 safety factor. It's not a guess.

3. Resource management: three independent faucets

An Agent service's resources should be managed as three independent faucets; any mixing will spiral out of control.

Faucet 1: model API concurrency and rate

Model vendors give you tiered limits: RPM (requests/min), TPM (tokens/min), concurrent connections. All three can become bottlenecks, and different models / different tiers have different quotas. A common mistake: you call both GPT-4o and Claude, think "that's 2000 RPM combined," and Claude alone blows up.

Engineering-wise, you must have a global, per-provider-independent token bucket + concurrency semaphore:

const providerLimits = {
  openai: { rpm: 1000, tpm: 2_000_000, maxConcurrent: 50 },
  anthropic: { rpm: 500, tpm: 800_000, maxConcurrent: 30 },
};

// one semaphore per provider
const llmSem: Record<string, Semaphore> = Object.fromEntries(
  Object.entries(providerLimits).map(([k, v]) => [k, new Semaphore(v.maxConcurrent)])
);

Faucet 2: tool concurrency

Tool resources are even easier to overlook than the model. Your Agent calls the database—the connection pool is 20; it calls browser automation—the sandbox instance cap is 10; it calls a third-party API—the other side limits you to 5 QPS. These must have a separate concurrency limit per tool and cannot share the semaphore with the model.

const toolLimits = {
  postgres: new Semaphore(20),
  browser_sandbox: new Semaphore(10),
  search_api: new RateLimiter({ qps: 5 }),
};

async function callTool(name: keyof typeof toolLimits, input: unknown) {
  const guard = name === 'search_api'
    ? await (toolLimits[name] as RateLimiter).acquire()
    : await (toolLimits[name] as Semaphore).acquire();
  try {
    return await tools[name].run(input);
  } finally {
    guard.release();
  }
}

Faucet 3: memory and KV cache

When self-hosting models, KV cache is a scarce resource—a single long session easily eats hundreds of MB of VRAM. The Agent service must either actively truncate/compress history when the prompt gets too long, or reuse common parts with a prefix cache. Hosted API users shouldn't be careless either: the longer the session, the more latency and cost rise linearly, so you need context window budget management.

4. Scheduling: priority, queues, rate limiting, reclamation

There are only so many resources—who goes first? That's scheduling.

Priority: paying users before free, interactive before background batch. Priority isn't lip service—it must be implemented with a priority queue:

const queue = new PriorityQueue<Job>((a, b) => a.priority - b.priority);
// priority: paid realtime (0) > free realtime (10) > background tasks (100)

Queue peak-shaving: the model API's quota is rigid, but user requests are spiky. The queue's job is to spread the spikes over time, not to pile up mindlessly. The queue length must have an upper limit—exceeding it means rejecting immediately (return 429 + Retry-After), not letting the user wait 10 minutes only to time out. Making users wait = making users curse; letting users know right away "busy now, try later" = still salvageable.

Rate limiting (anti-abuse): a per-user_id concurrency cap + QPS cap, preventing one user (or one crawler) from eating the whole pool.

const perUserConcurrent = new Map<string, Semaphore>();
function getUserGate(userId: string, max = 2): Semaphore {
  if (!perUserConcurrent.has(userId)) {
    perUserConcurrent.set(userId, new Semaphore(max));
  }
  return perUserConcurrent.get(userId)!;
}

Timeout and reclamation: this is the last gate of Agent service stability. Each Agent must have a hard timeout (e.g., 5 minutes); on timeout it must be forcibly cancelled—not just cancelling the LLM call, but also rolling back/marking the tool side effects it was executing. Otherwise a stuck Agent keeps hogging the pool until it bleeds you dry.

async def run_with_reclaim(agent, task, hard_timeout=300):
    try:
        return await asyncio.wait_for(agent.run(task), timeout=hard_timeout)
    except asyncio.TimeoutError:
        await agent.cancel()          # interrupt the LLM stream
        await agent.mark_orphaned()   # mark possibly unfinished tool side effects
        raise ServiceUnavailable()

5. Multi-user state isolation: no bleeding, or it's an incident

This is where Agent services most easily produce P0 incidents, and it deserves its own section.

Memory, conversation history, cache—everything is strictly partitioned by user_id + session_id. In the database this means enforced composite primary keys; in Redis this means key prefixes; in an in-memory Map this means never reading across users.

# correct: all memory queries are forced to carry user_id
async def load_memory(user_id: str, session_id: str) -> Memory:
    return await db.fetch_one(
        "SELECT * FROM memories WHERE user_id=$1 AND session_id=$2",
        [user_id, session_id],
    )

# wrong (will bleed): query by session_id only
#   if two users coincidentally or adversarially construct the same session_id, it bleeds

A few practical iron rules:

1. The key is always (user_id, session_id), not session_id alone. session_id can be forged and can collide; trusting only user_id won't do; user_id + session_id is the security boundary.
2. When a pooled Agent switches users, all context variables, connection handles inside tools, and caches must be fully reset. Cultivate the habit that "reset is a first-class method," rather than scattered assignments.
3. Tool returns must not contain other users' data—for example, a list_orders tool's query must forcibly inject WHERE user_id = current_user; you cannot trust the user_id the LLM passes up.
4. Logs and traces must not contain someone else's user_id either—isolation awareness must extend to the observability layer.

Treat state bleed as a P0 to design for, not a P0 to fix.

6. Long tasks: don't let the HTTP connection wait with you until the end of time

Agent tasks often exceed 30 seconds, while HTTP/gateway timeouts are often 30 seconds. Making the user hold the connection waiting for a result is an anti-pattern—the connection gets cut by the gateway, the experience is terrible, and you can't provide progress feedback.

The correct architecture is: request goes straight to a queue + background Agent execution + status query + progress push.

Client ──POST /jobs──> [priority queue] ──> [Worker: Agent]
           │                                   │
           └──returns job_id                   ├──writes status to Redis/DB
                                               │
Client <───GET /jobs/{id} (polling)            │
Client <───SSE/WebSocket (push progress) <─────┘

// enqueue
app.post('/jobs', auth, async (req, res) => {
  const job = await queue.enqueue({
    userId: req.user.id,
    prompt: req.body.prompt,
    priority: req.user.tier === 'paid' ? 0 : 10,
  });
  res.status(202).json({ jobId: job.id, status: 'queued' });
});

// status query
app.get('/jobs/:id', auth, async (req, res) => {
  const job = await jobs.get(req.params.id, req.user.id); // force user_id
  res.json({ status: job.status, progress: job.progress, result: job.result });
});

// progress push
app.get('/jobs/:id/stream', auth, (req, res) => {
  res.setHeader('Content-Type', 'text/event-stream');
  jobs.subscribe(req.params.id, req.user.id, (evt) => {
    res.write(`data: ${JSON.stringify(evt)}\n\n`);
  });
});

The background Worker is a long-running process that takes jobs from the queue, spins up an Agent, and writes intermediate events from agent.run() (step / tool_call / token) back to Redis; the frontend polls or uses SSE to fetch them. This architecture gives you peak-shaving, observability, and recoverability (if the worker dies, the job is still in the queue) all at once.

7. Observability: concurrency, queue time, failure rate

An Agent service in production without observability is running naked. You must at least see these metric groups:

• Concurrency metrics: current active Agent count, in-flight LLM requests per provider, in-flight calls per tool, queue depth. These are the basis for capacity planning.
• Latency breakdown: queue time (from enqueue to Agent execution start), Agent execution time, model call P95, tool call P95. Queue time is the number-one killer of user experience—it gets its own line, don't average it into total latency.
• Failures and reclamation: Agent failure rate (by phase: LLM failure / tool failure / timeout), hard-timeout trigger count, 429 rejection rate, pool exhaustion count.
• Cost: token consumption per task, distribution of call rounds, aggregated by user/tier. This directly relates to whether the bill holds up.

Each Agent task must have a trace ID, stringing together LLM calls, tool calls, and queue waits. When something goes wrong, you can see on one screen exactly which step an Agent is stuck on, how long it waited, and what tools it called. Tools like Langfuse, OpenTelemetry, and Arize can save you a lot of debugging time.

An underrated practice: tag each Agent task (plan rounds, tool call count, final success/failure); over time you'll see which kinds of prompts burn the most money and which are most likely to time out—this is ammunition for product iteration.

Summary

1. Agents are long tasks, resource-heavy, stateful—they are not HTTP requests; you can't brute-force them with the web-style "handle one as one comes" approach. You must manage resources independently across three dimensions: model API, tools, and memory.
2. Pick the concurrency model Per-request first, pooling later. The only fatal problem with pooling is state bleed; only adopt it when cold start genuinely becomes a bottleneck, and it must come with full reset tests.
3. The scheduling four-pack: priority, queue, rate limiting, reclamation. The queue must have an upper limit, the Agent must have a hard timeout, and the user must have a concurrency cap—any one of these three gates missing will cause a production incident.
4. State isolation strictly partitions by (user_id, session_id), running through database keys, tool SQL, and log tags. Bleed is P0; plug it at the design stage.
5. Long tasks go "enqueue + background + push"; don't let the HTTP connection wait with the Agent. This architecture gives you peak-shaving and recoverability as a bonus.

Punchline: Getting a single Agent running is a demo; getting a set of Agents scheduled together is a service. Only when you've nailed resources, scheduling, isolation, and observability does your Agent earn the right to talk about "bearing traffic."