LLM Batch Inference and Offline Jobs

Most teams treat LLMs as synchronous "request-response" services: the user clicks, the front end makes a call, waits for the model to finish generating, and returns the result. That works well for interactive scenarios such as chat, search, and copilots. But once the business starts producing jobs like "classify 100,000 support tickets," "generate summaries for 10,000 documents," or "tag all content added today every night," synchronous real-time calls become the most expensive and least efficient solution. You clearly have a window of several hours, yet you are paying for "return immediately" priority.

LLM inference effectively has two pricing tiers: real-time is expensive because you pay a premium for low latency and elastic compute, while offline is cheaper because providers can run your requests gradually on spare capacity. The essence of batching is trading latency for a lower price: accumulate requests whose results are not needed immediately, submit them asynchronously, and retrieve the results a few hours later at half the price. Not every task should use batching, but a substantial share of tasks use real-time inference when they should be batched.

1. Real Time vs. Batch: How Much Are You Paying for "Immediately"?

Real-time inference is expensive not because providers are greedy, but because its cost structure demands it. Real-time traffic requires low latency and elastic peak capacity. When your request arrives, a GPU must be ready to accept it immediately, with enough capacity to scale during traffic peaks. Providers therefore have to maintain a pool of compute that can never operate at full utilization, and the cost of that idle capacity is ultimately built into real-time pricing.

Batch requests have exactly the opposite characteristics: a delay of several hours is acceptable, and requests can wait in a queue. Providers can schedule them during compute troughs and use fragmented GPU time unavailable to real-time requests, maximizing GPU utilization. Providers pass those savings back as a discount: the OpenAI and Anthropic Batch APIs are both priced at 50% off list price. This is not a promotion; it is economics.

A rough cost curve:

延迟容忍度          推荐路径              相对成本
─────────────────────────────────────────────────
< 2 秒(交互)       实时 + 流式           1.0x
几秒~几分钟        实时(可缓存)         0.3~0.5x(命中缓存)
几十分钟~几小时    Batch API             0.5x
几个小时~隔夜      Batch + prompt cache  0.1~0.2x

The key question is not "How difficult is this task?" but "Is a user waiting for it?" If a user is waiting in the front end, it has to run in real time regardless of cost. Using real-time inference for unattended background processing simply burns money.

2. Five Scenarios Suited to Batching

Batching is not universal, but it can reliably handle these types of high-volume, non-interactive, rerunnable work:

1. Data classification and labeling. Route customer support tickets by category, assign risk levels to content, or classify the sentiment of user reviews. These tasks can easily involve hundreds of thousands of records while producing only a few tokens each. With real-time APIs, nearly all the cost comes from input tokens; batching cuts that cost in half.
2. Document summarization and information extraction. Extract key fields from contracts, generate summaries of long reports, or extract structured attributes from product pages. Long inputs, short outputs, and low latency sensitivity make these tasks natural candidates for batching.
3. Bulk content generation. Draft SEO landing-page copy, product descriptions, or email marketing content in bulk. Run the job overnight and have people review it the next day; trading latency for a lower price is highly cost-effective.
4. Evaluation runs. During development, compare models across thousands of examples in an eval set that users will never see. Running evaluations through a Batch API halves the cost and lets you compare multiple models or prompt versions in one run.
5. LLM steps in ETL pipelines. This is the stage in a data pipeline that uses an LLM to clean, enrich, or transform data, such as converting unstructured resumes into structured JSON for storage. It already runs in an offline workflow, so there is no reason to use real-time inference.

The common traits are clear: latency is tolerable, volume is high, failures can be rerun, and each item produces an independent result. The counterexamples are equally clear: multi-turn conversations, agents that call tools, and results users are actively waiting for. Keep those in real time instead of forcing them into batches.

3. Provider Batch APIs: Submit JSONL → Process Asynchronously → Retrieve Results, Usually at 50% Off

The OpenAI and Anthropic Batch APIs have highly similar designs: write a collection of Messages requests as JSONL, with one independent request per line and a custom custom_id; submit it and receive a batch ID; poll the status over the next several hours; then retrieve each result individually. Both offer a 50% discount.

Take Anthropic Message Batches as an example. Keep these constraints in mind first: each batch supports up to 100,000 requests or 256MB; most batches finish within one hour, with a maximum of 24 hours; results are retained for 29 days; and pricing is 50% off.

import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();

// 1. 提交:requests 里每个请求带 custom_id + 完整的 messages 参数
const batch = await client.messages.batches.create({
  requests: tickets.map((t, i) => ({
    custom_id: `ticket-${t.id}`,            // 结果按它对回原数据,别按顺序
    params: {
      model: "claude-haiku-4-5",
      max_tokens: 64,
      messages: [{ role: "user", content: `分类以下工单的类目:\n${t.text}` }],
    },
  })),
});

// 2. 轮询:processing_status 从 in_progress → ended
while (true) {
  const b = await client.messages.batches.retrieve(batch.id);
  if (b.processing_status === "ended") break;
  await new Promise(r => setTimeout(r, 60_000));
}

// 3. 取结果:顺序不保证!必须用 custom_id 做 key
const results = new Map<string, string>();
for await (const r of client.messages.batches.results(batch.id)) {
  if (r.result.type === "succeeded") {
    results.set(r.custom_id, r.result.message.content[0].text ?? "");
  } else if (r.result.type === "errored") {
    // 部分失败是常态,见下一节的管线设计
  }
}

OpenAI's /v1/batches endpoint is almost identical. The only difference is that you first upload a JSONL file, then use its file_id to create the batch. Both providers let you stack prompt caching on top of batch pricing: for stable prefixes such as system prompts and few-shot examples, cache hits can save even more beyond the batch discount, theoretically reducing the combined cost to 10%–20% of the original price. Another frequently overlooked point is that batch rate limits are much more generous than real-time API limits. Large workloads blocked by real-time TPM limits can often complete in a single batch.

4. Self-Hosted Batching: vLLM Continuous Batching for High Throughput

Calling proprietary Batch APIs is the easiest way to reduce costs, but two scenarios call for self-hosting: data compliance, when data cannot leave your VPC, and the lowest possible cost, when running open-source models yourself at massive scale is cheaper than any API. In those cases, you do not need a particular Batch endpoint; you need a high-throughput inference engine. vLLM is the first choice.

The key to vLLM's high throughput is continuous batching, also called in-flight batching or iteration-level scheduling. Traditional batch processing has to wait until a batch is full before it runs, adding delay while requests accumulate and wasting GPU resources when requests in the same batch have different lengths. Continuous batching lowers scheduling granularity to each forward pass: new requests can join a running batch at any time, and completed requests immediately vacate their slots for the next queued items. The GPU remains saturated, PagedAttention controls memory fragmentation, and VRAM utilization stays high.

传统 static batching:    [等凑满 8 条] → [8 条一起跑完] → [等凑满下一批]
                         空等 + 短请求被长请求拖累

vLLM continuous batching: [3 条在跑] → 第 4 条进 → 第 1 条完,立刻塞第 5 条 → ...
                          GPU 不空转,吞吐数倍于静态批

Only a few parameters are essential when starting a vLLM service for batch workloads:

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --max-num-seqs 256          # 并发请求数,显存允许就调大,吞吐基本线性涨
  --gpu-memory-utilization 0.9
  --enable-prefix-caching     # 复用稳定前缀,自部署版的 prompt caching

Then send concurrent requests directly to the OpenAI-compatible /v1/chat/completions endpoint. vLLM applies continuous batching automatically on the server; the client simply issues requests concurrently. The cost of self-hosting is managing GPUs, VRAM, and model weight upgrades. The benefits are that data stays within your environment and marginal costs approach electricity plus depreciation. For sensitive, high-volume data workloads that can accept open-source model quality, self-hosted batching is the preferred alternative to proprietary Batch APIs.

5. Build Batching as a Job Pipeline, Not a One-Off API Call

Beginners treat Batch as a one-off function call. Experienced teams operate it as an offline job pipeline. A production-grade batch pipeline needs at least these five stages:

组装 JSONL → 提交 batch → 轮询状态 → 下载结果 → 失败重试 + 成本统计
   │            │           │          │              │
   持久化       记录        指数退避    按 custom_id   单条降级实时
   原始请求     batch_id    不阻塞      对回原数据     重算 + 核账

Several practical points matter:

• Persist the original requests. Before submission, write every request, including custom_id and its payload, to a database or object store. Batch results can only be matched to the source data through custom_id; if the original requests are lost, you can never join them back together.
• Do not block the main workflow while polling. Batching is asynchronous, so the submission process should not wait synchronously. Use cron or a job queue to call retrieve periodically, and advance the workflow when the status changes.
• Retry individual failures instead of rerunning the whole batch. A batch will always contain a few requests that fail because a prompt is too long, content triggers a safety policy, or for similar reasons. Extract those errored entries and either revise the prompt and resubmit them as a batch or fall back to the real-time API. A few failures should not prevent the rest of the batch from being delivered.
• Reconcile costs immediately after downloading the results. The usage data in the result file contains token consumption. Calculate the cost using the "half of real-time pricing" rate and send it to the cost-attribution system. The batch discount must be visible on the dashboard, or nobody will know how much money it saved.
• Make the pipeline idempotent. Upstream retries may submit the same data twice. Each custom_id must be traceable to a business record, and results should be stored with an upsert.

The pipeline's purpose is to capture the benefits of trading latency for price while using engineering controls to contain the uncertainty of partial failures, latency variation, and out-of-order results. Without this pipeline, batching remains a demo rather than a production capability.

6. Real Time vs. Batch: Choose by Business Need, Not Habit

The tradeoff is simple and depends on three factors:

The decision rule is: whether a user is waiting in front of the screen determines whether to use real-time or batch inference. Front-end chat, search suggestions, and Copilot completions should be real-time. Nightly ETL, offline labeling, evaluation runs, and bulk email generation should be batched. When the answer is unclear, ask: "Would the business collapse if this result arrived two hours later?" If not, batch it.

7. Advanced: Shave Peaks by Moving the "Can Wait" Portion of Real-Time Traffic to Batches

The most advanced cost optimization is not simply using batching for new tasks. It is moving the "can wait" portion of existing real-time traffic into batches, a practice known as peak shaving.

Many real-time features have hidden opportunities for delay. Consider "generate a summary after a user uploads a document." The front end can immediately return "Processing," while the back end places the summary job in a batch queue and delivers the result a few hours later through a push message or notification. The user experience loses only the immediacy of seeing the result at once, while the cost of that workflow is cut in half. Content moderation provides another example: run only a lightweight classifier in real time for hard blocks, then move in-depth review to an overnight batch.

The engineering pattern for peak shaving:

实时路径(贵):    请求 → 实时 LLM → 返回      [高峰期扛不住,TPM 爆表]
削峰路径(省):    请求 → 轻量判断
                        ├ 紧急/关键 → 实时 LLM(只放行尾部少数)
                        └ 可延迟   → 入批量队列 → 完成后回调

Real-time compute is most expensive and most likely to hit rate limits during peak periods. Moving traffic that can wait into batches avoids both the real-time premium and real-time rate limits. Peak shaving applies the batch discount to traffic that would otherwise have paid full real-time prices, producing a much higher ROI than simply adding new batch workloads. This is the key step that turns batching from an offline tool into an online cost-control lever.

Summary

1. LLM inference has two pricing tiers: expensive real-time inference and cheaper offline inference. Batching trades latency for price. The OpenAI and Anthropic Batch APIs both offer 50% off, which reflects economics rather than a promotion.
2. Five types of work should use batching by default: classification and labeling, document summarization, bulk content generation, evaluation runs, and LLM steps in ETL. The criterion is whether a user is waiting, not how difficult the task is.
3. The Batch API pattern is consistent: submit JSONL → poll → retrieve results by custom_id. Out-of-order results and partial failures are normal. Build a job pipeline around them, including persistence, retries, fallback, and cost reconciliation, rather than treating the batch as a one-off function call.
4. For data compliance or the lowest possible cost, self-host vLLM with continuous batching. Iteration-level scheduling keeps GPUs saturated and delivers several times the throughput of static batching, making it the preferred alternative to proprietary Batch APIs.
5. The advanced technique is peak shaving: move the portion of real-time traffic that can wait into batches, avoiding real-time premiums and rate limits while applying batch discounts to traffic that would otherwise pay full price.

Key takeaway: Real time charges a premium for "now"; batch gives a discount for "can wait." A cost-efficient AI application does not force every request through the real-time path. It asks each request honestly: "How urgent are you, really?" Urgent requests run in real time, while nonurgent requests accumulate into batches. That one question can cut the bill in half.