Batch API discounts: when to use them

A batch API lets you submit a large set of requests as a single asynchronous job instead of calling the synchronous endpoint one request at a time. In exchange, the provider charges a flat discount on both input and output tokens. Results typically land within a defined processing window rather than instantly, but the model and output quality don't change. It's a rare deal in this market: same model, same quality, lower price, and the only real cost is latency you often didn't need in the first place.

What belongs in a batch job

• Bulk classification or tagging runs over an existing dataset — support ticket triage, content moderation sweeps, lead scoring.
• Offline summarization or extraction jobs that feed a report, not a live user session.
• Eval and regression runs against a fixed prompt set during development.
• Nightly re-processing jobs that already run on a schedule and don't need to be instant, such as re-embedding a corpus or re-scoring a queue.

What doesn't

Anything a user is waiting on in real time is disqualified by definition — chat interfaces, live agent tool calls, interactive coding assistants. Batch also isn't a fit for workloads with unpredictable, bursty submission patterns where you need to react within seconds of a new item arriving; the processing window makes that architecture impractical even if the discount is appealing.

The discount varies by provider: check the model page, not a rule of thumb

The batch discount isn't a fixed, provider-agnostic number. It's set per provider and sometimes varies by model within the same provider's lineup, so two providers advertising "batch discounts" can mean noticeably different savings once you compare the actual figure on the model page. Before you build a cost model around batch pricing, pull the discount percentage for the specific model you intend to use rather than assuming it matches what a competitor offers.

Processing windows aren't identical either

The window a provider commits to, meaning how long a batch job can take before results are guaranteed, differs by provider and isn't always published as a hard SLA. If your job has a real deadline (a report due by morning, a queue that needs to clear before the next business day), confirm the provider's stated window against that deadline rather than assuming "batch" means the same turnaround everywhere.

How the discount compounds with other levers

Batch pricing and prompt caching solve different problems and can often be combined: a batch job that also repeats a static prefix across its requests gets both discounts at once, since caching affects the input rate and batch affects the whole request. If you're running a large offline job with a shared system prompt or reference document across every item, check whether your provider allows caching within batch requests — it's the largest compounding saving available on this dataset.

The batch discount percentage is a fixed multiplier off standard pricing, not a separate rate — you can compute the batch cost of any workload yourself once you know the standard cost and the discount on the model page.

A simple test before you migrate a job to batch

Ask whether anyone is actually looking at the result within minutes of submission. If the real answer is hours or the next business day, it's a batch candidate regardless of how the code currently calls the API. Migrating an existing synchronous integration usually just means swapping the endpoint and adding a poll-or-webhook step to collect results. The request payload and model choice don't need to change at all.

Revisit the question periodically, not just once at launch. A workload that started as interactive can drift into batch-eligible territory as the product evolves. An internal tool that used to need instant answers might now feed a nightly digest instead. Whenever you're reviewing spend, that's the right moment to ask the same question again: does anything downstream of this job actually need it synchronously, or has that requirement quietly gone away?