Requests & priority
A service describes what can run. A request expresses how urgently its result is needed.
Infrastructure and request intent
Access, replica bounds, and batching belong to the deployed service. Urgency can vary between calls: a batch evaluation can wait longer than a live conversation on the same model.
The product contract separates per-call performance from infrastructure configuration. The current public client has no documented per-call priority method; confirm the interface supported by your environment before integrating.
Choose the right service objective
| Mode | Intent | Typical work |
|---|---|---|
| Deferred | Finish by an accepted deadline | Evaluation runs, bulk processing |
| Standard | Balanced responsiveness and cost | Interactive APIs |
| Priority | Favor faster response under demand | Live assistants, real-time experiences |
Use the right performance measure
For token generation, time to first token describes initial responsiveness; tokens per second describes generation speed. For streaming, measure time to first output and processing speed. A deferred deadline describes completion, not first response.
Targets and prices must come from the model and the service terms available in your environment. The landing page’s slider explains the tradeoff using illustrative values.
Integrate against a known contract
- Use the model’s published input and output schema.
- Confirm authentication, streaming protocol, and timeout behavior.
- Do not infer a production route or parameter from a browser example.
- Treat retries according to the actual endpoint’s idempotency behavior.