infimal
infimal.md
Docs/Build

Batching & streaming

Define how requests can share execution and how a model handles a live stream.

Configure dynamic batching

Batch sets the maximum group size and how long requests may wait to form a group. A larger group can improve throughput, while a wider window adds waiting time. Start with a small window and measure your workload.

Python
Batch(max_size=16, window_ms=20)

Keep compatible shapes together

bucket_by names the request attribute used to group compatible shapes. Requests that cannot share a forward pass should not be placed in the same batch.

A batched handler must return one answer per request. A mismatch produces BatchShapeError. Validate output ordering and counts using representative input before deployment.

Python
Batch(max_size=8, window_ms=10, bucket_by="shape")

Handle streaming frames

@app.stream defines a per-frame function for a live stream. Its first argument is the setup state. A stream-only app can also be invoked as a single-frame stream by the local harness.

Python
@app.streamdef process_frame(model, frame):    return model(frame)

Measure the tradeoff

  • Measure waiting time as well as execution time.
  • Check behavior for one request, a partial batch, and a full batch.
  • Confirm every request gets exactly one correctly ordered result.
  • Test streaming state across the lifecycle your model requires.