Scaling & residency
Set the range your service can use without predicting the shape of tomorrow’s traffic.
Declare the bounds
min_replicas=0 allows a service to scale down when idle. max_replicas limits how far it may scale out. These are service bounds, not advance capacity purchases.
Python
Scale( min_replicas=0, max_replicas=1000, target_queue_ms=500,)Respond to waiting work
The scale policy uses queue age rather than GPU utilization as its signal. Waiting time reflects the experience of incoming work even when the model has been parked. Choose target_queue_ms for your workload, then inspect actual behavior.
Move through residency states
| State | What is retained | Why it exists |
|---|---|---|
| Active | Model state in GPU memory | Serve incoming work |
| Parked | State in host memory | Release GPU memory while retaining a nearby image |
| Checkpointed | State on local storage | Retain a restorable image at a lower residency cost |
| Released | Published artifact; no local image | Recover local resources until staging is needed |
Idle policy fields
| Field | Default | Meaning |
|---|---|---|
| idle_park_ms | 60000 | Idle time before parking; 0 disables parking |
| parked_checkpoint_ms | 600000 | Parked time before local checkpoint; 0 disables it |
| cold_release_ms | 0 | Cold time before releasing the local image; 0 retains it |
Inspect scaling decisions
Inspect status alongside usage. A zero-replica state can be normal when there is no work. If requests are waiting, check the declared maximum, available placement capacity, and restore state.
Terminal / reference
infimal apps status asr-emotion-checkpoint4200
infimal usage show --since 1h --json