05. Online serving & promotion

Phase 3 — optional online HTTP serving as an ACA App that loads an exact model version, with promotion and rollback as version/definition changes.

Outcome

When a model needs low-latency online inference, it is served by an ACA App whose container loads an exact MLflow model version at startup. We bring our own image, so there is no framework/runtime lock-in; rollback is a version change, not a rebuild; and the running app’s model identity is unambiguous. This is Phase 3 (docs/05), together with the promotion/rollback mechanics of docs/06. Online serving is optional — batch-only deployments skip it.

## Design — serve an exact version, promote by definition

  • Own image. The serving container (FastAPI or similar) loads models:/<name>/<version> from the self-hosted registry at startup and exposes an inference endpoint. We control framework, runtime, and dependencies.
  • Exact version pinning. The served version is pinned in the App definition (env/config) — never latest, never a floating alias resolved per request. The resolved version is logged at startup and exposed on the readiness/health response, so the live version is externally verifiable.
  • Promotion = version + definition change. An evaluated registry version is marked promoted (MLflow stage/alias or tag) and the App definition is updated to reference it. Provenance is: registered version → its evaluation record → Git tag → deployed image digest — no hash-chained release ledger.
  • Rollback. Model rollback repoints the definition at the previous promoted version (still in the registry); code rollback repoints at the previous image digest (still in ACR). Both are reviewed IaC/definition changes, not portal edits.

Build in projects/ml-platform/

projects/ml-platform/
├── src/serving_app/
│   ├── Dockerfile              # BYO image: python:3.11-slim + FastAPI + MLflow
│   ├── requirements.txt        # mlflow/sklearn/fastapi/uvicorn pinned
│   └── app.py                  # startup loads models:/name/version → /healthz,
│                               #   /readyz (version-aware), /v1/predictions
└── infra/
    ├── main.tf                 # + module "serving_app" (two-pass on serving_image)
    ├── variables.tf            # + serving_image, serving_model_name, serving_model_version
    ├── outputs.tf              # + serving_url
    └── modules/serving_app/    # ACA App: id-serving, readiness/liveness probes
        ├── main.tf
        ├── variables.tf
        └── outputs.tf

Online serving is optional — batch-only deployments leave serving_image = "" and the module’s count = 0 keeps it absent. The serving identity (id-serving) is provisioned by the foundation regardless (Ch 02), so it can be wired up at any point without a re-provision.

How the pieces connect

Startup: load + canary

app.py uses FastAPI’s lifespan hook so the model is loaded once at container startup, not per request. On startup it:

  1. Reads MODEL_NAME and MODEL_VERSION from env (both required; the App refuses to start if MODEL_VERSION is empty — no floating aliases in production).
  2. Calls mlflow.sklearn.load_model(f"models:/{name}/{version}") via id-serving’s managed identity (DefaultAzureCredential).
  3. Runs a canary prediction on a fixed feature vector to confirm the loaded artefact is callable before marking the App ready.

Probes

/healthz is the liveness probe — it always returns 200 once the Python process is alive. /readyz is the readiness probe — it returns 503 until the canary passes, then 200 with {model_name, model_version, status}. ACA routes traffic only to ready replicas, so callers never see a half-loaded model.

# Promote: bump MODEL_VERSION in the App definition, redeploy → ACA rolls new revision
# Rollback: repoint MODEL_VERSION (or serving_image) at the prior value, redeploy

Inference endpoint

POST /v1/predictions accepts {"instances": [[f1, f2, ...]]} and returns {"predictions": [...], "model_name": "...", "model_version": "..."}. The response includes model_version so clients can verify which version they hit — the live version is always externally verifiable.

Infra (infra/modules/serving_app/)

An azurerm_container_app (not a Job) with: - revision_mode = "Single" — ACA manages one live revision; deploying is updating the template. - min_replicas = 1 by default — keeps the model warm (override to 0 for scale-to-zero if cold starts are acceptable). - readiness_probe on /readyz (10 s period, 10 failures before unhealthy) and liveness_probe on /healthz — ACA restarts unresponsive containers automatically. - MODEL_VERSION pinned in env — rollback is a definition change (repoint the variable), no image rebuild required.

Two-pass gating: count = var.serving_image == "" || var.mlflow_image == "" ? 0 : 1. The new serving_url output in infra/outputs.tf is empty on earlier passes and populated once the App is live.

## Golden-path position & acceptance evidence

This chapter builds the optional promote → serving App (loads models:/name/version, /health reports version) branch.

Acceptance evidence:

  • /readyz reports the exact loaded version and returns 503 until the model is loaded and a canary prediction passes.
  • Repointing the App definition at a prior version rolls back with no rebuild, and /readyz reflects the change.
  • The live version on /health matches the promoted registry version.

## Extensions (deferred from the MVP)

Deferred Contract MVP substitute
Token/scope auth docs/05 Simple API key or Easy Auth in front
Autoscaling on HTTP concurrency docs/05 Fixed / min replica count
LLM shared-budget partitioning docs/05 N/A (classical model)
Full four-part release provenance automation docs/06 Git tag + definition digest by hand

Next: 06 — Observability & dashboard makes the running platform visible and launchable.

Back to top