05. Online serving & promotion

A FastAPI serving container pinned to one registry version, with evaluation-gated promotion and rollback implemented as an alias update plus consumer redeploy.

Outcome

When a model needs low-latency inference, a long-running serving container loads an exact MLflow model version at startup and exposes an HTTP prediction endpoint. The image carries code only. Promotion first requires passing evaluation evidence for the candidate, then moves the production alias and recreates the consumer pinned to that version. Rollback repeats the operation for an older evaluated version. Batch-only deployments may skip serving entirely.

Serving and release decisions

  • Own image. src/serving_app/ builds a small FastAPI image that loads models:/<name>/<version> from the self-hosted registry at startup.
  • Exact version pinning. MODEL_VERSION must name one concrete registry version. The app refuses to become ready without it.
  • Verifiability. /readyz reports the loaded model name and version after a startup canary succeeds.
  • Evaluation gate. A candidate is releasable only when the configured MLflow evaluation experiment contains eval.model_uri=models:/<name>/<version> and eval.passed=True.

Release then follows this order: verify evaluation, move the production alias, repin serving to the exact version, wait for readiness. Ephemeral batch jobs receive a model version as an execution parameter and do not hold a live model. Consumers never poll the registry for changes.

Rollback uses the same path with a previous evaluated version. Nothing is rebuilt; the model version and image digest already identify immutable artifacts.

Build in projects/ml-platform/

projects/ml-platform/
├── demo/
│   ├── docker-compose.yml      # exact serving MODEL_VERSION pin
│   ├── promote.py              # eval lookup → alias flip → consumer redeploy
│   └── golden_path.py          # train → eval → promote → serve → batch
└── src/serving_app/
    ├── Dockerfile
    ├── requirements.txt
    └── app.py                  # startup load/canary; health, readiness, prediction

Compose publishes serving at localhost:18080. A fresh stack serves bootstrap version 1; demo/.env, written by promotion, supplies later exact pins. Part II runs the same image as an ACA App with managed identity, ingress, and the same probe paths.

How the pieces connect

Startup: load once, canary, then ready

app.py registers a FastAPI lifespan hook, so the model loads once at container startup, not per request. Startup is all-or-nothing:

  1. Read MLFLOW_TRACKING_URI, MODEL_NAME, and MODEL_VERSION from the environment. With MODEL_VERSION unset, the App records the error and refuses to serve: no floating aliases.
  2. Call mlflow.pyfunc.load_model(“models://”). This is the common loading interface for both the tabular sklearn artifact and the text-column LLM artifact.
  3. Run a canary prediction. Tabular models receive the fixed feature vector; text models receive a small input record. This proves the loaded artifact is callable before the readiness flag flips.

If any step raises, the process stays alive but never reports ready, and the reason is visible on /readyz.

Probes

/healthz is liveness: always 200 once the Python process runs (the Dockerfile HEALTHCHECK curls it). /readyz is readiness: 503 with the failure reason until the canary passes, then 200 carrying {status, model_name, model_version}. Only the ready state carries a version, so a green /readyz doubles as proof of which model is live.

Inference

POST /v1/predictions accepts tabular instances such as {“instances”: [[f1, f2, …]]}; text models also accept string instances or {“input”: “…”} records. The response includes predictions, model_name, and model_version. Every response self-identifies, so a client can tell which version served it without probing. Requests arriving before readiness draw a 503; an empty instances list draws a 422.

Promotion and rollback in practice

demo/promote.py performs the release in a fail-closed order:

  1. connect to the selected MLflow tracking/registry URI;
  2. find a passing eval run for the exact candidate in the configured evaluation experiment;
  3. remember the current production alias, then set it to the candidate;
  4. repin and recreate the serving consumer for the selected backend.
Backend Consumer update
local merge DEMO_MODEL_VERSION=N into demo/.env, then recreate only serving
aca update MODEL_VERSION=N, producing a new ACA revision; dry-run unless --execute is supplied

A missing experiment, missing exact-version pass, or failed alias update makes the command non-zero before serving changes. If the consumer update fails after the alias moves, the script makes a best-effort compensating update: it restores the previous alias, or removes the newly created alias if none existed. The gate therefore prevents an unevaluated candidate from changing either release surface, while compensation keeps the registry decision aligned with a failed deployment.

Rollback needs no separate tool. Promote an older version that already has passing evaluation evidence, then require /readyz to report it. The production workflow may later add an emergency override, but the baseline deliberately has none.

# Promote version 3 locally: alias flip, then only the serving container restarts.
python demo/promote.py --backend local --version 3

# The same promotion against Azure: alias flip, then a new serving revision.
# Prints the az command by default; add --execute to apply it.
python demo/promote.py --backend aca --version 3 \
    --resource-group rg-mlp --app-name app-serving

# Rollback: the identical operation aimed at the previous version.
python demo/promote.py --backend local --version 2

The command returning means the redeploy has been issued, not that the new version is being served: Compose still has to stop the old container, start the new one, load the model, and clear the canary. Promotion is complete only when GET /readyz answers {“status”: “ready”, “model_name”: …, “model_version”: N}. That exact-version property turns “which model is live?” from a question about deployment history into an observable fact.

It is also what makes the chapter mechanically testable. demo/golden_path.py drives the full path over plain HTTP against a running Compose stack. The Azure smoke adapters in deploy/ use Azure’s Job and App control-plane APIs separately, then assert the same terminal status, results, readiness, model identity, and prediction behavior.

# From projects/ml-platform/demo/, with the stack up:
python demo/golden_path.py

Each step asserts before the next begins:

  1. Train: trigger training through the dashboard API and poll the results row to SUCCESS.
  2. Resolve: read the newest registered version N from the MLflow REST API.
  3. Promote: invoke demo/promote.py in a subprocess, the real entrypoint rather than a reimplementation.
  4. Serve: poll /readyz until status is ready, then assert model_name and model_version equal the promoted values exactly; a stale version fails the suite.
  5. Score: trigger batch scoring pinned to N and poll to terminal status.
  6. Record: assert the batch parent row shows SUCCESS in the results API.

Acceptance evidence is the suite printing GOLDEN PATH: PASS: promotion lands the exact promoted version behind /readyz, rollback replays an older promotion cleanly, and every prediction response echoes the version that produced it.

Extensions (deferred from the MVP)

Deferred capability Current baseline
Token/scope auth on the prediction endpoint Trust the network boundary (Compose network now, ACA ingress later)
Autoscaling on HTTP concurrency Single warm replica
LLM shared-budget partitioning N/A (classical model)
Automated four-part release provenance Git tag plus recorded digest, by hand

Next: 06 — Observability & dashboard makes the running platform visible and launchable.

Back to top