13. Operations on Azure

Operate the deployed platform with results state, ACA execution status, readiness probes, alerts, dashboard controls, and promotion rollback.

Operations is where the shared contract becomes evidence. Azure supplies the control plane, identity, logging, and alerting adapters; the workload still reports the same results statuses, model version identity, readiness state, and prediction response that the local Compose golden path checks.

Four signals, one runbook

Use different signals for different questions:

Question Evidence
Did ACA start and finish the process? az containerapp job execution show, polled until Succeeded, Failed, Stopped, or timeout
Did the application complete its work? Parent/child rows in the results database and dashboard API
Which model is serving? /readyz returns status=ready, model_name, and exact model_version
Did inference work? /v1/predictions returns a prediction with the same model version

A green Job execution is necessary but not sufficient: a process can exit zero after recording an incorrect outcome. That is why the smoke adapter waits for the cloud execution and then checks the behavioral result row. The local golden path applies the same assertions through its runner.

Alerts and dashboard access

The observability module creates two Log Analytics scheduled-query rules backed by messages the batch code emits to ACA console logs:

  • permanent child failures above alert_failure_count_threshold;
  • a batch circuit-breaker event.

Both queries read ContainerAppConsoleLogs_CL and filter the batch Job container group before matching permanently failed or circuit breaking. Action groups are optional, so development can validate matching without paging anyone.

Failed ACA executions and missed schedules remain visible in ACA history and the results dashboard, but they are not baseline alerts. Their alert definitions need live platform-log schemas and per-workflow expected windows; guessed KQL is worse than an explicit deferred capability.

The dashboard uses Easy Auth for every route except /healthz. Authenticated users can inspect results. POST routes decode the trusted client-principal claims and require the configured Entra operator group; a viewer receives 403. The id-dashboard machine identity remains separately scoped to Postgres reads and ACA Job starts.

Evaluate, promote, verify, and rollback

Evaluate the exact candidate first, either from the dashboard or the ACA Job:

az containerapp job start --name <eval-job> --resource-group <resource-group> \
  --args --version 3 --registered-name wine-quality \
  --data-source <held-out-csv> --max-rmse 0.8

After that execution succeeds, promotion verifies the passing MLflow tags before changing the alias or serving App:

python demo/promote.py --tracking-uri https://<mlflow-app> \
  --backend aca --version 3 \
  --resource-group <resource-group> --app-name <serving-app> --execute

Serving receives exact MODEL_VERSION=3, starts a revision, and must report that version from /readyz. Rollback repeats the same gated operation for an older, already evaluated version. Nothing is rebuilt.

Cloud smoke tests and operating limits

Run the Azure adapter after deployment:

./deploy/smoke-tests.sh --tf-vars infra/environments/dev.tfvars

PowerShell users can run the matching .ps1. The local-only demo/golden_path.py remains separate because runner calls and Azure control-plane calls are different adapters.

The operating limits are explicit: public ingress is an MVP networking choice; Terraform state must protect the dashboard app-registration secret; failed-job and missed-schedule alerting waits for live telemetry validation; and the LLM evaluator needs an accessible dataset plus a Key Vault credential. None requires forking the shared model or job code.

Next: 14 — Multi-GPU training covers the admission-gated exception path.

Back to top