08. The environment contract

The variables, schemas, trigger APIs, and promotion rules that keep the Docker Compose POC and the Azure deployment behaviorally aligned while their control-plane adapters remain separate.

This chapter defines the contract that lets every Part I feature survive the transfer to Azure: the environment variables each image may read, who owns the results schema, equivalent trigger inputs, how promotion works, and the behavioral checks both validation adapters must preserve. Compose remains the complete local feature/demo environment; ACA supplies the cloud control plane and managed-identity adapter.

Why a contract

The platform exists in two deployment phases, not two implementations. Part I runs the complete feature/demo environment on one machine with Docker Compose: database, object storage, MLflow, training, batch scoring, serving, dashboard, and the local runner. Part II moves the same workload images and entrypoints to Azure Container Apps (ACA), where long-lived HTTP services run as Apps, run-to-finish workloads run as Jobs, and managed identities replace stored passwords.

The control-plane mechanisms are allowed to differ. The local runner accepts HTTP requests and starts subprocesses; the cloud dashboard starts ACA Job executions. What must not differ is the workload contract: environment fallbacks, results schema and statuses, model version identity, health/readiness routes, prediction shape, and the evidence used to decide that a run succeeded.

The design rule is therefore: maximize source reuse, isolate deployment adapters, and compare behavior rather than forcing local code to impersonate Azure APIs.

The environment-variable contract

Application images contain code only. Configuration arrives from Compose, Terraform-managed Job/App definitions, managed identity, or per-execution overrides.

Variable	Consumed by	Purpose	Compose	Azure
`MLFLOW_TRACKING_URI`	train, eval, batch, serving, dashboard	Tracking/registry URL	Compose MLflow DNS	MLflow ACA App URL
`MODEL_NAME`	eval, batch, serving, runner	Registered model name	`wine-quality`	Job/App definition or execution override
`MODEL_VERSION`	eval, serving, runner	Exact candidate/live version	local default or override	eval Job default/override; serving App pin
`DATA_SOURCE`	eval, batch	Input CSV	public sample or override	Job template or execution override
`EVAL_MAX_RMSE`	eval	Promotion threshold default	`0.8`	eval Job template
`BATCH_CHUNK_SIZE`	batch	Rows per chunk	`100`	default or override
`PGHOST`, `PGPORT`, `PGUSER`, `RESULTS_DB`	jobs, dashboard	Results-store connection	Compose Postgres	Azure PostgreSQL + identity principal
`PGPASSWORD`, `PGSSLMODE`	jobs, dashboard	Local password / TLS policy	demo password, local TLS choice	no stored DB password; Entra token and `require`
`TRIGGERED_BY`	jobs	Human or schedule attribution	runner value	Easy Auth caller or schedule
`RESULTS_RUN_ID`	jobs	Optional external result ID	local execution ID	unset by default
`IMAGE_DIGEST`	train/LLM entrypoints	Code lineage	`demo-local`	pinned ACR digest
`LLM_`, `MODEL_API_KEY`, `KEY_VAULT_URL`	LLM paths	LLM artifact/eval configuration	fixture and optional local key	Job template and Key Vault
`MLFLOW_UI_URL`	dashboard	Browser-reachable MLflow link	localhost port	public MLflow URL
`TRIGGER_BACKEND`, `RUNNER_URL`	dashboard	Execution adapter	`local`, runner DNS	`aca`, runner unset
`AZURE_SUBSCRIPTION_ID`, `AZURE_RESOURCE_GROUP`	dashboard	ACA API scope	unset	Terraform values
`TRAIN_JOB_NAME`, `EVAL_JOB_NAME`, `BATCH_JOB_NAME`	dashboard	Logical-to-resource mapping	unset	Terraform module outputs
`DASHBOARD_OPERATOR_GROUP_ID`	dashboard	Mutation authorization	unset	Entra group object ID

The Easy Auth client secret is an ACA app setting consumed by the platform auth module, not by Python. It comes from ignored secret.auto.tfvars, is marked sensitive, and requires protected Terraform state. Adding or changing a Python- read variable means updating the workload, Compose/Terraform providers, this table, and the executable environment check together.

Results schema ownership

One table, results, in a dedicated PostgreSQL database is the canonical operational state of every workflow: id, parent_id, name, status, output, error, attempts, triggered_by, and timestamps. Its DDL lives in src/ml_platform/results/schema.sql with IF NOT EXISTS guards.

Environment	Mechanism
Docker Compose	Postgres initialization creates the `results` database and applies the guarded DDL on a fresh volume
Azure	Deployment runs `infra/grants.sql`, maps managed identities to least-privilege roles, then applies `schema.sql`

The status vocabulary is PENDING, STARTED, SUCCESS, RETRY, FAILURE, and REVOKED. Batch fan-out uses one parent plus deterministic child rows. The continuation loop retries eligible children during one execution; a fresh invocation creates a fresh parent unless the caller explicitly reuses RESULTS_RUN_ID. Cross-execution crash resume is not a baseline guarantee.

Schema changes land in schema.sql first and update every bootstrap copy and reader in the same product change. The dashboard SQL depends on these columns; there is no separate documentation-defined schema.

Trigger API equivalence

The dashboard is a catalog and launcher, never an executor. Each launch goes to an execution-plane adapter:

Local: demo/runner/ accepts the caller and scalar parameters, validates a per-job allow-list, and starts train.py, evaluate.py, or score.py as an independent subprocess.
Azure: the dashboard resolves the logical job name through Terraform- supplied TRAIN_JOB_NAME, EVAL_JOB_NAME, or BATCH_JOB_NAME, reads that deployed ACA Job template, and starts a new execution after changing only the caller and allow-listed CLI arguments.

Operation	Dashboard route	Local adapter	Azure adapter
Train	`POST /api/runs/train/trigger`	runner `train` subprocess	training ACA Job execution
Evaluate	`POST /api/runs/eval/trigger`	runner `evaluate.py` subprocess	evaluation ACA Job execution from the train image
Batch score	`POST /api/runs/batch/trigger`	runner `score.py` subprocess	batch ACA Job execution
Parameters	typed request models	command-line flags after allow-list validation	same flags as execution-specific `args` on a copy of the deployed template
Poll	results URL and local execution route	runner status + results row	ACA execution API + results row

ACA replaces the whole execution template when overrides are supplied. Copying the deployed template first is therefore part of the adapter contract: image, resources, identity, and unchanged environment values survive parameterization. A local trigger looks like this:

curl -X POST http://localhost:18000/api/runs/train/trigger \
  -H 'content-type: application/json' \
  -H 'X-MS-CLIENT-PRINCIPAL-NAME: demo-user' \
  -d '{"parameters":{"alpha":0.25,"l1_ratio":0.8,"random_state":7}}'

The response names the execution and echoes the accepted parameters. Locally it also returns result_id and result_url; the runner exports its execution name as RESULTS_RUN_ID, so one ID chains from trigger response to results row. ACA returns its own execution name while the workload writes the application outcome to the results DB.

Azure attribution and authorization use different Easy Auth headers for different purposes. X-MS-CLIENT-PRINCIPAL-NAME supplies the display identity, while base64 X-MS-CLIENT-PRINCIPAL carries claims. Easy Auth authenticates all non-health routes, and the app requires the configured operator-group claim on POST triggers. Viewers can read but cannot launch. Local mode simulates the identity and trusts the laptop boundary.

Unknown, non-scalar, or job-inappropriate parameters are rejected before an execution starts. Separate requests are independent: parallelism = 1 controls replicas inside one ACA execution, not how many Job executions may coexist.

Promotion semantics

Promotion follows six rules:

Images carry code only. Model artifacts live in the registry, keyed by immutable version.
A passing evaluation is mandatory. Before changing anything, demo/promote.py searches the configured evaluation experiment for a run whose eval.model_uri names the exact candidate and whose eval.passed tag is True.
The registry alias records the release decision. After the gate passes, production → N is one MLflow alias update in either environment.
Only the long-running consumer is redeployed. Serving receives MODEL_VERSION=N and restarts; ephemeral jobs take a version as an ordinary execution parameter.
A failed consumer update compensates the alias change. The script restores the previous alias, or deletes the newly created alias if the candidate was the first promotion.
Consumers never poll for new models. /readyz reports the exact version loaded at startup, so the deployed state is directly verifiable.

Step	Local	Azure
Verify evaluation	MLflow client searches `wine-quality-eval`	same call against the cloud MLflow URL
Flip alias	`production → N`	same registry operation
Repoint serving	write `DEMO_MODEL_VERSION=N`; recreate Compose `serving`	update ACA `MODEL_VERSION=N`; create a revision
Wrapper	`python demo/promote.py --backend local --version N`	`python demo/promote.py --backend aca --version N ... --execute`

Rollback points the same operation at an older, already evaluated version. No image or model artifact is rebuilt. Promotion is complete only when /readyz reports the requested version after its startup canary passes.

Under the wrapper, the local backend edits demo/.env and shells out to docker compose up -d serving, so only the serving service is recreated; the ACA backend calls az containerapp update --set-env-vars MODEL_VERSION=N, which rolls a new revision with ACA handling the traffic switch. Either way the promotion is not finished when the command returns: it is finished when GET /readyz reports exactly the new version, which happens only after the model is loaded and the startup canary prediction has passed.

This is why /readyz exists as a version-aware probe. It turns “which model is live?” from a question about deployment history into an observable fact, and it gives the golden-path suite below something exact to assert on.

Compose ↔︎ Azure mapping

Platform piece	Part I: Docker Compose	Part II: Azure	Access
Registry + operational DB	Postgres container (`mlflow` and `results`)	Azure Database for PostgreSQL	static demo password vs managed identities with Entra token auth
Object storage	MinIO (S3 API)	Blob Storage	demo keys vs workload identity
Model registry + tracking UI	MLflow container	MLflow ACA App	Compose DNS name vs HTTPS ingress FQDN
Train/eval/batch execution	one-shot services plus the local runner	independent ACA Job executions	runner HTTP API vs ACA Jobs API; scalar CLI overrides in both
Online serving	serving container	serving ACA App (`id-serving`)	published port vs ingress and probes
Dashboard	dashboard container, `TRIGGER_BACKEND=local`	dashboard ACA App, `TRIGGER_BACKEND=aca`	laptop trust vs Easy Auth plus operator-group authorization
Operational logs	Compose stdout/stderr	ACA console logs in Log Analytics	direct CLI vs workspace queries and two batch alerts
Secrets	static demo credentials in Compose	managed identities plus one dashboard app-registration secret	ignored local config and protected Terraform state; never images or Git

The local runner has no Azure counterpart by design. It supplies the asynchronous execution shape a laptop needs; ACA supplies that control plane in the cloud. The workload entrypoints, parameter names, status vocabulary, and model identity remain shared.

Behavioral checks, with separate trigger adapters

The contract ends where it can be tested. The local demo/golden_path.py and the cloud deploy/smoke-tests.sh / .ps1 scripts intentionally use different trigger mechanisms, but preserve the same assertions:

Training reaches terminal success and produces a registered model version.
Evaluation of that exact version reaches terminal success and leaves the passing MLflow evidence required by promotion.
Promotion moves the production alias only after that evidence exists.
Serving /readyz reports the exact promoted version, and a prediction response echoes it.
Batch reaches terminal success and its parent result is successful.
The dashboard health endpoint remains available while non-health Azure routes require Entra authentication.

Only the adapter changes. Local calls use runner execution IDs and direct results polling; cloud calls use ACA execution names, authenticated dashboard access, and Log Analytics. Keeping the suites separate makes failures diagnostic while keeping their behavioral assertions aligned.