09. Just enough Terraform

Read and review the real Terraform root and modules that turn the local-first contract into Azure resources.

Terraform is the deployment adapter for Part II. It does not define a second ML workload: it declares the Azure resources, identities, schedules, and environment values needed to run the same images and entrypoints that Compose exercises in Part I. This chapter gives you enough Terraform to review that adapter confidently before chapter 10 applies it.

The root module is a composition

Start at projects/ml-platform/infra/main.tf. The root configures azurerm, random, and the small azapi surface needed for the Container Apps Easy Auth child resource. It calls module "foundation" first and passes foundation outputs into workload modules.

Layer Real code Responsibility
Foundation modules/foundation/ Resource group, ACR, ACA environment, Postgres, storage, Key Vault, Log Analytics, and workload identities
Workload adapters modules/mlflow_app/, train_job/, batch_job/, serving_app/, dashboard/, llm_job/ Map existing images and environment values onto ACA Apps or Jobs; train_job is instantiated separately for train and classical eval
Operations modules/observability/ Two scheduled queries over batch console signals
Optional path modules/aml/ Admission-gated multi-GPU training

The workload modules stay thin. Training remains in src/train_job/train.py, evaluation in evaluate.py, scoring in src/batch_job/score.py, and serving in src/serving_app/app.py. Terraform supplies identities, deployed resource names, schedules, auth configuration, and digest-pinned images.

The Terraform vocabulary that matters

A resource manages one remote object. A variable is a module input. An output exposes a value to a caller. A module groups resources, and a data block reads existing information without creating it. The azapi_resource in the dashboard module is still ordinary Terraform lifecycle management; it is used because Container Apps exposes authConfigs through ARM before the azurerm provider has an equivalent first-class resource.

count gates the two-pass image workflow. MLflow is created only when mlflow_image is non-empty; train, eval, batch, serving, and dashboard wait for the images and upstream endpoints they require. An empty image means the workload is not declared yet.

foundation
    └── mlflow_app
          ├── train_job
          ├── eval_job
          ├── batch_job
          ├── serving_app
          ├── dashboard
          └── llm_job instances

Terraform infers ordering from expressions such as module.mlflow_app[0].mlflow_url. The dashboard also receives the concrete Job names from module outputs, preventing logical names such as train from being mistaken for deployed names such as mlpdev-job-train.

State, plan, and apply

Terraform state is its record of which configuration objects correspond to which remote resources. plan compares the current state and configuration, while apply performs the approved change. State is deployment metadata, not a place for application secrets; the example secret values stay in the ignored secret.auto.tfvars file or an external state backend.

From the repository root, a review loop looks like this:

cd projects/ml-platform/infra
terraform init
terraform fmt -check -recursive
terraform validate
terraform plan -var-file environments/dev.tfvars

The repository currently uses local state for the development MVP. A production deployment should move state to a protected remote backend before multiple operators or CI jobs apply the same configuration. Never approve a plan that replaces a Postgres server or storage account without understanding the data consequence.

Why deployment is two-pass

The first apply creates the foundation, including ACR. The deploy script then builds the MLflow image, applies the database bootstrap, and runs a second Terraform apply with the MLflow digest. After the remaining images are built, the final apply enables the train, batch, serving, dashboard, and shared LLM Jobs.

The one non-declarative step is database grants. Azure creates the Postgres server and databases, but the managed-identity principals and table privileges are applied by infra/grants.sql as the Postgres administrator. The script runs grants before schema creation to create principals, applies the canonical src/ml_platform/results/schema.sql, then runs grants again so the results table receives explicit train, batch, and dashboard privileges.

This ordering matters: a grant on ALL TABLES issued before the table exists does not retroactively grant the table. The second pass closes that gap.

Review checklist

Before approving an infrastructure change, trace these questions from root to module:

  • Which image digest is being deployed, and which shared entrypoint does its container command run?
  • Which managed identity is attached, and does grants.sql give that principal only the needed database privileges?
  • Are MLFLOW_TRACKING_URI, PGHOST, PGUSER, RESULTS_DB, and IMAGE_DIGEST present?
  • Does a scheduled Job receive the same effective data source, model name, alias, and retry arguments as the local runner?
  • Is a long-lived App protected by readiness and liveness probes?
  • Does an output expose the value needed by the smoke tests or the next deploy pass?

The result should read as an adapter: the model, results schema, HTTP routes, retry behavior, and entrypoints stay in shared source; Azure-specific concerns remain identity bindings, resource definitions, ingress, schedules, and control-plane APIs.

Next: 10 — Azure foundation applies the foundation and follows the deploy script end to end.

Back to top