10. End-to-end integration

Walk the whole golden path on the deployed platform, verify the acceptance evidence for each phase, and catalog every deferred production extension.

Outcome

Everything built in Chapters 02–07 runs as one path: a tracked dataset becomes a trained, evaluated, registered model version; the version is promoted and then served online and/or scored in batch; and every step is visible in the results DB, Grafana, and the dashboard. This chapter ties the phases together, provides smoke tests, and consolidates the extension catalog.

## The full golden path

flowchart TD
    DATA["tracked dataset (Blob)"]
    TRAIN["train Job (ACA) — Ch 03"]
    VER["MLflow run + registered version"]
    EVAL["eval Job — metrics + results row"]
    PROMOTE["promote version — Ch 05"]
    BATCH["batch Job (scheduled/manual) — Ch 04<br/>parent/child rows"]
    SERVE["serving App (optional) — Ch 05<br/>/health reports version"]
    OBS["dashboard + Grafana + alerts — Ch 06"]

    DATA --> TRAIN --> VER --> EVAL --> PROMOTE
    PROMOTE --> BATCH --> OBS
    PROMOTE --> SERVE --> OBS

An LLM app (Ch 07) enters at registered version as a pyfunc and follows the same path. The exception (Ch 08) and broker (Ch 09) tracks are not on this path.

Build in projects/ml-platform/

projects/ml-platform/
├── deploy/
│   ├── deploy.ps1              # 4-pass full deploy: foundation → MLflow + DB setup
│   │                           #   → images → full platform; calls smoke-tests.ps1
│   └── smoke-tests.ps1         # end-to-end golden path verification (phases 0–5)
└── .github/workflows/
    └── ci.yml                  # build / Trivy scan / push / update ACA definitions
                                #   via OIDC as id-ci (no runtime data access)

deploy.ps1 now covers all phases: it builds every image, runs both grants.sql (Postgres principals) and schema.sql (results table DDL), and applies Terraform in a single choreographed sequence. smoke-tests.ps1 calls the live endpoints to verify acceptance evidence for each phase.

How the pieces connect

deploy.ps1 — four-pass orchestration

Pass What happens
1 terraform apply — foundation only (ACR, Postgres, storage, identities)
2 Build + push MLflow image; run grants.sql (principals) + schema.sql (results table); terraform apply with MLflow image
3 Build + push train / batch / serving / dashboard images
4 terraform apply with all four images pinned; optional smoke-tests.ps1

All image references are by digest (ACR show --query digest). The two-pass gating in each module (count = image == "" ? 0 : 1) means pass 1 produces only the foundation; everything else comes live in pass 4.

smoke-tests.ps1 — acceptance evidence

The script reads Terraform outputs to discover live endpoints, then: - Phase 0: GET /health on MLflow. - Phase 1: az containerapp job start on the train Job; polls until Succeeded or Failed. - Phase 2: triggers the batch Job (row verification noted as a manual DB step). - Phase 3: GET /readyz on the serving App; asserts status=ready and a non-empty model_version. - Phase 4: GET /healthz and /api/runs on the dashboard.

Each phase is skipped gracefully when its output is empty (partial deploy). The script exits non-zero if any assertion fails — runnable in CI post-deploy.

ci.yml — GitHub Actions (build/test/scan/deploy only)

Four jobs: 1. test — ruff check + pytest (no cloud). 2. build-and-push (matrix: all five images) — docker build, Trivy CRITICAL scan, docker push, az containerapp job/app update with the new digest.

CI authenticates via OIDC as id-ci (ACR push + ACA definition update only; no runtime data access — invariant 8). It never schedules or orchestrates workflows (invariant 7).

Module index — everything that now exists

Module / source Chapter Purpose
infra/modules/foundation/ 02 RG, ACR, ACA env, Postgres, storage, Grafana, 6 identities
infra/modules/mlflow_app/ 02 Self-hosted MLflow ACA App
infra/modules/train_job/ 03 Training/eval ACA Job (id-jobs-train)
infra/modules/batch_job/ 04 Batch scoring ACA Job (id-jobs-batch)
infra/modules/serving_app/ 05 Online serving ACA App (id-serving)
infra/modules/observability/ 06 4 Log Analytics alert rules
infra/modules/dashboard/ 06 Workflow catalog + launcher ACA App (id-dashboard)
infra/modules/aml/ 08 AML workspace + min-zero GPU cluster (exception only)
infra/modules/broker/ 09 Managed Redis + KEDA scale rule (upgrade only)
src/ml_platform/common/ 03 MLflow client, datasets, schemas, results context mgr
src/ml_platform/results/ 04 Results DB DDL, store API, continuation rule
src/ml_platform/llm/ 07 pyfunc model, artifact builder, evaluator
src/train_job/ 03 train.py, evaluate.py, register_llm.py
src/batch_job/ 04 score.py; worker.py (broker upgrade)
src/serving_app/ 05 FastAPI serving App
src/dashboard/ 06 FastAPI catalog + launcher
src/train_aml/ 08 Distributed training entrypoint + job.yml

## Acceptance evidence (per phase)

Phase Evidence
0 Foundation IaC-created footprint; least-privilege identities; MLflow reachable
1 Training Reproducible registered version with dataset + code lineage
2 Batch Parent/child rows; transient item re-dispatched to terminal state
3 Serving /readyz reports exact version; version-based rollback, no rebuild
4 Observability Dashboard lists runs + deep-links; alert fires on failure
5 LLM pyfunc version serves/batches through unchanged paths

A phase is “done” when its evidence is demonstrated — not when a local demo runs.

## Extension catalog

Consolidated from each chapter’s Extensions section — the production contract (docs/00–docs/08) beyond this MVP:

  • Foundation: private endpoints/VNet, Azure Policy, identity bootstrap ordering, multi-environment, Postgres topology split.
  • Training: rich stage-identity chain, distributed/multi-GPU (Ch 08).
  • Batch: broker/backpressure for large fan-out (Ch 09), event triggers.
  • Serving: token/scope auth, HTTP-concurrency autoscaling, LLM budget partitioning.
  • Observability: sampling, cross-plane tracing, SLOs, runbooks, budget alerts.
  • LLM: evaluator evidence contract, provider-upgrade workflow, retrieval index lifecycle.

Each is intentionally deferred: we add machinery only when a concrete need forces it. That is the whole design philosophy (docs/00), and it is why this platform stays small enough for a team that is not full-time platform engineers to operate.

Back to top