15. End-to-end integration

Run the Compose golden path and the Azure smoke adapter against the same behavioral definition of done.

Outcome

The local suite proves that the stages compose: tracked data, training, exact- version evaluation, gated promotion, serving, batch scoring, and results visibility. The cloud suite exercises the corresponding ACA resources and auth boundary.

The local instrument is demo/golden_path.py. The cloud instruments are deploy/smoke-tests.sh and deploy/smoke-tests.ps1. They use different control planes but preserve the same evidence: terminal execution status, passing evaluation, results state, exact model identity, readiness, and prediction.

The full golden path

flowchart TD
    DATA["tracked dataset"]
    TRAIN["train job"]
    VER["MLflow registered version"]
    EVAL["exact-version evaluation"]
    PROMOTE["gate + production alias + repin"]
    SERVE["serving /readyz + prediction"]
    BATCH["batch parent/child results"]
    CHECK["behavioral assertions"]

    DATA --> TRAIN --> VER --> EVAL --> PROMOTE
    PROMOTE --> SERVE --> CHECK
    PROMOTE --> BATCH --> CHECK

An LLM artifact enters at the registered-version step and uses its own evaluator; the registry, results, identity, and release shapes remain shared.

Adapter-specific execution

The local golden path:

  1. Triggers training and polls its results row.
  2. Resolves the newly registered version.
  3. Triggers evaluation for that exact version and requires SUCCESS.
  4. Calls demo/promote.py, which independently verifies the passing evaluation before changing the alias and recreating serving.
  5. Waits for /readyz, verifies one prediction, then runs batch scoring pinned to the same version.
  6. Confirms the batch parent result is SUCCESS.

The Azure smoke adapter starts ACA train, eval, and batch executions with az, polls terminal platform status, exercises serving, verifies the dashboard probe, and confirms Easy Auth blocks anonymous data access. Authenticated operator and viewer checks require real Entra principals and remain explicit deployment acceptance steps rather than simulated smoke-test headers.

A phase is done when its evidence exists. Local acceptance is demo/golden_path.py ending with GOLDEN PATH: PASS; cloud acceptance is a smoke script ending with all checks passed plus the two human-role checks.

Acceptance evidence

Phase Compose evidence Azure evidence
Foundation environment check passes; Compose config parses Terraform validates; expected resources and identities exist
Training registered version carries dataset and code lineage train ACA execution succeeds
Evaluation exact candidate records a passing threshold decision eval ACA execution succeeds for the candidate
Promotion unevaluated versions are rejected; evaluated version moves alias same MLflow gate before ACA serving repin
Batch parent/child rows settle; transient items retry within the execution batch ACA execution succeeds; application result is inspected with authenticated access
Serving /readyz and prediction echo the exact version same assertions through ACA ingress
Operations dashboard lists runs and logs are inspectable Log Analytics captures ACA logs, two batch alerts are deployed, Easy Auth protects data routes
LLM pyfunc registration/evaluation use the bundled fixture same image and entrypoints with configured dataset and Key Vault credential

The local and cloud scripts remain separate because their trigger and authentication mechanisms differ. Their common behavioral evidence, not shared test-driver code, is what makes the local-first build useful.

Back to top