06. Observability & dashboard

Canonical results state, container logs, and a small catalog/launcher dashboard, with an Azure adapter that adds Entra authentication and two alerts backed by logs the batch code actually emits.

Outcome

An operator can answer four practical questions: what ran, what is still running, what failed and why, and which workflow should be launched next. The baseline uses three surfaces that already exist in the platform:

  • the results DB for canonical application outcomes;
  • container stdout/stderr for diagnostic detail;
  • a small dashboard for browsing results and launching train, eval, and batch.

Part I exposes those surfaces through Postgres, docker compose logs, and the local dashboard/runner. Part II sends ACA logs to Log Analytics, adds two scheduled-query alerts for batch signals the code really emits, and puts Entra Easy Auth in front of the same dashboard image. There is no Managed Grafana or Application Insights dependency in the baseline.

Operational surfaces

Surface Answers Part I (Compose) Part II (Azure)
Results DB What ran, is running, failed, and why? Postgres results, read through the catalog and /api/results Same schema and readers
Container logs What did the process report around a failure? docker compose logs <service> ACA console logs in Log Analytics
Dashboard What can a person inspect or launch? Catalog, results API, MLflow link, local runner backend Same app, Easy Auth, ACA Jobs backend
Alerts Which emitted failures deserve attention now? Human inspection Permanent-child-failure threshold and batch circuit breaker

These surfaces have distinct responsibilities. A results row is the canonical application outcome; a container log explains execution detail; the dashboard is only a view and launcher. Correlation uses fields recorded once: workflow name, parent/child IDs, caller identity, MLflow run ID, and model version.

On Azure every authenticated tenant user may view the dashboard. Mutation routes perform a second check: only members of the configured Entra operator group may start jobs. The id-dashboard managed identity authorizes machine-to-machine calls to Postgres and the ACA Jobs API; the signed-in human principal supplies authorization and triggered_by attribution for a launch.

Build in projects/ml-platform/

projects/ml-platform/
├── src/dashboard/
│   ├── Dockerfile              # FastAPI dashboard image
│   ├── requirements.txt        # web, Postgres, identity, and ACA SDK dependencies
│   └── app.py                  # catalog/results reads; train/eval/batch triggers; authz
├── demo/
│   ├── docker-compose.yml      # TRIGGER_BACKEND=local, RUNNER_URL=http://runner:8090
│   └── runner/app.py           # independent parameterized subprocess executions
└── infra/modules/
    ├── observability/alerts.tf # two Log Analytics queries over ACA console logs
    └── dashboard/              # ACA App, probes, Easy Auth, job-name mapping

The same src/dashboard/app.py runs in both environments. Configuration selects the local runner or ACA Jobs adapter and supplies the deployed Azure Job resource names.

How the pieces connect

Dashboard API

Route Purpose
GET / HTML catalog of recent results, MLflow link, and local launch buttons
GET /api/results, GET /api/runs Results rows; /api/results/{id} fetches one
GET /api/jobs Triggerable jobs, accepted parameters, and examples
POST /api/runs/train/trigger Start parameterized training
POST /api/runs/eval/trigger Evaluate an exact registered version
POST /api/runs/batch/trigger Start parameterized batch scoring
GET /api/executions/{id} Local runner state plus the matching result row
GET /healthz Probe endpoint; the only Azure auth exclusion

With TRIGGER_BACKEND=local, the dashboard posts the caller and scalar parameters to {RUNNER_URL}/api/jobs/{job}/run. Each accepted request starts an independent subprocess, so two train runs with different hyperparameters may be in flight together.

With TRIGGER_BACKEND=aca, logical names map to the resource names Terraform actually deployed through TRAIN_JOB_NAME, EVAL_JOB_NAME, and BATCH_JOB_NAME. ACA execution overrides replace a complete template, so the app first reads the deployed Job template, then changes only TRIGGERED_BY and the allow-listed CLI arguments for this execution. Image, resources, identity, and every unchanged environment variable remain intact.

Human authentication and authorization

Terraform creates the Container Apps authConfigs/current child resource with Entra ID enabled. Unauthenticated non-health requests redirect to sign-in. Easy Auth injects a trusted base64 X-MS-CLIENT-PRINCIPAL header after successful authentication. The app decodes its claims and enforces DASHBOARD_OPERATOR_GROUP_ID on every POST trigger. An authenticated viewer can read; a non-member receives 403 on launch; a missing or malformed principal is rejected. Local mode deliberately treats the laptop boundary as trusted and uses demo-user when no simulated identity is supplied.

Logs and alerts

ACA sends container stdout/stderr to the foundation’s Log Analytics workspace. continuation.py already emits two stable messages:

  • permanently failed when a child settles as a non-retriable failure;
  • circuit breaking when a batch cannot make progress or reaches its iteration cap.

The observability module queries ContainerAppConsoleLogs_CL for those exact messages in the batch Job’s container group. An optional action group controls notification delivery. The baseline intentionally does not declare failed-job or missed-schedule alerts from guessed schemas: ACA execution history and the results dashboard expose those states, and alert rules should be added only after their live telemetry and per-workflow schedule expectations are verified.

Golden-path position & acceptance evidence

This chapter builds the batch / serve → operational visibility tail of the golden path. Every upstream job already writes results rows, so the dashboard mainly exposes state and launches new executions.

Acceptance evidence: Part I

  • http://localhost:18000 lists recent results and links to the MLflow UI.
  • A forced failure appears in GET /api/results?status=FAILURE with its error.
  • Train, eval, and batch triggers return independent execution IDs; the caller is recorded in triggered_by.
  • docker compose logs runner shows every launched subprocess’s output.

Acceptance evidence: Part II

  • /healthz remains available for probes, while an unauthenticated request to a results or launch route cannot reach the app.
  • An authenticated viewer can inspect runs but receives 403 on a trigger; a configured operator can launch a Job and is recorded in triggered_by.
  • A permanent child failure and a circuit-breaker event appear in ACA console logs and match the two deployed scheduled-query rules.
  • The dashboard starts the Terraform-produced resource name and preserves the deployed Job template while applying per-execution parameters.

Extensions

Deferred capability Baseline
Failed-execution and missed-schedule paging ACA history + results dashboard until live schemas and schedule windows are validated
Metrics/tracing backend Structured results and container logs
Trend dashboards SQL/API queries over results; add a product only when recurring analysis justifies it
SLOs and runbook catalog Two actionable batch alerts and explicit operating checks
Budget alerts Cost review and tear-down discipline

Next: 07 — LLM release artifacts ships an LLM app through this same registry, evaluation, results, and dashboard machinery.

Back to top