06. Observability & dashboard

Phase 4 — three non-overlapping observability layers plus a lightweight catalog/launcher dashboard over the four planes.

Outcome

The platform is observable through three non-overlapping layers plus one lightweight dashboard that unifies them. An operator can see what is scheduled, what is running, what failed and why, and can launch or rerun a workflow with an audit trail. This is Phase 4 (docs/06).

## Design — three layers, no duplication

Layer Answers Backed by
Results DB canonical run state (what ran / is running / failed and why) Postgres results
Log Analytics / App Insights infra telemetry + alerting (job failed, run missed, permanent-failures over threshold, batch stalled) Log Analytics
Azure Managed Grafana deep operational dashboards Log Analytics + Postgres

The dashboard is a read-and-launch ACA App that deep-links to Grafana and MLflow rather than re-implementing charts. It holds no authoritative state: it reads the results DB and job status, and starts Jobs via the ACA execution API (audited through triggered_by). Human access is gated by Entra groups (ml-platform-operators can launch; ml-platform-viewers read-only).

Correlation is by shared dimensions across layers: workflow_id, parent_id, name, image_digest, and the resolved model version.

Build in projects/ml-platform/

projects/ml-platform/
├── src/dashboard/
│   ├── Dockerfile              # python:3.11-slim + FastAPI + psycopg + azure-mgmt-appcontainers
│   ├── requirements.txt        # fastapi/uvicorn/psycopg/azure-identity/azure-mgmt-appcontainers
│   └── app.py                  # GET / catalog, GET /api/runs, POST /api/runs/{job}/trigger,
│                               #   GET /healthz; triggered_by from Easy Auth header
├── infra/modules/observability/  # alert rules (Log Analytics scheduled-query alerts)
│   ├── alerts.tf               # job-failed, run-missed, permanent-failures, batch-stalled
│   ├── variables.tf
│   └── outputs.tf
├── infra/modules/dashboard/    # ACA App: id-dashboard, Easy Auth, readiness/liveness probes
│   ├── main.tf
│   ├── variables.tf
│   └── outputs.tf
└── infra/
    ├── main.tf                 # + module "observability" (always), + module "dashboard" (two-pass)
    ├── variables.tf            # + dashboard_image, alert_action_group_id, alert_failure_count_threshold
    ├── outputs.tf              # + dashboard_url
    └── modules/foundation/
        └── outputs.tf          # + log_analytics_workspace_id (used by observability module)

The mockup for the dashboard surface lives at projects/ml-platform/docs/mockups/workflow-dashboard.html.

How the pieces connect

Dashboard app (src/dashboard/app.py)

The dashboard is decoupled from execution — it reads and triggers, stores nothing authoritative. Three routes drive the catalog:

  • GET / — HTML catalog reading the last 20 rows from the results DB; deep-links to Grafana (GRAFANA_URL env) and MLflow (MLFLOW_TRACKING_URI env). No charts are re-implemented; those live in Grafana.
  • GET /api/runs — raw JSON from the results DB, for scripted queries or a richer frontend later.
  • POST /api/runs/{job_name}/trigger — calls azure-mgmt-appcontainers’s jobs.begin_start with id-dashboard’s managed identity. Reads the caller’s Entra UPN from the X-MS-CLIENT-PRINCIPAL-NAME header (injected by Easy Auth) and passes it as a TRIGGERED_BY env override so it lands in the results DB — authorization by machine identity, attribution by human identity.

If the dashboard App is down, scheduled runs still fire (ACA cron) and in-flight executions keep running. The same workflow can always be triggered directly via az containerapp job start … with the same identity and audit trail.

Observability alerts (infra/modules/observability/)

Four Log Analytics scheduled-query alert rules (firing every 5–60 min):

Alert Condition Severity
job-failed ACA container exit code in logs 1 (critical)
run-missed No Job execution in a 2-hour window 2
permanent-failures FAILURE log messages exceed threshold 2
batch-stalled Circuit-breaker message in logs 1 (critical)

action_group_id is optional — leave it empty for dry-run (rules exist, no pages). The observability module is always provisioned (no count gate); it only needs the foundation’s Log Analytics workspace ID, which is now exposed as module.foundation.log_analytics_workspace_id.

Dashboard infra (infra/modules/dashboard/)

An azurerm_container_app running as id-dashboard: - Read-only on the results DB (SELECT granted in grants.sql). - ACA execution-start via the custom job_starter role (provisioned by foundation/identities.tf). - GRAFANA_URL and MLFLOW_TRACKING_URI are env values, not secrets — the links open in the operator’s browser, not in the container. - Easy Auth (Entra ID) sits in front at the ACA ingress level; the App itself reads X-MS-CLIENT-PRINCIPAL-NAME for the triggered_by attribution. - Two-pass gating: count = dashboard_image=="" || mlflow_image=="" ? 0 : 1.

## Golden-path position & acceptance evidence

This chapter wires the batch / serve → dashboard + Grafana + alerts tail of the golden path.

Acceptance evidence:

  • The dashboard lists scheduled/running/failed workflows read from the results DB and deep-links to the matching MLflow run and Grafana panel.
  • A forced job failure fires an alert; a missed scheduled run is detected.
  • An operator launches a Job from the dashboard and the run records their Entra identity in triggered_by; a viewer cannot launch.

## Extensions (deferred from the MVP)

Deferred Contract MVP substitute
Sampling strategy docs/06 Default App Insights sampling
Distributed tracing across planes docs/06 Shared correlation dimensions
SLOs (99.5% API, 99% workflow) + runbook catalog docs/06 Basic error alerts
Budget-monitoring alerts docs/06 Cost review at tear-down

Next: 07 — LLM release artifacts ships an LLM app through this exact same machinery.

Back to top