09. Broker upgrade (only if forced)

The conditional upgrade path: Celery-as-a-library on short-lived KEDA-triggered ACA Jobs backed by managed Redis — adopted only when batch fan-out outgrows the results-DB continuation rule.

Outcome

If batch fan-out routinely exceeds a few hundred concurrent units or needs queue backpressure, we adopt a broker — but never a long-running worker fleet. The documented upgrade is Celery-as-a-library with workers running as short-lived KEDA-triggered ACA Jobs (drain-and-exit, still image-pinned), backed by managed Redis. This is a conditional upgrade (docs/04), off the critical path; the baseline ships without it.

## Design — a broker without a fleet

  • Trigger to adopt. Only when the parent/child + continuation model of Ch 04 can no longer keep up (sustained fan-out beyond a few hundred concurrent units, or a real need for backpressure). Do not adopt preemptively.
  • Short-lived workers, not a fleet. Workers are Celery as a library inside ACA Jobs that KEDA scales up on queue depth and that drain and exit — so a code deploy is still just an image digest bump, preserving invariant 1 (no stale worker on old code).
  • Managed Redis as the broker. Redis is the queue; results still land in the same generic results DB with the same parent/child rows, so observability and the dashboard are unchanged.
  • What does not change. The results DB, model identity, promotion, serving, and batch semantics are all identical; only the dispatch mechanism changes.

Build (only when triggered)

projects/ml-platform/
├── src/batch_job/
│   └── worker.py               # Celery task: score_chunk — drain-and-exit;
│                               #   marks child rows via store.py (unchanged API)
└── infra/modules/broker/       # (applied only when fan-out forces it)
    ├── main.tf                 # managed Redis Cache + Key Vault secret for connection string
    ├── variables.tf
    └── outputs.tf              # redis_hostname, redis_url_secret_name

The existing infra/modules/batch_job/ gains a KEDA-triggered execution mode (documented in broker/main.tf as a manual az containerapp job update step until the azurerm provider exposes KEDA scale rules). Everything else — results DB, serving, dashboard, alerts — is unchanged.

How the pieces connect

Trigger to adopt

Only when sustained fan-out routinely exceeds a few hundred concurrent units and the results-DB continuation rule (Ch 04) demonstrably cannot keep up. Document the threshold and get explicit approval — do not adopt preemptively.

worker.py — Celery as a library, not a fleet

score_chunk is a Celery task whose lifecycle is: - KEDA detects queue depth on the Redis list → scales the ACA Job execution count. - Each Job execution starts a Celery worker process, drains tasks from the queue, and exits — no long-lived daemon. - On each task: store.mark(child_id, "STARTED") → predict → store.mark("SUCCESS"). On BatchItemFailure (permanent): mark FAILURE. On any other exception: Celery retries up to max_retries=3 with exponential backoff; after exhaustion the child is RETRY/FAILURE per the continuation rule. - task_acks_late=True means a crashed worker does not silently drop the task — it goes back on the queue.

Because the task marks results via store.py (same API as Ch 04), the dashboard, alerts, and finalize_parent work identically — only the dispatch mechanism changed.

infra/modules/broker/

azurerm_redis_cache (Standard C1 by default; upgrade at admission). The primary access key is stored in Key Vault (redis-url secret) so workers fetch it via DefaultAzureCredential at runtime — never in env or config files.

The KEDA ScaledJob annotation on the batch ACA Job is applied via a one-time az containerapp job update call after terraform apply (documented in broker/main.tf as a comment; will move into Terraform once the azurerm provider exposes it).

What does not change

Results DB schema, parent/child rows, finalize_parent, dashboard reads, alert rules — all unchanged. The broker is a drop-in dispatch replacement, not an architectural shift. If you later remove the broker (e.g., fan-out shrinks), the results-DB continuation rule in Ch 04 takes over again with no schema migration.

## Boundary

Boundary Contract
Long-running worker fleets docs/00 invariant 3 — never
Broker in the baseline docs/04 — only if forced by real load

Next: 10 — End-to-end integration walks the whole golden path and summarizes every deferred extension.

Back to top