03. Reproducible training & registry

Phase 1 — training and evaluation as ACA Jobs that log to self-hosted MLflow with reconstructable lineage and a results-DB record.

Outcome

Training and evaluation run as ordinary ACA Jobs from pinned images, log to the self-hosted MLflow, and produce a registered model version whose lineage — code digest, tracked dataset, params, metrics — can be reconstructed from MLflow alone. Re-running the same inputs yields the same registered artifact, and every run is also visible in the results DB. This is Phase 1 (docs/02).

What “model” means here: anything with learned weights — a scikit-learn/ XGBoost estimator and a fine-tuned transformer classifier — travels the identical registry path. An LLM app (prompt + config, no weights trained) is packaged differently but registered in the same registry (Ch 07).

## Design — training and evaluation are ACA Jobs

A training run is a Manual- or Schedule-triggered ACA Job execution from a pinned image digest. The script:

  1. Builds an MLflow dataset from its source with mlflow.data, capturing the source location, a content digest, and the schema — not just “the latest table”.
  2. Starts an MLflow run, logging params, the code image digest, and the dataset via mlflow.log_input(dataset, context="training").
  3. Trains, evaluates on a held-out set, logs metrics and artifacts.
  4. Registers the model as a new version in the self-hosted registry.
  5. Writes a results-DB record (name='train:<model>', status, output carrying the MLflow run ID and registered version).
import mlflow

dataset = mlflow.data.from_pandas(raw_df, source=blob_path, name="fraud-train")
with mlflow.start_run() as run:
    mlflow.log_input(dataset, context="training")
    mlflow.log_params(params)
    mlflow.set_tag("code.image_digest", image_digest)
    model = train(...)
    mlflow.log_metrics(evaluate(model, holdout))
    mlflow.sklearn.log_model(model, name="model",
                             registered_model_name="fraud")

Evaluation can run inline or as a separate eval Job that scores a candidate version and writes its own results-DB record; promotion requires meeting recorded thresholds.

## Build in projects/ml-platform/

projects/ml-platform/
├── src/train_job/
│   ├── Dockerfile              # pinned base + training deps + ml_platform
│   ├── requirements.txt        # mlflow pinned to the server's version
│   ├── train.py                # entrypoint: dataset → run → register
│   └── evaluate.py             # held-out metrics; separate eval Job
├── src/ml_platform/common/
│   ├── mlflow_client.py        # configure_mlflow(): tracking + registry URI
│   ├── datasets.py             # load_csv + tracked_dataset (mlflow.data)
│   ├── schemas.py              # Pandera contract, validated before training
│   └── results.py              # record_run(): results-DB row per Job (Ch 04)
└── infra/modules/train_job/    # ACA Job: image digest, env, id-jobs-train

The Job runs as id-jobs-train. Everything it needs is environment, not secrets: MLFLOW_TRACKING_URI (the self-hosted App), PGHOST/PGUSER/RESULTS_DB, and IMAGE_DIGEST (recorded on the run). Postgres auth is an Entra access token fetched at runtime by DefaultAzureCredential — there is no password in the image. results.record_run(...) is a no-op until Phase 2 wires PGHOST (Ch 04), so the training path is runnable now and gains its operational record later without a code change.

## How the pieces connect

The entrypoint is thin because the reusable machinery lives in common/, shared with every later Job (batch, eval):

  • common/mlflow_client.py — configure_mlflow(experiment) points both the tracking and registry URIs at the one self-hosted server and raises if MLFLOW_TRACKING_URI is unset. One call, and mlflow.* and the registry agree.
  • common/datasets.py — load_csv(source, delimiter=…) reads the source (URL or path); tracked_dataset(df, source=…, name=…, targets=…) wraps mlflow.data.from_pandas so the run captures source + content digest + schema, not “whatever the table held today”.
  • common/schemas.py — validate(df) runs the Pandera wine_quality_schema (11 physicochemical floats + integer quality) at the boundary, before training. Bad data fails the run early instead of poisoning a registered version.
  • common/results.py — record_run("train:<model>") is a context manager that opens a results-DB row (RUNNING → SUCCESS/FAILURE) and yields a mutable output dict. train.py fills it with the MLflow run id and the registered version, so the operational record points straight at the lineage.

train.py then just sequences them: configure_mlflow → load_csv → validate → tracked_dataset → train_test_split → inside record_run(...) and mlflow.start_run(): log_input the dataset, log_params, tag code.image_digest, fit an ElasticNet, log rmse/mae/r2, then log_model(..., registered_model_name=…). evaluate.py loads models:/<name>/<version>, scores the held-out split, and exits non-zero below the threshold so promotion (Ch 07) only proceeds on a passing gate.

Runnable now, no cloud: the defaults pull the UCI wine-quality (white) CSV, so python train.py produces a registered version against any reachable MLflow tracking server.

## Golden-path position & acceptance evidence

This chapter builds the tracked dataset → train Job → registered version → eval Job segment of the golden path.

Acceptance evidence:

  • A run appears in MLflow with its dataset digest, code image digest, params, and metrics; the model is a registered version (models:/fraud/<n>).
  • The same inputs re-run produce an equivalent registered artifact (reproducible lineage), and the run is queryable in the results DB by name/status.
  • Evaluation metrics are recorded and gate promotion.

## Extensions (deferred from the MVP)

Deferred Contract MVP substitute
Rich stage-identity chain (many lineage fields) docs/00 Dataset digest + code image digest + version
Distributed / multi-GPU training docs/08 Single-node ACA Job (see Ch 08)
Full evaluator package + evidence contract docs/03 Held-out metrics + threshold check
Scheduled retrains as a workflow docs/04 Manual az containerapp job start (see Ch 04)

Next: 04 — Results DB & batch workflows turns run-recording into the operational backbone and adds scheduling and batch fan-out.

Back to top