07. LLM release artifacts

Phase 5 — package an LLM app as an MLflow pyfunc version, evaluate it with a repeatable job, and release it through the same promotion path as any model.

Outcome

An LLM workflow (prompt + model + retrieval/config) is packaged as a single MLflow pyfunc model version, evaluated by a repeatable evaluator, and released by the same promotion mechanism as any other model. There is no separate LLM control path — an LLM app is just a kind of model version in the self-hosted registry. This is Phase 5 (docs/03), and it reuses the registry, promotion, serving, and batch paths built in Chapters 03–06.

## Design — an LLM app is a pyfunc version

The pyfunc artifact bundles everything that defines the app:

  • the prompt template(s) and configuration,
  • the model/endpoint reference and generation parameters,
  • any retrieval/index configuration,
  • the signature (inputs/outputs) and dependencies.

Registering it produces a version number in the same registry classical models use. Serving and batch inference load it by models:/<name>/<version> exactly like any other model.

  • Evaluation is a repeatable job. An ACA Job scores the candidate version against a fixed evaluation set with defined metrics (quality/faithfulness, plus cost/latency where relevant); results are logged to MLflow and summarized in a results-DB record. Promotion requires meeting recorded thresholds.
  • Config travels inside the artifact; secrets do not. Prompts/config are in the artifact so a version is self-contained. External model/API credentials resolve at runtime from Key Vault via managed identity — never embedded in the artifact or image.
  • No bespoke release ledger. A release is: registered version + its evaluation record + a Git tag + the Job/App definition digest that references it.

Build in projects/ml-platform/

projects/ml-platform/
├── src/ml_platform/llm/
│   ├── model.py                # LLMPyfunc(PythonModel): load prompt/config artifacts,
│   │                           #   call OpenAI-compat endpoint, return uniform response
│   ├── artifact_builder.py     # build_and_register(): log prompt.yaml + config.yaml,
│   │                           #   package + register the pyfunc version
│   └── evaluator.py            # score candidate vs fixed eval JSONL → exact_match,
│   │                           #   latency, token counts; gates promotion
└── src/train_job/
    └── register_llm.py         # CLI entrypoint: configure_mlflow → build_and_register
                                #   → record_run (same pattern as train.py)

No new serving, batch, or infra is needed — an LLM version is loaded by models:/name/version and served / batch-scored identically to a classical model. The LLM-specific code lives entirely in ml_platform/llm/; the rest of the platform is unchanged.

Credentials (API key) are resolved at runtime from Key Vault or the MODEL_API_KEY env var — never embedded in the artifact or image.

How the pieces connect

Pyfunc artifact (ml_platform/llm/model.py)

LLMPyfunc is an mlflow.pyfunc.PythonModel subclass with two methods:

  • load_context — reads prompt.yaml and config.yaml from the artifact directory, populates _system_prompt, _user_template, _endpoint, and _gen_params. No network calls at load time; credentials are resolved lazily.
  • predict — formats each input row via the user template, calls _call_openai_compat (a thin httpx.post against the /chat/completions endpoint), returns a DataFrame with content, model, prompt_tokens, completion_tokens.

The credential lookup chain: MODEL_API_KEY env → Key Vault secret named by MODEL_API_KEY_SECRET env, fetched with DefaultAzureCredential. The artifact contains no secret values.

Artifact builder (ml_platform/llm/artifact_builder.py)

build_and_register(registered_name, prompt_yaml_path=…, config_yaml_path=…) opens an MLflow run, logs the prompt/config files as lineage artefacts, records model_endpoint, model_id, and temperature as params (no secrets), and calls mlflow.pyfunc.log_model with the LLMPyfunc instance, both artifact paths, and the enforced input/output signature. Returns the ModelVersion.

A stub canary_predict(model_uri) loads the registered version and runs one prediction — the same pattern as evaluate.py’s held-out check.

Evaluator (ml_platform/llm/evaluator.py)

Runs as an ACA Job bound to id-jobs-train (shares the image with register_llm.py):

  1. Loads models:/<name>/<version> via mlflow.pyfunc.load_model.
  2. Reads a JSONL eval file ({"input": "…", "expected": "…"}).
  3. Runs all predictions, measures latency and token counts.
  4. Computes exact_match (where expected is provided).
  5. Logs all metrics + a per-row CSV to the MLflow run; sets gate_result=PASS|FAIL as a run tag.
  6. Applies threshold gates (--min-exact-match, --max-avg-tokens); exits non-zero on miss so promotion is blocked.
  7. Writes a results-DB record via record_run — the dashboard sees this job the same way it sees a training run.

Registration entrypoint (src/train_job/register_llm.py)

Thin CLI (mirrors train.py) that calls configure_mlflow → build_and_register → record_run. Runs inside the existing train-job image if ml_platform.llm is packaged there, or as a separate image if dependencies diverge.

No new infra

Serving (serving_app) and batch (batch_job) call mlflow.sklearn.load_model today but are easy to widen to mlflow.pyfunc.load_model — one-line change, same models:/name/version URI. The registry, promotion path, and results DB are identical for classical and LLM versions.

## Golden-path position & acceptance evidence

This chapter feeds a second kind of producer into the same registered version → eval → promote → serve/batch path — no new branch.

Acceptance evidence:

  • An LLM app is registered as a pyfunc version and loads via models:/name/version in both the serving App and a batch Job with no serving/batch code change.
  • The evaluator runs as a Job, logs metrics to MLflow + a results-DB record, and its thresholds gate promotion.
  • Credentials are resolved from Key Vault at runtime; the artifact contains no secrets.

## Extensions (deferred from the MVP)

Deferred Contract MVP substitute
Rich evaluator evidence contract docs/03 Threshold pass/fail + logged metrics
Provider/model upgrade workflow docs/03 Re-register a new pyfunc version
Retrieval index lifecycle docs/03 Static index config in the artifact

Next: the off-critical-path tracks — 08 — Multi-GPU training and 09 — Broker upgrade — then 10 — End-to-end integration.

Back to top