07. LLM release artifacts

Package an LLM app as an MLflow pyfunc version, evaluate it with a repeatable one-shot job on the local Compose stack, and release it through the same promotion path as any other model.

Outcome

An LLM workflow (prompt + external model endpoint + retrieval/config) is packaged as a single MLflow pyfunc model version, evaluated by a repeatable evaluator job, and released through the same promotion mechanism as any other model. There is no separate LLM control path: an LLM app is just another kind of version in the registry built in chapter 03, promoted exactly as chapter 05 promotes a classical model, and served and batch-scored by machinery chapters 04–06 already built. Every step in this chapter runs against the local Docker Compose stack from chapter 02.

Design — an LLM app is a pyfunc version

The pyfunc artifact bundles everything that defines the app:

  • the prompt template(s) and configuration,
  • the model/endpoint reference and generation parameters,
  • any retrieval/index configuration,
  • the signature (inputs/outputs) and dependencies.

Registering it produces a version number in the same registry classical models use. Serving and batch inference load it by models:/<name>/<version> exactly like any other model.

  • Evaluation is a repeatable job. A one-shot job scores the candidate version against a fixed evaluation set with defined metrics (exact match where expected outputs exist, plus latency and token cost); results are logged to MLflow and summarized in a results-DB record (chapter 04), and promotion requires meeting recorded thresholds. Locally it is the same one-shot pattern the demo stack runs train/batch with; Part II turns the same image into an ACA Job (chapter 11).
  • Config travels inside the artifact; secrets do not. Prompts/config live in the artifact so a version is self-contained. The external API key resolves at predict time from the environment (MODEL_API_KEY) when running locally; Part II replaces that variable with a Key Vault read via managed identity (chapter 10). Under either backend the key is never embedded in the artifact or image.
  • No bespoke release ledger. A release is: registered version + its evaluation record + a Git tag + the image digest that references it. Same provenance chain, same registry, same promotion semantics as every other model (chapter 08).

Build in projects/ml-platform/

projects/ml-platform/
├── src/ml_platform/llm/
│   ├── model.py                # LLMPyfunc(PythonModel): load prompt/config artifacts,
│   │                           #   call OpenAI-compat endpoint, return uniform response
│   ├── artifact_builder.py     # build_and_register(): log prompt.yaml + config.yaml,
│   │                           #   package + register the pyfunc version
│   └── evaluator.py            # score candidate vs fixed eval JSONL → exact_match,
│   │                           #   latency, token counts; gates promotion
└── src/train_job/
    └── register_llm.py         # CLI entrypoint: configure_mlflow → build_and_register
                                #   → record_run (same pattern as train.py)

No new service joins the Compose stack: an LLM version is loaded by models:/name/version and served / batch-scored identically to a classical model. The LLM-specific code lives entirely in ml_platform/llm/; the rest of the platform is unchanged. Both entrypoints (register_llm.py, evaluator.py) obey the same one-shot job contract as train/batch: read configuration from the environment, talk to MLFLOW_TRACKING_URI, write their results-DB record, exit non-zero on failure.

The external API key resolves at predict time: MODEL_API_KEY from the environment first (export it before triggering the job locally), then a Key Vault fallback gated on KEY_VAULT_URL that stays dormant until Part II wires it (chapter 11). With neither value present the call fails fast rather than issuing an unauthenticated request. The key is never embedded in the artifact or image.

How the pieces connect

Pyfunc artifact (ml_platform/llm/model.py)

LLMPyfunc is an mlflow.pyfunc.PythonModel subclass with two methods:

  • load_context reads prompt.yaml and config.yaml from the artifact directory, populates the prompt and generation configuration, and performs no network calls.
  • predict formats each input row via the user template, calls the OpenAI-compatible endpoint with httpx, and returns a DataFrame with content, model, prompt_tokens, and completion_tokens.

Credentials are resolved lazily. Local Compose supplies MODEL_API_KEY; Azure supplies KEY_VAULT_URL and MODEL_API_KEY_SECRET to the same pyfunc model, which resolves the secret through DefaultAzureCredential.

Artifact builder and evaluator

build_and_register logs the prompt/config and signature, then registers one pyfunc version. ml_platform.llm.evaluator loads that version with mlflow.pyfunc.load_model, evaluates JSONL rows, logs metrics, and writes the same results record used by training.

The local Compose profile uses the actual train image for both Jobs:

docker compose --profile llm run --rm llm-register
docker compose --profile llm run --rm llm-evaluate

The cloud llm_job Terraform adapter uses that same train image and changes only the command plus identity-backed environment. This is one workload path with two trigger/credential adapters, not a local-only LLM implementation.

One model-loading interface

Both classical and LLM consumers use mlflow.pyfunc.load_model. The shared adapter inspects the signature: tabular models receive feature frames with the target removed, while text models receive a DataFrame with an input column. Serving preserves tabular responses as scalar predictions and serializes pyfunc DataFrame responses as records. Batch behavior remains chunked and read-only to MLflow.

Golden-path position & acceptance evidence

This chapter feeds a second kind of producer into the same register → eval → promote → serve/batch path; no new branch appears. Promotion of an LLM version is deliberately not restated here: it is the alias flip plus pinned redeploy defined in chapter 05 and tabulated for both backends in chapter 08, wrapped by python demo/promote.py --model-name <llm-app> --version N (local backend; the same script’s --backend aca is the Part II form).

Acceptance evidence:

  • An LLM app registers as a pyfunc version and loads via models:/name/version in both the serving container and a batch job with no serving/batch code change.
  • The evaluator runs as a one-shot job against the local stack, logs metrics to MLflow plus a results-DB record, and its thresholds gate promotion.
  • Credentials enter through the environment at trigger time; the artifact and image contain no secrets. Part II changes only how that variable is delivered: managed identity plus Key Vault (chapter 10, chapter 11), same images otherwise.

Extensions (deferred from the MVP)

Deferred capability Current baseline
Rich evaluator evidence package Threshold pass/fail + logged metrics
Provider/model upgrade workflow Re-register a new pyfunc version
Retrieval index lifecycle Static index config in the artifact

Why not Azure Machine Learning’s built-in MLflow?

The demo pins mlflow==3.15.1. Moving up from 2.x buys newer lineage, evaluation, tracing, and prompt tooling; nothing taught here depends on those. Part II keeps the self-hosted MLflow container because aliases and the exact pyfunc surface are part of this course’s shared promotion interface.

The local Compose profile and the cloud llm_job adapter use the same registration and evaluation entrypoints. Their trigger mechanisms and credential delivery differ, but neither phase creates a local-only LLM implementation.

Off the critical path sits 14 — Multi-GPU training, the admitted escape hatch when an LLM is trained rather than wrapped and outgrows the laptop. 15 — End-to-end integration then compares the local and cloud behavioral checks described in chapter 08.

Back to top