Pretraining Evaluation and Analysis

Build a composable harness for perplexity, calibration, memorization, overlap, and generation diversity.

A lower next-token loss is evidence about one prediction task, not a complete description of a language model. This chapter builds the evaluation harness that later posttraining chapters can reuse. The NumPy functions keep calibration, canary, overlap, and diversity definitions inspectable; the shared ProofLM evaluators attach explicit checkpoint-stage identities to masked likelihood and verifier reports.

Each metric is a small function with explicit inputs and no hidden model state. That design makes it possible to rerun the same suite after a checkpoint, a data change, or a posttraining update and to report the measurements together rather than selecting a single favorable score.

Perplexity

For target tokens y_i and logits z_i, the token negative log-likelihood is

\operatorname{NLL}(z,y)=-\frac{1}{n}\sum_{i=1}^n\log\operatorname{softmax}(z_i)_{y_i},

and perplexity is \exp(\operatorname{NLL}). Aggregate the numerator and token count before exponentiating. Averaging sequence-level perplexities would give short sequences the same weight as long ones and would not equal the corpus-level likelihood.

The evaluator below accepts records with split, source, and token IDs, plus a predictor that returns one logit vector for each input token. The model is fitted only on training records so the grouping can expose both source and split effects.

import numpy as np


SEED = 71


def log_softmax(logits):
    logits = np.asarray(logits, dtype=float)
    shifted = logits - np.max(logits, axis=-1, keepdims=True)
    return shifted - np.log(np.exp(shifted).sum(axis=-1, keepdims=True))


def token_nlls(logits, targets):
    logits = np.asarray(logits, dtype=float)
    targets = np.asarray(targets, dtype=np.int64)
    if logits.ndim != 2 or len(targets) != len(logits):
        raise ValueError("logits must be (tokens, vocab) and targets must match")
    return -log_softmax(logits)[np.arange(len(targets)), targets]


def perplexity(total_nll, token_count):
    if token_count < 1:
        raise ValueError("token_count must be positive")
    return float(np.exp(total_nll / token_count))


def fit_bigram_predictor(records, vocab_size, smoothing=0.1):
    counts = np.full((vocab_size, vocab_size), smoothing, dtype=float)
    for record in records:
        tokens = np.asarray(record["tokens"], dtype=np.int64)
        np.add.at(counts, (tokens[:-1], tokens[1:]), 1.0)
    probabilities = counts / counts.sum(axis=1, keepdims=True)

    def predict(previous_tokens):
        return np.log(probabilities[np.asarray(previous_tokens, dtype=np.int64)])

    return predict


def evaluate_by_group(records, predictor):
    totals = {}
    for record in records:
        tokens = np.asarray(record["tokens"], dtype=np.int64)
        losses = token_nlls(predictor(tokens[:-1]), tokens[1:])
        key = (record["split"], record["source"])
        total_nll, count = totals.get(key, (0.0, 0))
        totals[key] = (total_nll + float(losses.sum()), count + len(losses))
    return {
        key: {
            "nll": total_nll / count,
            "tokens": count,
            "perplexity": perplexity(total_nll, count),
        }
        for key, (total_nll, count) in sorted(totals.items())
    }


records = [
    {"split": "train", "source": "prose", "tokens": np.array([0, 1, 2, 1, 3, 1, 2])},
    {"split": "train", "source": "code", "tokens": np.array([0, 4, 5, 4, 6, 4])},
    {"split": "validation", "source": "prose", "tokens": np.array([0, 1, 3, 1, 2, 1])},
    {"split": "test", "source": "code", "tokens": np.array([0, 5, 4, 6, 4, 5])},
]
predictor = fit_bigram_predictor(records[:2], vocab_size=7)
perplexity_report = evaluate_by_group(records, predictor)
for key, result in perplexity_report.items():
    print(f"{key}: tokens={result['tokens']}, nll={result['nll']:.4f}, ppl={result['perplexity']:.3f}")
assert set(perplexity_report) == {
    ("train", "prose"), ("train", "code"), ("validation", "prose"), ("test", "code")
}
assert all(result["perplexity"] >= 1.0 for result in perplexity_report.values())
assert all(result["tokens"] > 0 for result in perplexity_report.values())

('test', 'code'): tokens=5, nll=1.1925, ppl=3.295
('train', 'code'): tokens=5, nll=0.7129, ppl=2.040
('train', 'prose'): tokens=6, nll=0.6857, ppl=1.985
('validation', 'prose'): tokens=5, nll=0.7096, ppl=2.033

The grouped report keeps token counts beside NLL and perplexity, so a reader can see how much evidence supports each number. Training-like rows should usually be easier for a fitted predictor than shifted rows, but that difference is not itself a capability measure. Source grouping can reveal that an aggregate validation value is being driven by one easy source while another source has much higher uncertainty.

The predictor is a closure over a fixed probability table, while the evaluator knows nothing about how that table was made. Later chapters can replace the predictor with a Transformer checkpoint without changing the grouping logic.

Calibration

For a next-token prediction, let c_i be the maximum predicted probability and let r_i indicate whether the selected token is correct. Partition confidences into bins I_b. The expected calibration error is

\operatorname{ECE}=\sum_b\frac{|I_b|}{n} \left|\frac{1}{|I_b|}\sum_{i\in I_b}r_i -\frac{1}{|I_b|}\sum_{i\in I_b}c_i\right|.

A reliability table must retain count, mean confidence, and empirical accuracy for every nonempty bin. ECE compresses those discrepancies to one number, so the table remains the primary diagnostic.

def reliability_bins(confidences, correct, bin_count=5):
    confidences = np.asarray(confidences, dtype=float)
    correct = np.asarray(correct, dtype=float)
    if confidences.shape != correct.shape:
        raise ValueError("confidence and correctness arrays must match")
    if np.any((confidences < 0.0) | (confidences > 1.0)):
        raise ValueError("confidence values must be in [0, 1]")
    assignments = np.minimum((confidences * bin_count).astype(int), bin_count - 1)
    rows = []
    for bin_index in range(bin_count):
        selected = assignments == bin_index
        if not selected.any():
            continue
        rows.append({
            "bin": bin_index,
            "count": int(selected.sum()),
            "confidence": float(confidences[selected].mean()),
            "accuracy": float(correct[selected].mean()),
        })
    return rows


def expected_calibration_error(confidences, correct, bin_count=5):
    confidences = np.asarray(confidences, dtype=float)
    correct = np.asarray(correct, dtype=float)
    rows = reliability_bins(confidences, correct, bin_count)
    total = len(confidences)
    return float(sum(
        row["count"] / total * abs(row["accuracy"] - row["confidence"])
        for row in rows
    ))


confidences = np.array([0.12, 0.18, 0.32, 0.38, 0.62, 0.68, 0.82, 0.92])
correct = np.array([0, 1, 0, 1, 1, 1, 1, 0])
calibration_table = reliability_bins(confidences, correct, bin_count=5)
calibration_error = expected_calibration_error(confidences, correct, bin_count=5)
for row in calibration_table:
    print(row)
print("ECE:", round(calibration_error, 4))
assert sum(row["count"] for row in calibration_table) == len(confidences)
assert 0.0 <= calibration_error <= 1.0
assert np.isclose(
    expected_calibration_error(np.array([0.5, 0.5]), np.array([1, 0]), bin_count=2),
    0.0,
)

{'bin': 0, 'count': 2, 'confidence': 0.15, 'accuracy': 0.5}
{'bin': 1, 'count': 2, 'confidence': 0.35, 'accuracy': 0.5}
{'bin': 3, 'count': 2, 'confidence': 0.65, 'accuracy': 1.0}
{'bin': 4, 'count': 2, 'confidence': 0.87, 'accuracy': 0.5}
ECE: 0.305

The table distinguishes two kinds of calibration failure. A model can be overconfident when its accuracy is below its mean confidence, or underconfident when accuracy is higher. ECE weights each discrepancy by the number of predictions in its bin, but it does not identify which examples caused the discrepancy. Keep the bin rows and the prediction-level data when calibration affects a later call-or-no-call decision.

Memorization canaries

Insert a unique string into the training corpus and evaluate the conditional probability of its continuation from a prefix. Compare a model fitted with the canary against the same model fitted without it, holding the alphabet and smoothing constant. The probe is deliberately local: a high recovery probability shows that the model retained the sequence, not that it can generalize the surrounding task.

A character bigram is sufficient to make the exposure effect measurable. Repeating the canary increases the counts of its internal transitions, while the held-out control shares the general format but not the exact transition.

def fit_character_bigram(texts, alphabet, smoothing=0.05):
    symbol_to_id = {symbol: index for index, symbol in enumerate(alphabet)}
    counts = np.full((len(alphabet), len(alphabet)), smoothing, dtype=float)
    for text in texts:
        tokens = np.asarray([symbol_to_id[symbol] for symbol in text], dtype=np.int64)
        np.add.at(counts, (tokens[:-1], tokens[1:]), 1.0)
    return counts / counts.sum(axis=1, keepdims=True), symbol_to_id


def continuation_logprob(probabilities, prefix, continuation):
    if not prefix:
        raise ValueError("prefix must contain one symbol")
    previous = prefix[-1]
    total = 0.0
    for token in continuation:
        total += np.log(probabilities[previous, token])
        previous = token
    return float(total)


def greedy_continuation(probabilities, prefix, length):
    generated = list(prefix)
    for _ in range(length):
        generated.append(int(np.argmax(probabilities[generated[-1]])))
    return generated


base_text = "the model reads data and reports loss. " * 6
canary = "<Q7|zebra|X9>"
control = "<Q7|zebra|Y9>"
alphabet = sorted(set(base_text + canary + control))
canary_ids = np.asarray([alphabet.index(symbol) for symbol in canary], dtype=np.int64)
control_ids = np.asarray([alphabet.index(symbol) for symbol in control], dtype=np.int64)

without_canary, symbol_to_id = fit_character_bigram([base_text], alphabet)
with_canary, _ = fit_character_bigram([base_text, canary], alphabet)
prefix = canary_ids[:-2].tolist()
continuation = canary_ids[-2:].tolist()
seen_score = continuation_logprob(with_canary, prefix, continuation)
unseen_score = continuation_logprob(without_canary, prefix, continuation)
recovered = greedy_continuation(with_canary, prefix, len(continuation))
print("canary continuation log-prob, seen:", round(seen_score, 4))
print("canary continuation log-prob, unseen:", round(unseen_score, 4))
print("greedy recovery:", recovered == canary_ids.tolist())
print("control differs at final transition:", control_ids[-3:].tolist())
assert seen_score > unseen_score
assert recovered == canary_ids.tolist()

print("exposure by repetition count")
exposure_scores = []
for repetitions in (1, 3, 8):
    model, _ = fit_character_bigram([base_text] + [canary] * repetitions, alphabet)
    score = continuation_logprob(model, prefix, continuation)
    exposure_scores.append(score)
    print(f"repetitions={repetitions}: log-prob={score:.4f}")
assert exposure_scores[0] < exposure_scores[-1]

canary continuation log-prob, seen: -1.4793
canary continuation log-prob, unseen: -6.3561
greedy recovery: True
control differs at final transition: [8, 3, 5]
exposure by repetition count
repetitions=1: log-prob=-1.4793
repetitions=3: log-prob=-0.6399
repetitions=8: log-prob=-0.2671

The seen model assigns more probability to the canary continuation and greedily recovers it, while the model without the inserted sequence has only the smoothing floor for the rare final transitions. The repetition sweep makes exposure a graded quantity rather than a binary label. This result is a memorization measurement under a specified model and prefix; it does not establish that every occurrence of the string would be recoverable from every prompt.

N-gram overlap

For a generated sequence g and a set of training sequences D, define the n-gram overlap rate as

\operatorname{overlap}_n(g,D)= \frac{\#\{u\in\operatorname{ngrams}_n(g):u\in\operatorname{ngrams}_n(D)\}} {\#\operatorname{ngrams}_n(g)}.

Use unique candidate n-grams in the numerator so a repeated phrase is not counted several times merely because it appears repeatedly in the same generation. Report several values of n: unigram overlap mostly reflects vocabulary, while longer overlap is more sensitive to copied local sequences.

def ngram_set(sequence, n):
    sequence = tuple(sequence)
    if n < 1:
        raise ValueError("n must be positive")
    return {
        sequence[index:index + n]
        for index in range(max(0, len(sequence) - n + 1))
    }


def ngram_overlap(generated, training_sequences, n=3):
    candidate = ngram_set(generated, n)
    if not candidate:
        return 0.0
    training = set().union(*(ngram_set(sequence, n) for sequence in training_sequences))
    return len(candidate & training) / len(candidate)


def overlap_report(generations, training_sequences, orders=(1, 2, 3)):
    return {
        n: float(np.mean([
            ngram_overlap(generation, training_sequences, n)
            for generation in generations
        ]))
        for n in orders
    }


training_sequences = [
    "the cat sat on the mat".split(),
    "the dog ran to the park".split(),
    "a model predicts the next token".split(),
]
generations = [
    "the cat sat on the mat".split(),
    "the model predicts a token".split(),
    "new birds fly above water".split(),
]
overlap = overlap_report(generations, training_sequences)
for order, value in overlap.items():
    print(f"mean unique {order}-gram overlap: {value:.3f}")
assert ngram_overlap(training_sequences[0], training_sequences, n=3) == 1.0
assert ngram_overlap(generations[-1], training_sequences, n=3) == 0.0
assert set(overlap) == {1, 2, 3}

mean unique 1-gram overlap: 0.667
mean unique 2-gram overlap: 0.417
mean unique 3-gram overlap: 0.333

The overlap report preserves the order of the comparison. A generation can have high unigram overlap because it uses the training vocabulary while having low trigram overlap, or it can copy a long phrase while changing individual word frequencies. No order alone distinguishes useful reuse from memorization, so interpret the rates with canary probes and held-out loss.

Generation diversity

A generation set can be diverse because its samples differ, because its token inventory is broad, or because it avoids repeating short phrases. Measure these separately:

• distinct_n is the fraction of unique n-grams among all generated n-grams.
• unique_sequence_ratio is the fraction of samples that are distinct as whole sequences.
• mean_pairwise_jaccard compares n-gram sets between pairs and detects collections that reuse the same local phrases.

These metrics describe output variation. They do not establish correctness, factuality, or calibration.

def distinct_n(generations, n):
    all_ngrams = [ngram for generation in generations for ngram in ngram_set(generation, n)]
    total = sum(max(0, len(generation) - n + 1) for generation in generations)
    return len(set(all_ngrams)) / total if total else 0.0


def unique_sequence_ratio(generations):
    return len({tuple(generation) for generation in generations}) / len(generations)


def mean_pairwise_jaccard(generations, n=2):
    sets = [ngram_set(generation, n) for generation in generations]
    scores = []
    for left in range(len(sets)):
        for right in range(left + 1, len(sets)):
            union = sets[left] | sets[right]
            scores.append(len(sets[left] & sets[right]) / len(union) if union else 1.0)
    return float(np.mean(scores)) if scores else 0.0


def generation_diversity(generations):
    return {
        "distinct-1": distinct_n(generations, 1),
        "distinct-2": distinct_n(generations, 2),
        "unique-sequence-ratio": unique_sequence_ratio(generations),
        "mean-pairwise-2gram-jaccard": mean_pairwise_jaccard(generations, n=2),
    }


repetitive_generations = [
    "the cat sat on the mat".split(),
    "the cat sat on the mat".split(),
    "the cat sat near the mat".split(),
]
varied_generations = [
    "red birds fly above water".split(),
    "small models read heldout data".split(),
    "green fish swim below ice".split(),
]
repetitive_metrics = generation_diversity(repetitive_generations)
varied_metrics = generation_diversity(varied_generations)
print("repetitive:", {key: round(value, 3) for key, value in repetitive_metrics.items()})
print("varied:    ", {key: round(value, 3) for key, value in varied_metrics.items()})
assert repetitive_metrics["unique-sequence-ratio"] < varied_metrics["unique-sequence-ratio"]
assert varied_metrics["distinct-2"] > repetitive_metrics["distinct-2"]

repetitive: {'distinct-1': 0.333, 'distinct-2': 0.467, 'unique-sequence-ratio': 0.667, 'mean-pairwise-2gram-jaccard': 0.619}
varied:     {'distinct-1': 1.0, 'distinct-2': 1.0, 'unique-sequence-ratio': 1.0, 'mean-pairwise-2gram-jaccard': 0.0}

The repetitive set has duplicate whole sequences and shared local phrases, so its unique-sequence ratio and distinct-2 score are lower. The varied set scores better on those dimensions, but a diverse set could still be uniformly incorrect. Keep diversity as a companion to likelihood, calibration, and task-specific checks rather than using it as a quality proxy.

Evaluation suite

A reusable harness returns a structured report whose fields can be compared across checkpoints. The composition function below does not average unlike quantities. It keeps grouped perplexity, the calibration table and ECE, memorization evidence, overlap by order, and diversity metrics in separate namespaces. Later stages can add instruction accuracy or tool-call validity without changing these primitive functions.

def evaluation_report(records, predictor, confidences, correct, training_sequences, generations):
    return {
        "perplexity_by_group": evaluate_by_group(records, predictor),
        "calibration": {
            "ece": expected_calibration_error(confidences, correct),
            "bins": reliability_bins(confidences, correct),
        },
        "ngram_overlap": overlap_report(generations, training_sequences),
        "generation_diversity": generation_diversity(generations),
    }


suite_report = evaluation_report(
    records,
    predictor,
    confidences,
    correct,
    training_sequences,
    generations,
)
print("report sections:", list(suite_report))
print("calibration ECE:", round(suite_report["calibration"]["ece"], 4))
print("generation diversity:", {
    key: round(value, 3)
    for key, value in suite_report["generation_diversity"].items()
})
assert set(suite_report) == {
    "perplexity_by_group", "calibration", "ngram_overlap", "generation_diversity"
}
assert suite_report["calibration"]["bins"] == calibration_table

report sections: ['perplexity_by_group', 'calibration', 'ngram_overlap', 'generation_diversity']
calibration ECE: 0.305
generation diversity: {'distinct-1': 0.875, 'distinct-2': 1.0, 'unique-sequence-ratio': 1.0, 'mean-pairwise-2gram-jaccard': 0.0}

Shared evaluator identity

The pure metric functions above define the measurements. The package-backed evaluators add the artifact boundary: masked likelihood reports an evaluator identity, while proof validity is scored by the independent verifier rather than by the model’s own likelihood. These compact reports are the interface consumed by later posttraining chapters.

import torch

from proof_lm.data import batch_packed_examples, pack_causal_examples
from proof_lm.evaluation import evaluate_language_model, evaluate_proof_fixtures
from proof_lm.logic import generate_examples
from proof_lm.model import DecoderConfig, ProofLM

torch.manual_seed(71)
identity_model = ProofLM(
    DecoderConfig(
        vocab_size=16,
        context_length=8,
        n_layers=1,
        d_model=32,
        n_heads=4,
        d_ff=64,
    )
)
identity_examples = pack_causal_examples(
    [[1, 2, 3, 4, 5], [6, 7, 8]],
    eos_id=15,
    pad_id=0,
    context_length=8,
)
identity_batches = batch_packed_examples(identity_examples, batch_size=2)
likelihood_report = evaluate_language_model(identity_model, identity_batches)
proof_report = evaluate_proof_fixtures(generate_examples(count=6, seed=71))
print("likelihood evaluator:", likelihood_report["evaluator_id"])
print("masked perplexity:", round(float(likelihood_report["perplexity"]), 4))
print("proof evaluator:", proof_report["evaluator_id"])
print(
    "proof validity / rejection:",
    round(float(proof_report["positive_validity"]), 3),
    round(float(proof_report["negative_rejection"]), 3),
)
assert str(likelihood_report["evaluator_id"]).startswith("eval-")
assert str(proof_report["evaluator_id"]).startswith("eval-")
assert likelihood_report["tokens"] > 0
assert proof_report["positive_validity"] == 1.0
assert proof_report["negative_rejection"] == 1.0

likelihood evaluator: eval-4eea5ac0c26bf06f
masked perplexity: 17.2463
proof evaluator: eval-ab7e61e86fa5f72d
proof validity / rejection: 1.0 1.0

The final object is intentionally a report of vectors, tables, and rates rather than a leaderboard scalar. A later checkpoint can be evaluated with the same records, prompts, and generation settings, then compared field by field. That comparison makes a regression visible when validation perplexity improves but calibration, canary recovery, or copied n-grams worsen.

Summary

• Perplexity is computed from aggregate token NLL and is grouped by split and source so data volume and domain effects remain visible.
• Reliability bins pair mean confidence with empirical correctness; ECE summarizes but does not replace the bin table.
• A canary inserted into training provides a controlled conditional-recovery probe whose exposure can be varied by repetition count.
• N-gram overlap measures local reuse at several orders, while diversity metrics separate token variety, whole-sequence variety, and shared phrases.
• The evaluation suite is composed from small pure functions and returns separate namespaces, allowing later posttraining stages to rerun the same measurements.

Chapters 08 through 12 add posttraining objectives and tool-use behavior; each stage should rerun this base suite and add its own task-specific checks.

Exercises

Use the exercises to test the chapter’s invariants and connect the derivations to the reusable implementation. Solutions are hidden in the notebook source and are available through the course tooling when needed.

[P7.1] Expected calibration error

Calibration calculation. Three predictions have confidences [0.2, 0.4, 0.9] and correctness [1, 0, 0]. Using three bins with one prediction in each, compute the ECE and identify whether the high-confidence prediction is overconfident.

[P7.2] Generation memorization audit

Generation audit. Explain why a high distinct-2 score does not prove correctness. Then state how n-gram overlap and a canary probe provide different evidence about memorization.

[P7.3]

Evaluator identity. Explain why likelihood and proof validity need separate evaluator IDs even when they are reported for the same checkpoint. Give one example of a regression that a lower perplexity would fail to reveal.

def evaluator_regression(lower_perplexity, proof_validity):
    # Return whether the checkpoint passes both independent gates.
    pass