LLM Regression Testing: When Reasoning Ate Our Tokens

A player types "I cast Firebolt at the goblin". Our game turns that sentence into a structured action, the rules engine resolves it, and a narrator describes the result. After a routine model upgrade, a growing share of those spells quietly became plain weapon attacks. Nothing crashed, no error reached the player, and the unit tests stayed green.

This post covers the LLM regression testing setup we used to confirm the cause: a replay bench built on real D&D turns, a second model that filters out turns nobody could label, and the numbers before and after a one-line fix. If you parse player input with an LLM, your game can probably fail the same way.

Where an LLM sits in our combat loop

LoreKeeper is a tabletop RPG where an AI plays Dungeon Master. Combat follows d20 rules: initiative, conditions, criticals, area effects. Its core rule is mechanics first, narrative second. The engine resolves the dice and applies damage before any language model writes a word, and the narration wraps an outcome that is already confirmed.

Players can build their own worlds or play presets, including Mokgard, the setting of our turn-based tactics game. Either way, the mechanics-first rule needs a bridge between free text and the engine. Players don't press "Cast spell, slot 3, target 2". They write a sentence, and something has to read it.

A combat round in LoreKeeper. Before it can roll anything, the engine needs one concrete action: what, with which spell or item, at whom.

Campaign play in LoreKeeper. Players answer "What does Bromwe do?" in their own words, and the engine has to turn that sentence into a rule it can resolve.

The bridge is a combat-intent extractor. It takes the player's text plus the character's grimoire (the spells they know), inventory, allies and enemies, and returns a small JSON object:

type CombatIntent = {
  actionType: 'attack' | 'spell' | 'defend' | 'flee' | 'help' | 'use_item'
            | 'dash' | 'hide' | 'demoralize' | 'skip';
  spellId: string | null;   // resolved against the character's grimoire
  itemId: string | null;
  targetId: string | null;  // resolved against allies/enemies in the encounter
  confidence: number;       // 0..1
};

The extraction runs as a short chain:

1. A small, fast classifier (Jev, a decision model from TypeSafe) tries first, capped at 1.5 seconds. When it is confident, we use its answer.
2. When it declines, an LLM gets the same input with a strict JSON response format, temperature 0.1 and a 200-token output budget. If the answer comes back truncated, we retry once with double the budget (400).
3. If the LLM also fails (error, truncation or empty content), we fall back to a weapon attack with confidence 0.

Step 3 is the trap. A default attack is a reasonable fallback, because the turn always resolves and the game never stalls. It also means that from the player's side, a broken extractor looks almost exactly like a working one.

The regression: a model upgrade that turned spells into attacks

We moved the extractor's LLM step from Claude Sonnet 4.6 to Claude Sonnet 5, served through OpenRouter. Our token logs showed what happened next. For the week of 2026-11-02, out of 45 LLM calls for this task:

• 36 hit the 200-token ceiling,
• 8 also hit the 400-token ceiling on the retry and failed for good.

With Sonnet 4.6, the same call averaged about 100 output tokens. The JSON was the same size; the extra tokens went into reasoning. OpenRouter served this model with extended reasoning enabled even though our request never asked for it, so the model spent its entire output budget thinking and returned an empty content field. Empty content meant the fallback ran, and the fallback meant an attack.

A wizard who typed a spell got a staff swing. The narrator would then describe that staff swing, because narration follows mechanics. The combat looked coherent, just wrong.

We had seen this before. In October, Claude Haiku 5.5 on one OpenRouter provider did the same thing. That time we added the model to an app-wide list of models that reason without being asked, and our LLM client turns reasoning off for every model on that list. The obvious move was to add Sonnet 5 to the list. We didn't, because the same model serves several callers with different needs. Sonnet 5 also writes long-form DM narration and generates worlds, and those callers have large budgets and may benefit from reasoning. A global switch would have changed them without any test covering it.

The fix went on the one call that was broken:

const response = await ai.generateResponse({
  systemPrompt: COMBAT_INTENT_PROMPT,
  userMessage,
  model: MODELS.strategist,
  maxTokens: 200,
  temperature: 0.1,
  response_format: { type: 'json_object' },
  // The provider serves this model with reasoning nobody asked for: it spent the
  // whole budget thinking, returned empty content, and the turn fell back to attack.
  disableReasoning: true,
  retryOnTruncate: true,
});

Inside the client, that flag becomes reasoning: { enabled: false } in the request body when the call goes through OpenRouter. A unit test asserts the extractor always sends it, so the next refactor can't silently drop it. See OpenRouter's reasoning tokens documentation for the parameter and its variants.

The flag was the easy part. Before shipping it we needed proof that it fixed the problem and didn't introduce another one.

Building an LLM regression test from real player turns

Unit tests with mocked LLM responses can't catch this kind of bug. The mock returns whatever we wrote into it, and the problem was in what the real model and provider sent back. We needed to run the real extractor against realistic input and score it against labels we didn't write ourselves.

The dataset: FIREBALL

FIREBALL (Zhu et al., ACL 2023, CC-BY-4.0) is a dataset of real Dungeons & Dragons play on Discord, recorded from groups using Avrae, a Discord bot that rolls dice and runs D&D actions from typed commands. For each turn it contains what players wrote, the Avrae command that was actually run afterward, and the game state.

That pairing is exactly what we needed. The free text is our extractor's input, and the command is a gold label that nobody on our team chose. Two illustrative examples:

A case-builder script downloads 40 fixed sessions and builds 150 attack cases and 150 spell cases. It uses a seeded pseudo-random generator, so every run produces the same 300 cases. Each case maps the FIREBALL game state onto our extractor's input: the actor's spell list becomes a grimoire, combatants become allies and enemies.

Filtering with a judge from another model family

Our first scores were misleading because many Discord messages before a command don't say what the character does. They are table talk: rules questions, jokes, comments about another player. No reader, human or model, can recover "cast Fire Bolt" from "lol did you see that crit".

So the bench asks a second model, from a different family than the extractor, one question per text: does this text state the character's action, and is it an attack, a spell or something else? A different family matters because a judge from the same family tends to share the extractor's blind spots.

Of the 300 texts, 180 were table talk. Our headline metric, which we call clean accuracy, uses only the cases where the judge says the text declares an action and the kind of action it read matches the command. That left 83 cases where you can actually read the label from the text. Cases where text and command disagree, for example a player who describes one spell and casts another, are reported separately.

Making the bench cheap to re-run

The bench caches both the judge's answers and the extractor's answers on disk. After a change to the extractor, a refresh flag tells the bench to re-ask only the extractor and reuse the judge's labels. We paid for the judge once. Later runs only pay for the extractor calls.

Requests run through a small concurrency pool, and every extractor call is wrapped in a watchdog timer:

let watchdog: NodeJS.Timeout | undefined;
const result = await Promise.race([
  extractCombatIntent(testCase.input),
  new Promise<never>((_, reject) => {
    watchdog = setTimeout(
      () => reject(new Error(`extractor never settled on ${testCase.id}`)),
      60_000,
    );
  }),
]).finally(() => clearTimeout(watchdog));

We learned that one the hard way. A promise that never settles, and that holds no timer, lets Node drain its event loop and exit with code 0 halfway through the run. No report and no error, just a process that ended. The watchdog keeps the loop alive and names the case that hung.

Results: before and after the fix

Same commit, same 300 cases, reasoning on (the regression) versus reasoning off (the fix). The third row repeats the fixed run to check stability.

The three failures in the repeat run were provider timeouts at our 8-second limit. None of them were empty responses. Our target for this bench is at least 90% clean action accuracy and zero extractor failures, so the repeat run's 3 timeouts still leave work to do.

Spell recognition went from 76% to 88%. Latency at the 90th percentile dropped from 8.8 to 2.0 seconds. That gap is the time players spent waiting for a model to reason before the game ignored the result.

Why attack accuracy went down

Attack accuracy fell from 42/42 to 40/42 after the fix. We don't count that as a regression. Before the fix, every failed extraction became an attack. On attack cases the broken extractor was right by default: a 100% attack score partly measured how often the fallback fired.

This is worth checking in any system with a default class. When the fallback equals one of your labels, accuracy on that label is inflated by failures. Look at the failure count next to the per-class scores, or score with the fallback disabled.

What we'd do differently

Alert on the fallback rate like you alert on errors. The extractor logged every fallback with its reason (empty-response, llm-truncated, llm-error), but nothing alerted on them. A safe fallback that runs on 8 calls out of 45 is an incident.

Track output tokens against the budget per task. The clearest signal was in the token logs before anyone reported a bug: 36 of 45 calls ending exactly at the ceiling. A call whose output length sits at max_tokens is truncated or wasting the budget, and either way something changed.

Pin behavior per call, not per model. Providers change their defaults without telling you, and the same model ID can start reasoning, refuse differently or get slower. Settings that a call depends on (reasoning, JSON mode, temperature) belong in that call, with a test that asserts them.

Know the limits of your labels. FIREBALL is English-only, our bench omits some context the real game sends (the DM's last message, full inventories), and Avrae's attack command is also used for class features, so ally-targeting labels are noisy. We report those metrics but don't gate releases on them.

Run the bench when a model changes, not after a bug report. The upgrade that broke this was a configuration change, and model upgrades are where LLM features regress. We built this bench after the regression reached production. Built earlier and run on every model swap, it would have caught the problem before players did.

FAQ

What is LLM regression testing?

It is re-running a fixed set of realistic inputs through your LLM feature after any change (prompt, model, provider, parameters) and comparing scores with the previous run. Unlike unit tests with mocked responses, it exercises the real model, so it catches behavior changes that live outside your code.

Why did a reasoning model return empty content?

Reasoning models spend output tokens on internal reasoning before writing the answer. With a small max_tokens budget, reasoning can use all of it and leave nothing for the visible answer. Raise the budget, cap the reasoning, or turn reasoning off for short structured-extraction calls that don't need it.

Do I need a labeled dataset to start?

It helps a lot. Public datasets that pair natural language with the action actually taken, like FIREBALL for D&D, give you labels you didn't invent. Without one, log real inputs from your game, label a few hundred by hand, and freeze them as your bench.

Takeaway

The bug wasn't in our code, and our fallback made sure no one would notice it. We confirmed the cause and measured the fix with a bench built on real player text and labels we didn't write, with a judge filtering out the turns nobody could label. If an LLM sits between your players and your rules engine, build that bench before your next model upgrade.

LoreKeeper's combat runs on this extractor every turn. Try it at lore-keeper.com.

Resources

Libraries, datasets and docs mentioned in this post.

• FIREBALL paper (ACL 2023): Zhu et al., the dataset of real D&D play on Discord with Avrae commands and game state.
• FIREBALL dataset on Hugging Face: the raw sessions our bench downloads (CC-BY-4.0).
• OpenRouter: reasoning tokens: how to enable, cap or disable reasoning per request.
• TypeSafe docs (Jev): the decision model that runs before the LLM in our extractor.