[{"data":1,"prerenderedAt":1170},["ShallowReactive",2],{"blog-\u002Fblog\u002Fllm-regression-testing-game-ai":3,"blog-surround-\u002Fblog\u002Fllm-regression-testing-game-ai":1161},{"id":4,"title":5,"author":6,"body":7,"cover":53,"date":1145,"description":1146,"draft":1147,"extension":1148,"game":1149,"meta":1150,"navigation":1151,"path":1152,"seo":1153,"stem":1154,"tags":1155,"__hash__":1160},"blog\u002Fblog\u002Fllm-regression-testing-game-ai.md","LLM Regression Testing: When Reasoning Ate Our Tokens","Iron Lizard Games",{"type":8,"value":9,"toc":1125},"minimark",[10,14,22,27,39,47,58,67,70,290,293,321,324,328,332,341,352,355,362,365,536,549,552,556,559,564,572,575,619,622,626,629,632,643,647,650,653,862,869,873,876,971,974,977,981,984,991,995,1012,1022,1028,1034,1040,1044,1048,1051,1055,1061,1065,1068,1072,1075,1083,1087,1090,1121],[11,12,13],"p",{},"A player types \"I cast Firebolt at the goblin\". Our game turns that sentence into a structured action, the rules engine resolves it, and a narrator describes the result. After a routine model upgrade, a growing share of those spells quietly became plain weapon attacks. Nothing crashed, no error reached the player, and the unit tests stayed green.",[11,15,16,17,21],{},"This post covers the ",[18,19,20],"strong",{},"LLM regression testing"," setup we used to confirm the cause: a replay bench built on real D&D turns, a second model that filters out turns nobody could label, and the numbers before and after a one-line fix. If you parse player input with an LLM, your game can probably fail the same way.",[23,24,26],"h2",{"id":25},"where-an-llm-sits-in-our-combat-loop","Where an LLM sits in our combat loop",[11,28,29,34,35,38],{},[30,31,33],"a",{"href":32},"\u002Fgame\u002Flore-keeper","LoreKeeper"," is a tabletop RPG where an AI plays Dungeon Master. Combat follows d20 rules: initiative, conditions, criticals, area effects. Its core rule is ",[18,36,37],{},"mechanics first, narrative second",". The engine resolves the dice and applies damage before any language model writes a word, and the narration wraps an outcome that is already confirmed.",[11,40,41,42,46],{},"Players can build their own worlds or play presets, including Mokgard, the setting of our ",[30,43,45],{"href":44},"\u002Fgame\u002Fmokgard","turn-based tactics game",". Either way, the mechanics-first rule needs a bridge between free text and the engine. Players don't press \"Cast spell, slot 3, target 2\". They write a sentence, and something has to read it.",[11,48,49,54],{},[50,51],"img",{"alt":52,"src":53},"A LoreKeeper combat round on the tactical grid: two goblins next to the player's hero, the turn panel with movement and action bars, and the narration log on the right","\u002Fimg\u002Fblog\u002Fllm-regression-testing-game-ai\u002Fcombat-round-grid.webp",[55,56,57],"em",{},"A combat round in LoreKeeper. Before it can roll anything, the engine needs one concrete action: what, with which spell or item, at whom.",[11,59,60,64],{},[50,61],{"alt":62,"src":63},"A LoreKeeper campaign turn: the hero panel on the left, the Narrator's prose in the center, and the free-text box at the bottom asking \"What does Bromwe do?\"","\u002Fimg\u002Fblog\u002Fllm-regression-testing-game-ai\u002Ffree-text-turn-play-screen.webp",[55,65,66],{},"Campaign play in LoreKeeper. Players answer \"What does Bromwe do?\" in their own words, and the engine has to turn that sentence into a rule it can resolve.",[11,68,69],{},"The bridge is a combat-intent extractor. It takes the player's text plus the character's grimoire (the spells they know), inventory, allies and enemies, and returns a small JSON object:",[71,72,77],"pre",{"className":73,"code":74,"language":75,"meta":76,"style":76},"language-ts shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","type CombatIntent = {\n  actionType: 'attack' | 'spell' | 'defend' | 'flee' | 'help' | 'use_item'\n            | 'dash' | 'hide' | 'demoralize' | 'skip';\n  spellId: string | null;   \u002F\u002F resolved against the character's grimoire\n  itemId: string | null;\n  targetId: string | null;  \u002F\u002F resolved against allies\u002Fenemies in the encounter\n  confidence: number;       \u002F\u002F 0..1\n};\n","ts","",[78,79,80,100,167,210,233,249,268,284],"code",{"__ignoreMap":76},[81,82,85,89,93,97],"span",{"class":83,"line":84},"line",1,[81,86,88],{"class":87},"spNyl","type",[81,90,92],{"class":91},"sBMFI"," CombatIntent",[81,94,96],{"class":95},"sMK4o"," =",[81,98,99],{"class":95}," {\n",[81,101,103,107,110,113,117,120,123,125,128,130,132,134,137,139,141,143,146,148,150,152,155,157,159,161,164],{"class":83,"line":102},2,[81,104,106],{"class":105},"swJcz","  actionType",[81,108,109],{"class":95},":",[81,111,112],{"class":95}," '",[81,114,116],{"class":115},"sfazB","attack",[81,118,119],{"class":95},"'",[81,121,122],{"class":95}," |",[81,124,112],{"class":95},[81,126,127],{"class":115},"spell",[81,129,119],{"class":95},[81,131,122],{"class":95},[81,133,112],{"class":95},[81,135,136],{"class":115},"defend",[81,138,119],{"class":95},[81,140,122],{"class":95},[81,142,112],{"class":95},[81,144,145],{"class":115},"flee",[81,147,119],{"class":95},[81,149,122],{"class":95},[81,151,112],{"class":95},[81,153,154],{"class":115},"help",[81,156,119],{"class":95},[81,158,122],{"class":95},[81,160,112],{"class":95},[81,162,163],{"class":115},"use_item",[81,165,166],{"class":95},"'\n",[81,168,170,173,175,178,180,182,184,187,189,191,193,196,198,200,202,205,207],{"class":83,"line":169},3,[81,171,172],{"class":95},"            |",[81,174,112],{"class":95},[81,176,177],{"class":115},"dash",[81,179,119],{"class":95},[81,181,122],{"class":95},[81,183,112],{"class":95},[81,185,186],{"class":115},"hide",[81,188,119],{"class":95},[81,190,122],{"class":95},[81,192,112],{"class":95},[81,194,195],{"class":115},"demoralize",[81,197,119],{"class":95},[81,199,122],{"class":95},[81,201,112],{"class":95},[81,203,204],{"class":115},"skip",[81,206,119],{"class":95},[81,208,209],{"class":95},";\n",[81,211,213,216,218,221,223,226,229],{"class":83,"line":212},4,[81,214,215],{"class":105},"  spellId",[81,217,109],{"class":95},[81,219,220],{"class":91}," string",[81,222,122],{"class":95},[81,224,225],{"class":91}," null",[81,227,228],{"class":95},";",[81,230,232],{"class":231},"sHwdD","   \u002F\u002F resolved against the character's grimoire\n",[81,234,236,239,241,243,245,247],{"class":83,"line":235},5,[81,237,238],{"class":105},"  itemId",[81,240,109],{"class":95},[81,242,220],{"class":91},[81,244,122],{"class":95},[81,246,225],{"class":91},[81,248,209],{"class":95},[81,250,252,255,257,259,261,263,265],{"class":83,"line":251},6,[81,253,254],{"class":105},"  targetId",[81,256,109],{"class":95},[81,258,220],{"class":91},[81,260,122],{"class":95},[81,262,225],{"class":91},[81,264,228],{"class":95},[81,266,267],{"class":231},"  \u002F\u002F resolved against allies\u002Fenemies in the encounter\n",[81,269,271,274,276,279,281],{"class":83,"line":270},7,[81,272,273],{"class":105},"  confidence",[81,275,109],{"class":95},[81,277,278],{"class":91}," number",[81,280,228],{"class":95},[81,282,283],{"class":231},"       \u002F\u002F 0..1\n",[81,285,287],{"class":83,"line":286},8,[81,288,289],{"class":95},"};\n",[11,291,292],{},"The extraction runs as a short chain:",[294,295,296,307,314],"ol",{},[297,298,299,300,306],"li",{},"A small, fast classifier (",[30,301,305],{"href":302,"rel":303},"https:\u002F\u002Fdocs.typesafe.ai",[304],"nofollow","Jev",", a decision model from TypeSafe) tries first, capped at 1.5 seconds. When it is confident, we use its answer.",[297,308,309,310,313],{},"When it declines, an LLM gets the same input with a strict JSON response format, temperature 0.1 and a ",[18,311,312],{},"200-token output budget",". If the answer comes back truncated, we retry once with double the budget (400).",[297,315,316,317,320],{},"If the LLM also fails (error, truncation or empty content), we fall back to a ",[18,318,319],{},"weapon attack with confidence 0",".",[11,322,323],{},"Step 3 is the trap. A default attack is a reasonable fallback, because the turn always resolves and the game never stalls. It also means that from the player's side, a broken extractor looks almost exactly like a working one.",[23,325,327],{"id":326},"the-regression-a-model-upgrade-that-turned-spells-into-attacks","The regression: a model upgrade that turned spells into attacks",[11,329,330,331],{},"We moved the extractor's LLM step from Claude Sonnet 4.6 to Claude Sonnet 5, served through OpenRouter. "," Our token logs showed what happened next. For the week of 2026-11-02, out of 45 LLM calls for this task:",[333,334,335,338],"ul",{},[297,336,337],{},"36 hit the 200-token ceiling,",[297,339,340],{},"8 also hit the 400-token ceiling on the retry and failed for good.",[11,342,343,344,347,348,351],{},"With Sonnet 4.6, the same call averaged about 100 output tokens. The JSON was the same size; the extra tokens went into ",[18,345,346],{},"reasoning",". OpenRouter served this model with extended reasoning enabled even though our request never asked for it, so the model spent its entire output budget thinking and returned an empty ",[78,349,350],{},"content"," field. Empty content meant the fallback ran, and the fallback meant an attack.",[11,353,354],{},"A wizard who typed a spell got a staff swing. The narrator would then describe that staff swing, because narration follows mechanics. The combat looked coherent, just wrong.",[11,356,357,358,361],{},"We had seen this before. In October, Claude Haiku 5.5 on one OpenRouter provider did the same thing. That time we added the model to an app-wide list of models that reason without being asked, and our LLM client turns reasoning off for every model on that list. The obvious move was to add Sonnet 5 to the list. We didn't, because ",[18,359,360],{},"the same model serves several callers with different needs."," Sonnet 5 also writes long-form DM narration and generates worlds, and those callers have large budgets and may benefit from reasoning. A global switch would have changed them without any test covering it.",[11,363,364],{},"The fix went on the one call that was broken:",[71,366,368],{"className":73,"code":367,"language":75,"meta":76,"style":76},"const response = await ai.generateResponse({\n  systemPrompt: COMBAT_INTENT_PROMPT,\n  userMessage,\n  model: MODELS.strategist,\n  maxTokens: 200,\n  temperature: 0.1,\n  response_format: { type: 'json_object' },\n  \u002F\u002F The provider serves this model with reasoning nobody asked for: it spent the\n  \u002F\u002F whole budget thinking, returned empty content, and the turn fell back to attack.\n  disableReasoning: true,\n  retryOnTruncate: true,\n});\n",[78,369,370,401,414,421,438,451,463,488,493,499,513,525],{"__ignoreMap":76},[81,371,372,375,379,382,386,389,391,395,398],{"class":83,"line":84},[81,373,374],{"class":87},"const",[81,376,378],{"class":377},"sTEyZ"," response ",[81,380,381],{"class":95},"=",[81,383,385],{"class":384},"s7zQu"," await",[81,387,388],{"class":377}," ai",[81,390,320],{"class":95},[81,392,394],{"class":393},"s2Zo4","generateResponse",[81,396,397],{"class":377},"(",[81,399,400],{"class":95},"{\n",[81,402,403,406,408,411],{"class":83,"line":102},[81,404,405],{"class":105},"  systemPrompt",[81,407,109],{"class":95},[81,409,410],{"class":377}," COMBAT_INTENT_PROMPT",[81,412,413],{"class":95},",\n",[81,415,416,419],{"class":83,"line":169},[81,417,418],{"class":377},"  userMessage",[81,420,413],{"class":95},[81,422,423,426,428,431,433,436],{"class":83,"line":212},[81,424,425],{"class":105},"  model",[81,427,109],{"class":95},[81,429,430],{"class":377}," MODELS",[81,432,320],{"class":95},[81,434,435],{"class":377},"strategist",[81,437,413],{"class":95},[81,439,440,443,445,449],{"class":83,"line":235},[81,441,442],{"class":105},"  maxTokens",[81,444,109],{"class":95},[81,446,448],{"class":447},"sbssI"," 200",[81,450,413],{"class":95},[81,452,453,456,458,461],{"class":83,"line":251},[81,454,455],{"class":105},"  temperature",[81,457,109],{"class":95},[81,459,460],{"class":447}," 0.1",[81,462,413],{"class":95},[81,464,465,468,470,473,476,478,480,483,485],{"class":83,"line":270},[81,466,467],{"class":105},"  response_format",[81,469,109],{"class":95},[81,471,472],{"class":95}," {",[81,474,475],{"class":105}," type",[81,477,109],{"class":95},[81,479,112],{"class":95},[81,481,482],{"class":115},"json_object",[81,484,119],{"class":95},[81,486,487],{"class":95}," },\n",[81,489,490],{"class":83,"line":286},[81,491,492],{"class":231},"  \u002F\u002F The provider serves this model with reasoning nobody asked for: it spent the\n",[81,494,496],{"class":83,"line":495},9,[81,497,498],{"class":231},"  \u002F\u002F whole budget thinking, returned empty content, and the turn fell back to attack.\n",[81,500,502,505,507,511],{"class":83,"line":501},10,[81,503,504],{"class":105},"  disableReasoning",[81,506,109],{"class":95},[81,508,510],{"class":509},"sfNiH"," true",[81,512,413],{"class":95},[81,514,516,519,521,523],{"class":83,"line":515},11,[81,517,518],{"class":105},"  retryOnTruncate",[81,520,109],{"class":95},[81,522,510],{"class":509},[81,524,413],{"class":95},[81,526,528,531,534],{"class":83,"line":527},12,[81,529,530],{"class":95},"}",[81,532,533],{"class":377},")",[81,535,209],{"class":95},[11,537,538,539,542,543,548],{},"Inside the client, that flag becomes ",[78,540,541],{},"reasoning: { enabled: false }"," in the request body when the call goes through OpenRouter. A unit test asserts the extractor always sends it, so the next refactor can't silently drop it. See ",[30,544,547],{"href":545,"rel":546},"https:\u002F\u002Fopenrouter.ai\u002Fdocs\u002Fuse-cases\u002Freasoning-tokens",[304],"OpenRouter's reasoning tokens documentation"," for the parameter and its variants.",[11,550,551],{},"The flag was the easy part. Before shipping it we needed proof that it fixed the problem and didn't introduce another one.",[23,553,555],{"id":554},"building-an-llm-regression-test-from-real-player-turns","Building an LLM regression test from real player turns",[11,557,558],{},"Unit tests with mocked LLM responses can't catch this kind of bug. The mock returns whatever we wrote into it, and the problem was in what the real model and provider sent back. We needed to run the real extractor against realistic input and score it against labels we didn't write ourselves.",[560,561,563],"h3",{"id":562},"the-dataset-fireball","The dataset: FIREBALL",[11,565,566,571],{},[30,567,570],{"href":568,"rel":569},"https:\u002F\u002Faclanthology.org\u002F2023.acl-long.229",[304],"FIREBALL"," (Zhu et al., ACL 2023, CC-BY-4.0) is a dataset of real Dungeons & Dragons play on Discord, recorded from groups using Avrae, a Discord bot that rolls dice and runs D&D actions from typed commands. For each turn it contains what players wrote, the Avrae command that was actually run afterward, and the game state.",[11,573,574],{},"That pairing is exactly what we needed. The free text is our extractor's input, and the command is a gold label that nobody on our team chose. Two illustrative examples:",[576,577,578,591],"table",{},[579,580,581],"thead",{},[582,583,584,588],"tr",{},[585,586,587],"th",{},"Avrae command",[585,589,590],{},"Gold label",[592,593,594,607],"tbody",{},[582,595,596,602],{},[597,598,599],"td",{},[78,600,601],{},"!a longsword -t goblin",[597,603,604,606],{},[78,605,116],{},", target \"goblin\"",[582,608,609,614],{},[597,610,611],{},[78,612,613],{},"!cast fire bolt -t goblin",[597,615,616,618],{},[78,617,127],{},", spell \"Fire Bolt\", target \"goblin\"",[11,620,621],{},"A case-builder script downloads 40 fixed sessions and builds 150 attack cases and 150 spell cases. It uses a seeded pseudo-random generator, so every run produces the same 300 cases. Each case maps the FIREBALL game state onto our extractor's input: the actor's spell list becomes a grimoire, combatants become allies and enemies.",[560,623,625],{"id":624},"filtering-with-a-judge-from-another-model-family","Filtering with a judge from another model family",[11,627,628],{},"Our first scores were misleading because many Discord messages before a command don't say what the character does. They are table talk: rules questions, jokes, comments about another player. No reader, human or model, can recover \"cast Fire Bolt\" from \"lol did you see that crit\".",[11,630,631],{},"So the bench asks a second model, from a different family than the extractor, one question per text: does this text state the character's action, and is it an attack, a spell or something else? A different family matters because a judge from the same family tends to share the extractor's blind spots.",[11,633,634,635,638,639,642],{},"Of the 300 texts, ",[18,636,637],{},"180 were table talk",". Our headline metric, which we call clean accuracy, uses only the cases where the judge says the text declares an action ",[18,640,641],{},"and"," the kind of action it read matches the command. That left 83 cases where you can actually read the label from the text. Cases where text and command disagree, for example a player who describes one spell and casts another, are reported separately.",[560,644,646],{"id":645},"making-the-bench-cheap-to-re-run","Making the bench cheap to re-run",[11,648,649],{},"The bench caches both the judge's answers and the extractor's answers on disk. After a change to the extractor, a refresh flag tells the bench to re-ask only the extractor and reuse the judge's labels. We paid for the judge once. Later runs only pay for the extractor calls.",[11,651,652],{},"Requests run through a small concurrency pool, and every extractor call is wrapped in a watchdog timer:",[71,654,656],{"className":73,"code":655,"language":75,"meta":76,"style":76},"let watchdog: NodeJS.Timeout | undefined;\nconst result = await Promise.race([\n  extractCombatIntent(testCase.input),\n  new Promise\u003Cnever>((_, reject) => {\n    watchdog = setTimeout(\n      () => reject(new Error(`extractor never settled on ${testCase.id}`)),\n      60_000,\n    );\n  }),\n]).finally(() => clearTimeout(watchdog));\n",[78,657,658,683,705,720,757,770,814,821,828,837],{"__ignoreMap":76},[81,659,660,663,666,668,671,673,676,678,681],{"class":83,"line":84},[81,661,662],{"class":87},"let",[81,664,665],{"class":377}," watchdog",[81,667,109],{"class":95},[81,669,670],{"class":91}," NodeJS",[81,672,320],{"class":95},[81,674,675],{"class":91},"Timeout",[81,677,122],{"class":95},[81,679,680],{"class":91}," undefined",[81,682,209],{"class":95},[81,684,685,687,690,692,694,697,699,702],{"class":83,"line":102},[81,686,374],{"class":87},[81,688,689],{"class":377}," result ",[81,691,381],{"class":95},[81,693,385],{"class":384},[81,695,696],{"class":91}," Promise",[81,698,320],{"class":95},[81,700,701],{"class":393},"race",[81,703,704],{"class":377},"([\n",[81,706,707,710,713,715,718],{"class":83,"line":169},[81,708,709],{"class":393},"  extractCombatIntent",[81,711,712],{"class":377},"(testCase",[81,714,320],{"class":95},[81,716,717],{"class":377},"input)",[81,719,413],{"class":95},[81,721,722,725,727,730,733,736,738,740,744,747,750,752,755],{"class":83,"line":212},[81,723,724],{"class":95},"  new",[81,726,696],{"class":91},[81,728,729],{"class":95},"\u003C",[81,731,732],{"class":91},"never",[81,734,735],{"class":95},">",[81,737,397],{"class":377},[81,739,397],{"class":95},[81,741,743],{"class":742},"sHdIc","_",[81,745,746],{"class":95},",",[81,748,749],{"class":742}," reject",[81,751,533],{"class":95},[81,753,754],{"class":87}," =>",[81,756,99],{"class":95},[81,758,759,762,764,767],{"class":83,"line":235},[81,760,761],{"class":377},"    watchdog",[81,763,96],{"class":95},[81,765,766],{"class":393}," setTimeout",[81,768,769],{"class":105},"(\n",[81,771,772,775,777,779,781,784,787,789,792,795,798,801,803,806,809,812],{"class":83,"line":251},[81,773,774],{"class":95},"      ()",[81,776,754],{"class":87},[81,778,749],{"class":393},[81,780,397],{"class":105},[81,782,783],{"class":95},"new",[81,785,786],{"class":393}," Error",[81,788,397],{"class":105},[81,790,791],{"class":95},"`",[81,793,794],{"class":115},"extractor never settled on ",[81,796,797],{"class":95},"${",[81,799,800],{"class":377},"testCase",[81,802,320],{"class":95},[81,804,805],{"class":377},"id",[81,807,808],{"class":95},"}`",[81,810,811],{"class":105},"))",[81,813,413],{"class":95},[81,815,816,819],{"class":83,"line":270},[81,817,818],{"class":447},"      60_000",[81,820,413],{"class":95},[81,822,823,826],{"class":83,"line":286},[81,824,825],{"class":105},"    )",[81,827,209],{"class":95},[81,829,830,833,835],{"class":83,"line":495},[81,831,832],{"class":95},"  }",[81,834,533],{"class":377},[81,836,413],{"class":95},[81,838,839,842,844,847,849,852,854,857,860],{"class":83,"line":501},[81,840,841],{"class":377},"])",[81,843,320],{"class":95},[81,845,846],{"class":393},"finally",[81,848,397],{"class":377},[81,850,851],{"class":95},"()",[81,853,754],{"class":87},[81,855,856],{"class":393}," clearTimeout",[81,858,859],{"class":377},"(watchdog))",[81,861,209],{"class":95},[11,863,864,865,868],{},"We learned that one the hard way. A promise that never settles, and that holds no timer, lets Node drain its event loop and ",[18,866,867],{},"exit with code 0 halfway through the run",". No report and no error, just a process that ended. The watchdog keeps the loop alive and names the case that hung.",[23,870,872],{"id":871},"results-before-and-after-the-fix","Results: before and after the fix",[11,874,875],{},"Same commit, same 300 cases, reasoning on (the regression) versus reasoning off (the fix). The third row repeats the fixed run to check stability.",[576,877,878,903],{},[579,879,880],{},[582,881,882,885,888,891,894,897,900],{},[585,883,884],{},"Run",[585,886,887],{},"Clean action accuracy",[585,889,890],{},"Attack",[585,892,893],{},"Spell",[585,895,896],{},"Spell ID correct",[585,898,899],{},"Extractor failures",[585,901,902],{},"p90 latency",[592,904,905,928,951],{},[582,906,907,910,913,916,919,922,925],{},[597,908,909],{},"Baseline (reasoning on)",[597,911,912],{},"73\u002F83 (88%)",[597,914,915],{},"42\u002F42",[597,917,918],{},"31\u002F41 (76%)",[597,920,921],{},"23\u002F31",[597,923,924],{},"22\u002F300",[597,926,927],{},"8.8 s",[582,929,930,933,936,939,942,945,948],{},[597,931,932],{},"Fix (reasoning off)",[597,934,935],{},"76\u002F83 (92%)",[597,937,938],{},"40\u002F42",[597,940,941],{},"36\u002F41 (88%)",[597,943,944],{},"26\u002F36",[597,946,947],{},"0\u002F300",[597,949,950],{},"2.0 s",[582,952,953,956,959,962,964,966,969],{},[597,954,955],{},"Fix, repeat run",[597,957,958],{},"77\u002F83 (93%)",[597,960,961],{},"41\u002F42",[597,963,941],{},[597,965,944],{},[597,967,968],{},"3\u002F300",[597,970,950],{},[11,972,973],{},"The three failures in the repeat run were provider timeouts at our 8-second limit. None of them were empty responses. Our target for this bench is at least 90% clean action accuracy and zero extractor failures, so the repeat run's 3 timeouts still leave work to do.",[11,975,976],{},"Spell recognition went from 76% to 88%. Latency at the 90th percentile dropped from 8.8 to 2.0 seconds. That gap is the time players spent waiting for a model to reason before the game ignored the result.",[560,978,980],{"id":979},"why-attack-accuracy-went-down","Why attack accuracy went down",[11,982,983],{},"Attack accuracy fell from 42\u002F42 to 40\u002F42 after the fix. We don't count that as a regression. Before the fix, every failed extraction became an attack. On attack cases the broken extractor was right by default: a 100% attack score partly measured how often the fallback fired.",[11,985,986,987,990],{},"This is worth checking in any system with a default class. ",[18,988,989],{},"When the fallback equals one of your labels, accuracy on that label is inflated by failures."," Look at the failure count next to the per-class scores, or score with the fallback disabled.",[23,992,994],{"id":993},"what-wed-do-differently","What we'd do differently",[11,996,997,1000,1001,1004,1005,1004,1008,1011],{},[18,998,999],{},"Alert on the fallback rate like you alert on errors."," The extractor logged every fallback with its reason (",[78,1002,1003],{},"empty-response",", ",[78,1006,1007],{},"llm-truncated",[78,1009,1010],{},"llm-error","), but nothing alerted on them. A safe fallback that runs on 8 calls out of 45 is an incident.",[11,1013,1014,1017,1018,1021],{},[18,1015,1016],{},"Track output tokens against the budget per task."," The clearest signal was in the token logs before anyone reported a bug: 36 of 45 calls ending exactly at the ceiling. A call whose output length sits at ",[78,1019,1020],{},"max_tokens"," is truncated or wasting the budget, and either way something changed.",[11,1023,1024,1027],{},[18,1025,1026],{},"Pin behavior per call, not per model."," Providers change their defaults without telling you, and the same model ID can start reasoning, refuse differently or get slower. Settings that a call depends on (reasoning, JSON mode, temperature) belong in that call, with a test that asserts them.",[11,1029,1030,1033],{},[18,1031,1032],{},"Know the limits of your labels."," FIREBALL is English-only, our bench omits some context the real game sends (the DM's last message, full inventories), and Avrae's attack command is also used for class features, so ally-targeting labels are noisy. We report those metrics but don't gate releases on them.",[11,1035,1036,1039],{},[18,1037,1038],{},"Run the bench when a model changes, not after a bug report."," The upgrade that broke this was a configuration change, and model upgrades are where LLM features regress. We built this bench after the regression reached production. Built earlier and run on every model swap, it would have caught the problem before players did.",[23,1041,1043],{"id":1042},"faq","FAQ",[560,1045,1047],{"id":1046},"what-is-llm-regression-testing","What is LLM regression testing?",[11,1049,1050],{},"It is re-running a fixed set of realistic inputs through your LLM feature after any change (prompt, model, provider, parameters) and comparing scores with the previous run. Unlike unit tests with mocked responses, it exercises the real model, so it catches behavior changes that live outside your code.",[560,1052,1054],{"id":1053},"why-did-a-reasoning-model-return-empty-content","Why did a reasoning model return empty content?",[11,1056,1057,1058,1060],{},"Reasoning models spend output tokens on internal reasoning before writing the answer. With a small ",[78,1059,1020],{}," budget, reasoning can use all of it and leave nothing for the visible answer. Raise the budget, cap the reasoning, or turn reasoning off for short structured-extraction calls that don't need it.",[560,1062,1064],{"id":1063},"do-i-need-a-labeled-dataset-to-start","Do I need a labeled dataset to start?",[11,1066,1067],{},"It helps a lot. Public datasets that pair natural language with the action actually taken, like FIREBALL for D&D, give you labels you didn't invent. Without one, log real inputs from your game, label a few hundred by hand, and freeze them as your bench.",[23,1069,1071],{"id":1070},"takeaway","Takeaway",[11,1073,1074],{},"The bug wasn't in our code, and our fallback made sure no one would notice it. We confirmed the cause and measured the fix with a bench built on real player text and labels we didn't write, with a judge filtering out the turns nobody could label. If an LLM sits between your players and your rules engine, build that bench before your next model upgrade.",[11,1076,1077,1078,320],{},"LoreKeeper's combat runs on this extractor every turn. ",[30,1079,1082],{"href":1080,"rel":1081},"https:\u002F\u002Flore-keeper.com?utm_source=ironlizardweb&utm_medium=blog&utm_campaign=llm-regression-testing-game-ai",[304],"Try it at lore-keeper.com",[23,1084,1086],{"id":1085},"resources","Resources",[11,1088,1089],{},"Libraries, datasets and docs mentioned in this post.",[333,1091,1092,1099,1107,1114],{},[297,1093,1094,1098],{},[30,1095,1097],{"href":568,"rel":1096},[304],"FIREBALL paper (ACL 2023)",": Zhu et al., the dataset of real D&D play on Discord with Avrae commands and game state.",[297,1100,1101,1106],{},[30,1102,1105],{"href":1103,"rel":1104},"https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Flara-martin\u002FFIREBALL",[304],"FIREBALL dataset on Hugging Face",": the raw sessions our bench downloads (CC-BY-4.0).",[297,1108,1109,1113],{},[30,1110,1112],{"href":545,"rel":1111},[304],"OpenRouter: reasoning tokens",": how to enable, cap or disable reasoning per request.",[297,1115,1116,1120],{},[30,1117,1119],{"href":302,"rel":1118},[304],"TypeSafe docs (Jev)",": the decision model that runs before the LLM in our extractor.",[1122,1123,1124],"style",{},"html pre.shiki code .spNyl, html code.shiki .spNyl{--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA}html pre.shiki code .sBMFI, html code.shiki .sBMFI{--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B}html pre.shiki code .sMK4o, html code.shiki .sMK4o{--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF}html pre.shiki code .swJcz, html code.shiki .swJcz{--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178}html pre.shiki code .sfazB, html code.shiki .sfazB{--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D}html pre.shiki code .sHwdD, html code.shiki .sHwdD{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic}html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sTEyZ, html code.shiki .sTEyZ{--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8}html pre.shiki code .s7zQu, html code.shiki .s7zQu{--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#89DDFF;--shiki-default-font-style:italic;--shiki-dark:#89DDFF;--shiki-dark-font-style:italic}html pre.shiki code .s2Zo4, html code.shiki .s2Zo4{--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF}html pre.shiki code .sbssI, html code.shiki .sbssI{--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C}html pre.shiki code .sfNiH, html code.shiki .sfNiH{--shiki-light:#FF5370;--shiki-default:#FF9CAC;--shiki-dark:#FF9CAC}html pre.shiki code .sHdIc, html code.shiki .sHdIc{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#EEFFFF;--shiki-default-font-style:italic;--shiki-dark:#BABED8;--shiki-dark-font-style:italic}",{"title":76,"searchDepth":102,"depth":102,"links":1126},[1127,1128,1129,1134,1137,1138,1143,1144],{"id":25,"depth":102,"text":26},{"id":326,"depth":102,"text":327},{"id":554,"depth":102,"text":555,"children":1130},[1131,1132,1133],{"id":562,"depth":169,"text":563},{"id":624,"depth":169,"text":625},{"id":645,"depth":169,"text":646},{"id":871,"depth":102,"text":872,"children":1135},[1136],{"id":979,"depth":169,"text":980},{"id":993,"depth":102,"text":994},{"id":1042,"depth":102,"text":1043,"children":1139},[1140,1141,1142],{"id":1046,"depth":169,"text":1047},{"id":1053,"depth":169,"text":1054},{"id":1063,"depth":169,"text":1064},{"id":1070,"depth":102,"text":1071},{"id":1085,"depth":102,"text":1086},"2026-11-12","LLM regression testing for a game AI: how a model swap turned spells into attacks, the replay bench on real D&D turns that confirmed it, and the one-line fix.",false,"md","lore-keeper",{},true,"\u002Fblog\u002Fllm-regression-testing-game-ai",{"title":5,"description":1146},"blog\u002Fllm-regression-testing-game-ai",[1156,1157,1158,1159],"ai-systems","llm-evals","regression-testing","game-ai","cusgeG4Oj7V6uVtx1xYM7KRm85830IDygp8-LLuEogw",[1162,1166],{"title":1163,"path":1164,"stem":1165,"children":-1},"Game UI Redesign: From SaaS Dashboard to Title Screen","\u002Fblog\u002Fgame-ui-redesign-web-rpg","blog\u002Fgame-ui-redesign-web-rpg",{"title":1167,"path":1168,"stem":1169,"children":-1},"Mokgard Update 0.1.0 Beta: New UI and Campaign Mode","\u002Fblog\u002Fmokgard-update-0-1-0-beta","blog\u002Fmokgard-update-0-1-0-beta",1794149269924]