diff --git a/AGENTS.md b/AGENTS.md index f7469c9f51..4fe6dc1ca2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -92,6 +92,12 @@ Provides project-specific utilities including: Imports and synchronizes Ghidra-exported data. +### Compiler Internals (`tools/compiler-internals`) + +Notes on the bundled MSVC backend itself - what `c2.dll` does, how its symbols and source layout were recovered, +the decoded stack-slot rule, and a `cdb` recipe for observing one function's allocation from inside the compiler. +Read it when a diff is down to register or stack-slot allocation and no source-level hypothesis is paying. + ### Reimplementation Control (`tools/reimplementation-control`) Scripts for enabling implementations and maintaining project state. @@ -169,7 +175,10 @@ restyling anything large: flags a `sub esp` frame-size mismatch (`FRAME`), byte-sized stack slots we have and the original does not (`BYTE`), a cold return block we inline where the original outlines it (`RET`), and the case where only register and stack-slot *naming* is left (`alloc-only`). Chase a `FRAME` flag with `diff_slots.py NAME`, which lists every `[esp + N]` slot per -side: one extra slot is one local the original does not have, and removing it has been worth 10-30 points. +side: one extra slot is often one local the original does not have, and removing it has been worth 10-30 points - +but locals with disjoint live ranges share a slot, so a slot count is not a local count. `slot_weight.py NAME.cpp` +reads MSVC's own `_name$ = N` table out of a `/FA` listing and says which local actually got which slot, instead +of inferring it from the diff. `diff_jcc.py` finds comparisons whose operator is one strictness step off the original (`jl`/`jle`, `jg`/`jge`) - these are real off-by-one bugs where Ghidra decompiled the condition wrongly, so confirm a fix by re-running that tool rather than by the percentage. `diff_types.py` is its counterpart away from comparisons: `movsx` against `movzx` for one field @@ -179,7 +188,11 @@ sign. It carries only those two rules on purpose: a third, "same instruction wit dropped because every hit it produced was a loop-induction stride rather than a constant the source chose. `reorder_search.py FILE NAME` hill-climbs the match % by reordering independent statements, which is the only lever left once a function is `alloc-only`; it pays about one move in ten, and mostly where the -statements sit between two calls or on a loop back-edge. +statements sit between two calls or on a loop back-edge. `alloc-only` is worth reading literally: a register +mismatch is most likely an interference difference - a different set of values was live at that point - rather +than a preference the source could state. That is a hypothesis, not a decoded rule; +`tools/compiler-internals/msvc-backend.md` has what is actually established about the compiler's allocator and +the debugger recipe that can settle it per function. ### Binary Comparison (`reccmp`) @@ -205,6 +218,12 @@ nothing getting worse. It is also the missing precondition for `cmake/compiler-f does nothing while the callee is still proxied. `tools/reimplementation-control/Enable-Reimplemented-Functions.ps1` flips them; like the struct resolvers they are never committed. Only enable a callee that is itself finished. +`structural_diff.py --all` splits the build list by *what kind* of difference is left - wrong instruction mix +(`mix`), right mix in the wrong order (`order`), or register naming only (`allocation`). Check it before picking +a target: the reccmp percentage is dominated by allocation, so it will rank a structurally wrong function above a +structurally perfect one. On `AI::AICState` it puts `aiGiveRaidInstructions` (delta 21) at the top while marking +`sellExcessGoods` - a 62.4% function - as having nothing structural left to fix. + ## Working on Many Functions When restyling or improving a large set of functions, work in batches (e.g. per folder) and verify each batch: @@ -326,7 +345,10 @@ Diff patterns that were reliable (more in the cheat sheet): - `diff_triage.py`'s `FRAME` column is worth checking but, unlike `BYTE`, is not mechanically actionable. Every cheap hypothesis for the frame mismatches in `Map::Units::UnitsState` came back an exact tie or worse: local declaration order, array element type, widening unsigned locals, and splitting a local the decompiler had reused - for two values. `diff_slots.py` says the lowest-access extra slot is usually the culprit, so a function whose + for two values. Declaration order was never going to work: the stack colorer sorts its elements by **reference + weight per byte**, not by declaration order, and leaves only the ties to the order the symbols happen to arrive + in - see `tools/compiler-internals/msvc-backend.md` for the comparator and the evidence. The lever on a slot layout is + how often, and at what loop depth, each local is read. `diff_slots.py` says the lowest-access extra slot is usually the culprit, so a function whose extra slots are all heavily used has genuine live values rather than a stray local, and needs structural understanding rather than a rule. - `jmp dword ptr [reg*4 + table]` on one side only is a dispatch-form mismatch: a `switch` over contiguous values diff --git a/IMPLEMENTATION_CHEAT_SHEET.md b/IMPLEMENTATION_CHEAT_SHEET.md index 1e436f53c0..4e7a7593b4 100644 --- a/IMPLEMENTATION_CHEAT_SHEET.md +++ b/IMPLEMENTATION_CHEAT_SHEET.md @@ -111,11 +111,33 @@ matched instruction for instruction except that the parameter was loaded into `e for the parameter load and took it from 95.7% to 100%. Uninitialised declarations do not have this effect - a sweep of `quick_reorder.py -s` over the declaration block of 27 functions found no win at all. -Stack *slot* assignment for spilled locals is a different matter and is not reachable from the source. Where the -original lays its locals out in Ghidra order (`local_1c` at the lowest `esp` offset through `local_4` at the -highest) and our build permutes them, neither declaration order, initialisation order, nor renaming changes -anything; this was tried on `setupSkirmishLobby`, `initMultiplayerLobbyState`, `initializeFinalResultsForActivePlayers` -and `createCompoundAddressBasedOnSelectedProvider`. Spend the effort elsewhere. +Stack *slot* assignment for spilled locals is a different matter: it follows the **reference weight** of each +local, not the order they are declared in. Neither declaration order, initialisation order, nor renaming moves a +slot - that was observed on `setupSkirmishLobby`, `initMultiplayerLobbyState`, +`initializeFinalResultsForActivePlayers` and `createCompoundAddressBasedOnSelectedProvider`, and the compiler +itself says why: its stack colorer sorts the elements by weight per byte and hands out offsets in that order. The +mechanism, the evidence and the debugger recipe are in +[tools/compiler-internals/msvc-backend.md](tools/compiler-internals/msvc-backend.md); what matters here is how to +read a diff: + +* A slot-order mismatch means our locals carry **different reference counts** from the original's, so look for a + read the original has and we do not. It is not a declaration-order problem, which is why every declaration-order + hypothesis for the `Map::Units::UnitsState` frame mismatches came back a tie. +* Weight **ties** are left to the order the symbols arrive in, permuted by an unstable quicksort. That is the + whole reason `perm_decls.py` took `assignSelectionToKey` from 95.9% to 100% once and does nothing in most + functions: permuting declarations can only decide ties. +* Locals with disjoint live ranges **share** one slot, and a local can be placed on the slot of a parameter that + is already dead (`writeMapOrSaveFile` puts eight locals on the first parameter's slot). A slot count is + therefore not a local count, and Ghidra's "repurposed parameter" rendering - `playerID = 0` used as a loop + counter in `aiRecruitUnits` - is this reuse, not something the original source wrote. Transcribing it back cost + 7 points. +* Renaming locals is inert; the comparator never looks at a name. + +`slot_weight.py` prints the slot table for one function with each local's weight beside it, read out of MSVC's own +`/FA` symbol table rather than inferred from the diff. + +For **registers** there is no decoded rule yet. Treat a register-only mismatch as an interference difference - a +different set of values was live at that point - which is a working hypothesis, not an established mechanism. ### Statement Order diff --git a/tools/compiler-internals/asm_scan.py b/tools/compiler-internals/asm_scan.py new file mode 100644 index 0000000000..75e21e4826 --- /dev/null +++ b/tools/compiler-internals/asm_scan.py @@ -0,0 +1,95 @@ +"""Find functions whose ORIGINAL contains handwritten inline assembly. + +usage: asm_scan.py [SNAPSHOT.json] [SRCDIR ...] + +Without arguments it scans every .cpp in cmake/openshc-sources.txt.local; with a reccmp_report.py +snapshot it scans only the functions below normalized 100 in it, and prints their percentages. + +Why this exists: tracePathPlanToDestinationViaUnoccupiedTiles sat at 12.5% with a frame nine +locals short, and no C++ change could touch it, because the original's loop is an `__asm` block. +MSVC keeps an ebp frame in any function containing inline assembly, and every local the block +names has to live in memory - so the "missing locals" were the asm's operands, and the pragma +that gets the frame pointer (`optimize("y", off)`) gets nothing else. LinkageNeighbourAsm.hpp +documents the same discovery for three siblings. Such a function has to be written as asm, +instruction for instruction; tuning the C++ around it is wasted. + +The fingerprints, each a thing MSVC's own codegen never emits: + + ebp an ebp frame (push ebp / mov ebp, esp) WITHOUT the `and esp, -8` of stack alignment + zx `mov eax, 0` immediately followed by `mov ax, word ptr ...` (MSVC writes movzx) + c16 16-bit register compares, `cmp ax, cx` and the like + rld layer bases reloaded from their own stack slot, `mov esi/edi, dword ptr [ebp - N]` + al0 `cmp al/cl/dl, 0` (MSVC writes test) + +The score is 3 for the frame plus 2 per zero-extension plus the others; anything scoring 3 or +more is listed. An ebp frame alone is also what `__asm`-free functions with a `double` or an +alloca get, so read the other columns before believing it. Over the 97 sub-100 PathFindingState +functions this listed exactly the four that are asm; the other 93 are ordinary C++ problems. +""" + +import io +import json +import os +import re +import subprocess +import sys + +ROOT = os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', '..')) +MARK = re.compile(r'FUNCTION:\s*STRONGHOLDCRUSADER\s+(0x[0-9A-Fa-f]+)') + + +def original(addr): + out = subprocess.run([sys.executable, 'tools/reimplementation-helper/batch/orig_body.py', addr], + cwd=ROOT, capture_output=True, text=True).stdout + return [re.sub(r'^[0-9a-f]{8}\s+', '', l) for l in out.splitlines() if re.match(r'^[0-9a-f]{8}\s', l)] + + +def fingerprints(ins): + n = len(ins) + ebp = n > 2 and ins[0].startswith('push ebp') and ins[1].startswith('mov ebp, esp') \ + and not any('and esp' in x for x in ins[:5]) + zx = sum(1 for i in range(n - 1) if re.match(r'mov\s+eax, 0$', ins[i]) and re.match(r'mov\s+ax,', ins[i + 1])) + c16 = sum(1 for x in ins if re.match(r'cmp\s+[a-d]x, [a-d]x$', x)) + rld = sum(1 for x in ins if re.match(r'mov\s+(esi|edi), dword ptr \[ebp - 0x[0-9a-f]+\]$', x)) + al0 = sum(1 for x in ins if re.match(r'cmp\s+(al|cl|dl), 0$', x)) + score = (3 if ebp else 0) + zx * 2 + c16 + rld // 3 + al0 + return score, n, ebp, zx, c16, rld, al0 + + +def main(): + args = sys.argv[1:] + snap = None + if args and args[0].endswith('.json'): + snap = json.load(io.open(args[0], encoding='utf-8')) + args = args[1:] + files = [] + if args: + for d in args: + files += [os.path.join(d, f) for f in sorted(os.listdir(os.path.join(ROOT, d))) if f.endswith('.cpp')] + else: + files = [l.strip() for l in io.open(os.path.join(ROOT, 'cmake/openshc-sources.txt.local')) if l.strip()] + pct = {} + if snap is not None: + for k, (raw, norm) in snap.items(): + pct[k.split('::')[-1]] = (100 * raw, 100 * norm) + rows = [] + for p in files: + short = os.path.basename(p)[:-4] + if snap is not None and (short not in pct or pct[short][1] >= 99.95): + continue + m = MARK.search(io.open(os.path.join(ROOT, p), encoding='utf-8', errors='replace').read(4000)) + if not m: + continue + score, n, ebp, zx, c16, rld, al0 = fingerprints(original(m.group(1))) + rows.append((score, pct.get(short, (0, 0)), short, n, ebp, zx, c16, rld, al0)) + rows.sort(reverse=True) + print('%5s %5s %5s %-52s %4s ebp zx c16 rld al0' % ('score', 'raw', 'norm', 'function', 'insn')) + for score, (raw, norm), short, n, ebp, zx, c16, rld, al0 in rows: + if score >= 3: + print('%5d %5.1f %5.1f %-52s %4d %s %2d %3d %3d %3d' + % (score, raw, norm, short[:52], n, 'Y' if ebp else '-', zx, c16, rld, al0)) + print('(%d function(s) scanned; listed: score >= 3)' % len(rows)) + + +if __name__ == '__main__': + main() diff --git a/tools/compiler-internals/blocks_probe.py b/tools/compiler-internals/blocks_probe.py new file mode 100644 index 0000000000..94f54735fb --- /dev/null +++ b/tools/compiler-internals/blocks_probe.py @@ -0,0 +1,158 @@ +"""Dump a function's block list, with tuple kinds/opcodes, at the points that decide block layout. + +usage: blocks_probe.py FILE.cpp --name FUNCSUBSTRING (preferred: invocation numbers drift between runs) + blocks_probe.py FILE.cpp --invocation N + blocks_probe.py FILE.cpp --scan every invocation, both phases + +Phases printed (all inside the GlobalRegAlloc N..N+1 window): + PHASE regalloc-entry the block order c2 starts allocation with + CTM ... every FgDoCommonTailMerge call (0x10b307e0): its two tuple args + TMWC a=.. b=.. FgTailMergeWithoutCheck (0x10b30d90): a's tail folded onto b + LABEL into=.. the newlabel path (0x10b30c58): a label inserted mid-block + PHASE after-merge the block list after each successful fold (0x10b30d73) + PHASE color at ColorStackElements (after allocation and the late flow opts) + +Block layout (from CollectStackElements / FgEarlyTailDup): Function (c2!Function, 0x10d0d700) ++8 -> list node whose [0] is the first block; block+0 next, +4 prev, +8 preds, +0x1c -> head node +whose [0] is the first tuple, +0x20 the end tuple (exclusive). Tuple+0 next, +4 opcode, +8 kind, ++9 flags, +0x10 prev, +0x1c a 16-bit field the merge code compares and stores (line number?). +Each tuple prints as kind/opcode@field16. +""" +import os +import re +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from slot_probe import EQ, NL, GLOBAL_REG_ALLOC, COLOR_ENTRY, run + +FUNCTION = '0x10d0d700' + + +def dump(tag): + return ('.printf ' + EQ + 'PHASE ' + tag + NL + EQ + '; ' + 'r $t3=poi(poi(' + FUNCTION + ')+8); r $t3=poi(@$t3); r $t4=0; ' + '.while ((@$t3!=0) & (@$t4<0n90)) { ' + '.printf ' + EQ + 'BLK %d %x prev=%x preds=%x first=%x/%x last=%x/%x :' + EQ + ', @$t4, @$t3, poi(@$t3+4), poi(@$t3+8), ' + 'poi(poi(@$t3+0x1c)+8)&0xff, poi(poi(@$t3+0x1c)+4), poi(poi(@$t3+0x20)+8)&0xff, poi(poi(@$t3+0x20)+4); ' + 'r $t5=poi(poi(@$t3+0x1c)); r $t6=0; ' + '.while ((@$t5!=0) & (@$t6<0n400) & (@$t5!=poi(@$t3+0x20))) { ' + '.printf ' + EQ + ' %x:%x/%x@%d' + EQ + ', @$t5, poi(@$t5+8)&0xff, poi(@$t5+4), poi(@$t5+0x1c)&0xffff; ' + 'r $t5=poi(@$t5); r $t6=@$t6+1 }; ' + '.printf ' + EQ + NL + EQ + '; r $t3=poi(@$t3); r $t4=@$t4+1 }') + + +RAW = ('r $t3=poi(poi(' + FUNCTION + ')+8); r $t3=poi(@$t3); r $t4=0; ' + '.while ((@$t3!=0) & (@$t4<0n12)) { .printf ' + EQ + 'RAWBLK %d first' + NL + EQ + ', @$t4; dd poi(@$t3+0x1c) L10; ' + '.printf ' + EQ + 'RAWBLK %d last' + NL + EQ + ', @$t4; dd poi(@$t3+0x20) L10; r $t3=poi(@$t3); r $t4=@$t4+1 }') + + +def probe(src, invocation): + lines = ['sxe ld:c2', 'g', + 'bp1 0x10b307e0 ".printf ' + EQ + 'CTM eax=%x edx=%x s=%x %x %x' + NL + EQ + + ', @eax, @edx, poi(@esp+4), poi(@esp+8), poi(@esp+0xc); gc"', + 'bp2 0x10b30ac7 ".printf ' + EQ + 'TMWC-site1 a=%x b=%x' + NL + EQ + ', @eax, @edx; gc"', + 'bp7 0x10b30ae5 ".printf ' + EQ + 'TMWC-site2 a=%x b=%x' + NL + EQ + ', @eax, @edx; gc"', + 'bp8 0x10b307e3 "' + dump('before-merge') + '; gc"', + 'bp3 0x10b30c58 ".printf ' + EQ + 'LABEL into=%x cut=%x' + NL + EQ + ', @ebx, @edi; gc"', + 'bp4 0x10b30d73 "' + dump('after-merge') + '; gc"', + 'bp5 ' + COLOR_ENTRY + ' "' + dump('color') + '; gc"', + 'bp6 0x10c07351 "' + dump('regalloc-entry') + '; gc"', + 'bd 1-8', 'r $t0=0', + 'bp0 ' + GLOBAL_REG_ALLOC + ' "r $t0=@$t0+1; .if (@$t0==0n' + str(invocation) + + ') { be 1-6 }; .if (@$t0==0n' + str(invocation + 1) + ') { bd 1-6 }; gc"', + 'g', 'q', ''] + txt = run(lines, src, 'blocks-%s-%d' % (os.path.basename(src)[:-4], invocation)) + if 'Malformed' in txt or 'Syntax error' in txt: + sys.exit('cdb rejected a command - see the log') + return [l for l in txt.splitlines() if re.match(r'^(PHASE|BLK|CTM|TMWC|LABEL|RAWBLK|[0-9a-f]{8} )', l)] + + +def probe_named(src, name): + """Same as probe(), but the window is chosen by the function's (mangled) name instead of an + invocation number: GlobalRegAlloc invocation numbers are NOT stable across runs (433 and 425 + for the same file - function order depends on pointer values), so every probe that keys on + a number can land on a neighbour. The name is read at GlobalRegAlloc entry from the Function + record: eax -> symbol at +0, mangled name at poi(sym+4) ("?name@Class@...").""" + # aliases do not expand inside a breakpoint command, so match the first 8 bytes of the + # mangled name ("?" + function name) as two little-endian dwords instead + import struct + mangled = ('?' + name + ' ')[:8].encode('ascii') + w0, w1 = struct.unpack('lineHeight_0x14; + BOOL local_f0 = FALSE; + BGR24 color = 0xccfaff; + BGR24 linkColor = 0x8bcf84; + int imageCount = -1; + BOOL centreActive = FALSE; + BOOL centerCurrentLine = FALSE; + int lineStartY = 0; + int lineEndY = 0; + int local_11c = 0; + int local_100 = 0; + int local_10c = 0; + BOOL local_e4 = FALSE; + int local_134 = -10000; + if (this->helpDialogVariant == 2 || this->isCustomTextMode) { + color = 0xc2f0eb; + } + if (!this->isTextHelpDialogMode) { + return 1; + } + this->field49_0x23964 = -1; + this->currentLinkId = -1; + if (this->field51_0x2396c) { + DAT_00df334c::instance = 1; + } + if (this->activeHelpHotspotIndex < 0) { + this->activeHelpHotspotIndex = 0; + } + if (this->activeHelpHotspotIndex > this->customHelpTextLength) { + this->activeHelpHotspotIndex = this->customHelpTextLength; + } + int lineY = -1; + int leftPosition; + do { + leftPosition = this->lineLayoutTable[++lineY].leftBorder; + } while (leftPosition == -1); + + bool abortProcessing; + do { + abortProcessing = false; + + TextMemUnion textToRender; + int local_114; + BGR24 charColor; + // false if text debug view token handled (this->helpDialogSubMode == 1), maybe related to input? + BOOL local_12c; + + charColor = color; + local_12c = TRUE; + if (0 <= currentLinkId) { + charColor = linkColor; + } + local_114 = textMemIndex; + textToRender.value + = MACRO_CALL_MEMBER(TextEditorState_Func::getWideCharOrWideCharPointer, this)(&textMemIndex); + if (this->useAlternateHelpTab && textToRender.value < L' ' + && textToRender.helpToken != Enums::HTT_NEWPARAGRAPH && local_114 == this->activeHelpHotspotIndex) { + lineStartY = lineY; + lineEndY = lineY + currentFontLineHeight; + if (param_1 == 1) { + int xModify = 0; + if (centerCurrentLine) { + xModify = this->lineLayoutTable[lineY].widthForCentering / 2; + } + local_10c = local_114 - local_11c; + if (this->field51_0x2396c < 0) { + this->activeHelpHotspotIndex = local_100 + DAT_00df3348::instance; + if (this->activeHelpHotspotIndex >= local_11c) { + this->activeHelpHotspotIndex = local_11c - 1; + } + this->field51_0x2396c = 0; + } + local_134 = 0; + int xPos = this->dialogContentX + xModify; + int yPos = this->dialogContentY - this->helpContentScrollOffsetY; + MACRO_CALL_MEMBER(UI::Rendering::PencilRenderCore_Func::drawLine, DAT_PencilRenderCore::ptr)( + xPos + leftPosition - 1, yPos + lineY - 1, xPos + leftPosition - 1, yPos + lineEndY + 1, + COL_DARK_LIME::instance.shortValue); + xPos = this->dialogContentX + xModify; + yPos = this->dialogContentY - this->helpContentScrollOffsetY; + MACRO_CALL_MEMBER(UI::Rendering::PencilRenderCore_Func::drawLine, DAT_PencilRenderCore::ptr)( + xPos + leftPosition, yPos + lineY - 1, xPos + leftPosition, yPos + lineEndY + 1, + COL_DARK_LIME::instance.shortValue); + local_e4 = TRUE; + } + } + + switch (textToRender.helpToken) { + case Enums::HTT_PIC: { + if (this->helpDialogSubMode == 1) { + charColor = 0xff; + local_12c = FALSE; + MACRO_CALL(Global_Func::PrintToDestination)(wideTextBuffer, u__PIC__d__S__005a583c, + this->DAT_PointerToTemporaryTextMemory[textMemIndex], + this->graphicFileNames[this->DAT_PointerToTemporaryTextMemory[textMemIndex]]); + Enums::HelpTextPicturePositionToken positionToken + = (Enums::HelpTextPicturePositionToken)this->DAT_PointerToTemporaryTextMemory[textMemIndex + 1]; + ++textMemIndex; + switch (positionToken) { + case Enums::HTPPT_LEFT: + wcscat(wideTextBuffer, u_LEFT__005a5494); + textMemIndex += 2; + break; + case Enums::HTPPT_RIGHT: + wcscat(wideTextBuffer, u_RIGHT__005a5484); + textMemIndex += 2; + break; + case Enums::HTPPT_CENTRE: + wcscat(wideTextBuffer, u_CENTRE__005a5474); + textMemIndex += 2; + break; + case Enums::HTPPT_HERE: + wcscat(wideTextBuffer, u_HERE__005a5468); + textMemIndex += 2; + break; + default: + textMemIndex += 2; + break; + } + textToRender.text = wideTextBuffer; + break; + } + ++imageCount; + + bool validImageIndex = false; + int imageIndex; + if (param_1 == 1 && this->DAT_PointerToTemporaryTextMemory[textMemIndex + 1] == Enums::HTPPT_HERE) { + imageIndex = this->DAT_PointerToTemporaryTextMemory[textMemIndex]; + validImageIndex = true; + } else if (!param_1 + && (imageCount == this->imageHotspotCount + || this->DAT_PointerToTemporaryTextMemory[textMemIndex + 1] == Enums::HTPPT_HERE)) { + imageIndex = this->DAT_PointerToTemporaryTextMemory[textMemIndex]; + if (imageCount == this->imageHotspotCount) { + this->imageHotspotTable[this->imageHotspotCount].yPos = lineY + 25; + this->imageHotspotTable[this->imageHotspotCount].imageIndex = imageIndex; + this->imageHotspotTable[this->imageHotspotCount].linkId = currentLinkId; + } + validImageIndex = true; + } + if (validImageIndex) { + Enums::HelpTextPicturePositionToken positionToken + = (Enums::HelpTextPicturePositionToken)this->DAT_PointerToTemporaryTextMemory[textMemIndex + 1]; + ++textMemIndex; + int imageWidth = DAT_TextureRenderCoreObject::instance.loadedGfxArray[99 - imageIndex].width; + int imageHeight = DAT_TextureRenderCoreObject::instance.loadedGfxArray[99 - imageIndex].height; + switch (positionToken) { + case Enums::HTPPT_LEFT: { + this->imageHotspotTable[this->imageHotspotCount].xPos = this->lineLayoutTable[lineY].leftBorder; + int newLeftBorder = this->lineLayoutTable[lineY].leftBorder + 25 + imageWidth; + for (int i = 0; i < imageHeight + 50; ++i) { + this->lineLayoutTable[lineY + i].leftBorder = newLeftBorder; + } + break; + } + case Enums::HTPPT_RIGHT: { + this->imageHotspotTable[this->imageHotspotCount].xPos + = this->lineLayoutTable[lineY].rightBorder - imageWidth; + int newRightBorder = this->lineLayoutTable[lineY].rightBorder - imageWidth - 25; + for (int i = 0; i < imageHeight + 50; ++i) { + this->lineLayoutTable[lineY + i].rightBorder = newRightBorder; + } + break; + } + case Enums::HTPPT_CENTRE: + this->imageHotspotTable[this->imageHotspotCount].xPos + = (this->dialogContentWidth - imageWidth) / 2; + for (int i = 0; i < imageHeight + 50; ++i) { + this->lineLayoutTable[lineY + i].leftBorder = -1; + } + break; + case Enums::HTPPT_HERE: { + width = imageWidth + 3; + if (width + leftPosition > this->lineLayoutTable[lineY].rightBorder) { + local_f0 = FALSE; + if (centerCurrentLine) { + this->lineLayoutTable[lineY].widthForCentering + = this->lineLayoutTable[lineY].rightBorder - leftPosition; + centerCurrentLine = centreActive; + } + do { + lineY += currentFontLineHeight + 1; + do { + leftPosition = this->lineLayoutTable[++lineY].leftBorder; + } while (leftPosition == -1); + } while (leftPosition + width > this->lineLayoutTable[lineY].rightBorder); + if (this->useAlternateHelpTab && param_1 == 1) { + local_100 = local_11c; + local_11c = local_114; + ++local_134; + if (0 < this->field51_0x2396c && 2 <= local_134) { + local_134 = -10000; + this->activeHelpHotspotIndex = local_100 + local_10c; + if (this->activeHelpHotspotIndex >= local_11c) { + this->activeHelpHotspotIndex = local_11c - 1; + } + this->field51_0x2396c = 0; + } + } + currentFontLineHeight = currentFontClass->lineHeight_0x14; + } + if (imageHeight > currentFontLineHeight) { + currentFontLineHeight = imageHeight; + } + leftPosition += width; + if (param_1 != 1 && imageCount == this->imageHotspotCount) { + this->imageHotspotTable[this->imageHotspotCount].xPos = leftPosition - width; + this->imageHotspotTable[this->imageHotspotCount].yPos = lineY; + ++this->imageHotspotCount; + } + break; + } + } + textMemIndex += 2; + if (positionToken != Enums::HTPPT_HERE && !param_1) { + ++this->imageHotspotCount; + DAT_TextManagerObject::instance.helpTextLinkActive = FALSE; + abortProcessing = true; + // continue; + } + } else { + textMemIndex += 3; + } + break; + } + case Enums::HTT_FONT: { + if (this->helpDialogSubMode == 1) { + charColor = 0xff; + local_12c = FALSE; + MACRO_CALL(Global_Func::PrintToDestination)( + wideTextBuffer, u__FONT__d__005a5454, this->DAT_PointerToTemporaryTextMemory[textMemIndex]); + textMemIndex += 2; + textToRender.text = wideTextBuffer; + break; + } + int fontSize = this->DAT_PointerToTemporaryTextMemory[textMemIndex]; + if (fontSize < 15) { + fontSize += 5; + } + textMemIndex += 2; + currentFontClass = &DAT_TextManagerObject::instance.fontSizeClassArray[fontSize]; + if (currentFontClass->lineHeight_0x14 > currentFontLineHeight || !local_f0) { + currentFontLineHeight = currentFontClass->lineHeight_0x14; + } + break; + } + case Enums::HTT_COLOUR: { + if (this->helpDialogSubMode == 1) { + charColor = 0xff; + local_12c = FALSE; + MACRO_CALL(Global_Func::PrintToDestination)(wideTextBuffer, u__COLOUR__S__005a5824, + DAT_UserHelpDefinedData::instance + .field6_0x7a16c[this->DAT_PointerToTemporaryTextMemory[textMemIndex]] + .name_0x0); + textMemIndex += 2; + textToRender.text = wideTextBuffer; + break; + } + if (param_1 == 1) { + int const colorIndex = this->DAT_PointerToTemporaryTextMemory[textMemIndex]; + + uint colorRed; + uint colorGreen; + uint colorBlue; + if (this->helpDialogVariant == 2 && colorIndex == 1) { + colorRed = 0xd7; + colorGreen = 0xd2; + colorBlue = 0xa4; + } else { + colorRed = DAT_UserHelpDefinedData::instance.field6_0x7a16c[colorIndex].r_0x4; + colorGreen = DAT_UserHelpDefinedData::instance.field6_0x7a16c[colorIndex].g_0x8; + colorBlue = DAT_UserHelpDefinedData::instance.field6_0x7a16c[colorIndex].b_0xc; + } + textMemIndex += 2; + color = RGB(colorRed, colorGreen, colorBlue); + } else { + textMemIndex += 2; + } + break; + } + case Enums::HTT_LINKCOLOUR: { + if (this->helpDialogSubMode == 1) { + charColor = 0xff; + local_12c = FALSE; + MACRO_CALL(Global_Func::PrintToDestination)(wideTextBuffer, u__LINKCOLOUR__S__005a5804, + DAT_UserHelpDefinedData::instance + .field6_0x7a16c[this->DAT_PointerToTemporaryTextMemory[textMemIndex]] + .name_0x0); + textMemIndex += 2; + textToRender.text = wideTextBuffer; + break; + } + if (param_1 == 1) { + int const colorIndex = this->DAT_PointerToTemporaryTextMemory[textMemIndex]; + textMemIndex += 2; + linkColor = RGB(DAT_UserHelpDefinedData::instance.field6_0x7a16c[colorIndex].r_0x4, + DAT_UserHelpDefinedData::instance.field6_0x7a16c[colorIndex].g_0x8, + DAT_UserHelpDefinedData::instance.field6_0x7a16c[colorIndex].b_0xc); + } else { + textMemIndex += 2; + } + break; + } + case Enums::HTT_LINK: { + if (this->helpDialogSubMode == 1) { + charColor = 0xff; + local_12c = FALSE; + MACRO_CALL(Global_Func::PrintToDestination)(wideTextBuffer, u__LINK_S__005a57ec, + DAT_UserHelpDefinedData::instance + .HelpSections[this->DAT_PointerToTemporaryTextMemory[textMemIndex]]); + textMemIndex += 2; + textToRender.text = wideTextBuffer; + break; + } + currentLinkId = this->DAT_PointerToTemporaryTextMemory[textMemIndex]; + if (param_1 == 1) { + DAT_TextManagerObject::instance.helpTextLinkActive = TRUE; + } + textMemIndex += 2; + break; + } + case Enums::HTT_INCLUDE: { + if (this->helpDialogSubMode == 1) { + charColor = 0xff; + local_12c = FALSE; + MACRO_CALL(Global_Func::PrintToDestination)(wideTextBuffer, u__INCLUDE_S__005a57cc, + DAT_UserHelpDefinedData::instance + .HelpSections[this->DAT_PointerToTemporaryTextMemory[textMemIndex]]); + textMemIndex += 2; + textToRender.text = wideTextBuffer; + break; + } + textMemIndex += 2; + break; + } + case Enums::HTT_SOUND: { + if (this->helpDialogSubMode == 1) { + charColor = 0xff; + local_12c = FALSE; + MACRO_CALL(Global_Func::PrintToDestination)(wideTextBuffer, u__SOUND_S__005a57b0, + this->soundFileNames[this->DAT_PointerToTemporaryTextMemory[textMemIndex]]); + textMemIndex += 2; + textToRender.text = wideTextBuffer; + break; + } + if (param_1 == 1) { + int const visibleY = lineY - this->helpContentScrollOffsetY; + if (0 <= visibleY && visibleY < this->dialogContentHeight + && !this->soundFilePlayedFlags[this->DAT_PointerToTemporaryTextMemory[textMemIndex]]) { + this->soundFilePlayedFlags[this->DAT_PointerToTemporaryTextMemory[textMemIndex]] = 1; + } + } + textMemIndex += 2; + break; + } + case Enums::HTT_ENDLINK: { + if (this->helpDialogSubMode == 1) { + charColor = 0xff; + local_12c = FALSE; + wcscpy(wideTextBuffer, u__LINK__005a53d4); + textToRender.text = wideTextBuffer; + break; + } + currentLinkId = -1; + if (param_1 == 1) { + DAT_TextManagerObject::instance.helpTextLinkActive = FALSE; + } + break; + } + case Enums::HTT_STRING: { + if (this->helpDialogSubMode == 1) { + charColor = 0xff; + local_12c = FALSE; + MACRO_CALL(Global_Func::PrintToDestination)( + wideTextBuffer, u__STRING__d__005a5358, this->DAT_PointerToTemporaryTextMemory[textMemIndex]); + textMemIndex += 2; + textToRender.text = wideTextBuffer; + break; + } + textMemIndex += 2; + break; + } + case Enums::HTT_CENTRE: { + if (this->helpDialogSubMode == 1) { + charColor = 0xff; + local_12c = FALSE; + wcscpy(wideTextBuffer, u__CENTRE__005a539c); + textToRender.text = wideTextBuffer; + break; + } + centreActive = TRUE; + centerCurrentLine = TRUE; + break; + } + case Enums::HTT_ENDCENTRE: { + if (this->helpDialogSubMode == 1) { + charColor = 0xff; + local_12c = FALSE; + wcscpy(wideTextBuffer, u__CENTRE__005a5388); + textToRender.text = wideTextBuffer; + break; + } + centreActive = FALSE; + break; + } + case Enums::HTT_NEWPARAGRAPH: { + if (centerCurrentLine) { + this->lineLayoutTable[lineY].widthForCentering + = this->lineLayoutTable[lineY].rightBorder - leftPosition; + centerCurrentLine = centreActive; + } + if (this->useAlternateHelpTab && local_114 == this->activeHelpHotspotIndex) { + lineStartY = lineY; + lineEndY = currentFontLineHeight + lineY; + if (param_1 == 1) { + int xModify = 0; + if (centerCurrentLine) { + xModify = this->lineLayoutTable[lineY].widthForCentering / 2; + } + int xPos = this->dialogContentX + xModify; + int yPos = this->dialogContentY - this->helpContentScrollOffsetY; + MACRO_CALL_MEMBER(UI::Rendering::PencilRenderCore_Func::drawLine, DAT_PencilRenderCore::ptr)( + xPos - 1 + leftPosition, yPos - 1 + lineStartY, xPos - 1 + leftPosition, + yPos + 1 + lineEndY, COL_DARK_LIME::instance.shortValue); + xPos = this->dialogContentX + xModify; + yPos = this->dialogContentY - this->helpContentScrollOffsetY; + MACRO_CALL_MEMBER(UI::Rendering::PencilRenderCore_Func::drawLine, DAT_PencilRenderCore::ptr)( + xPos + leftPosition, yPos - 1 + lineStartY, xPos + leftPosition, yPos + 1 + lineEndY, + COL_DARK_LIME::instance.shortValue); + local_e4 = TRUE; + } + } + local_f0 = FALSE; + lineY += 3 + currentFontLineHeight; + do { + leftPosition = this->lineLayoutTable[++lineY].leftBorder; + } while (leftPosition == -1); + currentFontLineHeight = currentFontClass->lineHeight_0x14; + + if (this->useAlternateHelpTab) { + local_100 = local_11c; + local_11c = local_114; + if (param_1 == 1) { + ++local_134; + if (0 < this->field51_0x2396c && 2 <= local_134) { + local_134 = -10000; + this->activeHelpHotspotIndex = local_10c + local_100; + if (this->activeHelpHotspotIndex >= local_11c) { + this->activeHelpHotspotIndex = local_11c - 1; + } + this->field51_0x2396c = 0; + } + } + if (local_11c == this->activeHelpHotspotIndex && param_1 == 1) { + local_10c = 0; + if (this->field51_0x2396c < 0) { + this->activeHelpHotspotIndex = DAT_00df3348::instance + local_100; + if (this->activeHelpHotspotIndex >= local_11c) { + this->activeHelpHotspotIndex = local_11c - 1; + } + this->field51_0x2396c = 0; + } + local_134 = 0; + } + } + if (this->helpDialogSubMode == 1) { + charColor = 0xff; + local_12c = FALSE; + wcscpy(wideTextBuffer, u__NEWPARAGRAPH__005a5790); + textToRender.text = wideTextBuffer; + break; + } + break; + } + case Enums::HTT_TAB: { // TODO?: unknown token + if (this->helpDialogSubMode == 1) { + charColor = 0xff; + local_12c = FALSE; + wcscpy(wideTextBuffer, u__tab__005a5784); + textToRender.text = wideTextBuffer; + break; + } + width = 30; + if (leftPosition + width > this->lineLayoutTable[lineY].rightBorder) { + local_f0 = FALSE; + if (centerCurrentLine) { + this->lineLayoutTable[lineY].widthForCentering + = this->lineLayoutTable[lineY].rightBorder - leftPosition; + centerCurrentLine = centreActive; + } + do { + lineY += currentFontLineHeight + 1; + do { + leftPosition = this->lineLayoutTable[++lineY].leftBorder; + } while (leftPosition == -1); + } while (leftPosition + width > this->lineLayoutTable[lineY].rightBorder); + + if (this->useAlternateHelpTab && param_1 == 1) { + local_100 = local_11c; + local_11c = local_114; + ++local_134; + if (0 < this->field51_0x2396c && 2 <= local_134) { + local_134 = -10000; + this->activeHelpHotspotIndex = local_100 + local_10c; + if (this->activeHelpHotspotIndex >= local_11c) { + this->activeHelpHotspotIndex = local_11c - 1; + } + this->field51_0x2396c = 0; + } + } + currentFontLineHeight = currentFontClass->lineHeight_0x14; + } + leftPosition += width; + break; + } + case (wchar_t*)0: { // TODO?: unknown token, or just NULL, meaning end? + if (param_1 == 0) { + this->topVisibleLineIndex = lineY + currentFontLineHeight - 1; + do { + ++this->topVisibleLineIndex; + if (this->lineLayoutTable[this->topVisibleLineIndex].leftBorder == 25 + && this->lineLayoutTable[this->topVisibleLineIndex].rightBorder + == this->dialogContentWidth - 25) { + break; + } + } while (this->topVisibleLineIndex < 20000); + this->topVisibleLineIndex += 5; + if (this->useAlternateHelpTab) { + if (lineStartY < this->helpContentScrollOffsetY) { + this->helpContentScrollOffsetY = lineStartY; + } + if (lineEndY > this->helpContentScrollOffsetY + this->dialogContentHeight) { + this->helpContentScrollOffsetY = (lineEndY - this->dialogContentHeight) + 5; + DAT_TextManagerObject::instance.helpTextLinkActive = FALSE; + return TRUE; + } + } + } else if (this->useAlternateHelpTab) { + DAT_00df3348::instance = local_10c; + if (0 < this->field51_0x2396c && local_134 == 1) { + this->activeHelpHotspotIndex = local_10c + local_11c; + if (this->activeHelpHotspotIndex > this->customHelpTextLength) { + this->activeHelpHotspotIndex = this->customHelpTextLength; + } + this->field51_0x2396c = 0; + } + if (!local_e4 && this->activeHelpHotspotIndex && DAT_00df334c::instance) { + --this->activeHelpHotspotIndex; + this->field52_0x23970 = 0; + DAT_TextManagerObject::instance.helpTextLinkActive = FALSE; + return TRUE; + } + DAT_00df334c::instance = 0; + } + DAT_TextManagerObject::instance.helpTextLinkActive = FALSE; + return TRUE; + } + } + if (textToRender.value < L' ') { + continue; + } + + width = MACRO_CALL_MEMBER(FontSizeClass_Func::getWidthOfWideText, currentFontClass)( + textToRender.text, wcslen(textToRender.text)); + if (leftPosition + width > this->lineLayoutTable[lineY].rightBorder) { + local_f0 = FALSE; + if (centerCurrentLine) { + this->lineLayoutTable[lineY].widthForCentering + = this->lineLayoutTable[lineY].rightBorder - leftPosition; + centerCurrentLine = centreActive; + } + lineY += 1 + currentFontLineHeight; + do { + leftPosition = this->lineLayoutTable[++lineY].leftBorder; + } while (leftPosition == -1); + currentFontLineHeight = currentFontClass->lineHeight_0x14; + if (this->useAlternateHelpTab) { + ++local_134; + local_100 = local_11c; + local_11c = local_114; + if (0 < this->field51_0x2396c && 2 <= local_134) { + local_134 = -10000; + this->activeHelpHotspotIndex = local_100 + local_10c; + if (this->activeHelpHotspotIndex >= local_11c) { + this->activeHelpHotspotIndex = local_11c - 1; + } + this->field51_0x2396c = 0; + } + } + if (leftPosition + width > this->lineLayoutTable[lineY].rightBorder) { + local_f8 = 0; + if (centerCurrentLine) { + local_f8 = this->lineLayoutTable[lineY].widthForCentering / 2; + } + wordLength = wcslen(textToRender.text); + charIndex = local_114; + for (i = 0; i < wordLength; ++i) { + width = MACRO_CALL_MEMBER(FontSizeClass_Func::getWideCharWidth, currentFontClass)( + textToRender.text[i]); + while (leftPosition + width > this->lineLayoutTable[lineY].rightBorder) { + + if (centerCurrentLine) { + this->lineLayoutTable[lineY].widthForCentering + = this->lineLayoutTable[lineY].rightBorder - leftPosition; + centerCurrentLine = centreActive; + } + lineY += currentFontLineHeight + 1; + if (this->useAlternateHelpTab && param_1 == 1) { + local_100 = local_11c; + local_11c = charIndex; + ++local_134; + if (0 < this->field51_0x2396c && 2 <= local_134) { + local_134 = -10000; + this->activeHelpHotspotIndex = local_100 + local_10c; + if (this->activeHelpHotspotIndex >= local_11c) { + this->activeHelpHotspotIndex = local_11c - 1; + } + this->field51_0x2396c = 0; + } + } + do { + leftPosition = this->lineLayoutTable[++lineY].leftBorder; + } while (leftPosition == -1); + if (centerCurrentLine) { + local_f8 = this->lineLayoutTable[lineY].widthForCentering / 2; + } + } + if (this->useAlternateHelpTab && charIndex == this->activeHelpHotspotIndex) { + lineStartY = lineY; + lineEndY = lineY + currentFontLineHeight; + if (param_1 == 1) { + if (local_12c) { + local_10c = (i - local_11c) + local_114; + if (this->field51_0x2396c < 0) { + this->activeHelpHotspotIndex = DAT_00df3348::instance + local_100; + if (this->activeHelpHotspotIndex >= local_11c) { + this->activeHelpHotspotIndex = local_11c - 1; + } + this->field51_0x2396c = 0; + } + local_134 = 0; + int xPos = this->dialogContentX + local_f8; + int yPos = this->dialogContentY - this->helpContentScrollOffsetY; + MACRO_CALL_MEMBER(UI::Rendering::PencilRenderCore_Func::drawLine, + DAT_PencilRenderCore::ptr)(xPos - 1 + leftPosition, yPos - 1 + lineY, + xPos - 1 + leftPosition, yPos + 1 + lineEndY, + COL_DARK_LIME::instance.shortValue); + xPos = this->dialogContentX + local_f8; + yPos = this->dialogContentY - this->helpContentScrollOffsetY; + MACRO_CALL_MEMBER(UI::Rendering::PencilRenderCore_Func::drawLine, + DAT_PencilRenderCore::ptr)(xPos + leftPosition, yPos - 1 + lineY, + xPos + leftPosition, yPos + 1 + lineEndY, COL_DARK_LIME::instance.shortValue); + local_e4 = TRUE; + } + } + } + if (param_1 == 1) { + int charXPos = this->dialogContentX + local_f8 + leftPosition; + int charYPos = this->dialogContentY - this->helpContentScrollOffsetY + lineY; + + wchar_t currentChar = textToRender.text[i]; + int const newCharXPos = MACRO_CALL_MEMBER(FontSizeClass_Func::renderWideChar, + currentFontClass)(currentChar, charXPos, charYPos, charColor, 0); + local_f0 = TRUE; + if (MACRO_CALL_MEMBER(Input::MouseState_Func::isMouseInsideBox, DAT_MouseState::ptr)( + charXPos - this->helpContentScrollX, charYPos - this->helpContentScrollY, + newCharXPos - charXPos, currentFontLineHeight + 2)) { + if (local_12c) { + this->field49_0x23964 = local_114 + i; + this->currentLinkId = currentLinkId; + } else { + this->field49_0x23964 = local_114; + } + } + } + + leftPosition += width; + ++charIndex; + } + continue; + } + } + int charXPos = this->dialogContentX + leftPosition; + int charYPos = (this->dialogContentY - this->helpContentScrollOffsetY) + lineY; + if (centerCurrentLine) { + charXPos += this->lineLayoutTable[lineY].widthForCentering / 2; + } + wordLength = wcslen(textToRender.text); + int const hotspotBase = local_114 - local_11c; + for (i = 0; i < wordLength; ++i) { + if (this->useAlternateHelpTab && i + local_114 == this->activeHelpHotspotIndex) { + if (param_1 == 1 && local_12c) { + MACRO_CALL_MEMBER(UI::Rendering::PencilRenderCore_Func::drawLine, DAT_PencilRenderCore::ptr)( + charXPos - 1, charYPos - 1, charXPos - 1, charYPos + 1 + currentFontLineHeight, + COL_DARK_LIME::instance.shortValue); + MACRO_CALL_MEMBER(UI::Rendering::PencilRenderCore_Func::drawLine, DAT_PencilRenderCore::ptr)( + charXPos, charYPos - 1, charXPos, charYPos + 1 + currentFontLineHeight, + COL_DARK_LIME::instance.shortValue); + local_e4 = TRUE; + } + lineStartY = lineY; + lineEndY = lineY + currentFontLineHeight; + if (param_1 == 1) { + local_134 = 0; + local_10c = hotspotBase + i; + if (this->field51_0x2396c < 0 && local_12c) { + this->activeHelpHotspotIndex = local_100 + DAT_00df3348::instance; + if (this->activeHelpHotspotIndex >= local_11c) { + this->activeHelpHotspotIndex = local_11c - 1; + } + this->field51_0x2396c = 0; + } + } + } + + if (param_1 == 1) { + local_f0 = TRUE; + int oldCharPos = charXPos; + wchar_t const currentChar = textToRender.text[i]; + charXPos = MACRO_CALL_MEMBER(FontSizeClass_Func::renderWideChar, currentFontClass)( + currentChar, charXPos, charYPos, charColor, 0); + if (MACRO_CALL_MEMBER(Input::MouseState_Func::isMouseInsideBox, DAT_MouseState::ptr)( + oldCharPos - this->helpContentScrollX, charYPos - this->helpContentScrollY, + charXPos - oldCharPos, currentFontLineHeight + 2)) { + if (local_12c) { + this->field49_0x23964 = i + local_114; + this->currentLinkId = currentLinkId; + } else { + this->field49_0x23964 = local_114; + } + } + } else { + charXPos += MACRO_CALL_MEMBER(FontSizeClass_Func::getWideCharWidth, currentFontClass)( + textToRender.text[i]); + } + } + leftPosition += width; + } while (!abortProcessing); + return FALSE; + } +} +} diff --git a/tools/compiler-internals/finding-mismatches.md b/tools/compiler-internals/finding-mismatches.md new file mode 100644 index 0000000000..0a96790583 --- /dev/null +++ b/tools/compiler-internals/finding-mismatches.md @@ -0,0 +1,364 @@ +# Finding out why a function does not match + +This is a walkthrough of the tooling, written for someone who has never done this before. No +compiler theory needed. Two real functions are used throughout: one that went from 67% to 95%, +and one that is still stuck, because the stuck one teaches more. + +## The problem, in one paragraph + +The original game is a compiled `.exe`. We write C++ that is supposed to compile to **the exact +same machine code**. `reccmp` compares our compiled function against the original's bytes and +gives a percentage. When it says 56%, something about our C++ is different - but the percentage +does not say *what*, and that is the whole difficulty. The tools below turn "56%" into a +sentence like "the original computes this three-instruction sequence twice and we compute it +once". + +## The one idea you need + +Compiling happens in stages, roughly: + +1. decide which local variables exist and where they live on the stack +2. decide which machine instructions to emit, and in what order +3. decide which CPU register holds each value + +Stage 3 depends on stage 2, which depends on stage 1. The compiler is **deterministic**: same +input, same output, every time. So if our stage 1 and stage 2 are identical to the original's, +stage 3 comes out identical for free. + +That gives the order of work, and it is the opposite of what feels natural: + +| step | tool | what it answers | +| --- | --- | --- | +| 1 | `slot_map.py` | do we have the same local variables, in the same stack slots? | +| 2 | `structural_diff.py` | do we emit the same instructions, in the same order? | +| 3 | - | registers. **Do not work on this directly.** It usually follows from 1 and 2. | +| 4 | `reccmp_report.py` | the official percentage, to confirm at the end | + +**The percentage is a bad guide until the end.** Here is the proof, measured on +`renderDebugDataAiInfo`. Adding two local variables that the original definitely has made the +stack frame go from wrong to byte-perfect - and the percentage *dropped*: + +| version | reccmp | stack frame | +| --- | --- | --- | +| before | **68.4%** | `0x3f8` - wrong, 8 bytes too small | +| after | 67.5% | `0x400` - exactly right | + +Why? Adding a local shifts every single `[esp + something]` offset in the function, and reccmp +counts each shifted offset as a mismatch. The change was right and the score got worse. If you +steer by the percentage you will undo correct work. (That same function later reached 95.4%, +so the frame fix was indeed on the way there.) + +## Step 1: the stack frame + +``` +$ python tools/reimplementation-helper/batch/slot_map.py src/OpenSHC/AI/AICState/aiGiveRaidInstructions.cpp +frame original 0x14 ours 0x14 (+0) + + off span acc refs key | off size refs key name + -20 4 4 9 2250 | -20 4 9 2250 _i$ + -16 4 4 5 1250 | -16 4 5 1250 _tribeID$462193 + -12 4 4 3 750 | -12 4 3 750 tv694 + -8 4 4 2 500 | -8 4 2 500 _aicIndex$ + -4 - 4 2 500 | -4 4 2 500 _this$ +``` + +Left is the original, right is ours. `off` is where the variable sits relative to the stack +pointer on entry, `refs` is how many times it is read or written, `name` is our variable's name +(`tv694` is a compiler-made temporary). + +This is a perfect result: same frame size, same five slots at the same offsets with the same +reference counts. Nothing to do. + +The interesting case is when it is *not* perfect. `sub esp, N` in the original tells you +exactly how many bytes of locals the original had, with no guesswork - and a frame that is too +small means **we are missing a local variable**, which no amount of fiddling with instructions +will fix. A `?` row means one side has a slot the other does not: + +``` + off span acc refs key | off size refs key name + -1024 4 4 4 1000 | ? <-- a local we do not have + -1020 4 4 3 750 | ? <-- a local we do not have + -1016 4 4 3 750 | -1016 4 5 1250 _this$GSCopy$ +``` + +That is what `renderDebugDataAiInfo` looked like, and it is how we knew to go looking for two +missing variables. + +One subtlety worth knowing: ours comes from a list the compiler prints (`/FA`), which states +every slot outright. The original's has to be *worked out* from the machine code, by tracking +how deep the stack pointer is at each instruction. That is why the tool sometimes prints a +warning instead of a table - if it cannot follow the stack depth (an indirect call whose callee +pops an unknown number of bytes, say), it says so rather than printing numbers it does not +believe. + +## Step 2: the instructions + +``` +$ python tools/reimplementation-helper/batch/structural_diff.py aiGiveRaidInstructions + instructions original 241 ours 234 (-7) + kind 0.960 <- drive this to 1.000 first; every difference here is real + shape 0.939 <- operand form, once registers and constants are anonymised + skel 0.939 <- the same instructions with only REGISTER NAMES anonymised + raw 0.568 <- reccmp, mostly register and slot naming + delta 9 instruction(s) of mismatched kind +``` + +Four numbers, each deliberately blind to more than the last: + +* **`kind`** - just the instruction names, all operands thrown away. `mov`, `add`, `push`... + A difference here is always real. +* **`shape`** - operands kept, but register names turned into `r` and numbers into `k`. + Catches "we used a constant where the original used a register". +* **`skel`** - the full instruction, with *only* register names blanked. The gap between + `skel` and `raw` is purely register naming. +* **`raw`** - what reccmp says. + +**`delta` is the most useful number.** It counts instructions, not ratios: "we emit two `xor` +the original does not". Here it is 9, and we are 7 instructions short overall. + +Then the per-instruction table: + +``` + mnemonic original ours + je 13 11 -2 + movsx 7 5 -2 + add 8 7 -1 + imul 8 7 -1 + lea 6 5 -1 + test 14 13 -1 + jne 15 16 +1 +``` + +### The trap, and the detector that exists because of it + +Read that table as seven separate missing instructions and you will hunt for seven things. It +is not seven things. Look at the shape of it: one missing `test`, one `je`, one `imul`, **two** +`movsx`, one `lea`, one `add`. That is one missing **copy** of a seven-instruction sequence. + +I got this wrong the first time on `renderDebugDataAiInfo` - the table read `add -1, mov -1, +push -1` and I went looking for three unrelated problems. So the tool now counts how many times +each sequence occurs on each side, and says it outright: + +``` + DUPLICATION: the original computes this run 2 time(s), we compute it 1: + test r, r + je L + imul r, r, 0x32c + movsx r, word ptr [r + G] + movsx r, word ptr [r + G] + lea r, [r + r*0x2] + add r, dword ptr [r*0x4 + G] +``` + +Register names are blanked first, so two copies still count as "the same sequence" even when +the compiler put them in different registers. + +Translating that back to C++: `imul r, r, 0x32c` is multiplying by 812, which is the size of a +building record, so this is `buildings[something]`; the two `movsx` are reading two 16-bit +fields; `lea r, [r + r*2]` multiplies by 3. It is this line of our source: + +```cpp +tile = DAT_BuildingsState::instance.buildings[buildingID].buildingEntryX + + DAT_ViewportRenderState::instance + .translationMatrix[DAT_BuildingsState::instance.buildings[buildingID].buildingEntryY] + .addXgetTile; +``` + +which appears three times in the function, once in each branch of an `if`/`else if`/`else`. + +### Why "share less" is the fix, not "share more" + +Your instinct is that duplicated code is bad and sharing is good. For matching, that instinct +is backwards: we have to produce whatever the original produced, and here the original +duplicates. + +MSVC hoists a common expression out of both arms of an `if`/`else`, and - this is the key part +- it hoists **as much of the expression as is common**. On `renderDebugDataAiInfo` the original +did this: + +``` +imul eax, eax, 0x6d98 ; shared, before the branch +cmp dword ptr [esi - 0xdf4], 0 +jne arm_b +mov eax, dword ptr [eax + 0x1866ac0] ; arm A: duplicated +add eax, 1 +... +arm_b: +mov edx, dword ptr [eax + 0x1866ac0] ; arm B: the SAME load again +add edx, 1 +``` + +Only the multiply is shared; the identical load is duplicated. Four source versions were tried +and every one of them hoisted the whole thing: + +| what we wrote | result | +| --- | --- | +| the full expression written out in both arms | whole chain hoisted | +| one shared local before the `if` | hoisted, obviously | +| a local declared before, assigned inside each arm | still hoisted | +| a ternary picking the format string | still hoisted | + +The reason is the *extent* of what is common. Written out twice, the entire chain is common - +read the member, multiply, read the field, add one - so all of it gets hoisted. The original +shares only the **front** of that chain. So the source has to share exactly that front and +write the rest out: + +```cpp +int aivID = DAT_GameState::instance.playerDataArray[playerID].aivID; // share the index +if (...) sprintf(text, "AIV:n/a", DAT_AIVState::instance.aivs[aivID].castleID + 1); +else sprintf(text, "AIV: %d", DAT_AIVState::instance.aivs[aivID].castleID + 1); +``` + +The index is shared because *we* shared it; the two reads are left alone because the compiler no +longer sees them as part of one common expression. `delta` went to 0 and the score went +**67.5% -> 83.6%**. + +Sharing *more* than the original also costs you. Using a pointer to the element instead of the +index got within one instruction, because a pointer has to be fully computed (`base + index * +size`, an extra `add`), whereas the original keeps only `index * size` in a register and folds +the base into each read: + +| form | delta | instructions | +| --- | --- | --- | +| expression inline in both arms | 5 | 300 | +| shared local before the branch | 3 | 302 | +| pointer/reference to the element | 1 | 306 | +| **index in a local** | **0** | **305** (matches) | + +**The rule: share in the source exactly the prefix the original shares, and write the rest +out.** + +## Step 3: registers - usually not your problem + +``` + registers: 70 operand(s) differ only in which register was used. + substitution count + ebp -> ebx 27 + ebx -> ebp 19 + ecx -> edx 9 + + The dominant substitution is a permutation in 2 closed cycle(s), covering + 66 of 70 (94%): eax -> ecx -> edx -> eax, ebp -> ebx -> ebp +``` + +When the substitutions form **closed cycles** like that, it means the allocation is structurally +right and the registers were just handed out in a different order - not that anything is broken. + +The tool then tells you whether to care: + +``` + The first STRUCTURAL difference is at original instruction 1 and the first + REGISTER difference at 2, so the registers diverge at or after the structure + does. ... these register names are an effect of that structural defect - + fix it and they follow. +``` + +That is the useful verdict. If registers start differing *after* the instructions start +differing, they are a symptom. Fix step 2 and they sort themselves out - on +`renderDebugDataAiInfo`, once the instruction stream matched exactly, the register +substitutions dropped to **zero** with no register work at all. + +If you ever do need to look at registers, `reg_scores.py` reads the decisions out of the +compiler itself with a debugger: + +``` +$ python tools/compiler-internals/reg_scores.py FILE.cpp --whyreg this +candidate got eax ecx edx ebx esp ebp esi edi +tile eax [0] 0 0 0 0 0 0 0 +this ebp 12 -15 0 0 0 [6300] 0 0 +``` + +Each row is one value, `got` is the register it received, and the numbers are what each register +"cost" - lowest wins, brackets mark the winner. Candidates are served in score order and take +the cheapest register still available, so what matters is **who gets served first**. Nothing +about variable names or types enters into it; the inputs are the instruction stream and which +registers are still busy. + +## Where `aiGiveRaidInstructions` currently stands + +Being honest about an unfinished one, because the dead ends are informative. + +Known: the frame is **perfect** (step 1 done). `delta` is 9, of which 7 instructions are that +one missing copy of the tile computation. The registers are a clean two-cycle swap: the original +keeps `this` in `ebx` and `playerID * 0x39f4` (the player-record base) in `ebp`; we have them +the other way round. + +Tried and rejected: moving `i` into the `for` initialiser. It cut `delta` from 9 to 3 - it +actually fixed the missing duplicate - but the score fell from **56.8% to 44.1%**, because it +scrambled the register assignment far more than it gained. Reverted. + +Why the register swap resists: `ebx` comes before `ebp` in the compiler's preference list, so +whichever value is served first takes `ebx`. `reg_scores.py` shows our player-base temporary +scoring **82** against `this` at **60**, so the temporary is served first and takes `ebx`. In +the original, `this` won `ebx`. Those scores are weighted reference counts, and both sides use +these two values exactly 27 and 22 times - identical. So there is no expression to rewrite to +change the ranking, and no source change I can name that flips it. That is where it sits. + +The lesson is the one from the table at the top: `delta` and the percentage can disagree, and +when they do, neither one alone is the answer. `delta` is the right target while the frame or +the instruction mix is still wrong, because those are *causes*. Once they are right, the +percentage is all that is left to steer by. + +## Cheat sheet + +```bash +# 1. the stack frame - do we have the right local variables? +python tools/reimplementation-helper/batch/slot_map.py src/.../Thing.cpp + +# 2. the instructions - right ones, right order? read `delta` and any DUPLICATION block +python tools/reimplementation-helper/batch/structural_diff.py Thing + +# rank every function in the build list by how reachable its problem is +python tools/reimplementation-helper/batch/structural_diff.py --all + +# see the original's real machine code, whole function, correct ends +python tools/reimplementation-helper/batch/orig_body.py src/.../Thing.cpp + +# a fast register-sensitive score - a compile, not a build, so seconds per try +python tools/reimplementation-helper/batch/quick_diff.py src/.../Thing.cpp -q + +# 4. the official number, last +python tools/reimplementation-helper/batch/build_quiet.py +python tools/reimplementation-helper/batch/reccmp_report.py --run pct +``` + +Two traps worth repeating: + +* **Do not read instruction streams out of `reccmp/dll/diff.json`.** It stores only the diff + *view*, so the better a function matches the less of it is in there - a function at 99.7% + leaves 22 lines behind out of 322 instructions. `structural_diff.py` used to read it and its + numbers for good functions were meaningless. Both streams now come from the code itself. +* **A dropping percentage does not mean the change was wrong.** Check the frame and `delta` + first. The one worked example here got better three times while the score went down once. + +## When several functions stop on the same number, stop looking at the functions + +Everything above hunts for a defect *inside* one function. Some defects are not in any +function, and they have a signature you can spot in the `pct` table in a second: several +unrelated functions sitting on **exactly** the same percentage. + +`Text/FontSizeClass` looked like this: + +``` + 92.6 96.0 getWidthOfWideText + 95.6 96.0 renderMultilineWideTextUnk + 99.0 96.0 renderSomeSpecificTextUnk + 94.3 96.0 renderWideText +``` + +Four functions of very different sizes, all at 96.0 normalized. Running `structural_diff.py` +on each one said the same thing four times - `delta 0`, same instructions in the same order, +`kind 1.000`. That combination is the tell: **structurally perfect, yet not 100, and tied with +its neighbours.** No amount of rewriting the C++ can move a function that is already emitting +the right instructions in the right order. + +So go straight to `reccmp_report.py diff` on the *smallest* of the group, because whatever is +shared will be a larger fraction of it and easier to see. Here the first line of the diff was +the whole answer: our `/GS` prologue read `__security_cookie` from our own `.data`, the +original read it from 0x00B94220. The fix was one new file in `src/symbols/`, it was worth 4% +in *every* `/GS`-protected function in the project, and no amount of staring at +`getWidthOfWideText`'s source would ever have found it, because its source was already right. + +The general rule: `delta 0` plus a percentage below 100 means the cause is outside the +function - the build, the headers, a resolver, a calling convention. A tie across unrelated +functions tells you how far outside. diff --git a/tools/compiler-internals/merge_probe.py b/tools/compiler-internals/merge_probe.py new file mode 100644 index 0000000000..5af0ef006e --- /dev/null +++ b/tools/compiler-internals/merge_probe.py @@ -0,0 +1,189 @@ +"""Did MSVC merge blocks in THIS function? The discriminator for a POLARITY report. + +usage: merge_probe.py FILE.cpp --invocation N merge attempts/outcomes for that function + merge_probe.py FILE.cpp [FILE2.cpp ...] whole translation unit (noisy) + +`structural_diff.py` reports POLARITY when the original branches the other way round, and the +advice is to swap the arms. That is right when the two sides genuinely differ in source shape, +and wrong when the flip is MSVC having merged two blocks the original left separate - the arms +are then already correct and swapping them cannot help, because the merge happens in the late +flow optimisations, *after* register allocation, on whatever the allocator made identical. Those +two cases look identical in the diff. This tells them apart. + +Decoded from c2.dll (addresses are c2.pdb publics, see pdbnames.py): + +* Two merge families run for us, both driven from `@FgDoLateFlowOptsOnce@20` (and again from + `@FgTryEpilogMerge@20`): + - pairwise: `@FgTryTailMerge@16` 0x10b2e1b0, exits `mov eax,1` 0x10b2e6e5 (merged) + and `xor eax,eax` 0x10b2e6f7 (declined) + - common tail: `@FgTryCommonTailMerge@8` 0x10b306c0 -> `@FgDoCommonTailMerge@20` 0x10b307e0, + whose exits are `mov eax,1` 0x10b30d73 (merged) and `xor eax,eax` 0x10b30833 + (declined) + `@FgTryHeadMerge@16` / `@FgCompareMerge@12` are the head-merge counterpart and are counted + at their entries only. `@FgTailMergeWithoutCheck@8` is a per-instruction predicate, not an + action, and is not counted. +* Every one of these routines opens with `cmp dword ptr [_PogoFlag], 1` and returns 0 on the + `== 1` side. So `_PogoFlag == 1` (PGO instrumentation, which must keep blocks intact for its + counters) disables block merging entirely; in a build without a profile - every build here - + all of it is live. An earlier reading of FgCompareMerge as having the opposite polarity was + wrong and is corrected here. The script prints the flag so this is checked on every run. +* Both merge families have to be counted. `renderMultilineTextUnk` showed 20 pairwise attempts + all declined while the common-tail path merged - a tool reading FgTryTailMerge alone would + have called it "no merge". + +Per-function attribution uses the same trick as reg_scores.py: the 12 phases run per function, +so everything between `dop2_GlobalRegAlloc` invocation N and N+1 belongs to function N, and the +late flow optimisations fall in that window. Get N from `reg_scores.py FILE --find LOCAL` **in +the same session** - it depends on the translation unit's template population, which changes +with the resolver state and the compile command (renderMultilineTextUnk was #37 under one +configuration and #74 under another). + +The counts are exact for one function with --invocation. Without it they cover the whole +translation unit, where hundreds of resolver templates merge among themselves; those numbers +are only meaningful as a *difference between two variants of the same file*, where the +template noise cancels. That controlled difference is the decisive instrument: countOccupied- +PlayerSlots' else-if variant showed exactly +4 pairwise merges over the || variant - one per +unrolled body - and nothing else moved. + +Reading the result: attempts with zero merged in both families means every candidate pair was +rejected, so a POLARITY report is a real source-shape difference - swap the arms. Merges > 0 +means blocks were folded together after allocation and the polarity may be a downstream effect; +confirm with a two-variant difference before spending time on the arms. +""" + +import json +import os +import re +import subprocess +import sys + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), + '..', 'reimplementation-helper', 'batch')) +import common + +CDB = r"C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe" +GLOBAL_REG_ALLOC = '0x10c07350' # @dop2_GlobalRegAlloc@4 +POGO_FLAG = '0x10d0ffdc' # _PogoFlag + +# (event name, address). Exits are inside the routine, so they are only reached from it. +EVENTS = [ + ('pair_attempt', '0x10b2e1b0'), # @FgTryTailMerge@16 entry + ('pair_merged', '0x10b2e6e5'), # mov eax, 1 + ('pair_declined', '0x10b2e6f7'), # xor eax, eax + ('common_attempt', '0x10b307e0'), # @FgDoCommonTailMerge@20 entry (via FgTryCommonTailMerge) + ('common_merged', '0x10b30d73'), # mov eax, 1 + ('common_declined', '0x10b30833'), # xor eax, eax + ('head_attempt', '0x10b2eb00'), # @FgTryHeadMerge@16 entry + ('head_compare', '0x10b2f700'), # @FgCompareMerge@12 entry + ('epilog_attempt', '0x10b321e0'), # @FgTryEpilogMerge@20 entry +] + +out = (common.ROOT / common.TMP / 'mergeprobe').resolve() +out.mkdir(parents=True, exist_ok=True) +BS = chr(92) +Q = chr(34) +EQ = BS + Q # an escaped quote, for a string nested inside a bp command +NL = BS + BS + 'n' + + +def compile_command(src): + cc = json.loads(common.read_text(common.BUILD_DIR / 'compile_commands.json')) + ours = [e for e in cc if 'OpenSHC.dll.dir' in e['command'] + and '/src/OpenSHC/' in e['file'].replace('\\', '/')] + if not ours: + sys.exit('configure the build first: no OpenSHC.dll entry in compile_commands.json') + o = str(out).replace('\\', '/') + cmd = re.sub(r'/Fo\S+', lambda _m: '/Fo%s/r.obj' % o, ours[0]['command']) + cmd = re.sub(r'-c \S+$', lambda _m: '-c ' + os.path.abspath(src), cmd) + return cmd, ours[0]['directory'] + + +def run(lines, src, tag): + path = out / (tag + '.cdb') + common.write_text(path, '\r\n'.join(lines) + '\r\n') + cmd, cwd = compile_command(src) + env = common.build_env() + env['PATH'] = str(common.ROOT / 'MSVC1400-SP1' / 'Common7' / 'IDE') + os.pathsep + env['PATH'] + full = '"%s" -cf "%s" %s' % (CDB, path, cmd) + r = subprocess.run(full, cwd=cwd, env=env, capture_output=True, text=True, timeout=3600) + common.write_text(out / (tag + '.log'), r.stdout) + return r.stdout + + +def probe(src, invocation=None): + # Each breakpoint prints a NUMERIC event id. cdb -cf echoes every script line, so a textual + # tag would be counted once in the echoed `bp` command itself; "EV %d" cannot match "EV 3". + bps = [] + for i, (_name, addr) in enumerate(EVENTS, start=1): + bps.append('bp%d %s ".printf %sEV %%d%s%s, 0n%d; gc"' % (i, addr, EQ, NL, EQ, i)) + last = len(EVENTS) + lines = ['sxe ld:c2', 'g', + '.printf ' + Q + 'POGOFLAG %d' + NL + Q + ', poi(' + POGO_FLAG + ')'] + lines += bps + if invocation is not None: + lines += ['bd 1-%d' % last, 'r $t0=0', + 'bp0 ' + GLOBAL_REG_ALLOC + ' "r $t0=@$t0+1; .if (@$t0==0n' + str(invocation) + + ') { be 1-%d }; .if (@$t0==0n' % last + str(invocation + 1) + ') { bd 1-%d }; gc"' % last] + lines += ['g', 'q', ''] + tag = 'merge-%s-%s' % (os.path.basename(src), invocation if invocation is not None else 'all') + txt = run(lines, src, tag) + if 'Malformed' in txt: + sys.exit('cdb rejected a breakpoint command - see ' + str(out / (tag + '.log'))) + counts = {} + for m in re.finditer(r'^EV (\d+)\s*$', txt, re.M): + i = int(m.group(1)) + if 1 <= i <= len(EVENTS): + name = EVENTS[i - 1][0] + counts[name] = counts.get(name, 0) + 1 + fl = re.search(r'POGOFLAG (-?\d+)', txt) + return (fl.group(1) if fl else '?'), counts + + +def report(src, invocation): + flag, c = probe(src, invocation) + + def g(k): + return c.get(k, 0) + + scope = ('GlobalRegAlloc invocation %d' % invocation) if invocation is not None \ + else 'whole translation unit' + print('### %s (%s)' % (os.path.basename(src), scope)) + print(' _PogoFlag %s %s' % ( + flag, 'block merging is DISABLED (PGO instrumentation mode)' if flag == '1' + else 'block merging is live')) + print(' pairwise FgTryTailMerge attempts %4d merged %4d declined %4d' + % (g('pair_attempt'), g('pair_merged'), g('pair_declined'))) + print(' common FgDoCommonTailMerge attempts %4d merged %4d declined %4d' + % (g('common_attempt'), g('common_merged'), g('common_declined'))) + print(' head FgTryHeadMerge attempts %4d (FgCompareMerge %d, FgTryEpilogMerge %d)' + % (g('head_attempt'), g('head_compare'), g('epilog_attempt'))) + merged = g('pair_merged') + g('common_merged') + attempts = g('pair_attempt') + g('common_attempt') + print(' MERGES PERFORMED %d' % merged) + if attempts and not merged: + print('') + print(' Every candidate pair was rejected, so a POLARITY report for this function is a') + print(' real source-shape difference: swap the arms and invert the test.') + elif merged: + print('') + print(' Blocks were folded together after allocation. A POLARITY report here may be a') + print(' downstream effect; confirm with a two-variant difference before touching the arms.') + return c + + +if __name__ == '__main__': + inv = None + rest = [] + it = iter(sys.argv[1:]) + for a in it: + if a.startswith('--invocation='): + inv = a.split('=', 1)[1] + elif a == '--invocation': + inv = next(it, None) + elif not a.startswith('--'): + rest.append(a) + if not rest: + sys.exit(__doc__) + for p in rest: + report(p, int(inv) if inv else None) + print('') diff --git a/tools/compiler-internals/msvc-backend.md b/tools/compiler-internals/msvc-backend.md new file mode 100644 index 0000000000..78a35816ca --- /dev/null +++ b/tools/compiler-internals/msvc-backend.md @@ -0,0 +1,2734 @@ +# MSVC 14.00 backend notes (`c2.dll`) + +Notes on the **compiler** rather than the game: what the bundled optimizer does, how it was read, and how to +observe it. The reimplementation-facing conclusions live in +[IMPLEMENTATION_CHEAT_SHEET.md](../../IMPLEMENTATION_CHEAT_SHEET.md); this file is the evidence behind them and +the starting point for digging further. + +Everything here is about `MSVC1400-SP1/VC/bin/c2.dll`, `Microsoft (R) Optimizing Compiler Version +14.00.50727.762` - the toolchain this project builds with, so it is the authority on our codegen rather than a +proxy for it. + +For a step-by-step walkthrough of the tooling aimed at a newcomer - what to measure, in what +order, and how to read each report - see [finding-mismatches.md](finding-mismatches.md). + +## The spill-cost weight, read directly: it is `1 << loop_depth` + +This supersedes the inferred chain in the two sections below, which was wrong at both ends. Read +out of the code rather than reasoned from the names, `Spillcosts` computes its own weight and +never consults the branch-prediction machinery. + +### Where the weight comes from + +`Spillcosts` opens `sub esp, 0x30` followed by four pushes, so the `[esp+0x28]` that it multiplies +by is a **local** at frame offset 0x18, not an incoming argument - the first thing that ruled out +the idea that a weight is handed in from elsewhere. It is written at five sites, all inside this +block: + +``` +0x10b1066a cmp dword ptr [_PogoFlag], 2 +0x10b10671 jne 0x10b106d9 ; no profile + ; profile path: esi:edi = total count, [ebx+0x190] = this block's 64-bit count + ; total == 0 -> weight = 1 + ; count <= 0 -> weight = 1 + ; otherwise -> weight = count * 0x2710 / total + 1 + ; (0x2710 = 10000, two 64-bit helper calls) +0x10b106d9 mov cl, byte ptr [ebx + 0x17e] +0x10b106df mov eax, 1 +0x10b106e4 shl eax, cl ; weight = 1 << [ebx+0x17e] +0x10b106e6 mov dword ptr [esp + 0x28], eax +``` + +So: + +* **without a profile** - every build here - `weight = 1 << depth`, where depth is the `word` at + `+0x17e` of the block record; +* **with `_PogoFlag == 2`** the weight is the block's share of the total execution count scaled by + 10000, plus one. + +`+0x17e` is a **loop nesting depth**. Every write to it comes from loop machinery - +`@HoistOutOfLoop@16`, `@LpCanonicalizeLoops@4`, `@LgFindExitBlocks1@12`, `@LurNewBlock@12`, +`@FPHoistOutOfLoop@20` - and the block creators zero it or copy it from a parent: +`@FgSplitBlock@12`, `@FgInsertAirlockBlock@16`, `@PtInlDupInlineeBlock@8`, `@PogoOptNewBlock@4`. + +### So the full key is + +``` +key(+0x0c) = sum over regions where the value IS referenced ( refs * count * 2^depth ) + - sum over regions where it is live but UNREFERENCED ( count * 2^depth ) +``` + +with `2^depth` replaced by the scaled profile share under PGO. + +### The weight formula, verified by measurement + +The `1 << depth` reading above was tested on `addUnitToSmallestBehaviourTypeTribe` rather than left +as a reading of the instructions, and it holds quantitatively. + +**The spilled values are the lowest-keyed ones.** `reg_scores.py --find smallestSize` locates the +function at invocation #371 and gives its `+0x0c` keys: + +``` +aicIndex +87 slot +79 firstSlot +71 groupsCount +40 aiType +32 +playerID +30 selectedTribeID +28 this +23 selectedSlot +14 smallestSize +2 +``` + +`slot_map.py` independently reports exactly two values in stack slots - `_smallestSize$` and +`_selectedSlot$` - and those are precisely the two lowest keys, **+2** and **+14**. Two tools, one +reading the compiler's own records through a debugger and one inferring the frame from the +original's disassembly, agreeing without being fitted to each other. + +**Removing one in-loop reference costs exactly one `count * weight` term.** Rewriting the loop +from a bound to a counter, + +```cpp +for (int slot = firstSlot; slot < firstSlot + groupsCount; slot++) // bound in the condition +for (int slot = firstSlot, remaining = groupsCount; remaining > 0; slot++, remaining--) +``` + +takes the `firstSlot + groupsCount` expression out of the loop condition, so `firstSlot` and `slot` +each lose one depth-1 reference. Measured: + +| candidate | bound form | counter form | change | +| --- | --- | --- | --- | +| `firstSlot` | +71 | +57 | **-14** | +| `slot` | +79 | +65 | **-14** | +| `groupsCount` | +40 | +52 | +12 | +| `smallestSize` | +2 | -1 | -3 | +| `aicIndex`, `aiType`, `this` | +87, +32, +23 | unchanged | 0 | + +Two independent values each lose **exactly 14** for losing one reference in the same loop - the +same `count * weight` term, as the formula requires - while every candidate the edit does not touch +moves not at all. `groupsCount` gains because the counter form gives it a new reference in the loop +initialiser. + +**But the lever does not fix the function.** The point of the rewrite was that a counter is +referenced every iteration where a bound merely lives across the body, so its key should rise +enough to win a register and drop the frame from 12 to the original's 8. It does not: the frame +stays at `+4`, and the instruction mix gets much worse - delta 10 to 22, `kind` 0.893 to 0.846, +quick_diff 0.6803 to 0.6179 - because the counter form emits a decrement the original does not +have. The original keeps a genuine bound in `edx`; the counter shape is simply not what it +compiled. + +So the formula is now verified and usable for *predicting* which value is at risk of spilling, and +the measurement confirms the mechanism. What it does not provide is a way to rescue a particular +value here: the only moves available change the instruction stream more than they change the +ranking. Treat it as a diagnostic - "this value will spill because its key is lowest, and here is +the term responsible" - rather than as a fix. + +### What the Bp heuristics actually feed + +Not this. `@BpBlockProbabilities@4` has exactly one caller, +`@BpComputeFlowgraphProbabilities@4`, and that has exactly two: +`@PtInlFindInitialPartialInlineeAux@12` and `@PtInlComputeBenefit@4`. `@FgWeightFlowGraph@4`'s +five callers are `@DoInline@24`, `@PtInlCollectBUInfo@12`, `@LiListFunction@4`, +`@X87MapGFxToSTx@4` and `@PogoFgCreateScaledZeroCounts@20`. + +The static branch-prediction suite is an **inliner** input - it estimates how much a partial +inline is worth - plus an x87 mapping use and a profile path. It does not reach register +allocation, and `Spillcosts` does not call any of it. + +That is the whole reason the guard-inversion test came out flat: inverting +`if (aiType == AITA_NULL) return 0;` changes which arm is predicted taken, but it does not change +any block's **loop depth**, so no weight moves, so no key moves, so the allocation is untouched - +which is exactly what was measured, with the register substitution profile identical in kind and +count. It also explains, in passing, why `#pragma inline_depth(1)` perturbed codegen in the +earlier pragma sweeps while the branch-shape rewrites did not: inlining is the consumer of that +chain. + +### The lever this does give + +Loop depth, and it is predictive. A reference inside a loop at depth *d* contributes `2^d` to its +value's key; the same reference outside contributes 1. A value that is live across a loop without +being referenced in it is charged `2^d` per such region. So moving a reference into or out of a +loop changes that value's allocation priority by a factor of two per level, in a direction the +formula states in advance - which is a much sharper instrument than the branch-shape rewrites +tried above, and the one to reach for when a value is enregistered on one side and spilled on the +other. + +Worth noting what this predicts about the functions in this file: `playerID` in +`aiGiveRaidInstructions` keys at **-147** because it is live across a depth-1 loop while being +referenced almost entirely outside it, so it collects the `count * 2` penalty repeatedly - and +`slot_map.py` independently shows it in a stack slot. The model and the observation agree without +being fitted to each other. + +## Superseded: static branch prediction feeds register allocation (it does not) + +`Spillcosts` keys on `refs * count * weight`, and `weight` is a block frequency. Following where +that comes from leads to the one lever found so far that the source actually controls. + +Block weights come from `@FgWeightFlowGraph@4` / `@FgWeightGraph@12`, which consume edge +probabilities from `@BpBlockProbabilities@4`. With a profile loaded that is +`@BpComputeFlowgraphPogoProbabilities@4`; without one it is `@BpComputeFlowgraphProbabilities@4`, +which combines a suite of static heuristics - the classic Ball-Larus set, each one named: + +``` +@BpLoopBranchHeuristic@12 @BpReturnHeuristic@12 @BpCallHeuristic@12 +@BpPointerHeuristic@12 @BpOpcodeHeuristic@12 @BpGuardHeuristic@12 +@BpLoopExitHeuristic@12 @BpLoopHeaderHeuristic@12 @BpCmpHeuristicTemplate@8 +``` + +So the chain is + +``` +branch shape -> Bp*Heuristic -> BpBlockProbabilities -> FgWeightFlowGraph -> block weight + -> Spillcosts key -> InsertInList order -> ChoosePhysReg +``` + +and **which arm of a branch you write first can change the register allocation**. + +### Two of them decoded + +Each heuristic returns three values - `-1`, `0` or `+1` - built with the `sbb eax, eax` and +`lea ecx, [ecx + ecx - 1]` idioms, where `-1` means "this heuristic does not apply". + +`BpReturnHeuristic` at `0x10c72f00` walks each successor to its terminating instruction, chasing +up to five links through `+0xc`, and tests the tuple opcode at `+8` against **`0x10` and `0x19`**, +the two return forms: + +* both successors return, or neither does - `cmp esi, eax; jne` falls through to + `or eax, 0xffffffff` - the heuristic **abstains** +* exactly one returns - the **non-returning** arm is predicted taken + +`BpCallHeuristic` at `0x10c73030` is the same shape against opcode **`0x0f`**, a call: scan each +successor's instruction list, and if exactly one arm contains a call, the arm **without** the call +is predicted taken. (The opcode numbers are inferred from the function names plus the surrounding +code, not from a typed enum; the control flow is read directly.) + +### Why this is the useful one + +It is predictive rather than trial-and-error. A branch only has its probability - and therefore +its block weight, and therefore its contribution to every candidate's spill cost - decided by arm +order **when the heuristic discriminates**, which is when exactly one arm returns or exactly one +arm calls. Otherwise the heuristic abstains and arm order is free. + +That retro-explains both results this file already records, which is the reason to trust it: + +* `aiGiveRaidInstructions` gained **56.8% -> 60.3%** from swapping the arms of its final + `if`/`else`. Swapping changed which arm is the fallthrough, and with it the predicted + probabilities feeding the weights. +* `playNervous2BikFromPlayer` was **byte-identical** across four source forms, including + early-return against guarded-call. Under this model that is expected: both forms leave both + arms reaching a return, so `BpReturnHeuristic` abstains either way and nothing downstream + moves. + +### How to use it + +When a function is structurally correct but its registers or frame differ, look for branches where +exactly one arm returns or exactly one arm calls, and check our arm order against the original's. +`structural_diff.py` already reports polarity flips; this says which of them are worth acting on - +a flip on a branch where the heuristics abstain is cosmetic, a flip on a discriminating branch +moves the block weights and can move the allocation. + +Still unread, in rough order of likely value: `@BpGuardHeuristic@12` (guard clauses, which the AI +code is full of), `@BpLoopExitHeuristic@12` and `@BpLoopBranchHeuristic@12` (the loop-carried +values that several of these functions spill), and `@BpOpcodeHeuristic@12` with +`@BpCmpHeuristicTemplate@8`, which would say whether the *comparison operator* itself shifts a +probability - that would make `!= 0` against `> 0` a codegen-relevant choice rather than a +stylistic one. + +### Correction: tested, and it is not a usable lever + +The section above was written before the lever was tested. Two things it claimed are wrong, and +the practical conclusion does not survive. + +**`BpGuardHeuristic` is inert.** Decoding `0x10c72e70` shows every path, including the one that +reaches the computation at `0x10c72edb`, falling through to + +``` +0x10c72eec or eax, 0xffffffff +0x10c72ef1 ret 4 +``` + +`or eax, -1` discards whatever was computed, so the function **always returns -1**, which is the +abstain value. The guard heuristic never contributes in this build - the opposite of the +prediction above that it would be the most valuable one to read next. Whether that is deliberate +neutering or a discarded result, the behaviour is the same. + +**The retro-explanation of `aiGiveRaidInstructions` was wrong.** Its winning branch is +`if (canNavigate(...)) commandUnitsToLocation(...) else sendUnitsToCampfire(...)`. Both arms +contain a call, so `BpCallHeuristic` abstains; neither arm returns, so `BpReturnHeuristic` +abstains. Both heuristics abstain on the one branch whose arm order was worth 3.5 points, so that +gain was plain instruction ordering and had nothing to do with block weights. Fitting it to this +model was wishful. + +**The direct test shows no effect on allocation.** `addUnitToSmallestBehaviourTypeTribe` opens +with `if (aiType == AITA_NULL) return 0;` followed by the body - a branch where +`BpReturnHeuristic` genuinely discriminates, one arm being a return block and the other the rest +of the function. Rewriting it as `if (aiType != AITA_NULL) { body } return 0;` is semantically +identical and flips which arm the heuristic predicts taken. Measured: + +| | instructions | `kind` | mix delta | quick_diff | frame | register substitutions | +| --- | --- | --- | --- | --- | --- | --- | +| baseline | 122 | 0.893 | 10 | 0.6803 | +4 | 27 | +| inverted guard | 123 | 0.849 | 11 | 0.5878 | +4 | 29 | + +The code changed - one extra jump and a different block layout - but the **register allocation did +not move at all**. The substitution profile is identical in kind and count: `eax -> edi` 8 times, +`ecx -> edi` 5, `edi -> edx` 4, `eax -> edx` 3, `edi -> ecx` 1, `ebp -> edi` 1 in both, with the +totals differing only by the two added instructions. The frame stayed at `+4`. + +So what survives is only what was read directly off the code: the call chain from the `Bp*` +heuristics through `BpBlockProbabilities` and `FgWeightFlowGraph` into the weight that +`Spillcosts` multiplies, and the decoded behaviour of `BpReturnHeuristic` and `BpCallHeuristic`. +What does **not** survive is the claim that this gives the source a handle on allocation. On the +one function where the construction could be made cleanly, flipping a discriminating branch moved +the block structure and left every register where it was. + +Possible reasons, none of them established: the weight change may be too small to reorder any +candidate given the margins seen in the `+0x0c` keys (`tile` 212 against the next at 148); the two +source forms may present the same successor shapes to the heuristic despite looking different; +or the probability-to-weight step may compress differences more than the chain suggests. Settling +it would mean reading `FgWeightFlowGraph` and `BpBlockProbabilities` rather than inferring from +the ends of the chain, and measuring the weights directly under the debugger the way `Spillcosts` +was measured - which is the next thing to do before anyone trusts this path again. + +## The PGO flag: three gated behaviours, and the phase order + +The global at `0x10d0ffdc` that an earlier section could only describe as "a mode flag, read in +about 400 places, set at invocation level" is **`_PogoFlag`**, the per-function copy of +**`_PogoMasterFlag`** at `0x10d44b4c`. Pogo is Microsoft's internal name for PGO, and the flag +gates three separate behaviours that between them cover both the allocation and the scheduling +walls this file keeps running into: + +| value | what changes | +| --- | --- | +| 0 (our builds) | full tail merging; `Spillcosts` penalty is `count * weight` | +| 1 (instrument) | `FgTryTailMerge` returns immediately - **no tail merging at all** | +| 2 (optimise) | `Spillcosts` penalty is just `weight`; `PogoOptCodeMotion` runs as an extra pass | + +`FgTryTailMerge` at `0x10b2e1b0` opens with `if (_PogoFlag != 1)` wrapping its entire body, which +makes sense - merging blocks would corrupt the profile counters being collected. That is the pass +that merges the duplicate `aiGiveRaidInstructions` is missing. + +**The phase order**, from the twelve `dop2_*` entry points in address order: + +``` +GetInitialTuples -> InitialOptsAndSetup -> PreLowerGlobalOpts -> Codegen -> +PostLowerPreRegAllocGlobalOpts -> GlobalRegAlloc -> CodegenUpdate -> +PostCodegenFlowOpts -> CodegenFinish -> FinalFlowOpts -> PreEncode -> EncodeAndRelated +``` + +`GlobalRegAlloc` is phase 6 and `FinalFlowOpts` is phase 10, and `dop2_FinalFlowOpts` is where +`FgDoLateFlowOpts`, `FgCodeMotion` and - only when `_PogoFlag == 2` - `PogoOptCodeMotion` are +called. So **tail merging and code motion run after register allocation**, which independently +confirms the claim made earlier in this file: two arms whose instructions are identical except +for their registers are not mergeable, so the allocation can be what preserves a duplicate. +`FgCodeMotion` is also the pass that sinks a load past a branch, which is what no source form of +`playNervous2BikFromPlayer` could prevent. + +### Forcing the flag: a real lever, but not the explanation + +`pogo_probe.py` compiles one file twice, the second time with `_PogoFlag` forced across +`Spillcosts` only - an entry breakpoint sets it and the `ret` at `0x10b1153f` restores it - so the +allocation penalty changes while the profile-dependent passes never see a value the build was not +configured for. It does change allocation, measurably, on +`addUnitToSmallestBehaviourTypeTribe`: + +| | frame | distinct callee-saved | instructions | `kind` vs original | mix delta | +| --- | --- | --- | --- | --- | --- | +| original | **8** | **4** | **122** | - | - | +| baseline | 12 | 3 | 122 | **0.893** | **10** | +| `_PogoFlag=2` | **8** | **4** | 115 | 0.624 | 15 | + +The smaller penalty does exactly what the formula predicts - the frame drops from 12 to the +original's 8 and one more value is enregistered, which is the direction every register-pressure +difference in this namespace pointed. But the overall match gets **worse**, because other `== 2` +paths activate and strip seven instructions. On `aiGiveRaidInstructions` it moves the two +contested values from `ebx`/`ebp` - the original's pair, merely swapped - to `esi`/`edi`, which is +further away. + +So the spill-penalty term is a confirmed lever on allocation and on frame size, and it is not the +explanation for the gap; the original was not simply built in this mode. It is worth re-testing +per function when a frame comes out too large, since that is the one symptom it reliably fixes. + +A warning from getting this wrong once: the first attempt put the restore breakpoint on the last +*byte* of `Spillcosts` rather than on its `ret`, so it never fired, the flag stayed 2 into the +profile-dependent passes, and the compile died on a null dereference in `FgFindDominatorsGraph`. +Bracket on a real instruction address, and keep a second restore at the next phase entry. + +### Reading the symbol table without Ghidra + +`pdbnames.py` recovers 9687 public symbols from `c2.pdb` by scanning for `S_PUB32` records +(`u16 reclen | u16 0x110e | u32 flags | u32 offset | u16 segment | name`) and mapping +segment-plus-offset through the PE section table. It is the practical name-to-address route, +because ghidra-mcp's name searches ignore the `program` argument and query whatever program is +current, while only `get_function_by_address` honours it. PDB names are decorated +(`@Spillcosts@12`) where Ghidra shows them stripped; every address agrees. + +Pass names worth knowing, all from that map: + +| area | symbols | +| --- | --- | +| tail merge / cross jump | `@FgTryTailMerge@16`, `@FgDoCommonTailMerge@20`, `@FgTailMergeWithoutCheck@8`, `@FgCrossJumpRemove@4`, `@PpSsaMdSubstPreventsTailMerge@8` | +| code motion and sinking | `@FgCodeMotion@4`, `@FgChangeSink@12`, `@GoptInvariantCodeMotion@12`, `@GoptCodeMotionBlock@12` | +| scheduling | `@Schedule@8`, `@SchNeedSchedule@8`, `@SchSchedGraph@4`, `@SchmdBlockNeedPreSchedule@8`, `_preSched` | +| profile-driven | `@PogoOptCodeMotion@*`, `@PogoCreateFinalOrder@4`, `@PogoDecorateFG@4`, `@BpComputeFlowgraphPogoProbabilities@4` | + +`SchNeedSchedule` turns out to be a weak lever - it schedules any block with more than one real +instruction, gated only on `_preSched` for the pre-pass - but `@PogoCreateFinalOrder@4` shows that +final block order is profile-driven in PGO builds, which is a scheduling difference no source +change can reach. + +## The resolver switches change the score but not the code + +Both resolver switches are **off** in a fresh checkout - 829 `MACRO_STRUCT_RESOLVER(..., false, ...)` +in `src/OpenSHC/Globals` and every `MACRO_FUNCTION_RESOLVER` in the `.func.hpp` files - and every +measurement in this file was taken that way until it was checked. Turning both on +(`tools/reimplementation-control/enable_reimplemented_data.py` and +`tools/reimplementation-helper/batch/enable_local_functions.py`) moves the reccmp percentage a long +way: + +| function | resolvers off | resolvers on | +| --- | --- | --- | +| `computeAttackPossibilities` | 72.4 | **89.7** | +| `recruitHarrassingSiegeEngines` | 65.9 | **73.2** | +| `playNervous2BikFromPlayer` | 64.0 | **76.0** | +| `decideOnTribeAttackLocation` | 64.4 | **68.8** | +| `renderDebugDataAiInfo` | 95.4 | **99.0** | +| `updateAIPlayerState` | 78.6 | **81.7** | +| `aiShouldAttackOrWaitForTeamCoordination` | 80.2 | **83.2** | +| `aiRecruitUnits` | 48.0 | **50.0** | + +The whole build list averages **94.20 over 60** functions with them on. So any percentage quoted +from a default checkout is understated, sometimes by seventeen points, and a figure is only +comparable to another taken in the same configuration. + +**But the structural measures do not move at all.** With the resolvers on, `slot_map.py`, +`structural_diff.py` and `quick_diff.py` return byte-identical results - +`addUnitToSmallestBehaviourTypeTribe` still reports frame `+4`, `kind` 0.893 and mix delta 10, and +the quick_diff ratios stay at 0.8000, 0.6803 and 0.7284 to four decimals. The reason is that a +resolver changes an *operand*, not an instruction: `mov ecx, OFFSET ?instance@...StructResolver@@` +against `mov ecx, 0x2157578` is the same `mov`, and a call through a trampoline is still one +`call`. reccmp scores operands, so it sees the difference; a mnemonic-level or frame-level +comparison does not. + +That is a useful division of labour rather than a problem. The structural tools can be run in a +default checkout and trusted, and the resolvers only need flipping when a percentage is about to be +recorded. It also means the conclusions this file draws about frames, instruction mix and register +pressure are unaffected by the switch - they were checked. + +### /GL does not transfer to this namespace + +`enable_local_functions.py` documents the reason the function resolvers matter beyond runtime: +`/GL` buys nothing while a call still leaves the module, because LTCG cannot see the callee's +register use, and flipping `getSellResourceAmount` took `getSalesPrice` from 68.4% to 100% because +the original keeps `this` in the volatile `edx` across that call. That is exactly the symptom class +behind every register-pressure wall in this file, and no AICState file had ever been given `/GL` - +`cmake/compiler-flags-gl.txt` had 22 entries, none of them AICState. + +Adding all sixty, with the function resolvers on, and verifying `/GL` really reached the compile +command (120 of 120 AICState entries carry it; `file_dependent_read_list` in `CMakeLists.txt` +makes cmake reconfigure when that list changes) produced **no gain whatsoever** - every function +identical, and `planToBuyWhenLowOnResourceAndSnoozeBuildings` 0.8 worse. + +The likely reason is visible in the call sites: AICState's hot callees - +`checkTribeActivityPercentages`, `selectRandomBuildingIDFromListOfBuildingTypes`, and the +`TribesState`/`UnitsState` methods - are not in the build list at all, so those calls leave the +module whatever the flags say. `getSalesPrice` won because its callee was in the list beside it. +So the lever is real but its precondition is unmet here, and the concrete next step is to bring +those callees into the build list first and only then retest `/GL`. + +## Every decoded name, confirmed against c2.pdb + +The whole backend model in this file was decoded from unnamed code, so it is worth recording +that it has since been checked against the real symbols. Loading `c2.dll` **with its PDB** into +a Ghidra project confirms every function this file names, and resolves the one naming confusion +it had: + +| address | PDB name | what this file decoded it as | +| --- | --- | --- | +| `0x10c07350` | `dop2_GlobalRegAlloc` | the register-allocation driver | +| `0x10b10420` | **`Spillcosts`** | accumulates the `+0x0c` ordering key | +| `0x10b11970` | `InsertInList` | candidate priority list, sorts on `+0x0c` | +| `0x10b119f0` | **`InsertByBenefit`** | the other insert, sorts on `+0x40` | +| `0x10b14b70` | `ChoosePhysReg` | picks the cheapest free register | +| `0x10beaef0` | `ColorStackElements` | the stack-slot colourer | +| `0x10beb760` | `colorElemental` | per-element slot placement | +| `0x10beb730` | **`StkGSBufferLevel`** | the class byte at `+0x18` | + +Three of those are worth dwelling on. + +**`Spillcosts`** is the function whose tail accumulates `+0x0c`, and the name is the semantic: +the key is a spill cost. A value gains `refs * count * weight` in each region where it is +referenced and is charged `count * weight` for each region it is live across without being +referenced - which is exactly what "what would it cost to spill this" means. The formula was +derived from the arithmetic before the name was known, and they agree. + +**`InsertInList` against `InsertByBenefit`** explains the error this file used to contain. The +two are adjacent and nearly identical; the old note said "InsertInList sorts on `+0x40`, the +benefit" when in fact `InsertInList` sorts on `+0x0c` and it is `InsertByBenefit`, 0x80 bytes +later, that sorts on `+0x40`. The names make the distinction explicit, and the candidate list +the harness breakpoints is the former. + +**`StkGSBufferLevel`** settles the class byte outright. The `/GS` reading in the section above +was an inference from the arithmetic plus one observed layout; the name says it directly - `Stk` +for stack, `GS` for the buffer security check, `BufferLevel` for the classification. Its +decompilation is also identical to the hand-decode, including `sym+0x63 < 3` for `<= 2`: + +```c +char StkGSBufferLevel(sym) /* sym in EAX */ +{ + if (((sym+0x37 & 0x40000) != 0) && (sym+0x63 < 3) && (4 < sym+0x1c)) + return 3; + return ((sym+0x20 & 0x4000000) != 0) + 1; +} +``` + +The PDB does **not** apply struct types, so the operands are still raw offsets and the fields of +the symbol record remain unnamed. The function names are what the PDB buys. + +**How to query it.** `ghidra-mcp`'s read tools take a `program` argument, so a non-current +program can be queried without disturbing anything: + + get_function_by_address(address="0x10beb730", program="c2.dll") + +That matters because `switch_program` reports success but does **not** redirect the query tools - +after switching, `get_metadata` still named the game executable and symbol lookups still missed. +Pass `program` instead of switching, and a shared session's `currentProgram` is never touched. +REA remains the alternative when no PDB is wanted, but it refuses a caller-supplied PDB +companion, which is why it could only ever report `FUN_` names. + +## Getting symbols + +`c2.dll` references `c2.pdb`, and Microsoft still publishes it: + +``` +https://msdl.microsoft.com/download/symbols/c2.pdb/8E3C6AD714BE4E55B528B0DBEB3BABD81/c2.pdb +``` + +(947 KB; the path is `//` from the RSDS record in the debug directory.) It is a +publics-only PDB - no line numbers, no locals, no struct layouts - but it names every function, and mangled C++ +names carry parameter types for free, e.g. +`?ProcessFunc@CControlGraph@OptDbgNameSpace@@QAEXPAUILivenessCallback@2@@Z`. + +Tools that will not accept a caller-supplied PDB can still be used: parse the `S_PUB32` records out of the PDB +(`reclen, rectyp=0x110E, flags, off, seg, name`), map `(seg, off)` through the PE section table, and you have a +name-to-address table to drive an address-based disassembler or a debugger. `c2.dll`'s image base is +`0x10b00000`. + +The retail build has **no** internal dump switch - no `-d2*` option strings, no asserts, no trace output. The only +diagnostic strings are the PGO log (`Trace formation`, `Copy Elim`, `HCO Def Push`, `PEEP:`). There is no free +ground truth; it has to be read or traced. + +## The backend's own source layout + +The fatal-assert path passes `__FILE__`, so the binary leaks 66 source paths. Useful ones: + +| area | file | +|---|---| +| stack slot layout | `P2/stack.c` | +| coloring | `P2/color.c` | +| register assignment | `P2/regasg.c` | +| global optimizer | `P2/globopt.c`, `P2/globlopt.c`, `P2/globdf.c` | +| IR | `P2/tuple.c`, `P2/fg.c`, `P2/dag.c` | +| x86 lowering and emission | `P2/x86/lower.c`, `P2/x86/code.c`, `P2/x86/mdlist.c`, `P2/x86/fpreg.c`, `P2/x86/fpo.c` | + +An assert also gives a file *and line*, which anchors a function to its source: `ColorStackElements` reports +`P2/stack.c` line 1372. + +## Addresses + +VAs for an unrebased load at `0x10b00000`: + +| symbol | VA | +|---|---| +| `@dop2_GlobalRegAlloc@4` | `0x10c07350` | +| `@dop2_PostLowerPreRegAllocGlobalOpts@4` | `0x10c07350`-adjacent, see PDB | +| `@Color@4` | `0x10b0cf40` | +| `@SsaColor@4` | `0x10be2b30` | +| `@CanColorPseudoreg@12` | `0x10b1b920` | +| `@ColorStackElements@12` | `0x10beaef0` | +| `@StkColorCmp@12` | `0x10bebb30` | +| `@QSortColors@12` | `0x10bebbb0` | +| `@SwapColors@8` | `0x10bebcf0` | +| `@StkColorWeightForEBPAdj@12` | `0x10bec120` | +| `@StkInsertDummyColor@8` | `0x10bec3c0` | +| `@Spillcosts@12` | `0x10b10420` | +| `@SpillOrDefer@12` | `0x10b0d690` | +| `@RegSpill@12` | `0x10bdad70` | +| `@CgAnnotateEdgeWeight@12` | `0x10b061d0` | +| generic bitset iterator (`FUN_10b04fa0`) | `0x10b04fa0` | + +## Stack slot assignment - decoded + +`ColorStackElements` builds one 28-byte record per stack element, sorts the records with `StkColorCmp`, and hands +out offsets in the sorted order. Record fields in use: `+0x08` the local's size rounded up to 4, `+0x0C` its +accumulated reference weight (accumulated with `+=` when elements merge), `+0x18` a class byte. + +```c +// StkColorCmp +if (a->class != b->class) // byte at +0x18, unsigned + return (a->class > b->class) ? (dir ? +1 : -1) : (dir ? -1 : +1); +ka = (a->weight * 1000) / a->size; // +0x0C scaled, divided by +0x08 +kb = (b->weight * 1000) / b->size; +return ka == kb ? 0 : (ka >= kb ? +1 : -1); +``` + +So for a homogeneous block of same-class scalars the layout is **descending weight per byte**. The direction flag +is a function-level property; `ColorStackElements` walks the sorted elements `0..n` or `n-1..0` accordingly and +passes the same flag to `QSortColors`. + +Confirmed on `planToBuyWhenLowOnResourceAndSnoozeBuildings` (fourteen `int allowX = 1;` locals, eleven colored): + +| slot | local | refs | declared | +|---|---|---|---| +| -48 | allowBlacksmiths | 5 | 6th | +| -44 | allowTanners | 5 | 7th | +| -40 | allowQuarries | 4 | 8th | +| -36 | allowWoodcutters | 4 | **4th** | +| -32 | allowBakeries | 4 | 9th | +| -28 | allowBreweries | 4 | 10th | +| -24 | allowInns | 4 | 11th | +| -20 | allowWells | 3 | 13th | +| -16 | allowFarms | 3 | **12th** | +| -8 | allowIronMines | 3 | **1st** | +| +8 | allowArmourers | 5 | 1st, on `_playerID$`'s slot | + +Monotone in weight, scrambled in declaration order. Two further checks: adding **one** reference to a local +reshuffled the whole table - another local left the stack for a register, a third took a vacated slot - and the +result was again exactly monotone (`5 5 5 4 4 4 4 3 3 3`); and renaming `allowQuarries` to `zzz` and +`allowWoodcutters` to `aaaaaaaaaa`, inverting both alphabetical order and length for an adjacent equal-weight +pair, left the function byte-identical. + +The **class byte at `+0x18`** is the primary key, and it is no longer unknown. A twelve-instruction +classifier at `0x10beb730` computes it, and the placement pass at `0x10beb760` inlines the same thing: + +``` +0x10beb730 test dword ptr [eax + 0x37], 0x40000 +0x10beb737 je 0x10beb748 +0x10beb739 cmp dword ptr [eax + 0x63], 2 +0x10beb73d jg 0x10beb748 +0x10beb73f cmp dword ptr [eax + 0x1c], 4 +0x10beb743 jle 0x10beb748 +0x10beb745 mov al, 3 +0x10beb747 ret +0x10beb748 test dword ptr [eax + 0x20], 0x4000000 +0x10beb74f setne al +0x10beb752 add al, 1 +0x10beb754 ret +``` + +that is + +``` +class(symbol) = 3 when (symbol+0x37 & 0x40000) + and symbol+0x63 <= 2 + and symbol+0x1c > 4 + = 1 + ((symbol+0x20 & 0x4000000) != 0) otherwise +``` + +so the class is **1, 2 or 3**, and the only size-dependent term is `symbol+0x1c > 4`: anything wider +than a dword takes class 3 and sorts as a group apart from the scalars. The byte is written only when +a gate holds - + +``` +((flags(DAT_10d195cc + 0x44c) & 0x20000) && DAT_10d0d700[0x17]) || (byte(DAT_10d0d700 + 0x2b) & 1) +``` + +- and otherwise stays 0, so with that feature off every slot shares one class and the key degenerates +to the weight alone. + +**What it is.** Read off the arithmetic rather than from a symbol name, this is the `/GS` +buffer-ordering rule: arrays are what can overflow, so they are classed separately and placed nearest +the canary. `renderDebugDataAiInfo` corroborates it - `char text[1000]` sits at `-1004`, above every +int scalar (`-1024` through `-1008`) and immediately below `__$ArrayPad$` at `-4`, which is the layout +that rule produces. The `/GS` reading is an inference from the arithmetic plus that one observation, +not a proven name; the arithmetic itself is read directly off the instructions. + +That explains the symptom this entry used to record as unexplained - arrays and address-taken locals +interleaving with scalars for reasons the weight model does not cover - because the class outranks the +weight. + +**It does not unblock `planToBuyWhenLowOnResourceAndSnoozeBuildings`**, which is what prompted the +work, and the prediction that it would was wrong. All ten of that function's locals are `int`, so +`symbol+0x1c` is 4 for each, none reaches class 3, and the class is uniform unless the `0x4000000` +flag differs between them. That leaves the weight, where both sides agree, and then declaration +order, which a hill-climb over all fourteen declarations plus three hand-made swaps showed is already +a local optimum. The byte discriminates by width, and that function has nothing but dwords. + +Finding this file's functions, since it is not obvious: the four `P2/stack.c` assert sites come from +searching the image for the UTF-16 path and then for pushes of its address. Their line numbers are +1372, 2533, 2534 and 2573, and 1372 is the one already attributed to `ColorStackElements`, which +fixes that function at `0x10beaef0`. + +`slot_weight.py` in `../reimplementation-helper/batch/` reports this per function. + +## Registers - what is and is not known + +**Established.** `.rdata` holds the register names in three parallel 4-byte-stride tables terminated by +`"noreg"`. The table is indexed **backwards from its end**, and the numbering is 1-based in x86 encoding order: + +``` +eax=1 ecx=2 edx=3 ebx=4 esp=5 ebp=6 esi=7 edi=8 (0 = none) +``` + +An earlier revision of this file read the table forwards and recorded the reverse (`edi=0 ... eax=7`). That was +wrong; reading `0x10cfc190 - (n-1)*4` reproduces the names above exactly. + +**Established.** A set is a 64-bit bitvector with set/clear mask tables indexed by `n & 0x3f` +(`DAT_10cf65e8`/`DAT_10cf67e8`), and the iterator at `0x10b04fa0` takes the lowest set bit, adds the set's base, +and clears the bit - so a set is always walked in ascending order: + +```c +for (; (set >> bit & 1) == 0; bit++) {} // low dword first, then the high one +return set_base + bit + dword_offset; +``` + +**Not established.** That iterator is *generic*, not a register allocator. Where its use could be observed it was +walking sets of symbols - `ColorStackElements` feeds its result straight into a symbol array as +`DAT_10d0faa4 + n * 4` - and tracing `sellExcessGoods`'s allocation returned members numbered 0 to 40 across 140 +distinct set objects, which is a symbol or pseudo-register space rather than the eight GPRs. An earlier claim here +that register choice is "the lowest-numbered free register" was an inference from the name table plus the +ascending-order mechanic, and is retracted. + +`Spillcosts` is 854 lines of live-set bitvector manipulation with no cost arithmetic in it, which suggests the +cost model decides *who gets spilled* rather than *which register a winner gets*, but that is also not traced. + +### Where registers are actually chosen + +Tracing one function's allocation answered this. During `sellExcessGoods` (invocation #613) the colouring entry +points ran as follows - `Color` once, `SsaColor` twice, `ColorStackElements` once, `Spillcosts` six times, +`SpillOrDefer` seven times, and `CanColorPseudoreg` and `RegSpill` not at all. So `@Color@4` is the per-function +register colourer, and bounding it with the next public symbol gives its 29 direct callees, among them: + +| callee | VA | +|---|---| +| `@ChoosePhysReg@12` | `0x10b14b70` | +| `@OrderGlobregs@12` | `0x10b11770` | +| `@SubsumeGlobregs@8` | `0x10b0fdb0` | +| `@InsertNewGlobregs@8` | `0x10b133e0` | +| `@UpdateAffectedGlobregs@20` | `0x10b15470` | +| `@HandleCurrUncolorable@8` | `0x10b0d830` | +| `@BetterToDefer@4` | `0x10b19de0` | +| `@InitialSplitPoints@16`, `@InsertSplitPoints@28` | `0x10b0ded0`, `0x10b13460` | + +**`@ChoosePhysReg@12` is where a physical register is picked.** Its second argument (`edx`, it is `__fastcall`) +is the **mask of still-available registers**, and watching that mask across calls gives the consumption order +directly. During `sellExcessGoods` it ran 18 times and the mask walked: + +``` +ffffffff -> fffffffd -> ffffffed -> fffffeed -> fffffeec -> fffffee8 + bit 1 +bit 4 +bit 8 +bit 0 +bit 2 + esi ebx (non-GPR) edi ebp +``` + +which is exactly our generated prologue, `push esi; push ebx; push ebp; push edi`. The original consumes the same +four registers in the opposite order, `ebp -> edi -> ebx -> esi`. That difference is now an **observable**, not an +inference: run the trace, read the mask progression, compare with the original's prologue. + +`eax` is not the chosen register - it returns pointers or 0 - so the assignment is recorded in a structure rather +than returned. + +Internally `ChoosePhysReg` does use `bsf` (lowest set bit) over a mask, at `0x10b14d03` and `0x10b151de`, and the +mask is first narrowed with the same 64-bit set/clear tables (`and`/`or` against `DAT_10cf65e8`/`DAT_10cf67e8`). +So "lowest-numbered free register" holds **within the already-filtered candidate set**, which is why the observed +consumption order (1, 4, 0, 2) is not ascending. + +### Which candidate asks first: OrderGlobregs and InsertInList + +`@OrderGlobregs@12` (`0x10b11770`, 512 bytes) runs once per function and does **not** sort - it builds an ordered +list by repeated insertion through `@InsertInList@8` (`0x10b11970`), which is 128 bytes and fully readable. In it +`eax` is the candidate being inserted and `edi` the list head, `+0x14`/`+0x18` are next/prev, and there are two +sort keys: + +```c +// InsertInList - list kept in descending (key, key2) order +if (new->key > p->key) insert new before p; // +0x0c, compared SIGNED +else if (new->key == p->key && + new->key2 >= p->key2) insert new before p; // +0x44, compared UNSIGNED +else advance to p->next; +``` + +with an insert-at-front fast path when the new node outranks the head, so among equal keys the newer candidate +goes first. `+0x44` is a small symbol id used only as a tiebreak. + +`+0x0c` is **not** the benefit, though an earlier reading of these notes said so. The sibling inserter +`@InsertByBenefit@8` (`0x10b119f0`) orders on `+0x40` and *copies* `+0x0c` from the list neighbour +(`mov ecx,[ecx+0xc]; mov [edx+0xc],ecx`), and `@BetterToDefer@4` sets `+0x0c` to `min(components) - 1`, so +`+0x0c` is a rank derived while ordering. The benefit is **`+0x40`**, which `BetterToDefer` also gates on +(`cmp dword ptr [eax+0x40], 0; jle` - not worth it). Tracing all three fields for `sellExcessGoods`: + +| name | `+0x0c` rank | `+0x40` benefit | `+0x44` id | +|---|---|---|---| +| `amount` | +104 | **70** | 111 | +| `variance` | -4 | **22** | 110 | +| `resourceType` | -34 | **20** | 39 | +| `j` | +88 | **20** | 108 | +| `resourceOffset` | -26 | **20** | 35 | +| `playerResourceBase` | -52 | **12** | 17 | +| `aicIndex` | +88 | **8** | 20 | +| `this` / `aiType` | -134 / +42 | **6** | 8 / 4 | +| `playerID` | -134 | **4** | 7 | + +The benefit is a small positive **weighted use count**: `amount`, assigned in eight branches and read by the +`sellGoods` call, scores 70; `playerID` scores 4. So register candidacy is driven by the same quantity as stack +slot order - how often, and at what loop depth, a value is used. + +Traced over `sellExcessGoods`, the candidates and their scores look like this (40+ insertions, abbreviated): + +``` +key=+104 key2=6f key=+97 key2=11 key=+96 key2=22 key=+88 key2=14 +key= +88 key2=6c key=+85 key2=74 key=+72 key2=6b key=+64 key2=27 +key= +42 key2=04 key=+40 key2=0b key=+32 key2=08 key=+32 key2=07 ... +key= -4 key2=6e key=-20 key=-26 key2=23 key=-34 key2=27 ... +key=-155 key=-156 key=-160 +``` + +So the pass scores every candidate and orders them by descending benefit; the positive-scoring ones are the +handful that actually win registers, and they ask `ChoosePhysReg` in that order. Candidates recur with different +scores across the run (`key2=0x27` appears at -34, then +64, then -2), which is the colourer iterating - `Color` +runs once but `Spillcosts` and `SpillOrDefer` run six and seven times for this function, re-scoring between +rounds. + +**The whole chain, then:** `OrderGlobregs` scores and orders the candidates, `Color` walks that order, and +`ChoosePhysReg` serves each from the still-available mask. Since the order is a benefit score derived from how +values are used, the source-level lever on register assignment is the same one as for stack slots - how often and +where each value is read - which is why permuting statements sometimes moves a register and usually does not. + +### Reading the scores per source variable + +The candidates *can* be mapped back to source names. A candidate node's `+0x00` is its symbol record (0 for a +compiler temporary), the symbol record's `+0x04` points one byte before the variable's name, and the names sit in +a string pool with a small header packed after each string: + +``` +name = poi(poi(poi(candidate) + 4)) + 1 +``` + +`reg_scores.py` does this. `--probe` counts candidate insertions per `GlobalRegAlloc` invocation so the function +can be picked out (ours is normally the largest - for `sellExcessGoods` 42 candidates against the +resolver-template flood at 33, which identifies it in one run with no source edits), and `--invocation N` dumps +the table. For `sellExcessGoods`: + +| score | name | | score | name | +|---|---|---|---|---| +| +104 | `amount` | | -4 | `variance` | +| +88 | `aicIndex` | | -26 | `resourceOffset` | +| +88 | `j` | | -34 | `resourceType` | +| +42 | `aiType` | | -52 | `playerResourceBase` | +| +72, +96, +85 | temporaries | | -134 | `this`, `playerID` | + +and then a second round re-scores several of them - `this` and `playerID` from -134 to +32, +`playerResourceBase` from -52 to +97, `resourceType` from -34 to +64 and then to -2 - because the colourer +iterates, which is also why `Spillcosts` and `SpillOrDefer` run six and seven times here. + +Run across the functions that had resisted every source-level hypothesis, the scores give each one a +recognisable signature: + +| function | match | scores | +|---|---|---| +| `aiRecruitUnits` | 50.0% | `buildingID` +188, `unitID` +174, `unitType` +51, `recruitInterval` +47 against `recruitDiggingUnit` **-335**, `i` **-279**, `behaviourType` -251, `recruitCount` -197 | +| `decideOnTribeAttackLocation` | 68.8% | `destinationTile` +122, `tribeID` +93, `destinationY` +58, `destinationX` +42, `buildingID` +42, `targetPlayer` +40, `this` -1 | +| `recruitHarrassingSiegeEngines` | 73.2% | everything positive and bunched: `unitID` +44, `tribeID`/`engineIndex`/`buildingID` +36, `aiType` +32, `siegeEngineToBuild` +26, `harrassingSiegeEnginesMax`/`this` +21, `aicIndex` +11, `playerID` +9, `engineersGuild` +2 | +| `planToBuy...SnoozeBuildings` | 90.3% | the fourteen `allowX` locals all **negative** at first (-20 to -94) with only `playerID` +46 and `wood` +26 positive; a later round flips some to +48 while `allowWells` stays worst (-94, -93, -76, -43) | + +Three different diagnoses fall out, where before all four looked like the same "allocator noise": + +* `recruitHarrassingSiegeEngines` is decided by **near-ties** - four candidates at +36, two at +21. With equal + scores the order falls to the symbol-id tiebreak, so no source change states a preference and small edits flip + the result arbitrarily. That matches it resisting every targeted attempt. +* `aiRecruitUnits` has a **structural** difference: our loop counter `i` is a candidate at -279, where the + original reuses the dead `playerID` parameter as its counter and so has no such candidate competing at all. +* `planToBuy` is **iteration-dependent** - the locals that matter start negative and only become positive in a + later colourer round, so its layout is decided by the spill rounds rather than by a single ordering. + +That makes the register side measurable per variable. `playerID` scoring -134 in the first round is why our build +serves it late and leaves it in a volatile register (`edx`), where the original keeps it in `ebp` from the start; +the question "did this source change move `playerID` up the list?" can now be answered with one trace instead of +being inferred from a register permutation in the reccmp diff. + +### ChoosePhysReg decoded: a cost minimisation + +The pick is not a bit-scan over a narrowed mask - that reading was wrong, and so was the note here calling +`0x10ce5a08` "not a per-register preference table". It is a table of **per-class candidate lists**, which is why +it holds pointers; class 0, the integer class, reads `eax, ecx, edx, esi, edi, ebx, ebp` - volatiles first, then +callee-saved, `esp` excluded. The rule, from `P2/color.c`: + +```c +cost[] = 0 // array at 0x10d14f10, indexed reg*4, per pick +for each interfering candidate g with g->benefit > 0: + for each (reg, count) preference pair of g: + cost[reg] += count; // a neighbour wanting reg makes reg dearer + if g has exactly one legal register R: + cost[R] += g->benefit * 100; // taking R from a single-choice neighbour is expensive +for each (reg, count) preference pair of this candidate: + cost[reg] -= count; // our own affinity makes reg cheaper +pick = the register in classList[class] that is legal here with the lowest cost + (ties to the earlier class-list entry) +``` + +`reg_scores.py --whyreg` dumps that array at each pick. For `sellExcessGoods`: + +``` +candidate got eax ecx edx ebx esp ebp esi edi +aicIndex eax [0] 2 0 0 0 0 0 0 +j edx 0 2 [0] 0 0 0 0 0 +variance edi 0 2 0 0 0 0 0 [0] +resourceOffset ebx 0 2 0 [0] 0 0 0 0 +playerResourceBase edx 0 2001 [0] 0 0 0 0 0 +this ecx 0 [-1] 1 0 0 0 0 0 +playerID edx 0 0 [-1] 0 0 0 1200 0 +resourceType ebx -1 0 0 [-1] 0 0 0 0 +``` + +Every entry is accounted for: `ecx` costs 2 for everyone because `this` has a thiscall affinity to it, and `this` +takes it at -1 because its own affinity subtracts; `playerID` sees `esi` at 1200, which is +`playerResourceBase`'s benefit of 12 times 100, a neighbour with one register left; the zeros are ties and fall +to class-list order, which is why `aicIndex` lands in `eax`. + +So the counterfactual is arithmetic. To move `playerID` out of `edx` into `ebp`, `ebp` needs a cost below -1, +i.e. `playerID` needs an affinity to `ebp` worth 2 or more. + +### Attributing each cost term: `--whycost` + +"`ebx` costs 1100" is not yet a statement about the source. Three instructions do the accumulating - +`0x10b14c2c` adds a neighbour's preference count, `0x10b14caa` adds `benefit * 100` when a neighbour has one +legal register left (`ebx` holds that neighbour), and `0x10b14d59` subtracts this candidate's own affinity - so +breaking on them and reading the neighbour's name attributes every term. `--whycost` does that: + +``` +aiRecruitUnits + playerID -> ebx + ebx total +13800 PIN +800 by (temp); PIN +300 by (temp); PIN +100 by (temp) + (temp) -> edx + ebp total +4700 PIN +300 by (temp); PIN +1400 by behaviourType; PIN +1000 by i + edx total +1600 PIN +1600 by this +``` + +The useful line is the second: **`ebp` is expensive for everyone because `behaviourType` pins it with 1400 and +`i` with 1000**. Those are named source variables, and it is a stronger statement than their own bad scores - +they do not merely lose their own pick, they make `ebp` cost 2400 more for every other candidate. It also +explains why removing `i` by counting `recruitCount` down lost ground: that just moves the pin to +`recruitCount`. + +The same run on `sellExcessGoods` is the cautionary half: + +``` + playerID -> edx + esi total +1200 PIN +1200 by (temp) + edx total -1 OWN -1 by itself +``` + +The 1200 is charged by a **compiler temporary**, not by a named local - an earlier note here guessed it was +`playerResourceBase` because its benefit of 12 times 100 matches, and that was a coincidence. Attribution is +what tells the two apart, and only a named source variable is something a programmer can act on. + +### Acting on the attribution: measured, and it does not pay + +`--whycost` named `behaviourType` (+1400) and `i` (+1000) as the two source variables pinning `ebp` in +`aiRecruitUnits`, so the indicated edit is to stop them being separate candidates. Every form of that was tried +against reccmp, from a 50.0% baseline: + +| edit | match | +|---|---| +| baseline | **50.0%** | +| `recruitDiggingUnit` as `bool` | 49.5% | +| count `recruitCount` down, drop `i` | 48.7% | +| narrow `behaviourType` into the loop | 48.9% | +| narrow both `behaviourType` and `unitType` | 36.6% | + +All five lose, and the tool explains why - which is the part worth keeping. Re-running `--whycost` on the +narrowed variant shows `behaviourType`'s pin went **up**, from +1400 to +1600, and `ebp`'s total from 4700 to +4900. Benefit is uses per unit of live range, so shortening a live range with the same uses *raises* the +benefit, and the pin is `benefit * 100`. The three levers therefore all fail for the same reason, coherently: + +* narrowing a range raises the benefit, so the pin gets dearer; +* deleting a candidate moves the pin to whatever replaces it (`recruitCount` took it over); +* widening a range adds interference instead. + +So the pin cannot be reduced from the source at all: only *fewer uses* would lower it, and the uses are what +the function has to do. `aiRecruitUnits` at 50.0% is a local optimum under every lever this model exposes, and +that is now a derived statement rather than five failed guesses. + +See [reading-reg-scores.md](reading-reg-scores.md) for a plain-language walkthrough of the output. + +So across the three functions measured: `recruitHarrassingSiegeEngines` charges nothing at all (pure class-list +order, nothing to act on), `sellExcessGoods` is pinned by temporaries (nothing to act on), and `aiRecruitUnits` +is pinned by `behaviourType` and `i` (something to act on, if the computation can be restructured so those +values do not exist as separate candidates). + +### Why there is still no source-level fixer + +The benefit is a property of what the code *does*, not of how it is written. It counts uses weighted by block +frequency, so none of the levers that cost nothing semantically - declaration order, statement order, renaming, +redeclaring - can move it. That is the reason every syntactic hill-climb in this repo has failed to shift a +register assignment, and it means a tool cannot be asked to "fix" a register mismatch by rewriting syntax. + +Affinities are the only term a programmer could hope to influence, and they come from **copies between a value +and a fixed register**: a parameter arriving, an argument passed in a particular slot, a return value in `eax`, +`this` in `ecx`. That is a semantic property - where a value crosses a call boundary - and for the +*callee-saved* registers it is barely reachable at all, because affinities to `ebx`, `ebp`, `esi` and `edi` come +from copies the allocator itself inserts rather than from anything the source states. The interference term is +likewise a property of which values are simultaneously live. + +What the numbers do support is a sharper statement of the problem. For an `alloc-only` function either + +* our use counts differ from the original's, in which case the fix is to restructure the computation so each + value is used the same number of times in the same places - `aiRecruitUnits` carrying a loop counter the + original does not have is this case - or +* our use counts already match, in which case what remains is the symbol-id tiebreak and the function is + unreachable from the source. `recruitHarrassingSiegeEngines`, with four candidates tied at +36, is this case + and should be abandoned. + +`reg_scores.py --rankdiff` tells you which of the two you are in, per variable. It pairs our benefit ranking +with the callee-saved save order on both sides - registers are consumed in the order they are first needed, so +the push order *is* the consumption order, and it comes free from the last reccmp run. For `sellExcessGoods`: + +``` +our benefit ranking: 1 amount 70 2 variance 22 3 resourceType 20 4 resourceOffset 20 ... +consumption order original: ebp edi ebx esi + ours: esi ebx ebp edi +``` + +Both sides use the same four registers, so the candidate *set* matches and neither side carries a value the other +does not - that much is solid, and it rules out the `aiRecruitUnits` case for this function. What the permutation +*means* cannot be settled from here: either our candidates are ranked differently, or they are ranked the same +and `ChoosePhysReg` narrowed the mask differently. The tool says so and labels the pairing a hypothesis rather +than a finding, because the mask narrowing is undecoded. It also flags the degenerate case where the two orders +are exact reverses, which would be systematic rather than per-candidate. + +So the honest scope of the "fixer" is: it decides set-mismatch against order-mismatch, names the candidate that +would have to be re-ranked under the ranking hypothesis, and states what that would cost (a change in weighted +use counts, which is a change in what the code does). It is a decision procedure, not an automatic rewrite, and +the last step from "this candidate is mis-ranked" to "therefore edit this line" needs the mask narrowing decoded +first. + +### Do the suggestions prescribe a fix? Measured: no + +`reg_scores.py --advise` names the patterns above and suggests what to try. Those suggestions were then tested +against reccmp on `aiRecruitUnits` (50.0%), the function with the strongest structural signal - a loop counter `i` +at -279 and a flag `recruitDiggingUnit` at -335 against a +188 top candidate: + +| variant | suggestion acted on | match | +|---|---|---| +| baseline | - | **50.0%** | +| count `recruitCount` down, drop `i` | DISFAVOURED loop counter | 48.7% | +| `recruitDiggingUnit` as `bool` | DISFAVOURED flag, and Ghidra renders it `bool` | 49.5% | + +Both lost ground. Removing a disfavoured candidate changes the whole scoring landscape - fewer candidates means +different spill rounds - so "delete the worst one" is not a monotone improvement, and the scores diagnose without +prescribing. + +What the scores are reliably good for is **triage**, including the negative verdict. A function whose candidates +share scores (`recruitHarrassingSiegeEngines`: four at +36, two at +21) cannot be steered from the source at all, +because the order falls to the symbol-id tiebreak - so the correct action is to stop working on it. That is worth +knowing before spending an afternoon on hill-climbing, even though it never adds a percent. + +Still not decoded: how the score itself is computed, and what narrows `ChoosePhysReg`'s mask before the `bsf`. +One dead end recorded so it is not retried: the table at `0x10ce5a08`, indexed by `eax * 4` in `ChoosePhysReg`, +holds pointers and runs into unrelated data after a few entries, so it is not a per-register preference table. + +## Debugger harness + +`cl.exe` is 32-bit, so use the x86 `cdb` (`C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe`). +`c2.dll` loads **unrebased** at `0x10b00000`, so PDB addresses work directly as breakpoints and cdb never needs to +load symbols. `c1xx.dll` and `c2.dll` run in-process, so no child-process debugging is needed. + +``` +sxe ld:c2 +g +r $t0=0 +bp1 0x10b04fa0 ".printf \"IT arg=%08x\\n\", @eax; gc" +bp2 ".printf \" ret %d\\n\", @eax; gc" +bd 1-7 +bp0 0x10c07350 "r $t0=@$t0+1; .if (@$t0==0n613) { be 1-7 }; .if (@$t0==0n614) { bd 1-7 }; gc" +g +q +``` + +Four things learned the hard way: + +* One TU of ours costs about **1455 `GlobalRegAlloc` invocations and 118k iterator calls**. Leave the inner + breakpoints **disabled** and let the outer one enable them for the invocation of interest; seven enabled + conditional breakpoints blew a 900-second timeout, the disabled form finished comfortably. +* `gu` inside a breakpoint command makes cdb skip the rest of that command ("previous commands caused target + execution inside an event handler"). To read a return value, break on the function's `ret` instead. +* Nested `.printf` quoting breaks the outer parse. Keep every breakpoint command one level deep and gate it on a + pseudo-register flag that the outer breakpoint sets. +* When taking `ret` addresses from a disassembly, bound the scan to the function. Reading 400 bytes and taking the + first six `ret`s crossed into neighbouring functions and produced 1210 "returns" from 532 calls, with pointer + values posing as register numbers. + +**Identifying the invocation** is the fiddly part: hundreds of resolver-template instantiations allocate +identically (exactly 463 iterator calls each), so size alone does not single out the function. Compile the file +twice, with the function's body replaced by `return;` the second time, and find the invocation whose count drops +by the total difference. For `sellExcessGoods` the total fell by 524 and the matching invocation was #613 at 532 +calls; re-running with the breakpoints armed captured exactly 532, which confirms it. + +## Order of work, and where reccmp belongs in it + +The match percentage is the goal, but it is a poor instrument for most of the way there, because it is +dominated by register and stack-slot *naming* - terms the source cannot state. Worse, it moves against +real progress: adding a local that the original has shifts every `[esp + N]` displacement in the function, +and each of those is scored as a mismatch until the layout lines up, so a correct change reads as a +regression. Work in this order instead, and each step has an instrument that is immune to the next step's +noise: + +1. **Frame size and slot layout** - `slot_map.py FILE.cpp`. Compares our `/FA` slot table against the + layout inferred from the original's disassembly: offsets, sizes, reference counts and the sort key. + A frame that is too small is a missing local, and the reference counts say which local is in the wrong + place. Independent of every later step. +2. **Instruction mix** - `structural_diff.py NAME`. `delta` names concrete defects ("we emit two `xor` + the original does not"); `kind` is the bare mnemonic sequence. Independent of register naming. +3. **Statement order** - the same `kind` tier, once `delta` is zero. This is what took the sixteen + `setAICParameters_NN` functions to 100%. +4. **Register and slot naming** - only now is the percentage the right objective, because it is all that + is left. `reorder_search.py` and `perm_decls.py` hill-climb on it. + +The reason this ordering works is that the earlier steps are *causes* and the later ones are *effects*: +register allocation is deterministic given the compiler's IR, so an identical instruction stream in an +identical frame allocates identically. Chasing the effect while the cause is still wrong is what makes a +function feel stuck. + +A worked case of the instruments disagreeing, and of trusting the right one, is below: on +`renderDebugDataAiInfo` the frame went from wrong to byte-exact - all seven slots, every offset and every +reference count - while reccmp *fell* by 0.9 points. + +## Frame size as a local-count oracle + +`sub esp, N` in the prologue is the one number in the original that states, directly and without any +allocation noise, **how much stack the original function's locals occupy**. It is worth checking first, +because a mismatch there is a missing or surplus local and no amount of statement reordering will fix it. + +Worked example, `AICState::renderDebugDataAiInfo` (`0x004d1320`). The original opens `sub esp, 0x400`; our +version opened `sub esp, 0x3f8`, so the original had 8 bytes - two dwords - of locals we did not. Two +independent instruction-mix defects pointed at the same two values: + +* `this->aics[aiType - 1]` appeared at three call sites, each emitting its own `imul reg, reg, 0x2a4`. + The original emits that `imul` **once**, and earlier stores `aiType - 1` to a slot (`add ecx, -0x1; + mov [esp+N], ecx`). Caching the index in a local collapsed three `imul` to one. +* the original *also* caches the resulting element address: `add eax, ecx; mov [esp+N], eax`, read back + later as `mov ecx, [esp+N]; add eax, [ecx + 0x1f4]`. That is the second dword. + +Adding the two locals took the frame from `0x3f8` to exactly `0x400`, the instruction-mix delta from 13 to +3 and the mnemonic-sequence similarity from 0.943 to 0.979. The frame size is what told us *how many* +locals to look for; the mix told us *which*. + +The caveat is that the frame also moves when the compiler spills, so a frame that is too small is strong +evidence of a missing local while a frame that is too large can be either a surplus local or a spill. + +## The mode flag at 0x10d0ffdc is not a lever, and neither pragma pays here + +`FUN_10b10420` switches the penalty on `cmp dword ptr [0x10d0ffdc], 2` - `key -= weight` when it +equals 2, `key -= count * weight` otherwise - which raised the obvious question of whether the +original was simply compiled in the other mode. It was not, and the flag is not reachable from the +source: + +* measured during our own compile, with a breakpoint on `dop2_GlobalRegAlloc` printing it: + **`MODEFLAG 0`**. So we take the larger `count * weight` penalty. +* `#pragma optimize("s", on)` does **not** change it - still 0 - so it is not the per-file + size/speed switch. It is read in about 400 places and is set at invocation level. +* and size-favoured code is plainly not what the original used anyway: that pragma takes + `aiGiveRaidInstructions` from 241-vs-234 instructions and a mix delta of 7 to 205 instructions + and a delta of **52**, with the quick_diff ratio collapsing 0.728 -> 0.372. + +So both sides compute the key with the same formula, and the ranking difference is not a flag. + +The two pragmas that the cheat sheet records as sometimes winning were also measured here, and both +**do** fix the structural defect while losing more than they gain. `aiGiveRaidInstructions` is +missing one copy of its tile computation because MSVC cross-jumped three identical `if`/`else` arm +tails into one, and the original kept two: + +| variant | quick_diff ratio | mix delta | `imul 0x32c` (original has 4) | `this` lands in | +| --- | --- | --- | --- | --- | +| as committed | **0.728** | **7** | 3 | ebp | +| `optimize("y", off)` | 0.262 | 10 | **4** | edi | +| `inline_depth(1)` | 0.708 | 15 | **4** | ebp | +| both | 0.253 | 26 | **4** | edi | +| `optimize("s", on)` | 0.372 | 52 | 4 | - | + +`optimize("y", off)` restores the duplicate exactly as its cheat-sheet entry predicts - it is the +pragma that stops `/Oy` cross-jumping identical sequences - but it also reserves `ebp` as a frame +pointer, which moves `this` to `edi` and rearranges everything. The original uses `ebp` as a +general register (`mov ebp, [esp+N]; imul ebp, ebp, 0x39f4`), so `/Oy` has to stay on, and with it +on the cross-jump happens. + +Why it cannot be separated: the original's two copies are the same instructions in *different* +registers - + +``` +copy A lea edx, [ecx + ecx*2] add eax, dword ptr [edx*4 + 0x2337300] +copy B lea ecx, [ecx + ecx*2] add eax, dword ptr [ecx*4 + 0x2337300] +``` + +Cross-jumping runs after register allocation, so in the original it was the differing registers +that made the two tails non-identical and saved them from being merged. Ours were allocated the +same registers and merged. The duplicate and the `ebx`/`ebp` swap are therefore one coupled +phenomenon, not cause and effect - an earlier note in this file claimed the duplicate was upstream +of the register swap, which this corrects. Rewriting the expression does not break the tie either: +commuting the addition in one arm only, so its text differs, changes nothing because MSVC +canonicalises it. + +What did pay on that function was an ordinary source defect the mix delta pointed at by +subtraction: delta 9 decomposed as the seven-instruction duplicate plus a leftover `je -1, jne +1`, +which is one inverted branch. The final `if` tested the navigation result the opposite way round, +and swapping its arms took it **56.8% -> 60.3%**. `diff_jcc.py` does not catch that because it +covers strictness steps and operand order, not a plain polarity flip - worth adding. + +## The ordering key decoded: a per-region benefit/penalty balance + +`+0x0c` is accumulated at the tail of `FUN_10b10420`, which the colourer calls for each register +class just before it builds the ordered list. The call chain is + +``` +dop2_GlobalRegAlloc 0x10c07350 driver + -> colourer 0x10b0cf40 loops register classes 6..0 + -> 0x10b0ded0 live ranges + -> 0x10b10420 ** accumulates +0x0c ** + -> 0x10b11560, 0x10b11770 walks the symbol hash table, calls InsertInList per candidate + -> 0x10b16720 interference, per block + -> 0x10b16bd0 interference, per instruction + -> ChoosePhysReg 0x10b14e2a picks the cheapest free register +``` + +and the accumulation itself is: + +``` +0x10b1145b add edi, 1 ; count every element of list A +0x10b11476 add edi, 1 ; plus list B elements carrying flag 0x10 +0x10b11484 imul edi, ebp ; ebp: a weight passed in - block frequency +0x10b11489 mov esi, edi ; esi = count * weight + +; a candidate REFERENCED in this region: +0x10b11495 mov ecx, [eax+0x18] ; its reference multiplier for the region +0x10b11498 imul ecx, esi +0x10b1149b add [eax+0xc], ecx ; key += refs * count * weight +0x10b114a2 mov [eax+0x18], edx ; reset the multiplier + +; a candidate LIVE here but NOT referenced: +0x10b114d2 cmp dword ptr [0x10d0ffdc], 2 +0x10b114db sub [eax+0xc], ebp ; key -= weight (when that global is 2) +0x10b114e0 sub [eax+0xc], esi ; key -= count * weight (otherwise) +``` + +So the key is **not** a use count. It is a balance taken region by region: a value gains +`refs * count * weight` where it is used, and is **penalised** `count * weight` for every region +it is live across without being referenced. A register held open through code that does not touch +it is charged for. + +That predicts the numbers we measured, which is the check worth doing. On +`aiGiveRaidInstructions` `playerID` comes out at **-147** - live across the whole loop and barely +read inside it - and `i` at **1**; both are among the locals `slot_map.py` independently shows in +stack slots, and the negative-key candidates are exactly the spilled set. + +**The consequence is that the key depends on the block structure, not just on how often a value +is named.** Two source versions with identical reference counts can rank differently if their +regions differ. That resolves the `aiGiveRaidInstructions` question left open above: our base +temporary outranks `this` (128 against 93) while the original's `this` won `ebx`, and the two +binaries reference those values an identical 27 and 22 times - but our block structure still +differs, because we emit one copy of the tile computation where the original emits two, which is +one region fewer. The register swap is downstream of that mix defect, which is what +`structural_diff.py` reports from the divergence positions alone. + +So the lever for that function is the duplication, not anything register-side. Moving `i` into +the `for` initialiser reached delta 3 but disturbed too much else (56.8% -> 44.1%); what is wanted +is the missing duplicate on its own. + +A note on tooling for this kind of chase: REA runs Ghidra in an ephemeral isolated project, which +decompiles `c2.dll` without touching a shared Ghidra session's `currentProgram`. It will not load +a caller-supplied PDB, though, so symbol names are absent - `FUN_10b10420` and friends are as good +as it gets, and the `P2/*.c` assert strings are what anchor a function to its source file +(`FUN_10b133e0` reports `P2/color.c` line 0x1b37). For named symbols a separate Ghidra project +with `c2.pdb` alongside would be the escalation. + +## Correction: the candidate ordering key is +0x0c, not +0x40 + +An earlier version of these notes said `InsertInList` sorts candidates on `+0x40`, a weighted use +count, and that `+0x0c` was only a tie-break. That is backwards, and it sent a whole afternoon of +work on `aiGiveRaidInstructions` chasing the wrong number. Disassembling `@InsertInList@8` at +`0x10b11970` settles it: + +``` +0x10b1197f mov ecx, dword ptr [edi + 0xc] ; the candidate already in the list +0x10b11983 mov esi, dword ptr [eax + 0xc] ; the one being inserted +0x10b11986 cmp esi, ecx +0x10b11988 jg 0x10b119e3 ; greater goes first - descending on +0x0c +0x10b1198a jne 0x10b11994 +0x10b1198c mov ecx, dword ptr [eax + 0x44] ; tie on +0x0c: compare the symbol id +0x10b1198f cmp ecx, dword ptr [edi + 0x44] +``` + +**Primary key `+0x0c`, descending; tie-break `+0x44`, the symbol id.** `+0x40` is a separate +weight that does not decide the order. + +Where the old claim came from: there is a *second*, very similar insert function immediately +after this one, at `0x10b119f0`, and that one does sort on `+0x40` - and it *writes* `+0x0c`, +copying it from a neighbouring record: + +``` +0x10b11a49 mov edi, dword ptr [edx + 0x40] +0x10b11a4c cmp edi, dword ptr [ecx + 0x40] +... +0x10b11a63 mov eax, dword ptr [eax + 0xc] +0x10b11a67 mov dword ptr [edx + 0xc], eax +``` + +The two are easy to confuse. The one the harness breakpoints, and the one whose candidates match +the function being compiled, is `0x10b11970`. + +**How to verify which field orders the picks**, rather than trusting either note: dump both +fields per candidate and compare each ordering against the actual sequence of register +assignments from `--whyreg`. On `aiGiveRaidInstructions` the picks went `tile`, temp, +`targetBuildingID`, temp, temp, `this`, `buildingID` x3, `y`, `x`: + +| field | order it predicts | matches the picks? | +| --- | --- | --- | +| `+0x0c` | tile 212, temp 148, targetBuildingID 140, temp 128, temp 108, this 93, buildingID 88, y 80 | **yes, exactly** | +| `+0x40` | temp 82, this 60, tile 52, i 46, temp 44, tribeID 44 | no - puts `this` and a temp ahead of `tile` | + +`+0x0c` reads as a *net* benefit: the values that come out low or negative (`i` at 1, +`playerID` at -147) are exactly the ones `slot_map.py` shows living in stack slots, so the two +tools cross-check each other. + +A dead end worth not repeating: trying to find the code that *computes* `+0x0c` by taking a +candidate record's address from one run and arming `ba w4 +0xc` in the next. The records +are heap-allocated and the addresses are **not** stable between runs - the watchpoint fired +inside `ntdll!RtlpFreeHeap` and the record read back a symbol id of `311b11c`. The address has +to be captured and the watchpoint armed within a single run, which needs a breakpoint command +that stops rather than continuing (omit the trailing `gc`) so the script regains control. + +Also: a linear capstone sweep of `c2.dll`'s `.text` stops at the first byte it cannot decode, +which gave 12106 instructions out of a 1.9 MB section and made a scan for field writes look +almost empty. Resync by skipping one byte and continuing; the real count is 647653, with 291 +writes to `[reg + 0x40]` and 2543 to `[reg + 0xc]`. + +## What this does and does not unlock for aiGiveRaidInstructions + +The function's remaining register gap is a clean two-cycle: the original keeps `this` in `ebx` +and `playerID * 0x39f4` (the player-record base) in `ebp`; ours has them swapped, which accounts +for 46 of its 70 differing operands. `ebx` precedes `ebp` in the class list, so whichever is +served first takes it, and with the corrected key our base temporary sits at **128** against +`this` at **93** - so the temporary is served first and takes `ebx`. + +For the original to have given `this` `ebx`, its `this` must have outranked its base temporary. +Both binaries reference those two values an identical 27 and 22 times, and the call pattern is +identical on both sides (14 `mov ecx, ` plus 3 `mov ecx, `), so there is no +difference in use counts to exploit. `this` is a genuine `__thiscall` parameter in both - the +original does `mov ebx, ecx`, ours `mov ebp, ecx` - so this is not a case of the object being +reached as a global instead. + +Three source forms were measured and all rejected: + +| change | effect | +| --- | --- | +| `PlayerData& pd = ...playerDataArray[playerID]` used throughout | base moves to `esi`, `this` stays in `ebp`; quick_diff ratio 0.695 -> 0.499 | +| `AICState* self = this;` for the member calls | byte-identical output, the optimiser folds it | +| `i` moved into the `for` initialiser | mix delta 9 -> 3, but 56.8% -> **44.1%** | + +So the ordering key is now known and measurable, and it says the two values are ranked the way +they are for reasons that are not expressible in the source. What would settle it is the formula +behind `+0x0c`, which is still open - the write that computes it has not been located, only the +two places that copy it. + +## Register allocation is an effect, not a cause + +The register names in a near-matching function are not a problem to be attacked directly, and +`ChoosePhysReg` says why. Dumping the per-register cost array at every pick in +`renderDebugDataAiInfo` (invocation #800, 41 candidates) gave **all zeros** for every volatile +pick: + +``` +candidate got eax ecx edx ebx esp ebp esi edi +(temp) esi 0 0 0 0 0 0 [0] 0 +y edi 0 0 0 0 0 0 0 [0] +x ebx 0 0 0 [0] 0 0 0 0 +color ebp 0 0 0 0 0 [7500] 0 0 +siegeTroops eax [0] 0 0 0 0 0 -1 0 +castleID eax [0] 0 0 0 0 0 0 0 +``` + +With the costs tied at zero the minimisation is vacuous and the winner is simply the first free +entry in the class list - `eax, ecx, edx, esi, edi, ebx, ebp`. Nothing about how the source is +spelled or typed enters into it. The only inputs left are the order candidates are served +(score descending, then the symbol-id tiebreak) and which registers are still live at that +point, and both of those are decided by the instruction stream. + +That makes the useful question not *which* registers differ but **where** they start to +differ, which `structural_diff.py` now reports: + +| | `renderDebugDataAiInfo` | `aiCreateSiegeUnits` | +| --- | --- | --- | +| instructions aligned on the register-stripped skeleton | 293 of 305 | 319 of 322 | +| first structural difference | instruction 139 | instruction 30 | +| first register difference | instruction 152 | instruction 223 | +| dominant substitution | `eax -> edx -> ecx -> eax`, 90% of 59 | `esi <-> ecx`, 2 | + +In both cases the registers diverge *after* the structure does, and in +`renderDebugDataAiInfo` the first 149 instructions have identical registers while the +callee-saved assignments - `ebx` for `x`, `edi` for `y`, `esi` for the player base, `ebp` for +the colour - are identical throughout. A substitution set that closes into a cycle is the +signature of this: the allocation is structurally right and the scratch registers were handed +out in a different order, which happens as soon as one extra or missing instruction changes +what is live. + +So the rule is: if the first register difference is at or after the first structural +difference, the register names are an effect of that structural defect and will follow once it +is fixed. Spending effort on them while a mix defect remains is spending it on a symptom. The +converse case - registers diverging *before* any structural difference - is the one worth +investigating, and there the lever is statement order, because that is what sets both the +serve order and the liveness. + +## MSVC hoists the maximal common expression, which decides how to write it + +When the same expression appears in both arms of an `if`/`else`, MSVC's global optimiser hoists it +above the branch - and it hoists as *much* of it as is common, which is the part that matters when +matching. `renderDebugDataAiInfo` has two `sprintf` calls differing only in the format string. The +original duplicates the whole argument setup in each arm: + +``` +imul eax, eax, 0x6d98 ; shared: the AIV index, scaled +cmp dword ptr [esi - 0xdf4], 0 +jne arm_b +mov eax, dword ptr [eax + 0x1866ac0] ; arm A: load castleID, base folded in +add eax, 1 +push eax +push 0x5aad50 ; "AIV:n/a" +lea ecx, [esp + 0x2c] +push ecx +jmp shared_call +arm_b: +mov edx, dword ptr [eax + 0x1866ac0] ; arm B: the SAME load again +add edx, 1 +push edx +push 0x5aad48 ; "AIV: %d" +lea eax, [esp + 0x2c] +push eax +shared_call: +call sprintf +add esp, 0xc +``` + +Only the `imul` is shared; the identical load is not. Writing the full expression out in both arms +does **not** reproduce that - MSVC hoists the load, the `add`, the value `push` and the `lea` of the +buffer above the `cmp`, leaving just `push OFFSET ` per arm. Four forms were tried and all +four hoisted: the expression inline in both arms, a shared local assigned before the branch, a local +declared before and assigned separately inside each arm, and a ternary on the format string. (The +`push` moving above the branch is worth noticing on its own - a post-allocation peephole cannot move +a push across a branch, so this is the global optimiser, before register allocation.) + +The reason is the *extent* of the common expression. Written out twice, the whole chain +`playerDataArray[playerID].aivID` -> `* 0x6d98` -> load `castleID` -> `+ 1` is common, so all of it +is hoisted. The original shares only the front of that chain, which is what you get when **the source +holds the index in a local** and writes the array access out in each arm: + +```cpp +int aivID = DAT_GameState::instance.playerDataArray[playerID].aivID; +if (...) sprintf(text, "AIV:n/a", DAT_AIVState::instance.aivs[aivID].castleID + 1); +else sprintf(text, "AIV: %d", DAT_AIVState::instance.aivs[aivID].castleID + 1); +``` + +The index is shared because the source shares it; the two loads are left alone. That took the function +from 305-vs-302 instructions and a mix delta of 3 to **305 against 305 with a delta of 0**, and +reccmp from 67.5% to 83.6%. + +A pointer or reference local to the element gets close but not there - delta 1, because it keeps the +address as a value and pays an `add` to materialise `base + index * size`, where the original keeps +only `index * size` in the register and re-folds the array base into each load's displacement: + +| form | mix delta | instructions | note | +| --- | --- | --- | --- | +| expression inline in both arms | 5 | 300 | whole chain hoisted | +| shared local before the branch | 3 | 302 | hoisted by construction | +| `AIVSpec*` / `AIVSpec&` to the element | 1 | 306 | extra `add` for the address | +| **index in a local** | **0** | **305** | matches | + +The general rule: to get a computation duplicated per arm, share in the source exactly the prefix the +original shares, and write the rest out. Sharing less gets the whole thing hoisted; sharing more - a +pointer where the original had an index - adds the instruction that materialises what you shared. + +## Settled: the aiRecruitUnits recruitCount tension + +The tooling notes have carried an unresolved observation for a while - that `recruitCount++` gets +the original's `neg`/`sbb`/`jmp` counts right and scores 47.3%, while +`recruitCount = interval != 0 ? 2 : 3` gets them wrong and scores 50.0%, and that this is why +`structural_diff.py` exists. The duplication detector now explains it, and the answer is that the +structurally correct form cannot be recovered from. + +**The duplicate is real and the detector names it.** On the committed form the report reads + +``` +DUPLICATION: we compute this run 2 time(s), the original computes it 1: + mov r, r + neg r + sbb r, r +``` + +`neg r; sbb r, r` is the convert-to-boolean idiom, and the source says why - the test is written +twice: + +```cpp +int recruitCount = recruitInterval != 0 ? 1 : 2; +if (isExtremeTrail) + recruitCount = recruitInterval != 0 ? 2 : 3; // second `!= 0` +``` + +Three forms were measured against it: + +| form | instructions | `kind` | mix delta | quick_diff | +| --- | --- | --- | --- | --- | +| committed ternary | 304 | 0.9305 | 6 | **0.6821** | +| `if (isExtremeTrail) recruitCount++;` | 303 | **0.9320** | **5** | 0.6600 | +| shared `int hasInterval = interval != 0;` | 305 | 0.9322 | 11 | 0.6545 | +| `(interval != 0 ? 1 : 2) + isExtremeTrail` | 301 | 0.9185 | 7 | 0.6556 | + +The increment form is the structurally correct one: the duplication disappears, `neg` and `sbb` +drop to the original's counts, delta falls 6 to 5. Note that sharing the boolean in a named local +makes things markedly *worse* - delta 11 - which is the same "share exactly what the original +shares, no more" rule seen in the AIV block. + +**But it costs 2.7 points with nothing to recoup them.** Measured, not recalled: the increment +form builds at **45.3 / 47.3** against the committed ternary's **48.0 / 50.0**. + +What decides it is what is left afterwards. At delta 5 the remainder is `xor +2`, `cmp -1`, +`jmp +1`, `test +1`, and reading the streams shows all of it to be one idiom: + +``` +original ours +mov eax, dword ptr [esi + G] mov eax, dword ptr [esi + G] +xor ebx, ebx test eax, eax +cmp eax, ebx +``` + +The original keeps 0 in `ebx` and compares against the register where we emit `test`, and it +materialises zero twice where we do it four times. That is exactly the class recorded above for +`updateAIPlayerState` - which constant occupies a register - and it has no source handle: +`aiType == AIT_NULL`, `aiType == 0` and `!aiType` all compile to `test`. + +So this is the opposite case to `renderDebugDataAiInfo`, where a frame fix cost 0.9 points and was +worth committing because the frame was provably byte-exact and a next step existed. Here the +structural gain is one instruction, the cost is 2.7 points, and the remaining defects are in a +class already shown to be unreachable - so there is no path that recovers it. Left at the ternary +form, 50.0%, with this recorded so the question does not get reopened a third time. + +## A defect class: which constant gets the callee-saved register + +`updateAIPlayerState` (78.6% with resolvers off, 81.7% with them on) is structurally almost +perfect - 483 instructions against 486, `kind` 0.989 - and its whole mix delta of 3 is +`xor +2, mov +1`. The cause is worth recording because it is a distinct class from the defects +already in this file, and because the obvious source fix does not work. + +A census of the two streams: + +| | immediate dword stores | `xor r, r` zeroings | `mov r, 1` | +| --- | --- | --- | --- | +| original | `0x9` x3, `0x5` x2, `0x1` x1, `0x8`, `0x7`, `0x2` - **no `0x0` at all** | **1** | 1 | +| ours | `0x9` x3, **`0x0` x3**, `0x5` x2, `0x8`, `0x7`, `0x2` | **3** | 2 | + +The source has fourteen `= 0;` assignments and one `= 1;`. The original keeps **0** in `ebp` for +essentially the whole function and never stores it as an immediate; we keep **1** there and +materialise 0 three separate times. + +Where it starts is exact. Both sides are identical for 45 instructions, including + +``` +36 mov ebp, dword ptr [edi + G] ; ebp = _aiPlayerState +37 test ebp, ebp +40 jne L ; so on the fallthrough ebp is known to be 0 +``` + +and then diverge immediately: + +``` +original ours +46 mov dword ptr [edi + G], ebp 46 xor eax, eax +47 mov dword ptr [edi + G], 0x1 47 mov ebp, 0x1 +48 mov dword ptr [edi + G], ebp 48 mov dword ptr [edi + G], eax +49 mov dword ptr [edi + G], ebp 49 mov dword ptr [edi + G], ebp +``` + +The original reuses the register that already holds `_aiPlayerState` - known zero on this path - +for all three zero stores, and writes the literal 1 as an immediate. We materialise a fresh zero +in `eax` and **clobber** the known-zero `ebp` with the constant 1. The same inversion then runs +through the whole function: every `xor ebp, ebp` in the original is a `mov ebp, 0x1` in ours, and +the original's `add dword ptr [...], 0x1` is our `add dword ptr [...], ebp`. + +**The obvious fix fails.** Inside `if (_aiPlayerState == 0)` the local is zero, so writing +`attackTicker = _aiPlayerState;` instead of `= 0` is semantically identical and ought to force the +reuse. It produces **byte-identical** output - quick_diff 0.8669 either way, delta still 3. +Constant propagation replaces `_aiPlayerState` with 0 before allocation ever runs, so the source +cannot express the difference; the compiler knows the value is zero in both spellings and still +chooses to dedicate the callee-saved register to 1. + +So this is a third symptom of the same underlying thing as the frame and pressure walls: the +allocator's choice of what to keep in a callee-saved register, decided by the `+0x0c` keys, with +no source handle. What is new is the *signature* - a small `xor` surplus with matching immediate +stores, and an `xor r,r` against `mov r,1` at the same instruction on both sides - which is worth +recognising, because it looks like a code-shape defect and is not one. If a function shows +`xor +N` with the stores otherwise matching, check whether the original is reusing a known-zero +variable's register before looking for a source cause. + +## Two slot-layout leads that were not source defects + +Both functions whose frames still disagreed after the jump-table fix turned out not to be +source problems, and the way each failed is worth keeping. + +**`aiCreateSiegeUnits` (96.9%) was a tool artifact.** It reported 13 slots against our 12, with +one at **-52 inside a 0x30 frame** - an offset past the end of what the prologue allocated, +which is arithmetically impossible. The cause is visible in its own warning: two paths reach +`0x4d23a7` with esp depths 4 apart, so the depth tracking drifted and invented a slot. +`slot_map.py` now excludes references below `-frame` and says so, with the explicit note that a +slot-count difference on a function reporting a depth conflict must not be read as a defect. +With the phantom gone both sides show 12 slots at matching offsets, and the function's real +residue is the 2-instruction `jmp` delta it already had. + +That is the third artifact of this kind this session - an unreferenced slot symbol MSVC emits +but never uses, switch cases unreachable without following the jump table, and now depth drift - +and they share a cause: our side is *stated* by the `/FA` listing while the original's is +*inferred*, so the inferred side needs sanity checks before any difference it reports is +believed. + +**`planToBuyWhenLowOnResourceAndSnoozeBuildings` (89.9%) is a real slot permutation with no +reachable lever.** 248 instructions on both sides, frame `0x30` on both, 10 slots on both, the +same reference weights - but the locals sit in different slots, and the zero-initialising stores +come out in a different order: + +``` +original: mov [esp+0x3c], edi / mov [esp+0x48], edi / mov [esp+0x38], edi +ours: mov [esp+0x48], edi / mov [esp+0x3c], edi / ... +``` + +With several locals tied on weight (three at 5 references, four at 4, two at 3) this looked like +the textbook declaration-order tie-break, which is the one case `perm_decls.py` and +`quick_reorder.py` exist for. It is not: a `quick_reorder -s` hill-climb over all fourteen +declarations kept **no** move, start and final both 0.9234. + +`final == start` is also what the documented concurrent-run corruption looks like, so it was +checked rather than believed - three hand-made swaps of equal-weight pairs score **0.9194**, +**0.9153** and **0.9113**, all worse. The scoring is live, and our declaration order is already +a local optimum. The residue is therefore neither weight nor declaration order, which leaves the +class byte at `+0x18` - the part of `StkColorCmp` that is still undecoded. + +So the earlier guess that "a slot-count difference is more likely source-reachable than a +pressure difference" was wrong in both directions: one difference was not real, and the other is +real but sits in the one input to the sort that has not been decoded. + +## Two things that do not work + +**Declaration order, when there are no ties.** `StkColorCmp` sorts by class and weight and uses declaration +order only to break ties, so where the weights are distinct the declaration order is simply inert. On +`renderDebugDataAiInfo`, moving both new locals to the top of the loop body and then swapping them against +each other gave byte-identical output three times over - `67.5 / 71.0` in every case. Permuting declarations +is worth trying only once `slot_weight.py` shows an equal-weight group. + +**Forcing the compiler to duplicate a computation.** The original duplicates the whole argument setup in +both arms of one `if`/`else`, using `eax` in one arm and `edx` in the other - two separate computations the +compiler never merged. Ours hoists the common part above the branch. Three source forms were tried: the +expression written out inline in both arms, a shared local assigned before the branch, and a local declared +before the branch but assigned separately inside each arm. MSVC CSE'd all three. Whatever produces the +original's duplication is not expressible by moving that expression around, so this is a dead end to stop +pushing on rather than a defect to keep chasing. + +## Structural similarity and the match percentage can move in opposite directions + +Measured on `renderDebugDataAiInfo`, three source forms, each a full build and reccmp run: + +| form | reccmp raw / normalised | mnemonic similarity | mix delta | frame | +| --- | --- | --- | --- | --- | +| committed baseline | **68.4 / 72.6** | 0.943 | 13 | `0x3f8` | +| index cached | 66.2 / 69.7 | 0.972 | 5 | `0x3fc` | +| index and element address cached | 67.5 / 71.0 | **0.979** | **3** | **`0x400`** | + +The percentage is monotone *against* the structural measures across all three points, while the frame size +says the third form is the one that matches the original's locals. The mechanism is that adding a local +shifts every `[esp + N]` displacement in the function, and reccmp scores each of those as a mismatch until +the whole layout happens to line up - so a structurally correct change can cost points for as long as the +slot layout stays wrong. + +The practical consequence is that **the percentage is not a safe objective for hill-climbing while the frame +size is still wrong**, and a drop of a point or two against a confirmed frame match is not evidence the +change was wrong. It also means a correct intermediate state can look like a regression, which is a reason +to record the frame size and the mix delta in the commit message alongside the percentage, not just the +percentage. + +## The /GS cookie was costing every protected function its first instruction + +This one is not a codegen subtlety, it is a build defect, and it had been capping a whole +class of functions below 100% for as long as `/GS` has been on. + +**The signal.** `Text/FontSizeClass` has four functions that `structural_diff.py` calls +perfect - `delta 0`, same instructions in the same order, `kind 1.000` - and which all +scored *exactly* 96.0% normalized. Four unrelated functions stopping on the same number is +a shared cause, not four coincidences. `reccmp_report.py diff` on the smallest of them, +`getWidthOfWideText` at 27 instructions, named it on the first line: + +``` +-mov eax, dword ptr [StructResolver::Instance::instance (DATA)] ++mov eax, dword ptr [__security_cookie (DATA)] +``` + +12141088 is 0x00B94220. Every `/GS`-protected function in the original reads its stack +cookie from that address; ours read the copy of `__security_cookie` that LIBCMT links into +our own `.data`. Two instructions per function - the prologue load and the epilogue +`call __security_check_cookie` - never matched, in *every* protected function in the +project. In a 27-instruction function that is 4%. + +**The fix is a MASM absolute, which the project was already doing for data.** `src/symbols/data.asm` +already gives every game global an address with `EQU`, so the mechanism existed; it simply had +never been pointed at the CRT's own symbols. `src/symbols/crt-gs.asm` now declares + +``` +___security_cookie EQU 00B94220h +___security_cookie_complement EQU 00B94224h +``` + +An absolute symbol makes `mov eax, [__security_cookie]` assemble to the literal address +rather than a relocation into our `.data`, and the instruction then matches the original +byte for byte. Verified in the linked image, not inferred: the built DLL contains **348** +occurrences of `A1 20 42 B9 00`, `mov eax, [0x00B94220]`. + +**Two things have to come along with it, and both are load-bearing.** + +*`__security_cookie_complement`.* Declaring only the cookie still fails with +`LNK2005: ___security_cookie already defined`. All three symbols live in one CRT object, +LIBCMT's `gs_cookie.obj`, so leaving the complement undefined pulls that object in on its +own account and brings its definition of the cookie with it. Defining all three means the +object is never linked and there is no duplicate - no `/FORCE:MULTIPLE` needed. (That flag +*does* make it link and was used to measure the upside first, but it is not a mechanism: +it silently keeps whichever definition came first.) + +*`__security_init_cookie`, stubbed to `ret`.* This is a correctness requirement, not a +matching one. The real one generates a fresh random cookie and stores it to the cookie +variable - which is now 0x00B94220, *the game's own cookie*. Running it at our DLL's +startup would replace the value that every already-live game stack frame is going to check +when it returns, and those frames would take `__report_gsfailure`. The game initialises the +cookie in its own startup long before we are injected, so for us the right initialisation +is to do nothing. + +**What is deliberately left alone.** The epilogue's `call __security_check_cookie` still +goes to our own copy. The game's is at 0x0057F755, but that reference is a *relative* call, +and a displacement computed at link time into a fixed foreign address is only correct while +the DLL loads at its preferred base - it cannot be relocated. Our copy is functionally +identical, and a differing call target is exactly what the normalized percentage forgives. + +**Result.** Five of the fourteen functions in `Text/FontSizeClass` went to normalized +100.0: `getWidthOfWideText`, `renderMultilineWideTextUnk`, `renderSomeSpecificTextUnk` and +`renderWideText` from 96.0, and `wrapTextIntoLines` from 98.4. Raw percentages do not move, +and that is expected rather than disappointing - reccmp cannot *name* an address outside our +module, so it prints `` and still scores the operand as different, while the +normalized measure treats it as a resolver address and forgives it. The instruction is +nonetheless correct in the image. + +The gain is not specific to this namespace. It applies to every `/GS`-protected function in +the project - any function with a local array - so it is worth re-measuring namespaces that +were previously parked just short of 100. + +**Measured with a controlled A/B, because a build-level change deserves one.** Same 207-file +list (all of `AI/AICState` plus `Text/FontSizeClass`), same resolver state, toggling only this +one file: + +``` +207 functions: 200 identical, 7 improved, 0 worse +normalized 100 count: 181 -> 186 +average normalized: 97.340 -> 97.432 +``` + +`renderDebugDataAiInfo` also picked up 95.1 -> 96.3, so the fix reaches AICState too, not only +the namespace that exposed it. + +**The change provably cannot perturb codegen**, which is worth knowing because it means an A/B +like the above can only ever come back "identical or better". `crt-gs.asm` is its own +translation unit and changes no `.cpp` compile command, and the compiled objects reference +`___security_cookie` as an *unresolved external* with none of the address bytes present - check +it with `grep -c ___security_cookie` on any `/GS` object. Only the link-time value of that +operand moves, so instruction selection, scheduling and register allocation are untouched. The +200 byte-identical functions are the confirmation. + +**A warning about what is NOT a valid baseline.** The first attempt at this comparison used the +recorded percentages in `status/addresses-SHC-3BB0A8C1.txt` as the "before", and it reported 11 +regressions and 37 improvements - all of it noise. Those numbers were recorded with a *different* +`cmake/openshc-sources.txt.local` (60 AICState files against this run's 193), and widening the +list changes cross-TU inlining enough to move real percentages in both directions, by as much as +46 points in one case. **Recorded status percentages are only comparable against the build list +they were taken under.** For any change that could affect more than one function, build the +control yourself. + +**A note on regression-testing it.** Building every implemented function is not available as +a check: a list of all 860 `.cpp` files with a `FUNCTION:` marker does not compile, because +`AI/AIVState/aiPlaceAIVBuilding.cpp` references `Building::field240_0x2c8`, which does not +exist. That is pre-existing and unrelated, but it does mean the curated per-developer +`cmake/openshc-sources.txt.local` is the only build that works, and a project-wide before/after +has to be assembled from namespace-sized runs. + +## Settled: renderMultilineTextUnk is an edi/ebx two-cycle, priced by c2.dll + +58.7% looks like a badly wrong function and is not one. The frame is exact (`0x84` both sides, +all 18 distinct slot offsets matching), `kind` is 0.953, and `delta 8` is almost all scheduling. +The percentage is low because **one pair of values is swapped between two registers**, and that +swap renames 34 operands: + +| | original | ours | +| --- | --- | --- | +| `this` | **edi** | ebx | +| `{_runXPos, possibleWidth}` | **ebx** | edi | +| `singleLineHeight` | esi | esi | +| `xPos` | ebp | ebp | + +`esi` and `ebp` agree. It is a clean 2-cycle on `edi`/`ebx`, and `reg_scores.py` reads the whole +decision out of the running compiler. + +**The two values are one unit on both sides.** The original computes `possibleWidth` +*destructively into `_runXPos`'s own register* at all three sites - `sub ebx, eax` at 0x4731ad, +`sub ebx, ebp` at 0x47321f and 0x4732a3 - then restores it with `mov ebx, ebp`. That is exactly +what the source says, because `_runXPos = xPos` follows the comparison, so `_runXPos` is dead and +its register is reusable. We coalesce the same pair; we just coalesce it into `edi`. + +**The pick, with the cost array that decided it** (`--whyreg`, `--whycost`): + +``` +_runXPos -> edi + eax total +2 PREF +1 by charWidth; PREF +1 by (temp) +... +this$GSCopy -> ebx + ecx total -6 OWN -6 by itself + edi total +4 PREF +4 by possibleWidth +``` + +`_runXPos` is served while `eax`, `ecx`, `edx` and `esi` are already consumed, so the class list +(eax, ecx, edx, esi, edi, ebx, ebp) hands it **edi** as the first free entry, at zero cost. +`possibleWidth`, coalesced with it, then carries an affinity for `edi`. Seven picks later +`this$GSCopy` is served, finds `edi` charged **+4** by that affinity and `ebx` at 0, and takes +`ebx`. Remove the +4 and `edi` ties with `ebx` at zero, ties go to the earlier class entry, and +`edi` (5th) beats `ebx` (6th) - the original's assignment. The whole 58.7% hangs on four cost +units. + +**What actually sets the order, and the correction that matters.** The serve order is the +`+0x40` benefit, *not* the `+0x0c` spill cost. Read the wrong column and the function looks +reachable, because by `+0x0c` `this$GSCopy` (149) appears to outrank `_runXPos` (135). By the +column that decides: + +``` +wt+0x40 key+0xc name + 112 392 length + 104 135 _runXPos <- served first + 66 -65 this$GSCopy <- served later +``` + +`_runXPos` outranks `this` by **38 units**, decisively rather than marginally. Note also that +`this$GSCopy` recurs later in the dump at wt 16 and 20: those are the colourer's re-scorings in +subsequent rounds, **not** additional regions, so they must not be summed. Summing them gives 102 +against `_runXPos`'s 104 and invents a near-tie that does not exist. + +For the original to put `this` in `edi`, its `this` had to outrank `_runXPos`. The benefit is a +block-frequency-weighted use count, and `_runXPos` genuinely has far more uses than `this` in any +source spelling of these semantics - `this` appears only as the `ecx` setup before nine member +calls, while `_runXPos` is read and written throughout both loops. + +**Three levers tested, all refuted:** + +| lever | why it should have worked | result | +| --- | --- | --- | +| `/GL` on the file and its four callees | the documented `ecx` lever; LTCG sees the callees do not clobber `ecx`, which should change `this`'s reference count | **58.7 -> 45.4**, much worse: it inlines the callees and rewrites the IR | +| one shared `possibleWidth` instead of three scoped ones | collapses three candidates each carrying an `edi` affinity into one | **byte-identical** (quick_diff 0.7492 either way) | +| hoisting the `/GS`-protected `_wordReceiver[64]` to function scope | the `/GS` shadow copy fragments `this`; a different protection scope might not | **byte-identical** | + +So this is the pressure class after all, but now with a number on it: not "the allocator decided" +but "`_runXPos`'s benefit beats `this`'s by 38, which no spelling of equivalent code changes". +Left at 58.7%. The useful general lesson is the middle column above - `--whyreg` naming a single +`PREF +4` makes a function look one edit away from matching, and the benefit table is what tells +you whether any edit can reach it. + +**Correction (next day).** The paragraph above says the serve order is the `+0x40` benefit and +that `_runXPos` beat `this$GSCopy` 104 to 66. The call graph read since (see "The two keys, +settled by the call graph") shows `Color` walks `InsertInList`, which sorts by **`+0x0c`**; `+0x40` +is what `SpillOrDefer` ranks by. The decision was still made in round one and still against +`this`: `_runXPos` keys **135** and `this$GSCopy`'s *first-round* key is **-65** - the +149 that +appeared to put it ahead is a split region from a later round, exactly the "re-scoring, not +regions" trap the paragraph warns about one line later, applied to the wrong column. The +conclusion is unchanged (and stronger: 200 units, not 38). The mechanism statement was wrong, +and the `--rankdiff` output quoted above labels the `+0x0c` column "benefit", which is what +started it. + +## The two keys, settled by the call graph: `+0x0c` orders colouring, `+0x40` orders spilling + +This document has flipped twice on which field orders the register candidates. The call graph +settles it, and both earlier readings were partly right about different things. + +**The static evidence** (`call` scan of `.text`, names from c2.pdb): + +``` +@Color@4 10b0d155 call @Spillcosts@12 (and again at 10b0d2ab - rounds) + 10b0d3aa call @InsertInList@8 sorts by +0x0c desc, tie-break +0x44 + 10b0d37e call @SpillOrDefer@12 + 10b0d471 call @ChoosePhysReg@... walks the list +@SpillOrDefer@12 10b0d6e5 call @InsertByBenefit@8 sorts by +0x40 +@OrderGlobregs@12 10b118c6 call @InsertInList@8 +@StackPacking@4 10be8c8a call @ColorStackElements@12 -> @colorElemental (STACK SLOTS, not registers) +``` + +So: **`Color` colours in `InsertInList` order, which is `+0x0c` descending** (the spill-cost key +decoded earlier as `refs x count x 2^depth` per region). **`+0x40` is the benefit `SpillOrDefer` +ranks by** when a candidate cannot be coloured and something has to be spilled or deferred to a +later round. And `ColorStackElements` / `colorElemental`, whose names suggest a Chaitin-Briggs +colouring stack, are the *stack-slot* allocator - they hang off `StackPacking`, which is the +frame layout `slot_map.py` already models. A Chaitin-style stack is not how registers are +served here. + +**The dynamic evidence agrees once rounds are respected.** `recruitHarrassingSiegeEngines` serves +`unitID` (key 44) at pick 5 and `tribeID` (key 36) at pick 11 - descending `+0x0c`, not +descending `+0x40` (12 vs 20, which would be the other way). `renderMultilineTextUnk` serves +`_runXPos` (key 135) before `this$GSCopy`, whose *first-round* key is **-65**; the +149 entry +that made it look like it should have won is a later-round split region. The decision was made +in round one, by 200 units, and the section on that function below has been corrected to say so. + +### `+0x40` decoded anyway: `CmdLoadCost x 2^depth` per reference + +`Spillcosts` writes both fields. Four sites for `+0x40`, two `add`/`sub` pairs (uses at +`0x10b10957`/`0x10b109c5`, definitions at `0x10b10f85`/`0x10b11007` with the signs swapped): + +``` +if CmdCanDirectSubstitute(tuple): eax = CmdSubCost(...) ; folds into the instruction +else: eax = CmdLoadCost(operand) ; a load would be needed +[ebp+0x18] += eax ; raw sum, unweighted +[ebp+0x24] += 1 ; reference count +[ebp+0x40] += eax * weight ; weight = 1 << loop_depth without PGO +``` + +`@CmdLoadCost@4` (0x10c23740) with `_Speed != 0` - every `/O2` build - returns **2**, or **1** +for an operand whose type byte at `+4` is `0xd`. `@CmdSubCost@8` (0x10c237c0) returns **0** on +every path read. The subtract side charges `2`, `6`, or `_regAvail[+0x18] + 3` by opcode class, +times weight. So the spill benefit is two points per depth-weighted reference that would +otherwise be a load, nothing for references that fold into their instruction, less a small +per-kind penalty. It does not multiply by the reference count the way `+0x0c` does, which is +why the two columns can rank the same candidates differently - and why reading the wrong one +produces confident wrong answers. + +### What this says about `(temp)` candidates + +The open question was why an unnamed `(temp)` in `renderMultilineTextUnk` sat above every named +local and consumed `eax/ecx/edx/esi` before `_runXPos` was served. By the right column it is no +mystery: that temp's `+0x0c` key is **552**, the highest in the function. Temps are CSE'd +subexpressions and go through the same `refs x count x 2^depth` as everything else; one used in +the innermost loop keys enormously. (Which subexpression it is cannot be read out of the +candidate record - temps carry no name - so the natural guess, the `_wordReceiver` address used +at depth 2 and 3, stays a guess.) + +That answers "does expression shape control temp benefits" with a narrow yes. A temp's key is +a function of **the set of uses and their depths**, not of spelling. Naming a subexpression, +splitting it into three scoped copies, or hoisting a buffer declaration changes none of that - +which is exactly why three such variants came back byte-identical. What moves a temp is whether +MSVC treats two uses as the same subexpression at all, which is the "share exactly the prefix +the original shares" rule already recorded for `renderDebugDataAiInfo`. No second lever here. + +## Block merging: what runs, what it costs to misread, and `merge_probe.py` + +`structural_diff.py`'s POLARITY report says the original branches the other way round and +suggests swapping the arms. That was right for `aiGiveRaidInstructions` (+3.5) and would have +been wrong for `renderMultilineTextUnk`, where the flip was MSVC folding our two return blocks +together after allocation - the arms were already in the original's order. The two cases are +indistinguishable in the diff. This is about telling them apart without guessing, and it took two +corrections to get right, both recorded because both are easy to repeat. + +**The call graph, from c2.pdb names and a `call` scan of `.text`:** + +``` +@FgDoLateFlowOptsOnce@20 ---+--> @FgTryTailMerge@16 0x10b2e1b0 pairwise tail merge + +--> @FgTryCommonTailMerge@8 0x10b306c0 + '--> @FgDoCommonTailMerge@20 0x10b307e0 common tail of all preds +@FgTryEpilogMerge@20 -------+--> both of the above again +@FgTryHeadMerge@16 -------------> @FgCompareMerge@12 0x10b2f700 head-merge comparison +``` + +Two merge *families*, both live for us. Each of the two "Try/Do" routines has exactly two exits, +which is what makes the whole thing instrumentable without decoding a predicate: + +``` +FgTryTailMerge mov eax, 1 @ 0x10b2e6e5 merged xor eax, eax @ 0x10b2e6f7 declined +FgDoCommonTailMerge mov eax, 1 @ 0x10b30d73 merged xor eax, eax @ 0x10b30833 declined +``` + +**Correction 1 - `_PogoFlag` does not select an implementation, it disables all of them.** Every +routine above opens with `cmp dword ptr [_PogoFlag], 1` and returns 0 on the `== 1` side. A first +reading had `FgCompareMerge` with the opposite polarity, which would have meant two mutually +exclusive implementations with one of them dead for us. It does not: `jne` skips *past* the +`xor eax, eax; ret` in all three. `_PogoFlag == 1` is PGO instrumentation, which has to keep +blocks intact for its counters, so it turns block merging off wholesale. Without a profile all of +it runs. The flag is printed on every probe run so this stays checked. + +**Correction 2 - both families must be counted, or the tool lies.** The first version of the +probe read `FgTryTailMerge` alone and reported `renderMultilineTextUnk` as "20 attempts, 0 +merged - a real source-shape difference, swap the arms". Wrong: in the same window +`FgDoCommonTailMerge` ran 17 times. The pairwise path declined everything and the common-tail +path did the folding. A single-routine reading of a two-family mechanism produced a confident, +specific, incorrect instruction. + +**Two instrument pitfalls, since both bit here.** `cdb -cf` echoes every script line, so a +textual tag inside a breakpoint command is counted once in the echo - the first run reported +exactly 2 of everything, which was the echoed command plus the echoed error. Print `EV %d` with +a numeric id instead; the echo says `%d`, the hit says `3`. And a string nested inside a `bp` +command needs `\"` - plain quotes are "Malformed string", silently setting no breakpoint. + +### `merge_probe.py` + +``` +python tools/compiler-internals/reg_scores.py FILE.cpp --find LOCAL # invocation N +python tools/compiler-internals/merge_probe.py FILE.cpp --invocation N +``` + +Counts entries and both exits of both families inside the `GlobalRegAlloc` N..N+1 window (the +same per-function trick `reg_scores.py` uses; the late flow optimisations fall in the window). +Get N **in the same session**: it depends on the translation unit's template population, which +moves with the resolver state and compile command - `renderMultilineTextUnk` was #37 under one +configuration and #74 under another, and the stale number silently measured a different function. + +Without `--invocation` the counts cover the whole translation unit and are dominated by the +resolver templates merging among themselves. Those numbers are meaningful only as a **difference +between two variants of the same file**, where the template noise cancels exactly. + +### Validation + +**Ground truth, by difference.** `countOccupiedPlayerSlots` was known to fold: the `else if` +form with two `++_sum` sites compiles to one `add eax, 1` per unrolled body. Whole-TU probe of +both variants: + +``` + || form else-if form delta +pairwise attempts 1653 1657 +4 +pairwise merged 439 443 +4 +pairwise declined 1214 1214 0 +common-tail attempts 300 300 0 +``` + +Exactly +4 merges for exactly 4 unrolled bodies, all pairwise, nothing else moved. That is the +mechanism confirmed quantitatively, and it also confirms the earlier reading of *why* the else-if +form matches: the body is larger at unroll time (two increment blocks), so the unroller stops at +x4 instead of x8, and tail merge then folds the duplicate back - once per body - in +FinalFlowOpts. The duplication exists only inside the window where the unroll factor is chosen. + +**The live case.** `renderMultilineTextUnk`, invocation #74: + +``` +pairwise FgTryTailMerge attempts 20 merged 0 declined 20 +common FgDoCommonTailMerge attempts 17 merged 2 declined 15 +``` + +So the pairwise family rejected everything and the common-tail family did the work. The +polarity flip in this function is a merge artefact, as suspected from reading the streams, and +the instrument now says so directly instead of by inference. + +## Triage of every sub-100 function in AICState + FontSizeClass, and what it turned up + +The point of decoding the allocator was never to fix functions one at a time; it was to know +which ones *can* be fixed before spending an afternoon on one that cannot. This is that pass, +run over all 21 functions below normalized 100 in the 207-file build (not the 51 the status file +suggested - see "Four stale 100s" below). Per function: `structural_diff.py`, then +`reg_scores.py --whyreg` for who holds which register, then `--invocation N` for the +first-round `+0x0c` keys of the two candidates in each swapped register pair. + +**One rule fell out of it that is worth more than the list.** When two candidates are swapped +between two registers and their first-round keys are within a few units, one statement reorder +can flip them. When the gap is tens of units, nothing can. The gap is the triage. + +### Fixed during the pass + +**`decideOnTribeAttackLocation`: 68.8 -> 91.6 raw, 68.2 -> 85.7 normalized, one line moved.** +The swapped pair was `destinationX` (key 42, in `ebp`) against `targetPlayer` (key 40, in `ebx`) +- the original has them the other way round, and 2 units is one depth-1 reference. The source +had a store sandwiched between the two campground reads: + +```cpp +int destinationX = ...playerDataArray[targetPlayer].campground.xEntry; +tribes[tribeID].targetBuildingID = 0; // <- between the two reads +int destinationY = ...playerDataArray[targetPlayer].campground.yEntry; +``` + +Moving the store above both reads: quick_diff 0.748 -> 0.960, register operand differences +58 -> 6. Two further variants (inlining the reads, a `tile` local before the call) were both +worse, so it stays there. The residue is `delta 2` - the `this` reload scheduled before the +argument computation in the original and after it in ours - and a volatile `eax/ecx/edx` chain. + +### The work-list: reachable, in order + +| function | now | signal | the lever | +| --- | --- | --- | --- | +| `recruitHarrassingSiegeEngines` | 73.2 / 64.5 | delta 0, `ebx<->edi` 14/11 = `tribeID` (36) vs `unitID` (44) | key gap 8; `unitID` is served first and takes `edi`. One or two depth-1 references moved would flip it. Untested. | +| `aiShouldAttackOrWaitForTeamCoordination` | 83.2 / 78.1 | delta 2, first structural difference at instruction 8, 5 register operands | small and structural-first; read the two mismatched instructions before anything else | +| `computeAttackPossibilities` | 89.7 / 85.7 | delta 0, different ORDER, a DUPLICATION report | statement order, untested | +| `updateAIStrengthState` | 97.2 / 83.3 | `kind 1.000`, same order, 12 register operands | pure register permutation, not yet analysed - `--whyreg _aicIndex` and read the gap | +| `smallestTribeOfUnitType` | 96.2 / 93.4 | same instructions, 3 register operands | tiny; same recipe | + +### The stop-list, with the reason each time + +| function | now | why no source edit reaches it | +| --- | --- | --- | +| `aiRecruitUnits` | 50.0 | settled earlier: zero-register idiom plus the recruitCount increment; both forms measured | +| `renderMultilineTextUnk` | 58.7 | settled: `_runXPos` (135) vs `this$GSCopy` (first-round -65); 200 units | +| `addUnitToSmallestBehaviourTypeTribe` | 80.1 / 59.1 | settled: frame +4, loop-depth lever measured and refuted | +| `sellExcessGoods` | 62.4 | two 2-cycles (`edi<->esi`, `ebp<->ebx`), each a named local against a *temp*; the named keys are -4 / -26 / -34 against `j` 88 and `amount` 104 - the temps are what win the callee-saved registers | +| `aiGiveRaidInstructions` | 62.0 / 63.3 | `this` (93, `ebp`) swapped with a temp in `ebx`; the polarity lever here is already spent (+3.5) | +| `chooseRandomBuildingID...` | 66.0 | delta 0, but the original computes the `buildings` pointer in the loop preheader *after* the `jle` guard and we hoist it above, which frees `ecx` and moves the tracker from `esi` to `ecx`; `[i]` indexing (0.83 -> 0.47) and a `for(...; i++, buildings++)` clause (-> 0.66) both worse | +| `playNervous2BikFromPlayer` | 76.0 / 75.0 | 25 instructions, delta 0; the original loads `aiType` before the countdown store, we sink it past the team compare. Three orderings of the source (early `aiIndex`, `currentPlayerID` first, load after the store) all **byte-identical** - the scheduler decides, not the declaration order. `structural_diff`'s "statement order, reachable" label is wrong for this one. | +| `sendWallPatrolUnitTribesToAIVLocations` | 77.2 / 69.4 | settled earlier: slot permutation, declaration order already optimal | +| `updateAIPlayerState` | 81.7 / 79.9 | settled: which constant gets the callee-saved register | +| `planToBuyWhenLowOnResource...` | 90.3 / 86.5 | settled: slot permutation | +| `renderDebugDataAiInfo` | 98.4 / 96.3 | settled; the cookie fix gave it the last +1.2 | + +### Four stale 100s: header drift, not source + +`countSiegeEngineUnits` (now 53.8), `teamIsWeakRelativeToEnemy` (90.1), `setTribe0x11...` (96.7) +and `someChatMessageSelection` (99.2) are all recorded as **100%** in the status file, all +committed as such on 2026-10-18 in #261, and all have **source identical** to that commit. +`countSiegeEngineUnits` is the clean case: `kind 1.000`, same instructions in the same order, a +pure `ecx<->edx` swap of two temps - and it calls nothing, so no resolver flip can be the cause. +Measured directly: data resolvers on or off, quick_diff 0.5385 either way. What *has* changed +since that commit is 22 struct headers under `Game/` and `Map/Units/` (909 lines, the upstream +Ghidra imports). A field that changes type or offset changes the address arithmetic, the +address arithmetic is the temps, and the temps are what swapped. + +Two consequences. The status file's recorded percentages are only valid for the headers they +were measured against, and nothing re-measures a function once it reads 100 - so a Ghidra import +can silently regress any number of "done" functions. And the count of sub-100 functions I started +this pass with (51, from the status file) was wrong in the other direction too: with resolvers +on and the current headers, it is 21. **Re-measure the 100s after every upstream import.** + +### Two tool verdicts to distrust + +`structural_diff.py`'s "same instructions in a different ORDER - that is statement order, and it +is reachable" fired on five functions here. It was right for `decideOnTribeAttackLocation` and +wrong for `playNervous2BikFromPlayer` and `chooseRandomBuildingID`, where the reorder is the +scheduler's and no declaration order reaches it. The label should say *may be*. And its +DUPLICATION report pointed the wrong way on `countOccupiedPlayerSlots` (repetition from unrolling +read as repetition from branch arms). Both are counts without a cause attached; the cause is +what the second step - the register picks and their keys - is for. + +## The work-list, worked: one fix, four measured stops + +The triage above left five functions ranked by how reachable they looked. All five were taken to +the point of a measured verdict. One moved; the other four each refuted a specific, plausible +lever, and the refutations are worth more than the labels that sent us to them. + +### `recruitHarrassingSiegeEngines`: 73.2 / 64.5 -> 85.0 / 84.7 + +The swapped pair was `tribeID` (key 36, ours `ebx`) against `unitID` (key 44, ours `edi`). Two +things had to be read correctly first. There are *two* `unitID`s in the source, and the one in +the pair is the `nonEuroRecruit` result, not the loop variable - the loop is byte-identical on +both sides, with its counter in the volatile `edx` and `edi` holding the constant `0x15` for a +16-bit compare. And the original assigns `tribeID` at instruction 31, before `unitID` exists, and +gives it `edi` - so the original *served* `tribeID` first. + +The lever that worked was live range, not reference count. Moving the two tribe stores to after +the `addUnitToTribe` call: + +``` +ST1 ST2 call BEH (as written) 0.7317 +call ST1 ST2 BEH 0.9358 +ST1 call ST2 BEH <- committed 0.9419 +BEH call ST1 ST2 0.8232 +call ST2 ST1 BEH 0.7317 +uid read into a local before the call 0.8354 +``` + +`tribeID` referenced after the call gains a region, outranks `unitID`, is served first, takes +`edi`. The residue is one instruction (the uid reload moved past the call) and a volatile +`eax/ecx/edx` rotation. Normalized went up twenty points. + +### `aiShouldAttackOrWaitForTeamCoordination`: 83.2 / 78.1, stop + +The original pushes `ebp`/`esi` at entry and `ebx`/`edi` only just before the loop, so its early +exits pop two registers fewer - that is the `pop +2`. Ours keeps the constant `1` in `edi` from +instruction 10 and reuses it for the early `cmp ..., edi` and `mov eax, edi`, which forces the +early save. The original uses immediates there and materialises `mov edi, 1` at the loop. + +The obvious lever is to stop `int otherPlayerID = 1;` from being the first statement. Four +placements, all worse: in the `for` header **0.51**, assigned just before the loop **0.51**, after +the `aiType` check **0.81**, after the `requestState` check **0.61**. The side-by-side of the 0.51 +variants shows why: with the shared constant gone the compiler moved the `playerID` *base* from +`esi` to `ebx` and still put the constant `1` in a register (`esi`) for the two early uses. The +constant is materialised because it has two nearby uses regardless of the loop; where the +declaration sits only decides which other value gets displaced. + +### `computeAttackPossibilities`: 89.7 / 85.7, stop + +The original takes the address of the `attackedPlayerID` field once (`lea edi, [eax+0x115e9d0]`) +before the first call and reloads `[edi]` four times; we fold the first use and share the other +three. `int* attackedPlayerID = &...;` reproduces the shape exactly - four `[edi]` reloads, +duplication count 4 = 4 - and scores **79.3 / 80.4**, ten points *down*. The pointer is formed +with `mov edi, esi; imul edi; add edi, G` where the original keeps the product in `eax` and +`lea`s, and the four reloads rotate through `eax/ecx/edx` differently. Matching the duplication +count is not the same as matching the code, and `quick_diff` had already said so (0.90 -> 0.79) +before the build confirmed it. + +### `updateAIStrengthState`: 97.2 / 83.3, stop - with the mechanism named + +Twelve operands `ebp` where the original has `ebx`, one value, same instructions, same order. +`--whycost` names the cause exactly: + +``` +_recruitProbDefensive -> ebp + ebx total +6500 PIN +700 by (temp); PIN +800 by (temp); PIN +400 by (temp) ... +_totalRecruitProb -> ecx + ebx total +2300 ... PIN +800 by aiRecruitUnitChoicePreference +``` + +The parameter is used three times: `(char)x == 0` at the top, `= x` as an int after the random +call, `(char)x == 0` again near the end. The two byte tests CSE into one byte-class temp that +lives across the call, and a byte-class value that must survive a call has exactly one legal +register on x86 - `ebx`. It pins `ebx` with `benefit x 100`, `_recruitProbDefensive` takes +`ebp`, and every later `ebp`/`ebx` choice follows. + +Seven variants. Changing the casts (`unsigned char`, `!(char)x`, both tests, one test) - +**byte-identical**, four times. Reading the byte through `*(char*)&x` to take the address and +force memory residence - **byte-identical**, three forms; MSVC folds it straight back. The one +form that dissolves the pin, casting the *assignment* too so all three uses are the same byte +expression, scores 0.969 -> 0.994 on quick_diff and 83.3 -> 86.4 normalized, and is **wrong**: the +original assigns with `mov ebp, dword ptr [esp+0x20]`, a dword load, and the variant emits a byte +load that is one byte longer, shifting every later branch target by one. Raw stayed at 97.2 for +exactly that reason. Rejected. The original's compiler did not create the cross-call byte temp +from the same source, and no spelling of ours stops it. + +### `smallestTribeOfUnitType`: 96.2 / 93.4, stop + +Three operands, one visible difference: we store `smallestTribeID = 0` before the `aiType` +check, the original stores both zeros after it. Moving the initialiser below the check: +**0.961 -> 0.517**. Left alone. + +### What the five say together + +* **Live range beats reference count as a lever.** `recruitHarrassingSiegeEngines` moved on a + statement reorder that gave a value a region after a call; `decideOnTribeAttackLocation` moved + on a store moved out from between two reads. Neither changed what was computed. +* **A reorder that fixes the pair can displace a third value.** Three of the four stops failed + because the compiler re-spent the freed register elsewhere (`esi` -> `ebx` for the base in + `aiShouldAttack`, the frame in `smallestTribeOfUnitType`). Check the whole side-by-side, not the + pair. +* **Matching a detector's count is not matching the code**, twice now (`computeAttackPossibilities` + here, `countOccupiedPlayerSlots`' DUPLICATION earlier). +* **A byte-class value live across a call pins `ebx`.** New, named, and the first allocator + mechanism found here that a source `(char)` cast creates directly. When `--whycost` shows a + PIN by a parameter or a `(char)`-typed local, look for a second use across a call. + +## The three branches: UnitsState, AIVState-leftover, PathFindingState-development + +The same workflow applied to three branches outside AICState, with the measurement done fresh +(resolvers on, the namespace's own files as the build list) rather than taken from the status +file. These namespaces are in a different regime from AICState: `Map/Units/UnitsState` has 82 of +114 functions below normalized 100, averaging 76, with 29 under 60. The AICState pass was a +clean-up of a nearly finished namespace; this is triage of an unfinished one, and the useful +tools change accordingly - `structural_diff.py --all` for the ranking, a frame scan over every +sub-100 function, and the Ghidra decompile for the frame cases, because a missing local is +usually a *named* thing the decompiler already shows. + +### `reimpl/UnitsState` - four commits + +| function | before | after | what it was | +| --- | --- | --- | --- | +| `queueDisbandAndAttackCommand5Params` | 57.1 | **100.0** | store order | +| `selectionContainsCombatUnit` | 88.8 | **98.5 / 92.0** | a `bool` flag that the original keeps as `int` | +| `recomputeTroopValuesForPlayer` | 51.7 / 50.3 | 54.1 / 52.0 | `this->units` vs the global `units` split | +| `setDestinationForUnit` | 44.3 / 44.6 | 44.5 / 44.9 | word stores re-read the field | + +**Store order, and a detector that cannot see it.** The five parameter-to-global stores were in +the wrong order. `quick_diff` reports 1.0000 for *every* ordering, because it collapses globals +to `G`; only reccmp's raw diff shows the displacements. But `quick_diff -s` keeps the `[esp+N]` +*load* offsets, and the loads follow the stores, so it does see the order - an exhaustive search +over all 120 permutations took eight minutes and found the plain `Param0..Param4` order at 1.0000. +Confirmed 100.0 by reccmp. Worth knowing: a first, hand-read ordering of the stores scored +*worse* (57.1 -> 28.6), because the scheduler reorders independent stores and the emitted order +is not the source order. Search it, do not reason it. + +**`bool` vs `int`.** The original holds the flag in `esi` as a dword; ours was `bool`, which put it +in `bl` and changed every test and set of it. 88.8 -> 98.5 on the type alone. The residue is the +320-byte array sitting at the other end of the frame, which is slot ordering of a `/GS` buffer. + +**`this->` vs global.** `recomputeTroopValuesForPlayer` had a frame 12 bytes short. The decompile +shows why: the original walks the unit array with *two* pointers, one relative to `this` for +`logicalState/dying/owner/isSelectable/unitType/...` and one absolute into the global for +`x/y/unitType-as-value/owner-as-index/uid`, plus the counter in a slot. Ours used the global for +everything, so there was one pointer and nothing to split. Routing the accesses the way the +decompile shows recovered two of the three slots (-12 -> -4). The third is the original spilling +its `this`-relative pointer, and writing the pointers explicitly as locals is byte-identical to +the indexed form - the compiler strength-reduces either way. Allocator; left at 54.1. + +**Re-reading a field for a 16-bit store.** `setDestinationForUnit`'s original loads `units[].x` +with `movzx` and sign-extends from `ax` because it also stores that same load as a word +(`ladderExitXPosition = units[].x`); ours routed every use through an `int` local. Re-reading the +field for the word stores is the "share only what the original shares" rule and is correct, but +it is worth +0.2 here against a 0x14-vs-0x8 frame and a tail-merged early exit, and the +`(short)(ushort)` double cast - the type-based guess - is byte-identical to baseline. Ghidra and +our header agree `x` is `short`. + +**Not taken, with the reason.** `processEntityDamageToUnit` (+12, three *extra* slots) and +`acquireShootTarget` (-4) are 300-line functions where the slots in question are real locals the +two compilers chose to spill differently - pressure class. `selectionContainsCombatUnit`'s array +placement, `createTribeFromSelectedUnits`' zero register, `changeDestinationByAmount`'s one copy +and `getFirstSelectedSiegeEngineID`'s one `jle` are all single-instruction scheduling or +allocation residues. + +### `reimpl/AIVState-leftover` - nothing to commit, and one measurement artefact + +38 functions, 7 below normalized 100, every frame exact. Four of the seven are a single +scheduling residue: one `mov ebp, [esp+0x10]` one position away (`findAppropriateGridLocation- +ForBuilding`, 99.3), two `lea`s swapped (`rotateAIV`, 97.2), a store moved across a `jle` +(`aiDecideOnNewBuildings`, 99.0). Nothing in the source reaches those. + +**`aiPlaceFlag` (98.2 / 94.7) is done and the tool does not know it.** Its one differing line is + +``` +-mov ebx, dword ptr [ecx + I::instance+705348 (OFFSET)] ++mov ebx, dword ptr [ecx + cached+24 (OFFSET)] +``` + +Same instruction, a resolver address on both sides - with the data resolver on, `instance` is +our own `Instance::instance` object, 700 KB of it, and reccmp labels an address +705348 bytes into it by the nearest symbol it can find, which turns out to be some unrelated +`cached` static. The original's side gets the proper `instance+705348` label from our GLOBAL +marker, the two labels differ, and the normalized score - which forgives resolver addresses +only when it recognises them - docks it 5 points. Checked at the byte level: the original's +encoding is in the exe once and in our DLL not at all, so the displacement really is a DLL +address, exactly as every other resolved global is; this one is just mislabelled. A tooling +fix (label addresses inside a known instance object by that object) would reclaim it and any +other large-struct access like it. + +`executeDefaultCastleAIV` (53.6, 971 instructions, delta 39, a `/GL` file) is the only real +work in the namespace and is a project on its own; not started. + + +### `reimpl/PathFindingState-development` - one commit, and the richest frame signal of the three + +114 functions, 97 below normalized 100, average 71 - the least finished of the three - and +**13 frame mismatches**, nine of them +-4 or +-8, which is one or two locals each and the most +mechanical case the workflow has. The slot tables name them; the decompile usually says what they +are. + +| function | before | after | what it was | +| --- | --- | --- | --- | +| `setWalkabilityBorderLogicLayerForSmallerMapSizes` | 99.2 / 92.3 | **100.0** | three zero-stores moved *before* the increment | + +The original compares `searchGeneration` in memory (`cmp dword ptr [esi+4], 0x7d00`) straight +after incrementing it; ours loaded it into `eax` first because three stores to other `this` +fields sat between the increment and the compare in the source. Moving the stores *after* the +`if` made it worse (0.987 -> 0.971); moving them *before* the increment made it exact. The +scheduler places them between the compare and the branch either way; what the source order +decides is whether the compare can read memory. + +**`findNextTileInExistingSearchThatIsModuloDistanceAway` (56.5, frame -8): the hoist is right +and not yet enough.** The original's two missing slots are `WalkLayer[tilesQueue[0]]` hoisted out +of the loop (compared against the current tile's `WalkLayer` each iteration) and `abs(dx)` held +across `abs(dy)`. Hoisting the first into a local recovers one slot (-8 -> -4), takes `delta` +7 -> 5 and the instruction count 60 -> 62 of 63 - every cause metric improves - and reccmp falls +56.5 -> 45.2 because the allocation reshuffles around it. The second local did not come from +reordering the `abs` terms. Not committed, because the frame is still not exact and the drop is +eleven points, but this is the function's next step and the hoist should be the first edit of it. + +**Byte-identical levers, recorded so they are not retried.** `igniteFireAtTilesDistanceAway` +(+4: `_tile` appears as both a local and a temp) - declaring `_tile` inside the loop instead of +in a comma-assignment condition changes nothing. `pathPlanningForTribe` (-4) - the clamped +`unitCount` is a parameter overwritten in place, and giving it a local changes nothing either; the +missing slot is something else. `updatePathLinkageTileMapRelatedToSiegeTower`'s one moved `or` +is scheduling; so are `isTileInRangeOfKeepRange`'s `mov edi, imm` and +`updateWalkLayerAndAIPathCostLayer`'s `add edi`. + +**Ranked but not opened:** `updatePathLinkageLayerAtTileForSomeLogicalReason` (-8: the original +holds the loop counter and one of the `BitFlagHelper` pointer temps in slots; the source is still +decompiler-shaped), `pathFindingWithBuildingsIncluded` (the original's `push ecx` frame is 4 bytes, +not 0 - slot_map does not count a `push ecx` frame - so the real difference is +4, one extra local +of ours), `commitUnitPathPlanUsingWalkLayer` (+4 at 19.9%, `__direction` is the extra), +`findBuildingAccessPoint` (+8: `__accessibleIndex` and `__validAccessPointsCount` where the +original has one counter), `setClimbBasedOnClosestClimbData` (75.7: the walking pointer is biased +to the field 4 bytes before the original's, `lea esi,[ebx+0x1288]` against `+0x128c`, and the loop +shape differs with it). + +### What the three branches say together + +* **The frame scan earns its place in unfinished namespaces.** In AICState every frame was exact + and the scan was a formality; here 13 of 97 were off, nine by one or two locals, and the slot + table plus the decompile named the local in every case that was opened. +* **Fresh measurement, not the status file.** UnitsState's 82 and PathFindingState's 97 sub-100 + counts are measured with resolvers on; the recorded numbers were taken under other + configurations and are only comparable to themselves. +* **The near-done tail is almost all scheduling.** Of the functions above 95 on all three + branches, one in five had a source lever (`bool` vs `int`, store order, a store moved across an + increment); the rest are a single instruction one position away, and nothing in the source + reaches that. + + +## The deep work: four PathFindingState functions, and the inline-assembly discovery + +The branch pass left the sub-60 functions as "big, multi-defect, take them one at a time with +the frame signal first". This is that pass. Four functions moved, three of them by a mechanism +that no amount of C++ tuning could ever have reached. + +| function | before | after | what it was | +| --- | --- | --- | --- | +| `tracePathPlanToDestinationViaUnoccupiedTiles` | 12.5 | **85.7 / 83.6** | the whole walk is handwritten assembly | +| `findSuitableSpawnLocationUnk` | 52.6 | **82.5 / 81.3** | the loop head is assembly too, and the indices live in parameter slots | +| `findLinkageBasedPathOrWalkRadius` | 50.5 | **79.7** | same | +| `findNextTileInExistingSearchThatIsModuloDistanceAway` | 56.5 | **66.1 / 65.6** | two source defects: a global written every iteration, and a result routed through a temp | + +### `findNextTileInExistingSearch...`: the two missing slots were two source defects + +The hoisted `WalkLayer[tilesQueue[0]]` recovered one slot (recorded in the branch pass). The +side-by-side of the result showed the other two differences were not allocation at all. Ours +stored the global `ALG_ResultTileIndex` on *every iteration* (`mov [G], ebx` inside the loop, +ours only) and compared it unsigned (`ja` against the original's `jg`); the original keeps the +counter in a signed local and writes the global once at the end. And the missing slot was +`local_10`: the original stores `0` to it at entry and the index into it on the found path +(`mov [esp+0x10], ebx`); ours routed the result through `iVar2` and assigned `local_10` after +the loop, so the compiler folded it into a register and materialised the zero late. A local +counter and a direct assignment on the found path: frame exact, 63 = 63 instructions, `delta 0`, +reccmp 56.5 -> 66.1. The remainder is an `ebx`/`ebp` 2-cycle between `_index` (key 34) and +`_dist` (key 38) - a 4-unit gap that four spellings did not move - plus epilogue volatiles. + +### The inline-assembly discovery + +`tracePathPlanToDestinationViaUnoccupiedTiles` sat at 12.5% with a frame of 0x18 against the +original's 0x3c - nine locals short - and the original's prologue was `push ebp / mov ebp, esp / +sub esp, 0x3c`. The file already carried `#pragma optimize("y", off)` with the comment "original +keeps an ebp frame", which got the frame pointer and nothing else. The body told the rest: + +``` +mov ax, word ptr [ebp-0x30] ; a 16-bit load no compiler writes for an int local +mov dword ptr [ebp-0xc], 6 ; the direction stored as a constant... +mov eax, dword ptr [ebp-0xc] ; ...and read back three instructions later +add dword ptr [ebp-8], 1 ; the step budget counted in memory +cmp dword ptr [ebp-8], 0x190 +mov esi, dword ptr [ebp-0x24] ; every layer base reloaded from its slot before each use +``` + +Every optimisation level was tried first - `("g", off)`, `("", off)`, `("gt", off)` - and all +made it *worse* (0.13 -> 0.05): the surrounding code (bounds checks, the `searchGeneration` +bump) is plainly optimised, so this is not a function compiled at `/Od`. It is a function whose +loop is an `__asm` block, exactly as `LinkageNeighbourAsm.hpp` already documents for three +siblings: MSVC keeps an `ebp` frame in any function containing inline assembly, and every local +the block names has to live in memory. The nine missing locals *were* the asm's operands. + +Rewriting the loop as an `__asm` block mirroring the original instruction for instruction - x, y, +the tile and the walk layer kept in `ebx/ecx/edx/edi` across iterations, everything else +reloaded, the direction stored as a constant, the three-way compare on y then x - gave frame +0x3c = 0x3c, 182 = 182 instructions and 85.7%. The residue is two things: `_budget` and +`_direction` occupy each other's slots (`-8`/`-0xc`), which neither declaration nor +initialisation order moves, so it is weight-driven and the original's weights came from a shape +not reproduced; and the second early-return epilogue is laid out at the end instead of inline. A +mixed form - the budget loop in C with the asm as its body, to give the budget a depth-1 weight - +costs nine instructions and scrambles the slots (0.43). The pure-asm form is right. + +**A scanner for the signature.** Over every sub-100 PathFindingState original: an `ebp` frame +without `and esp` alignment, `mov eax, 0` followed by `mov ax, ...`, 16-bit compares, and bases +reloaded from `[ebp-N]`. Exactly four functions carry it: `tracePathPlan...` and the three +`LinkageNeighbourAsm.hpp` users. The other 93 low scores on the branch are ordinary C++ problems. +It also corrected a mistake in the branch write-up above: those three macro users are **not** at +100 - a lookup had defaulted to it - they were at 50.5, 52.6 and 66.7. + +### The two macro users: the loop head is assembly too + +`findLinkageBasedPathOrWalkRadius` and `findSuitableSpawnLocationUnk` use the push macros for +the eight-neighbour expansion but had the loop *head* in C. The original's head is in the same +hand as the pushes: + +``` +mov ecx, [ebp+0x10] ; the read index - in the x2 PARAMETER's slot +cmp ecx, [ebp+0xc] ; the write index - in y's slot +mov eax, [ebp+0x1c] ; the depth - in continuePreviousSearch's slot +cmp eax, [ebp+0x18] ; maxIterations +mov bx, word ptr [esi+edx*2] +add bx, 1 ; the distance kept in bx (and in findLinkage..., cmp bx, 0x320) +mov eax, 0 +mov ax, word ptr [esi+ecx*2] ; the row, zero-extended the asm way +mov edi, _walkLayer / mov edx, _tile / mov esi, _linkage / mov bl, byte ptr [esi+edx] +``` + +Three consequences for the source. The read index is the `x2` parameter, so our `_readIndex` +local was the extra frame slot (+4 on both). There is no `_cLink` local - the original loads `bl` +straight from the layer - so the shared `MACRO_LINKAGE_EXPAND_NEIGHBOURS`, whose first three +instructions load `_cLink`, is not what these two functions do before the pushes; the pushes +themselves (`MACRO_LINKAGE_PUSH` / `_FREE`) match exactly and are reused. And `_curGen` and +`_cCardinalDistance` are dword slots written as words, so they are `int`s in C with `word ptr` +in the asm. With a function-specific asm head wrapped around the push macros and the indices +back in the parameters, both frames are exact and **every slot of both functions sits at the +original's offset** - twelve in one, thirteen in the other - without any ordering work: the +asm-named locals come out in declaration order. 50.5 -> 79.7 and 52.6 -> 82.5. + +What remains in both is the C prologue around the `fillMemory` call: the original keeps `y` in +`edi` and `x2` in `ebx` across it and rematerialises the constant `1` in `eax`; ours parks the +`1` in `edi` and reloads the parameters. That follows from the original pushing `edi` at entry +and ours delaying the push past the first bounds check, which is the same early-exit layout +difference as the two functions above, and the same wall. + +**`updateSeparateAreaTileMap` (66.7)** is the third macro user and is frame-exact, but its slot +layout is not: the original has *two* copies each of the `yQueue` and `tilesQueue` pointers +(`lea eax, [esi+0x6a380]` stored to both `[ebp-0x24]` and `[ebp-0x34]`), one per loop, where +ours has one local and two compiler temps. Its residue is ~55 slot stores and `esi`/`eax` renames +on top of a zero/one-register prologue (`xor ecx,ecx; lea ebx,[ecx+1]`). The duplicate pointer +locals are the next step for it; not started. + +### What the deep work says + +* **A function stuck below ~20% with an `ebp` frame and no `and esp` is probably inline + assembly**, and the pragma that gets the frame pointer gets nothing else. The scanner above + finds the rest in a minute; the LinkageNeighbourAsm header already said how to write them. +* **For asm-bound functions the frame is not a signal, it is the specification.** Every named + local is a slot, in declaration order; getting the local set right got all 25 slots across two + functions to the exact offsets with no tuning. +* **The remaining residues are all one class** - early-exit layout, which decides where `push + edi` lands, which decides which value gets a callee-saved register across the first call. + Four functions on this branch stop at the same wall, 80-86%. + +### `updateSeparateAreaTileMap`: 66.7 -> 68.0, and the slot tiebreak the model does not have + +The third macro user's residue was predicted above to be the duplicate queue pointers. Reading +the original properly, both sides have them: the asm-named `_tilesQueue`/`_yQueue` in slots, and +a second copy of each address that the C loop uses for `tilesQueue[0] = first` / `yQueue[0] = row` +and that gets spilled across the asm block (the block clobbers every register, so anything live +across it is spilled - which is also why `_first`, `_row`, `_rowEnd` and `_areaID` are all slots). +Ours has exactly the same sixteen slots. What differs is the *order* of two adjacent pairs: + +``` + original ours + -0x14 / -0x18 _cY / _tile _tile / _cY + -0x28 / -0x24 queue-start copies (ours puts them at -0x3c / -0x40, below everything) +``` + +Six source edits brought the other three ordering differences into line (`_readIndex` zeroed in +the prologue at function scope like the original, `forceUpdate = 1` set before the loop as well +as inside it, the row test as `_first >= *_rowEnd`, the cap as `>= 1000`): `row/rowEnd/readIndex` +now sit at `-0xc/-8/-4` exactly, instruction count 284 vs 281. 66.7 -> 68.0. The two pairs did not +move, and the reccmp diff is 182 lines of nothing but those four slots' displacements through the +asm block. + +**What the pairs are not.** Not weight: the original references `_tile` 3 times and `_cY` 11, +identical to ours, and places them the other way round. Not declaration order: swapping the +declarations changes nothing (also tested on `_budget`/`_direction` in `tracePathPlan...`, where +neither declaration nor initialisation order moved them). Not first mention in the asm text: both +originals store `_tile` before `_cY` and still disagree with each other about which goes higher. +Not alphabetical (the layer pointers rule that out). Across the three asm functions, every slot +landed at the original's offset except exactly one adjacent pair per function plus, here, the +two compiler temps - so the frame model in `slot_map.py` (class byte, then `weight*1000/size`, +then declaration order) is right about everything except a final tiebreak among equal-weight +asm-referenced locals, and that tiebreak is what decides the last 10-15 points of all three. +Finding it is a c2.dll question (`StackPacking` / `ColorStackElements`, already located) rather +than a source one, and it is the single most valuable open item on this branch. + +### The slot tiebreak, decoded: for small frames there is no sort, and the asm's def-set is name-hashed + +`slot_probe.py` (tools/compiler-internals) traces the three c2.dll routines that decide a frame +and prints their state for one GlobalRegAlloc invocation. What it showed on +`tracePathPlanToDestinationViaUnoccupiedTiles` (invocation 204) rewrites the frame model for the +inline-assembly functions: + +**1. `QSortColors` is conditional, and does not run for these functions.** `ColorStackElements` +(0x10beaef0) only sorts when the frame is larger than 0x80 bytes, or the function has a /GS buffer +(`Function+0xAC & 1`, or `Function+0x5c` with module flag 0x20000). A 0x3c-byte frame with no +array is handed out **unsorted**: slot order is purely the order of the element list, head at the +highest address (`-4`) and tail at the lowest. The `(class, weight*1000/size)` key of +`StkColorCmp` never enters. That is why `slot_map.py`'s key column explained the large frames and +the /GS functions and was silent on these. + +**2. The element list is built by `AddStackElement` (0x10be9190), once per operand reference, +in tuple order.** `CollectStackElements` walks the final tuple stream, block by block, dst +operands then src operands, and calls `AddStackElement` for every stack symbol it meets. The first +call inserts the element after every element of smaller-or-equal size (so the list is size +ascending, FIFO among equals) with weight 1. Every later call adds 1 to the weight and moves the +element forward past the same-size elements whose weight is *strictly* lower, stopping behind the +first with weight >= the new one. Net effect: size ascending, weight descending, and among equal +weights **the element that reached its final weight first comes first**, i.e. gets the higher +address. Declaration order and initialisation order are nowhere in it (both re-tested: no change). + +**3. An `__asm` block is one tuple whose operand sets supply the references, so each symbol +counts once per set, not once per mention.** The asm tuple (opcode 0x2ce) carries a points-at list +that `CollectStackElements` flattens as: the *def-set* (every local the block writes) as a nested +set, then the *use-list* (every local the block reads, in reverse order of first mention), then the +def-set again. For the walk in `tracePathPlan...` that is 4 + 17 + 4 = 25 references for 18 +symbols. The written locals therefore get +3 from the block, read-only ones +1, write-only ones +2; +a local's C-side references add 1 each before and after. + +**4. The def-set is iterated in hash order of the symbol name.** The second def-set pass is the +*last* reference of every asm-written local that the C code does not touch afterwards, so their +relative order in it is the tiebreak among equal weights - and that order is not textual. Renaming +`_direction` to `_dirn` with no other change moved it from first to last in the set and flipped the +`_budget`/`_direction` pair into the original's order; swapping declarations or initialisations +did nothing. A synthetic file with one block writing 39 candidate names +(`scratchpad/var/nameRank.cpp`) gives the whole ranking in one probe run, and it is consistent +with every real function traced so far: + + _best _direction _count _found _distance _dir _cTile _tileIndex _facing _first _step + _ppIndex _result _budget _links _link _dist _tileNo _areaID _cY _tile _way _course _walk + _tilePos _dirIndex _readIndex _octant _dirn _heading _curTile _rowEnd _turn _radius _cX + _moveDir _n _cLink _row + +Earlier in this list = earlier last-bump = earlier in the element list = higher address, for +asm-written locals of equal weight whose last reference is the asm block. So the "tiebreak" is +our own choice of variable names, and the original's layout is recovered by picking names in the +right hash order - `_budget` before the direction local here, which `_heading` (or `_dirn`, +`_octant`, `_course`) satisfies and `_direction`/`_dir` do not. + +Caveats: the ranking only orders *these* names against each other (a different name may land +anywhere, and two names can share a bucket), so a new name is confirmed with one `slot_probe.py` +run on the real function, not assumed. And this only governs equal-weight ties: a pair with +different weights is still a reference-count problem, and the trace shows exactly which references +each side has. + +How to use it: `slot_probe.py FILE.cpp --find LOCAL` prints the invocation(s) whose frame holds +LOCAL; `slot_probe.py FILE.cpp --invocation N` prints the AddStackElement sequence with names, +tuple and nested-set markers, the final element list, whether the sort ran, and the colour records. + +**Applied to the four functions (one build each, declaration swaps and a rename only):** + +| function | before | after | what moved | +|---|---|---|---| +| tracePathPlanToDestinationViaUnoccupiedTiles | 85.7 | 94.5 | `_direction` -> `_heading`: def-set order now `_budget` first | +| findSuitableSpawnLocationUnk | 82.5 | 86.6 | `_tile`/`_cY` declared the other way round (they share a bucket, so the chain keeps declaration order) | +| findLinkageBasedPathOrWalkRadius | 79.7 | 84.3 | same swap | +| updateSeparateAreaTileMap | 68.0 | 91.9 | the swap the other way (its original has `_cY` higher), plus the asm locals initialised *before* the fillMemory calls so the two queue-pointer copies die in registers and are reloaded at the loop head - their second reference then precedes the asm's use-list and they land at `-0x24/-0x28` as in the original; `*_yQueueStart` stored before `*_tilesQueueStart` | + +Every frame of the four is now at the original's offsets. What remains is the early-exit epilogue +layout in the first three (the known wall) and register choice for the constant 1 in the fourth. +The `_tile`/`_cY` case is the warning about the ranking: in the three macro users the pair's order +followed declaration order, which only happens when two names hash to the same bucket - the +`name_rank.cpp` ranking puts them adjacent for that reason, and the probe on the real function is +what settled each one. + +### The rule, as code: `simulate()`, `regime()`, and the regression tests + +`slot_probe.simulate(adds)` replays `AddStackElement` over a recorded reference sequence and +`slot_probe.regime(frame, gs)` says whether `QSortColors` runs. `tests/test_slot_rules.py` holds +nine recorded c2.dll traces as fixtures (the four asm functions, the rename / declaration-swap / +split-block variants of tracePathPlan, and the 39-local name_rank run where the sort did run) and +checks that the simulation reproduces every recorded list element for element, that the rename +and not the declaration swap moves the pair, that list order maps onto `-4, -8, ...` when the +sort is skipped, and that the two `regime()` copies agree. No compiler needed: +`python tools/compiler-internals/tests/test_slot_rules.py`. If the rule is ever refined, a +recorded compile that stops replaying is the regression. + +`slot_map.py` now prints the regime for the function and, in the list-order regime, shows the +bare reference count as the key and says that ties go by the position of the last reference. + +**Sweep of the 60 AICState files (reimpl/AICState-leftover), same build A/B against a saved +baseline:** every sub-100 function with a frame already has the original's slot count, order +and reference counts (`aiRecruitUnits`, `sellExcessGoods`, `smallestTribeOfUnitType`, +`sendWallPatrolUnitTribesToAIVLocations`, `aiShouldAttackOrWaitForTeamCoordination`) - their +residue is register allocation, and `diff_triage.py` already said so (`alloc-only`). The one +FRAME mismatch, `addUnitToSmallestBehaviourTypeTribe` (0xc vs 8), is not an ordering problem: +the original keeps two locals in dead *parameter* homes (`[esp+0x1c]` a pointer iterator over +the tribe-ID row, `[esp+0x24]` the running minimum), which is `ColorStackElements` phase 2 +sharing a local into a parameter colour. Rewriting the loop with the pointer iterator made it +worse (80.1 -> 77.3) and was reverted; what decides the sharing is liveness of the parameter, +i.e. where the last read of each argument sits, and that is the next thing to look at there. + +### The early-exit epilogue wall: what it is and is not (first probes) + +The residue left in `tracePathPlan...` (94.5), both `find*` asm functions and earlier +`decideOnTribe`/`aiShouldAttack`: the original's second guard pair has its `pop edi / pop esi / +xor eax, eax / ... / ret` inline as the fallthrough of its last test; ours appends that block at +the end and reaches it by three jumps. Everything else in the prologue is identical, including +the lazily placed `push edi` between the two pairs. + +**Not `FgEarlyTailDup`.** It is called first thing in `GlobalRegAlloc` (0x10c0738c) under /O2 +and copies the block *immediately before the exit block* into its predecessors when that block +starts with a label, holds at most two real tuples, none a call/asm, and `preds x tuples <= 8`. +`taildup_probe.py` shows it entering and finding `TAIL 0` on every function tried: the block +before the exit is an empty landing pad (label plus one pseudo tuple, kind 0x17 opcode 0x2f9), +so there is never anything to duplicate. The inline epilogues come from elsewhere. + +**What the listing experiments say** (20 s each, `/FA` of the file alone, no debugger): + +| guards written as | instructions | FALSE epilogues | +|---|---|---| +| 2 + 2 statements (ours, `tppA`) | 182 | pair 1 inline, pair 2 at the end | +| the pairs swapped (`tppI`) | 182 | first pair inline, second at the end - positional | +| `goto fail` to one `return FALSE` at the end | 176 | one, at the end; `push edi` moves to entry | +| either or both pairs as one `a \|\| b \|\| c` statement | 176 | one, at the end; `push edi` at entry | + +So only the 2+2 shape keeps `push edi` between the pairs like the original, and in that shape +`merge_probe.py` counts exactly two post-allocation `FgDoCommonTailMerge` folds (R1+R2, R3+R4) +and nothing pairwise. The fold of the first pair keeps the fallthrough block in place; the fold +of the second pair ends up at the end of the function. `FgDoCommonTailMerge` does not create +blocks (it calls `FgTailMergeWithoutCheck` from two sites, 0x10b30ac7 / 0x10b30ae5, or inserts +a `newlabel` and retargets), so the second pair's survivor must already have been at the end +before the fold - i.e. the last guard's return block is placed after the body by something +earlier than GlobalRegAlloc, and the original's was not. + +Next probes, in order: (1) the condition choosing between the two `FgTailMergeWithoutCheck` +sites (which of the pair survives); (2) a block-order dump at GlobalRegAlloc entry for tppA to +see whether the fourth return block is already last; (3) the front end's handling of a +`return` whose fallthrough successor is the main body - the one thing that differs between the +third and fourth guards is that the fourth is followed by the body. + +### The early-exit epilogue wall, second session: the mechanism, down to the two folds + +Tools: `blocks_probe.py` dumps the function's block list / tuple stream (kind/opcode per tuple, +tuple addresses) at chosen points; `--name FUNC` gates on the function's mangled name read at +`GlobalRegAlloc` entry (`poi(poi(eax)+4)` = "?name@Class@..."), because **GlobalRegAlloc +invocation numbers are not stable across runs** (433 and 425 for the same file: function order +depends on pointer values) - every earlier `--invocation N` probe risked landing on a neighbour. +`--passes` dumps at the entry of each late pass; `--scan` dumps every invocation. + +Tuple legend read off the dumps: kind 0x18/0x2f4 OPENTRY, 0x19/0x2f5 OPEXIT, 0x1a/0x2f3 block +boundary, 0x1b/0x2f2 label, 0x12/0xf conditional branch, 0x12/0x10 goto, 0xf/0xe call, +0x16/0x2ce the `__asm` block (one opaque tuple), 0xd/* ordinary instructions (0x34 = the +materialised `xor eax, eax`, 0xba/0xb9 push/pop), 0x15/0x2d9 and 0x17/0x2f7-0x2f9 pseudo. + +What happens to `tracePathPlan...`'s four `return FALSE` (2+2 guard shape, the only shape that +keeps `push edi` between the pairs like the original): + +1. **The front end makes two kinds of return block.** `if (map == 0) return FALSE;` gives an + inline block (fallthrough of the test). `if (399 < x || 399 < y) return FALSE;` gives a + *labelled block at the end of the function*, the shared short-circuit target of both `ja`s. + At GlobalRegAlloc entry the function has 16 blocks: BLK 4 (pair 1 inline), BLK 8 (pair 2 + inline), BLK 12 and BLK 13 (the two `||` blocks, after the body). `FgEarlyTailDup` sees only + the empty landing pad before OPEXIT and does nothing. +2. **Prologue/epilogue insertion** (between `MscFixOpexitGoto` and the first + `FgDoLateFlowOpts`) linearises the stream, places `push edi` lazily after pair 1, and adds a + pop-edi exit stub (`label; pop edi; goto EXIT`) for the paths that pushed it. The gotos' + 16-bit field at +0x1c (what the merge code compares and propagates) is this exit-depth. +3. **`FgDoLateFlowOptsOnce` materialises `xor eax, eax` and `FgTryCommonTailMerge` folds + identical `xor eax, eax; goto EXIT` tails**, pairing the exit block's predecessors in + predecessor-list order, outer first: the earlier predecessor survives, the later one is cut + to a jump (`newlabel` into the survivor, `TupReplaceBranchTarget`, `TupRemoveRange`). + Observed, with tuple addresses: fold #1 cut pair 2's *inline* copy and pointed it at pair 1's + *end* block; fold #2 removed pair 2's *end* block and moved its label onto pair 1's *inline* + copy. The depth-1 predecessors are then re-routed to a pop-edi epilogue that is emitted at + the end - which is exactly our `$LN3` and the 3 jumps to it. +4. The original's layout is what you get if pair 2's inline copy survives its fold. Same + source shape, same compiler: so the input that differs is the exit block's predecessor + order (or the weights `FgWeightFlowGraph` gives the copies), not the statements. + +Listing experiments (20 s each, in `scratchpad/var/tpp*.cpp`, all against A = committed): +`[x||y],[map]` + `[destX],[destY||map2]` (M2) = A's layout; six simple `if`s (L), or any +pair-1 split (M1, M3): the pushes shrink-wrap per guard (`push esi` before test 1) - not the +original; `goto fail` (B) and `||`-collapsed pairs (C, D, E): one epilogue at the end and +`push edi` at entry. So the guard statements are A's; what is left is the predecessor order. + +Next, if resumed: print the exit block's predecessor list (block+0x28 chain, `[edge+0xc]` the +branch tuple) at `FgTryCommonTailMerge` entry for A, and look at what `FgWeightFlowGraph` +(`FgTryEpilogMerge+0x55`) writes into the two copies' +0x1c words; then look for the source +property that reorders them (candidates: the order the front end emits the `||` block versus +the inline block, i.e. whether the `||` statement comes first or second in the pair). + +### Applying the workflow to a 1300-instruction function: `processHelpRichTextTokens` (70.2%, not solved) + +What the c2 workflow gave in six builds, and where it stopped. + +**Frame (`slot_probe.py --name`, `slot_map.py --end=0x46137c`).** Sorted regime (0x13c frame, /GS +cookie). `orig_body` stops this function 0x140 bytes early (its main switch's cases sit after the +first `ret` path, the jump table at 0x46137c), hence `--end`. With it the original has 30 slots +and we had 37 symbols for the same 28 dword colours. Mapping the original's slots from the +prologue stores and the reference sites gave the original's local set exactly: + +* textMemIndex -316, leftPosition -312, local_134 -308, currentFontLineHeight -304, + local_12c -300, textToRender -296, charColor -292, centerCurrentLine -288, local_11c -284, + **one `i` for both character loops** -280 (14 refs; ours had two), local_114 -276, + currentFontClass -272, local_10c -268, lineEndY -264, **a loop-1 `charIndex` + (= local_114 + i, incremented) sharing its slot with loop-2's charYPos** -260, local_100 -256, + centreActive -252, **local_f8 sharing with a hoisted `local_114 - local_11c`** -248, **one + `width` for the PIC-HERE width, the TAB advance (30), the word width and the char width, + sharing with positionToken** -244, local_f0 -240, currentLinkId -236, lineStartY -232, + local_e4 -228, charXPos -224, imageCount -220, color -216, **one `wordLength`** -212, + linkColor -208, wideTextBuffer -204, cookie -4. +* Our `textToRender` had c2 weight 22 (rank 4) where the original's rank is 6 (between 19 and + 17). Hoisting the five identical `textToRender.text = wideTextBuffer` of the PIC debug + sub-switch to one after it gave exactly 18 and rank 6; the original keeps `textMemIndex += 2` + per branch (each has its own `add ebx, 2`), hoisting that too cost 4 points (build 5). + +**What did not work.** A shared debug tail after the main switch (one `textToRender = +wideTextBuffer`, normal paths `continue`) is what the original's *code* looks like (every debug +case jumps to 0x460777, every normal case to the loop head) - but it is c2's late tail merge +doing that, not the source: written that way, textToRender leaves the frame entirely and the +function drops to 41%. The shape that keeps the original's frame is per-case assignments. + +**Where it ended.** Build 6 (`examples-processHelpRichTextTokens-frame-variant.cpp`): frame +identical to the original through rank 9, structural `kind` 0.835 -> 0.868, instruction count ++63, reccmp 67.9 against the committed 70.2 - the five weight-11 locals (i, local_114, +currentFontClass, local_10c, lineEndY) land in quicksort tie order and the loop-1/loop-2 +temporaries take the -280 colour the original gives `i`; the rest is register allocation across +1300 instructions. Not committed. Next: break the weight-11 ties the way the original has them +(i above local_114: the original references `i` 14 times, ours 11), then the allocation. diff --git a/tools/compiler-internals/name_rank.cpp b/tools/compiler-internals/name_rank.cpp new file mode 100644 index 0000000000..71abf1693e --- /dev/null +++ b/tools/compiler-internals/name_rank.cpp @@ -0,0 +1,64 @@ +// Which order does c2 iterate a set of asm-written locals in? One __asm block that writes every +// candidate name; `slot_probe.py name_rank.cpp --invocation 1` then prints the def-set in the +// order AddStackElement receives it, which is the hash order that breaks equal-weight slot ties +// (see msvc-backend.md, "The slot tiebreak, decoded"). Add names, rerun, read the order. It is +// only a ranking of the names present against each other: two names in one bucket keep their +// declaration order, so a new name is confirmed with a probe on the real function. +int nameRank(int p) +{ + int _budget; int _result; unsigned _ppIndex; int _direction; int _dirn; int _dir; int _heading; + int _moveDir; int _dirIndex; int _step; int _facing; int _course; int _octant; int _way; int _turn; + int _tile; int _cY; int _cX; int _tileIndex; int _curTile; int _cTile; int _tileNo; int _tilePos; + int _distance; int _dist; int _radius; int _walk; int _first; int _row; int _rowEnd; int _readIndex; + int _areaID; int _cLink; int _link; int _links; int _best; int _found; int _count; int _n; + _budget = p; _result = p; _ppIndex = p; _direction = p; _dirn = p; _dir = p; _heading = p; + _moveDir = p; _dirIndex = p; _step = p; _facing = p; _course = p; _octant = p; _way = p; _turn = p; + _tile = p; _cY = p; _cX = p; _tileIndex = p; _curTile = p; _cTile = p; _tileNo = p; _tilePos = p; + _distance = p; _dist = p; _radius = p; _walk = p; _first = p; _row = p; _rowEnd = p; _readIndex = p; + _areaID = p; _cLink = p; _link = p; _links = p; _best = p; _found = p; _count = p; _n = p; + __asm { + mov dword ptr _budget, 1 + mov dword ptr _result, 1 + mov dword ptr _ppIndex, 1 + mov dword ptr _direction, 1 + mov dword ptr _dirn, 1 + mov dword ptr _dir, 1 + mov dword ptr _heading, 1 + mov dword ptr _moveDir, 1 + mov dword ptr _dirIndex, 1 + mov dword ptr _step, 1 + mov dword ptr _facing, 1 + mov dword ptr _course, 1 + mov dword ptr _octant, 1 + mov dword ptr _way, 1 + mov dword ptr _turn, 1 + mov dword ptr _tile, 1 + mov dword ptr _cY, 1 + mov dword ptr _cX, 1 + mov dword ptr _tileIndex, 1 + mov dword ptr _curTile, 1 + mov dword ptr _cTile, 1 + mov dword ptr _tileNo, 1 + mov dword ptr _tilePos, 1 + mov dword ptr _distance, 1 + mov dword ptr _dist, 1 + mov dword ptr _radius, 1 + mov dword ptr _walk, 1 + mov dword ptr _first, 1 + mov dword ptr _row, 1 + mov dword ptr _rowEnd, 1 + mov dword ptr _readIndex, 1 + mov dword ptr _areaID, 1 + mov dword ptr _cLink, 1 + mov dword ptr _link, 1 + mov dword ptr _links, 1 + mov dword ptr _best, 1 + mov dword ptr _found, 1 + mov dword ptr _count, 1 + mov dword ptr _n, 1 + } + return _budget + _result + _ppIndex + _direction + _dirn + _dir + _heading + _moveDir + _dirIndex + _step + + _facing + _course + _octant + _way + _turn + _tile + _cY + _cX + _tileIndex + _curTile + _cTile + + _tileNo + _tilePos + _distance + _dist + _radius + _walk + _first + _row + _rowEnd + _readIndex + + _areaID + _cLink + _link + _links + _best + _found + _count + _n; +} diff --git a/tools/compiler-internals/pdbnames.py b/tools/compiler-internals/pdbnames.py new file mode 100644 index 0000000000..8c5a6224e3 --- /dev/null +++ b/tools/compiler-internals/pdbnames.py @@ -0,0 +1,90 @@ +"""Name -> virtual address for every public symbol in c2.pdb, by scanning S_PUB32 records. + +Ghidra has the PDB loaded, but only `get_function_by_address` honours ghidra-mcp's `program` +argument - the name searches ignore it and query whatever program is current - so a local map is +the practical way to go from a name to an address. + +An S_PUB32 record is + + u16 reclen | u16 rectyp = 0x110e | u32 flags | u32 offset | u16 segment | zero-terminated name + +so the records can be found by scanning for the type word and validating the length against the +name that follows. Segment is a 1-based index into the PE section table, which turns +(segment, offset) into a virtual address. + +usage: pdbnames.py [SUBSTRING ...] case-insensitive; no argument dumps the count only +""" + +import re +import struct +import sys + +PDB = r'C:\Users\Gynt\Projects\sourcehold\ghidra\symbol-servers\c2.pdb\8E3C6AD714BE4E55B528B0DBEB3BABD81\c2.pdb' +DLL = r'C:\Users\Gynt\Projects\sourcehold\rea\OpenSHC-2\MSVC1400-SP1\VC\bin\c2.dll' +S_PUB32 = 0x110e + + +def sections(path): + d = open(path, 'rb').read() + pe = struct.unpack_from(' ebx + ebx total +13800 PIN +800 by (temp); PIN +300 by (temp) +(temp) -> edx + ebp total +4700 PIN +1400 by behaviourType; PIN +1000 by i +``` + +`PIN` means "a stuck neighbour charged this". The second block is the useful kind: **`ebp` is expensive for +everyone because the variables `behaviourType` and `i` are stuck on it**, charging 1400 and 1000. Those are +names from our source code, so they are things a person could act on. + +`(temp)` means the charge came from a value the compiler invented itself - an intermediate result with no name +in our code. You cannot act on those, because there is no line of source to change. **The whole point of +`--whycost` is telling those two cases apart.** + +An empty table means nothing was charged at all: every register was free, the prices were all zero, and the +tie-break list alone decided. That function cannot be steered from the source, so stop working on it. + +## The catch, which is important + +Benefit counts *uses*. Cost comes from *which variables are alive at the same time* and from *where values +cross function-call boundaries*. All of those are facts about **what the code does**, not about how it is +typed. Renaming a variable, moving its declaration, or reordering statements does not change any of them - +which is why those tricks never move a register assignment, and why a tool cannot just rewrite your syntax to +fix a mismatch. + +There is also a trap worth knowing. "This variable is stuck on `ebp` and charging 1400, so I will give it a +shorter life" sounds right and is wrong: a shorter life with the same number of uses means *more uses per unit +of life*, so the benefit goes **up**, so the charge goes **up**. Measured on `aiRecruitUnits`: moving +`behaviourType`'s declaration inside the loop raised its charge from 1400 to 1600 and the match fell from 50.0% +to 48.9%. + +So treat the output as an explanation, not an instruction. It reliably tells you *why* a register went where it +did, and it reliably tells you when to **give up on a function** - which saves real time. Turning it into a fix +means restructuring what the function computes, and every edit has to be measured with +`reccmp_report.py --run` afterwards, because the price list changes the moment you touch anything. diff --git a/tools/compiler-internals/reg_scores.py b/tools/compiler-internals/reg_scores.py new file mode 100644 index 0000000000..08e9c4bd85 --- /dev/null +++ b/tools/compiler-internals/reg_scores.py @@ -0,0 +1,507 @@ +"""Print the register-candidate scores MSVC computes for one of our functions. + +usage: reg_scores.py FILE.cpp --probe + reg_scores.py FILE.cpp --find [LOCALNAME ...] + reg_scores.py FILE.cpp --advise LOCALNAME + reg_scores.py FILE.cpp --rankdiff LOCALNAME + reg_scores.py FILE.cpp --whyreg LOCALNAME + reg_scores.py FILE.cpp --whycost LOCALNAME + reg_scores.py FILE.cpp --invocation N + +`OrderGlobregs` orders the candidate values and `Color` walks that order, calling +`ChoosePhysReg` to serve each from the still-available registers. The ordering key is the +candidate's **benefit** at `+0x40`, which is a weighted use count - `amount` in +`sellExcessGoods`, assigned in eight branches and read by the call, scores 70 where +`playerID` scores 4. This script reads those numbers out of the running compiler with the +candidates resolved back to source variable names. + +The benefit is a property of what the code *does*, not of how it is written: it counts uses +weighted by block frequency. No reordering, renaming or redeclaring moves it, which is why +syntactic hill-climbing does not shift a register assignment. What can move it is changing +how many times, and at what loop depth, a value is actually used. + +It drives `cl.exe` under the x86 `cdb` with breakpoints taken from `c2.pdb`'s public +addresses - see msvc-backend.md for how that works and why no symbol loading is needed. + +Two steps, because the compiler allocates for every function in the translation unit and +ours is not distinguishable by size (hundreds of resolver-template instantiations score +identically): + +* `--probe` prints how many candidates each `GlobalRegAlloc` invocation inserted, biggest + first. Our function is normally the largest, but only because the resolver templates all + score identically (33 candidates in a typical TU) - a function with fewer candidates than + that cannot be picked out this way. +* `--find LOCALNAME` prints every *named* candidate with its invocation index, so the + function can be identified by one of its locals instead of by size. This is the reliable + mode; `--probe` is the quick one. +* `--invocation N` dumps that invocation's candidates as `score, symbol id, name`, in the + order the compiler scored them. `(temp)` is a compiler temporary with no source name. +* `--advise LOCALNAME` prints the score table per candidate and names the patterns in it. +* `--rankdiff LOCALNAME` pairs our benefit ranking with the callee-saved save order on both + sides, taken from the last reccmp run, and names the candidate that is served later than + the original serves it - and which candidates it would have to outrank. Needs a current + `reccmp_report.py --run`. +* `--whyreg LOCALNAME` dumps the per-register cost array at each pick, which is the whole + reason one register was chosen over another, with the taken register bracketed. + +On what `--advise` is worth: its suggestions were tested on `aiRecruitUnits` (50.0%), whose +scores show the strongest structural signal of any function tried - a loop counter `i` at +-279 and a flag `recruitDiggingUnit` at -335 against a +188 top. Both suggested fixes made +the match *worse*: eliminating the counter by counting `recruitCount` down gave 48.7%, and +declaring the flag `bool` as Ghidra renders it gave 49.5%. Removing a disfavoured candidate +changes the whole scoring landscape - fewer candidates means different spill rounds - so +"delete the worst one" is not a monotone improvement. + +Treat the output as triage, not prescription. The reliably useful verdict is the negative +one: when several candidates share a score, the order falls to the symbol-id tiebreak, no +source change can state a preference, and the right move is to stop working on that +function. That saves effort rather than adding points. + +Candidates recur with different scores because the colourer iterates, re-scoring between +rounds; the first block is the initial scoring. + +Reading the output: a value with a high score is served early and gets its pick of the +registers, a negative one is likely to be spilled or to land in whatever is left. Comparing +two source variants of the same function shows whether a change moved the value you care +about up or down the list, which is otherwise only visible as an unexplained register +permutation in the reccmp diff. +""" + +import json +import os +import re +import subprocess +import sys + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), + '..', 'reimplementation-helper', 'batch')) +import common + +CDB = r"C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe" +GLOBAL_REG_ALLOC = '0x10c07350' # @dop2_GlobalRegAlloc@4 +INSERT_IN_LIST = '0x10b11970' # @InsertInList@8, the candidate priority insert +ASSIGN_REG = '0x10b14e2a' # in ChoosePhysReg: mov [cand+0x10], rec; ebx = register number +COST_ARRAY = '0x10d14f10' +COST_PREF = '0x10b14c2c' # add [eax*4+cost], ecx - neighbour ebx's preference pair +COST_PIN = '0x10b14caa' # add [esi*4+cost], ecx - ebx has one legal register: benefit*100 +COST_OWN = '0x10b14d59' # sub [ecx*4+cost], edx - this candidate's own affinity # per-register cost, indexed reg*4, rebuilt for every pick +REGNAMES = {1: 'eax', 2: 'ecx', 3: 'edx', 4: 'ebx', 5: 'esp', 6: 'ebp', 7: 'esi', 8: 'edi'} +CLASS0 = [1, 2, 3, 7, 8, 4, 6] # integer class candidate list, in preference order +BS = chr(92) + +args = [a for a in sys.argv[1:] if not a.startswith('--')] +probe = '--probe' in sys.argv +find = '--find' in sys.argv +rankdiff = '--rankdiff' in sys.argv +whyreg = '--whyreg' in sys.argv +whycost = '--whycost' in sys.argv +advise = '--advise' in sys.argv +inv = None +for a in sys.argv[1:]: + if a.startswith('--invocation'): + inv = a.split('=', 1)[1] if '=' in a else None +if inv is None and '--invocation' in sys.argv: + i = sys.argv.index('--invocation') + if i + 1 < len(sys.argv): + inv = sys.argv[i + 1] +if not args or (not probe and not find and not advise and not rankdiff and not whyreg + and not whycost and not inv): + sys.exit(__doc__) +src = os.path.abspath(args[0]) +out = (common.ROOT / common.TMP / 'regscores').resolve() +out.mkdir(parents=True, exist_ok=True) + + +def compile_command(): + cc = json.loads(common.read_text(common.BUILD_DIR / 'compile_commands.json')) + ours = [e for e in cc if 'OpenSHC.dll.dir' in e['command'] + and '/src/OpenSHC/' in e['file'].replace('\\', '/')] + if not ours: + sys.exit('configure the build first: no OpenSHC.dll entry in compile_commands.json') + o = str(out).replace('\\', '/') + cmd = re.sub(r'/Fo\S+', lambda _m: '/Fo%s/r.obj' % o, ours[0]['command']) + cmd = re.sub(r'-c \S+$', lambda _m: '-c ' + src, cmd) + return cmd, ours[0]['directory'] + + +def run(script_lines, tag): + path = out / (tag + '.cdb') + common.write_text(path, '\r\n'.join(script_lines) + '\r\n') + cmd, cwd = compile_command() + env = common.build_env() + env['PATH'] = str(common.ROOT / 'MSVC1400-SP1' / 'Common7' / 'IDE') + os.pathsep + env['PATH'] + full = '"%s" -cf "%s" %s' % (CDB, path, cmd) + r = subprocess.run(full, cwd=cwd, env=env, capture_output=True, text=True, timeout=3600) + common.write_text(out / (tag + '.log'), r.stdout) + return r.stdout + + +def advise_on(rows): + """Turn one invocation's (score, name) list into suggestions worth measuring.""" + first, allscores = {}, {} + for sc, nm in rows: + allscores.setdefault(nm, []).append(sc) + first.setdefault(nm, sc) + named = {n: v for n, v in first.items() if n != '(temp)'} + print('') + print('%-28s %-8s %s' % ('candidate', 'first', 'all rounds')) + for n in sorted(named, key=lambda k: -named[k]): + print('%-28s %-8d %s' % (n, named[n], ' '.join('%+d' % x for x in allscores[n]))) + print('') + out = [] + + # near-ties: with equal scores the order falls to the symbol-id tiebreak + buckets = {} + for n, v in named.items(): + buckets.setdefault(v, []).append(n) + ties = {v: ns for v, ns in buckets.items() if len(ns) > 1} + tied = sum(len(ns) for ns in ties.values()) + if tied >= 3: + detail = '; '.join('%+d: %s' % (v, ', '.join(ns)) for v, ns in sorted(ties.items(), reverse=True)) + out.append(['NEAR-TIES: %d candidates share a score (%s).' % (tied, detail), + 'With equal scores the order falls to the symbol-id tiebreak, so no source change', + 'states a preference here. Expect small edits to flip the result arbitrarily, and', + 'do not spend hill-climbing effort on this function.']) + + # a strongly disfavoured candidate is a local the original may not carry at all + spread = (max(named.values()) - min(named.values())) if named else 0 + for n, v in sorted(named.items(), key=lambda kv: kv[1])[:3]: + if v < -100 and spread > 150: + msg = ['DISFAVOURED: %s scores %+d against a spread of %d.' % (n, v, spread), + 'A candidate this far down is one the original may not carry separately.', + 'Try removing it, or letting a value that already exists carry it.'] + if n in ('i', 'j', 'k', 'index', 'idx', 'counter'): + msg.append('It is a loop counter: the original often reuses something it already has -') + msg.append('a dead parameter, or counting an existing local down - costing no candidate.') + out.append(msg) + + # sign flips mean the colourer's later rounds decide, not one ordering + flips = sorted(n for n, v in allscores.items() if n != '(temp)' and min(v) < 0 < max(v)) + if len(flips) >= 3: + out.append(['ITERATION-DEPENDENT: %s start negative and only turn positive later.' % ', '.join(flips[:6]), + 'The layout is settled by the spill rounds rather than by one ordering, so a single', + 'declaration permutation will not move it.']) + + if not out: + out.append(['No pattern stands out: the scores are spread and untied, so the ordering does follow', + 'from how the values are used. Changing how often or where a value is read should move it.']) + for i, msg in enumerate(out, 1): + print('%d. %s' % (i, msg[0])) + for extra in msg[1:]: + print(' ' + extra) + print('') + print('Every suggestion above is a hypothesis about codegen, not a verified fix.') + print('Measure each one with reccmp_report.py --run before keeping it.') + + +def push_orders(funcname): + """(original, ours) callee-saved save order for one function, from the last reccmp run. + + Registers are consumed in the order they are first needed, and MSVC saves a callee-saved + register where it first needs it, so the push order is the consumption order. + """ + import common as _c + try: + byaddr = _c.load_diff() + except Exception: + return None, None, None + push = re.compile(r'^push (ebx|esi|edi|ebp)$') + for _f, e, _a in _c.entries(byaddr): + if e is None or not e.get('diff') or not e['name'].endswith('::' + funcname): + continue + orig, ours = _c._asm_lines(e) + seqs = [] + for stream in (orig, ours): + seq = [] + for line in stream: + m = push.match(line.strip()) + if m and m.group(1) not in seq: + seq.append(m.group(1)) + seqs.append(seq) + return seqs[0], seqs[1], float(e['matching']) * 100 + return None, None, None + + +def rank_diff(funcname, rows): + """Pair our benefit ranking with each side's register consumption order.""" + best = {} + for benefit, name in rows: + if name != '(temp)': + best[name] = max(best.get(name, benefit), benefit) + ranked = sorted(best.items(), key=lambda kv: -kv[1]) + orig, ours, pct = push_orders(funcname) + print('') + print('our benefit ranking (candidates, highest first):') + for i, (n, b) in enumerate(ranked[:12], 1): + print(' %2d %-28s %d' % (i, n, b)) + if orig is None: + print('') + print('No reccmp entry for %s, so the original ordering is unknown.' % funcname) + print('Run reccmp_report.py --run first.') + return + print('') + print('callee-saved consumption order (= push order):') + print(' original: %s' % ' '.join(orig)) + print(' ours: %s' % ' '.join(ours)) + if orig == ours: + print('') + print('The two orders agree, so our candidates are served in the original\'s order.') + print('Whatever is left in this function is not a candidate-ranking difference.') + return + if len(orig) == len(ours) and orig == list(reversed(ours)) and len(orig) > 2: + print('') + print('The two orders are exact reverses of each other. That is a systematic difference -') + print('every position is wrong in the same way - which points at the register pick rather') + print('than at the candidate ranking: ChoosePhysReg narrows the available mask before it') + print('takes the lowest bit, and that narrowing is not decoded. A per-candidate ranking') + print('difference shows up as one candidate out of place, not as a reversal.') + print('Changing use counts will not fix this; do not read the pairing below as advice.') + if set(orig) == set(ours) and orig != ours: + print('') + print('Both sides use the same registers, just in a different order, so neither side') + print('carries a value the other does not - the candidate *set* matches. A permutation') + print('like this has two possible causes and they cannot be told apart from here: our') + print('candidates are ranked differently, or they are ranked the same and ChoosePhysReg') + print('narrowed the available mask differently (that narrowing is not decoded). The') + print('pairing below assumes the first. Treat it as a hypothesis to test, not a finding.') + print('') + print('Pairing position k of the ranking with position k of the save order - candidates are') + print('served in benefit order, so the k-th register saved goes to the k-th ranked candidate:') + print('') + print(' %-4s %-28s %-9s %-9s' % ('pos', 'our candidate', 'ours', 'original')) + for i in range(max(len(orig), len(ours))): + nm = ranked[i][0] if i < len(ranked) else '-' + o = ours[i] if i < len(ours) else '-' + g = orig[i] if i < len(orig) else '-' + flag = '' if o == g else ' <- differs' + print(' %-4d %-28s %-9s %-9s%s' % (i + 1, nm, o, g, flag)) + first = next((i for i in range(min(len(orig), len(ours))) if orig[i] != ours[i]), None) + if first is None: + return + print('') + print('First divergence at position %d: we save %s there, the original saves %s.' + % (first + 1, ours[first], orig[first])) + where = next((j for j, r in enumerate(ours) if r == orig[first]), None) + if where is not None and where > first: + print('We save %s at position %d instead, so the candidate we rank %d - %s - is served' + % (orig[first], where + 1, where + 1, ranked[where][0] if where < len(ranked) else '?')) + print('later than the original serves it. To match, it has to outrank the %d candidate(s)' + % (where - first)) + print('above it: %s.' % ', '.join(n for n, _b in ranked[first:where])) + print('') + print('That means raising its weighted use count above %d, or lowering theirs - a change to' + % (ranked[first][1] if first < len(ranked) else 0)) + print('how often and at what loop depth those values are used. No reordering or renaming') + print('will do it, and whether it is reachable without changing behaviour is a judgement') + print('call about the function, not something this tool can decide.') + else: + print('The original uses a register we do not use at all here, so the two sides do not') + print('have the same candidate set - look for a value one side carries and the other does not.') + + +def why_cost(invocation): + """Attribute every term of the cost array to the candidate that charged it. + + Each pick scans the interfering candidates. A neighbour's preference pair adds its + count to that register; a neighbour left with a single legal register adds its + benefit*100, which is what produces the large numbers; this candidate's own affinities + subtract. Breaking on the three accumulation sites and reading the neighbour's name + turns "ebx costs 1100" into "ebx is pinned by ". + """ + q = BS + '"' + name_of = lambda reg: ('.if (poi(poi(@' + reg + ')) != 0) { da poi(poi(poi(@' + reg + '))+4)+1 L28 } ' + '.else { .printf ' + q + '(temp)' + BS + BS + 'n' + q + ' }') + lines = ['sxe ld:c2', 'g', 'r $t0=0', + 'bp1 ' + COST_PIN + ' ".printf ' + q + 'PIN reg=%d delta=%d by=' + q + + ', @esi, @ecx; ' + name_of('ebx') + '; gc"', + 'bp2 ' + COST_PREF + ' ".printf ' + q + 'PREF reg=%d delta=%d by=' + q + + ', @eax, @ecx; ' + name_of('ebx') + '; gc"', + 'bp3 ' + COST_OWN + ' ".printf ' + q + 'OWN reg=%d delta=-%d' + BS + BS + 'n' + q + + ', @ecx, @edx; gc"', + 'bp4 ' + ASSIGN_REG + ' ".printf ' + q + 'PICK reg=%d ' + q + ', @ebx; ' + + name_of('eax') + '; gc"', + 'bd 1-4', + 'bp0 ' + GLOBAL_REG_ALLOC + ' "r $t0=@$t0+1; .if (@$t0==0n' + str(invocation) + + ') { be 1-4 }; .if (@$t0==0n' + str(invocation + 1) + ') { bd 1-4 }; gc"', + 'g', 'q', ''] + txt = run(lines, 'whycost' + str(invocation)) + ev = re.findall( + r'(PIN|PREF) reg=(\d+) delta=(-?\d+) by=(?:\(temp\)|[0-9a-f]{8}\s+"([^"]*)")' + r'|OWN reg=(\d+) delta=(-\d+)' + r'|PICK reg=(-?\d+) (?:\(temp\)|[0-9a-f]{8}\s+"([^"]*)")', txt) + picks, pending = [], [] + for kind, reg, delta, by, oreg, odelta, preg, pname in ev: + if kind: + pending.append((kind, int(reg), int(delta), by or '(temp)')) + elif oreg: + pending.append(('OWN', int(oreg), int(odelta), 'itself')) + else: + picks.append((pname or '(temp)', int(preg), pending)) + pending = [] + if not picks: + print('no picks seen in invocation %d' % invocation) + return + print('') + for name, reg, terms in picks: + got = REGNAMES.get(reg, reg) + charged = {} + for kind, r, delta, by in terms: + charged.setdefault(r, []).append((kind, delta, by)) + print('%s -> %s' % (name, got)) + for r in sorted(charged, key=lambda k: -sum(abs(d) for _k, d, _b in charged[k])): + total = sum(d for _k, d, _b in charged[r]) + who = '; '.join('%s %+d by %s' % (k, d, b) for k, d, b in charged[r][:3]) + star = ' <- taken' if r == reg else '' + print(' %-5s total %+7d %s%s' % (REGNAMES.get(r, r), total, who, star)) + if not charged: + print(' (no cost charged: every register was free, so the class list decided)') + print('') + print('A PIN term is a neighbour with one legal register left charging benefit*100 for it.') + print('That names a value to look at: if our source carries it and the original does not,') + print('the register it pins is free in the original. OWN terms are this candidate\'s own') + print('affinities. An empty table means nothing was charged and the pick was class order.') + + +def why_register(invocation): + """For each pick in one invocation: the candidate, the register taken, and the cost array. + + ChoosePhysReg (P2/color.c) zeroes a per-register cost array, adds each interfering + candidate's preference counts, adds benefit*100 for any neighbour left with a single + legal register, subtracts this candidate's own preference counts, and then takes the + cheapest register from its class list. The cost array is therefore the whole + explanation of one pick, and the counterfactual is arithmetic. + """ + q = BS + '"' + costs = ', '.join('poi(' + COST_ARRAY + '+' + hex(4 * n) + ')' for n in range(1, 9)) + cmd = ('.printf ' + q + 'PICK reg=%d costs=%d,%d,%d,%d,%d,%d,%d,%d ' + q + ', @ebx, ' + costs + '; ' + '.if (poi(poi(@eax)) != 0) { da poi(poi(poi(@eax))+4)+1 L28 } ' + '.else { .printf ' + q + '(temp)' + BS + BS + 'n' + q + ' }; gc') + lines = ['sxe ld:c2', 'g', 'r $t0=0', + 'bp1 ' + ASSIGN_REG + ' "' + cmd + '"', + 'bd 1', + 'bp0 ' + GLOBAL_REG_ALLOC + ' "r $t0=@$t0+1; .if (@$t0==0n' + str(invocation) + + ') { be 1 }; .if (@$t0==0n' + str(invocation + 1) + ') { bd 1 }; gc"', + 'g', 'q', ''] + txt = run(lines, 'why' + str(invocation)) + rows = [] + for m in re.finditer(r'PICK reg=(-?\d+) costs=([\d,-]+) (?:\(temp\)|[0-9a-f]{8}\s+"([^"]*)")', txt): + rows.append((int(m.group(1)), [int(x) for x in m.group(2).split(',')], m.group(3) or '(temp)')) + if not rows: + print('no register assignment seen in invocation %d' % invocation) + return + print('') + print('%-22s %-5s %s' % ('candidate', 'got', 'cost per register (lowest wins)')) + print('%-22s %-5s %s' % ('', '', ' '.join('%-7s' % REGNAMES[n] for n in range(1, 9)))) + for reg, cost, name in rows: + cells = [] + for n in range(1, 9): + c = str(cost[n - 1]) + cells.append('%-7s' % (c if n != reg else '[' + c + ']')) + print('%-22s %-5s %s' % (name, REGNAMES.get(reg, reg), ' '.join(cells))) + print('') + print('The bracketed cost is the register that was taken. To move a candidate elsewhere,') + print('that register needs a cost below the bracketed one. Cost is built from interference') + print('- a neighbour left with one legal register charges benefit*100 for it - minus this') + print('candidate\'s own affinities. Affinities come from copies between the value and a') + print('fixed register: a parameter arriving, an argument passed in a particular slot, a') + print('return value in eax. So the lever is where the value crosses a call boundary, not') + print('how the source is spelled. Ties go to the earlier class-list entry: ' + + ', '.join(REGNAMES[n] for n in CLASS0) + '.') + + +if find or advise or rankdiff or whyreg or whycost: + # every named candidate with its invocation index: identification by name, which works + # for functions smaller than the resolver-template flood that --probe ranks by size + q = BS + '"' + cand = ('.if (poi(poi(@eax)) != 0) { .printf ' + q + 'N %d %d %d ' + q + + ', @$t0, poi(@eax+0x40), poi(@eax+0xc); da poi(poi(poi(@eax))+4)+1 L28 }; gc') + lines = ['sxe ld:c2', 'g', 'r $t0=0', + 'bp1 ' + INSERT_IN_LIST + ' "' + cand + '"', + 'bp0 ' + GLOBAL_REG_ALLOC + ' "r $t0=@$t0+1; gc"', + 'g', 'q', ''] + txt = run(lines, 'find') + rows = {} + for m in re.finditer(r'N (\d+) (-?\d+) (-?\d+) [0-9a-f]{8}\s+"([^"]*)"', txt): + # group(2) is +0x40 and group(3) is +0x0c. The ORDERING key is +0x0c - see the note + # in the header - so that is what goes in the ranking. + rows.setdefault(int(m.group(1)), []).append((int(m.group(3)), m.group(4))) + want = [a for a in sys.argv[1:] if not a.startswith('--') and not a.endswith('.cpp')] + hits = [i for i in sorted(rows) + if not want or any(w in [n for _s, n in rows[i]] for w in want)] + if whycost: + if not hits: + sys.exit('no invocation holds a candidate named %s - try another local' % ', '.join(want)) + pick = max(hits, key=lambda i: len(rows[i])) + print('### invocation #%d, cost attribution' % pick) + why_cost(pick) + sys.exit(0) + if whyreg: + if not hits: + sys.exit('no invocation holds a candidate named %s - try another local' % ', '.join(want)) + pick = max(hits, key=lambda i: len(rows[i])) + print('### invocation #%d, register picks' % pick) + why_register(pick) + sys.exit(0) + if rankdiff: + if not hits: + sys.exit('no invocation holds a candidate named %s - try another local' % ', '.join(want)) + pick = max(hits, key=lambda i: len(rows[i])) + fname = os.path.basename(src)[:-4] + print('### invocation #%d of %s, %d candidates' % (pick, fname, len(rows[pick]))) + rank_diff(fname, rows[pick]) + sys.exit(0) + if advise: + if not hits: + sys.exit('no invocation holds a candidate named %s - try another local' % ', '.join(want)) + if len(hits) > 1: + print('note: %d invocations match; advising on the largest' % len(hits)) + pick = max(hits, key=lambda i: len(rows[i])) + print('### invocation #%d, %d candidates' % (pick, len(rows[pick]))) + advise_on(rows[pick]) + sys.exit(0) + print('invocations holding a named candidate: %d' % len(rows)) + for i in hits: + print(' #%-5d %s' % (i, ', '.join('%s(%+d)' % (n, s) for s, n in rows[i]))) + sys.exit(0) + +if probe: + # count candidate inserts per invocation, print only the busy ones + lines = ['sxe ld:c2', 'g', 'r $t0=0', 'r $t1=0', + 'bp0 ' + GLOBAL_REG_ALLOC + ' ".if (@$t1 > 0n15) { .printf ' + BS + '"RA %d %d' + + BS + BS + 'n' + BS + '", @$t0, @$t1 }; r $t1=0; r $t0=@$t0+1; gc"', + 'bp1 ' + INSERT_IN_LIST + ' "r $t1=@$t1+1; gc"', + 'g', 'q', ''] + txt = run(lines, 'probe') + rows = [(int(b), int(a)) for a, b in re.findall(r'RA (\d+) (\d+)', txt)] + if not rows: + sys.exit('no invocation inserted more than 15 candidates - check the build and the file') + print('%-6s %s' % ('cands', 'invocation')) + for n, i in sorted(rows, reverse=True)[:15]: + print('%-6d #%d' % (n, i)) + print('\nnow: reg_scores.py %s --invocation ' % args[0]) + sys.exit(0) + +q = BS + '"' +cand = ('.printf ' + q + 'CAND %4d %4d %02x ' + q + ', poi(@eax+0xc), poi(@eax+0x40), poi(@eax+0x44); ' + '.if (poi(poi(@eax)) != 0) { da poi(poi(poi(@eax))+4)+1 L28 } ' + '.else { .printf ' + q + '(temp)' + BS + BS + 'n' + q + ' }; gc') +lines = ['sxe ld:c2', 'g', 'r $t0=0', + 'bp1 ' + INSERT_IN_LIST + ' "' + cand + '"', + 'bd 1', + 'bp0 ' + GLOBAL_REG_ALLOC + ' "r $t0=@$t0+1; .if (@$t0==0n' + str(int(inv)) + + ') { be 1 }; .if (@$t0==0n' + str(int(inv) + 1) + ') { bd 1 }; gc"', + 'g', 'q', ''] +txt = run(lines, 'inv' + str(int(inv))) +rows = [] +for m in re.finditer(r'CAND\s+(-?\d+)\s+(-?\d+)\s+([0-9a-f]+)\s*(?:\(temp\)|[0-9a-f]{8}\s+"([^"]*)")', txt): + rows.append((int(m.group(1)), int(m.group(2)), m.group(3), m.group(4) or '(temp)')) +if not rows: + sys.exit('no candidates seen for invocation %s - run --probe to pick one' % inv) +print('%-8s %-8s %-5s %s' % ('key+0xc', 'wt+0x40', 'id', 'name')) +for key, wt, sid, name in rows: + print('%-8d %-8d %-5s %s' % (key, wt, sid, name)) +print('') +print('Candidates are served in `key+0xc` order, highest first, and each takes the cheapest') +print('register still free, so this column decides who gets ebx and who gets ebp. A low or') +print('negative key is a value headed for the stack - compare with slot_map.py.') +print('\n%d candidates; higher score is served first and gets its pick of the registers.' % len(rows)) diff --git a/tools/compiler-internals/slot_probe.py b/tools/compiler-internals/slot_probe.py new file mode 100644 index 0000000000..2860c092e8 --- /dev/null +++ b/tools/compiler-internals/slot_probe.py @@ -0,0 +1,238 @@ +"""Trace how c2 decides a function's stack slot order, for one GlobalRegAlloc invocation. + +usage: slot_probe.py FILE.cpp --name FUNCTIONNAME (preferred; invocation numbers drift between runs) + slot_probe.py FILE.cpp --invocation N + slot_probe.py FILE.cpp --find LOCALNAME which invocation(s) hold that local + +What it prints, and why each part matters (msvc-backend.md, "The slot tiebreak, decoded"): + +* every `AddStackElement` (0x10be9190) call, in order, with the element's name, size, weight + before the call, the caller, the tuple and opcode, and for an inline-asm tuple (opcode 0x2ce) + whether the reference came from a nested set. `CollectStackElements` makes one call per operand + reference while walking the tuple stream; the first call inserts the element by size, every + later one adds 1 to its weight and moves it ahead of same-size elements with strictly lower + weight. So the list ends up size ascending, weight descending, and among equal weights in the + order the elements reached their final weight. An __asm block is a single tuple whose operand + sets are: its def-set (locals it writes, iterated in name-hash order), its use-list (locals it + reads, reverse order of first mention), and the def-set again - each symbol counts once per set. +* the `symStackElements` list at `ColorStackElements` (0x10beaef0) entry, head to tail. Without + the sort (below) this IS the slot order: head at the highest address. +* whether `QSortColors` ran. It runs only for frames over 0x80 bytes or functions with a /GS + buffer; the (class, weight*1000/size) key that slot_map.py models never applies to a small + frame without an array. +* the colour records after the (optional) sort: size, weight, class byte, alignment. + +The record array is `Colort` (poi(0x10d0fa9c)), its count `numOfColors` (poi(0x10d0fa98)), the +element list head `symStackElements` (poi(0x10d0fab0)); an element's symbol is at +0, its size at ++0x20, weight at +0x38, next at +0x30, and the symbol's name is `da poi(sym+4)+1`. c2.pdb names +all of these. name_rank.cpp is a ready-made input for ranking candidate local names. +""" + +import json +import os +import re +import subprocess +import sys + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), + '..', 'reimplementation-helper', 'batch')) +import common + +CDB = r"C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\cdb.exe" +GLOBAL_REG_ALLOC = '0x10c07350' +BEFORE_SORT = '0x10beb34f' +AFTER_SORT = '0x10beb354' +REC_BASE = '0x10d0fa9c' +REC_COUNT = '0x10d0fa98' +ELEM_HEAD = '0x10d0fab0' +ADD_STACK_ELEMENT = '0x10be9190' +COLOR_ENTRY = '0x10beaef0' + +out = (common.ROOT / common.TMP / 'slotprobe').resolve() +out.mkdir(parents=True, exist_ok=True) +BS = chr(92) +Q = chr(34) +EQ = BS + Q +NL = BS + BS + 'n' + + +def compile_command(src, tag='r'): + cc = json.loads(common.read_text(common.BUILD_DIR / 'compile_commands.json')) + ours = [e for e in cc if 'OpenSHC.dll.dir' in e['command'] + and '/src/OpenSHC/' in e['file'].replace('\\', '/')] + if not ours: + sys.exit('configure the build first') + o = str(out).replace('\\', '/') + cmd = re.sub(r'/Fo\S+', lambda _m: '/Fo%s/%s.obj' % (o, tag), ours[0]['command']) # one .obj per run, so runs can overlap + cmd = re.sub(r'-c \S+$', lambda _m: '-c ' + os.path.abspath(src), cmd) + return cmd, ours[0]['directory'] + + +def run(lines, src, tag): + path = out / (tag + '.cdb') + common.write_text(path, '\r\n'.join(lines) + '\r\n') + cmd, cwd = compile_command(src, tag) + env = common.build_env() + env['PATH'] = str(common.ROOT / 'MSVC1400-SP1' / 'Common7' / 'IDE') + os.pathsep + env['PATH'] + full = '"%s" -cf "%s" %s' % (CDB, path, cmd) + r = subprocess.run(full, cwd=cwd, env=env, capture_output=True, text=True, timeout=3600) + common.write_text(out / (tag + '.log'), r.stdout) + return r.stdout + + +NAME = ('.if (poi(@$t2) != 0) { .if (poi(poi(@$t2)+4) != 0) { da poi(poi(@$t2)+4)+1 L28 } .else { .printf ' + + EQ + '(temp)' + NL + EQ + ' } } .else { .printf ' + EQ + '(nosym)' + NL + EQ + ' }') + + +def dump_cmd(tag): + # one line per colour record: REC idx size weight class align + return ('r $t1=0; .while (@$t1 < poi(' + REC_COUNT + ')) { r $t2 = poi(' + REC_BASE + ') + @$t1*0n28; ' + '.printf ' + EQ + 'REC ' + tag + ' %d sz=%d wt=%d cls=%d al=%d' + NL + EQ + + ', @$t1, poi(@$t2+8), poi(@$t2+0xc), (poi(@$t2+0x18) & 0xff), poi(@$t2+0x14); r $t1=@$t1+1 }') + + +def list_cmd(): + # the symStackElements list at ColorStackElements entry, head to tail: LIST idx elem size weight kind name + return ('r $t2=poi(' + ELEM_HEAD + '); r $t1=0; .while (@$t2 != 0) { .printf ' + EQ + + 'LIST %d %x sz=%d wt=%d kind=%d ' + EQ + ', @$t1, @$t2, poi(@$t2+0x20), poi(@$t2+0x38), (poi(@$t2+4) & 0xff); ' + + NAME + '; r $t1=@$t1+1; r $t2=poi(@$t2+0x30) }') + + +def add_cmd(): + # every AddStackElement call: ADD elem size weight-before caller name + return ('r $t2=@edx; .printf ' + EQ + 'ADD %x sz=%d wt=%d ret=%x tup=%x op=%x esi=%x edi=%x ' + EQ + + ', @edx, poi(@edx+0x20), poi(@edx+0x38), poi(@esp), @ebx, poi(@ebx+4), @esi, @edi; ' + NAME) + + +ADD_RE = re.compile(r'^ADD ([0-9a-f]+) sz=(\d+) wt=(-?\d+) ret=([0-9a-f]+)(?: tup=([0-9a-f]+) op=([0-9a-f]+))?' + r'(?: esi=([0-9a-f]+) edi=([0-9a-f]+))? \s*(?:[0-9a-f]{8}\s+"([^"]*)"|\((temp|nosym)\))', re.M) +LIST_RE = re.compile(r'^LIST (\d+) ([0-9a-f]+) sz=(\d+) wt=(\d+) kind=(\d+) \s*(?:[0-9a-f]{8}\s+"([^"]*)"|\((temp|nosym)\))', re.M) + + +def parse(txt): + """The ADD sequence and the final LIST of a probe log, as plain tuples.""" + adds = [(m.group(1), int(m.group(2)), int(m.group(3)), m.group(9) or m.group(10)) for m in ADD_RE.finditer(txt)] + lst = [(m.group(2), int(m.group(3)), int(m.group(4)), m.group(6) or m.group(7)) for m in LIST_RE.finditer(txt)] + return adds, lst, bool(re.search(r'^SORTED', txt, re.M)) + + +def simulate(adds): + """Replay AddStackElement (c2.dll 0x10be9190) over an ADD sequence of (elem, size, ...). + + First call for an element: insert it after every element whose size is <= its own, weight 1. + Later calls: weight += 1, then move it forward past the same-size predecessors whose weight + is strictly lower, stopping behind the first whose weight is >= the new one. Returns the + list head-to-tail as [(elem, size, weight)]. test_slot_rules.py checks this against the + recorded lists; slot_probe.py prints a mismatch if the live list ever disagrees. + """ + order = [] # elem ids head-to-tail + size, weight = {}, {} + for elem, sz in ((a[0], a[1]) for a in adds): + if elem not in weight: + size[elem], weight[elem] = sz, 1 + i = 0 + while i < len(order) and size[order[i]] <= sz: + i += 1 + order.insert(i, elem) + continue + weight[elem] += 1 + i = order.index(elem) + j = i + while j > 0 and size[order[j - 1]] == sz and weight[order[j - 1]] < weight[elem]: + j -= 1 + if j != i: + order.pop(i) + order.insert(j, elem) + return [(e, size[e], weight[e]) for e in order] + + +def regime(frame, gs_buffer): + """Does ColorStackElements sort the colours? Only for a frame over 0x80 bytes or a /GS + buffer (Function+0xAC & 1); otherwise the element list order IS the slot order.""" + return 'sorted' if (frame > 0x80 or gs_buffer) else 'list-order' + + +def gate_for(name): + """bp0 command that enables bps 1-4 only for the function whose mangled name starts with + "?NAME": GlobalRegAlloc invocation numbers drift between runs (function order depends on + pointer values), so a --invocation N probe can land on a neighbour; the name cannot.""" + import struct + w0, w1 = struct.unpack(' 8)') diff --git a/tools/compiler-internals/tests/fixtures/declD-204.json b/tools/compiler-internals/tests/fixtures/declD-204.json new file mode 100644 index 0000000000..7e8426e67b --- /dev/null +++ b/tools/compiler-internals/tests/fixtures/declD-204.json @@ -0,0 +1,416 @@ +{ +"source": "slots-declD-204.log", +"sorted": false, +"adds": [ +[ +"4268224", +4, +0, +"this" +], +[ +"42683a4", +4, +0, +"x" +], +[ +"4268344", +4, +0, +"y" +], +[ +"42682e4", +4, +0, +"destX" +], +[ +"4268284", +4, +0, +"destY" +], +[ +"42682e4", +4, +1, +"destX" +], +[ +"42683a4", +4, +1, +"x" +], +[ +"4268404", +4, +0, +"_budget" +], +[ +"4268464", +4, +0, +"_result" +], +[ +"42684c4", +4, +0, +"_direction" +], +[ +"4268524", +4, +0, +"_gen" +], +[ +"4268584", +4, +0, +"_fireCount" +], +[ +"42685e4", +4, +0, +"_tile" +], +[ +"4268644", +4, +0, +"_destTile" +], +[ +"42686a4", +4, +0, +"_occupancy" +], +[ +"4268704", +4, +0, +"_walkLayer" +], +[ +"4268764", +4, +0, +"_linkage" +], +[ +"42687c4", +4, +0, +"_dirMatrix" +], +[ +"4268824", +4, +0, +"_cardinal" +], +[ +"4268884", +4, +0, +"_pPathPlan" +], +[ +"42688e4", +4, +0, +"_ppIndex" +], +[ +"42684c4", +4, +1, +"_direction" +], +[ +"42688e4", +4, +1, +"_ppIndex" +], +[ +"4268464", +4, +1, +"_result" +], +[ +"4268404", +4, +1, +"_budget" +], +[ +"4268884", +4, +1, +"_pPathPlan" +], +[ +"42688e4", +4, +2, +"_ppIndex" +], +[ +"4268824", +4, +1, +"_cardinal" +], +[ +"42687c4", +4, +1, +"_dirMatrix" +], +[ +"42684c4", +4, +2, +"_direction" +], +[ +"42682e4", +4, +2, +"destX" +], +[ +"4268284", +4, +1, +"destY" +], +[ +"4268764", +4, +1, +"_linkage" +], +[ +"42686a4", +4, +1, +"_occupancy" +], +[ +"4268584", +4, +1, +"_fireCount" +], +[ +"4268644", +4, +1, +"_destTile" +], +[ +"4268524", +4, +1, +"_gen" +], +[ +"4268404", +4, +2, +"_budget" +], +[ +"42685e4", +4, +1, +"_tile" +], +[ +"4268344", +4, +1, +"y" +], +[ +"42683a4", +4, +2, +"x" +], +[ +"4268704", +4, +1, +"_walkLayer" +], +[ +"42684c4", +4, +3, +"_direction" +], +[ +"42688e4", +4, +3, +"_ppIndex" +], +[ +"4268464", +4, +2, +"_result" +], +[ +"4268404", +4, +3, +"_budget" +], +[ +"42688e4", +4, +4, +"_ppIndex" +], +[ +"4268224", +4, +1, +"this" +], +[ +"4268464", +4, +3, +"_result" +] +], +"list": [ +[ +"42688e4", +4, +5, +"_ppIndex" +], +[ +"42684c4", +4, +4, +"_direction" +], +[ +"4268404", +4, +4, +"_budget" +], +[ +"4268464", +4, +4, +"_result" +], +[ +"42682e4", +4, +3, +"destX" +], +[ +"42683a4", +4, +3, +"x" +], +[ +"4268884", +4, +2, +"_pPathPlan" +], +[ +"4268824", +4, +2, +"_cardinal" +], +[ +"42687c4", +4, +2, +"_dirMatrix" +], +[ +"4268284", +4, +2, +"destY" +], +[ +"4268764", +4, +2, +"_linkage" +], +[ +"42686a4", +4, +2, +"_occupancy" +], +[ +"4268584", +4, +2, +"_fireCount" +], +[ +"4268644", +4, +2, +"_destTile" +], +[ +"4268524", +4, +2, +"_gen" +], +[ +"42685e4", +4, +2, +"_tile" +], +[ +"4268344", +4, +2, +"y" +], +[ +"4268704", +4, +2, +"_walkLayer" +], +[ +"4268224", +4, +2, +"this" +] +] +} \ No newline at end of file diff --git a/tools/compiler-internals/tests/fixtures/findLinkageBasedPathOrWalkRadius-202.json b/tools/compiler-internals/tests/fixtures/findLinkageBasedPathOrWalkRadius-202.json new file mode 100644 index 0000000000..65d3193fc7 --- /dev/null +++ b/tools/compiler-internals/tests/fixtures/findLinkageBasedPathOrWalkRadius-202.json @@ -0,0 +1,476 @@ +{ +"source": "slots-findLinkageBasedPathOrWalkRadius-202.log", +"sorted": false, +"adds": [ +[ +"41d11cc", +4, +0, +"this" +], +[ +"41d140c", +4, +0, +"x" +], +[ +"41d13ac", +4, +0, +"y" +], +[ +"41d134c", +4, +0, +"x2" +], +[ +"41d12ec", +4, +0, +"y2" +], +[ +"41d128c", +4, +0, +"maxIterations" +], +[ +"41d128c", +4, +1, +"maxIterations" +], +[ +"41d122c", +4, +0, +"continuePreviousSearch" +], +[ +"41d134c", +4, +1, +"x2" +], +[ +"41d13ac", +4, +1, +"y" +], +[ +"41d146c", +4, +0, +"_curGen" +], +[ +"41d14cc", +4, +0, +"_walkLayer" +], +[ +"41d152c", +4, +0, +"_certainPath" +], +[ +"41d158c", +4, +0, +"_tilesQueue" +], +[ +"41d15ec", +4, +0, +"_yQueue" +], +[ +"41d164c", +4, +0, +"_dirMatrix" +], +[ +"41d16ac", +4, +0, +"_linkage" +], +[ +"41d170c", +4, +0, +"_cCardinalDistance" +], +[ +"41d176c", +4, +0, +"_result" +], +[ +"41d140c", +4, +1, +"x" +], +[ +"41d13ac", +4, +2, +"y" +], +[ +"41d122c", +4, +1, +"continuePreviousSearch" +], +[ +"41d140c", +4, +2, +"x" +], +[ +"41d12ec", +4, +1, +"y2" +], +[ +"41d140c", +4, +3, +"x" +], +[ +"41d134c", +4, +2, +"x2" +], +[ +"41d122c", +4, +2, +"continuePreviousSearch" +], +[ +"41d134c", +4, +3, +"x2" +], +[ +"41d13ac", +4, +3, +"y" +], +[ +"41d176c", +4, +1, +"_result" +], +[ +"41d194c", +4, +0, +"_cY" +], +[ +"41d188c", +4, +0, +"_tile" +], +[ +"41d170c", +4, +1, +"_cCardinalDistance" +], +[ +"41d164c", +4, +1, +"_dirMatrix" +], +[ +"41d194c", +4, +1, +"_cY" +], +[ +"41d170c", +4, +2, +"_cCardinalDistance" +], +[ +"41d146c", +4, +1, +"_curGen" +], +[ +"41d16ac", +4, +1, +"_linkage" +], +[ +"41d188c", +4, +1, +"_tile" +], +[ +"41d14cc", +4, +1, +"_walkLayer" +], +[ +"41d15ec", +4, +1, +"_yQueue" +], +[ +"41d152c", +4, +1, +"_certainPath" +], +[ +"41d140c", +4, +4, +"x" +], +[ +"41d158c", +4, +1, +"_tilesQueue" +], +[ +"41d128c", +4, +2, +"maxIterations" +], +[ +"41d122c", +4, +3, +"continuePreviousSearch" +], +[ +"41d13ac", +4, +4, +"y" +], +[ +"41d134c", +4, +4, +"x2" +], +[ +"41d122c", +4, +4, +"continuePreviousSearch" +], +[ +"41d134c", +4, +5, +"x2" +], +[ +"41d13ac", +4, +5, +"y" +], +[ +"41d176c", +4, +2, +"_result" +], +[ +"41d194c", +4, +2, +"_cY" +], +[ +"41d188c", +4, +2, +"_tile" +], +[ +"41d170c", +4, +3, +"_cCardinalDistance" +], +[ +"41d13ac", +4, +6, +"y" +], +[ +"41d11cc", +4, +1, +"this" +], +[ +"41d134c", +4, +6, +"x2" +], +[ +"41d122c", +4, +5, +"continuePreviousSearch" +], +[ +"41d176c", +4, +3, +"_result" +] +], +"list": [ +[ +"41d13ac", +4, +7, +"y" +], +[ +"41d134c", +4, +7, +"x2" +], +[ +"41d122c", +4, +6, +"continuePreviousSearch" +], +[ +"41d140c", +4, +5, +"x" +], +[ +"41d170c", +4, +4, +"_cCardinalDistance" +], +[ +"41d176c", +4, +4, +"_result" +], +[ +"41d128c", +4, +3, +"maxIterations" +], +[ +"41d194c", +4, +3, +"_cY" +], +[ +"41d188c", +4, +3, +"_tile" +], +[ +"41d12ec", +4, +2, +"y2" +], +[ +"41d164c", +4, +2, +"_dirMatrix" +], +[ +"41d146c", +4, +2, +"_curGen" +], +[ +"41d16ac", +4, +2, +"_linkage" +], +[ +"41d14cc", +4, +2, +"_walkLayer" +], +[ +"41d15ec", +4, +2, +"_yQueue" +], +[ +"41d152c", +4, +2, +"_certainPath" +], +[ +"41d158c", +4, +2, +"_tilesQueue" +], +[ +"41d11cc", +4, +2, +"this" +] +] +} \ No newline at end of file diff --git a/tools/compiler-internals/tests/fixtures/findSuitableSpawnLocationUnk-202.json b/tools/compiler-internals/tests/fixtures/findSuitableSpawnLocationUnk-202.json new file mode 100644 index 0000000000..a2d3dd598e --- /dev/null +++ b/tools/compiler-internals/tests/fixtures/findSuitableSpawnLocationUnk-202.json @@ -0,0 +1,494 @@ +{ +"source": "slots-findSuitableSpawnLocationUnk-202.log", +"sorted": false, +"adds": [ +[ +"42b1144", +4, +0, +"this" +], +[ +"42b1384", +4, +0, +"x" +], +[ +"42b1324", +4, +0, +"y" +], +[ +"42b12c4", +4, +0, +"x2" +], +[ +"42b1264", +4, +0, +"y2" +], +[ +"42b1204", +4, +0, +"param_5" +], +[ +"42b1204", +4, +1, +"param_5" +], +[ +"42b11a4", +4, +0, +"param_6" +], +[ +"42b12c4", +4, +1, +"x2" +], +[ +"42b1324", +4, +1, +"y" +], +[ +"42b13e4", +4, +0, +"_curGen" +], +[ +"42b1444", +4, +0, +"_tilesQueue" +], +[ +"42b14a4", +4, +0, +"_certainPath" +], +[ +"42b1504", +4, +0, +"_yQueue" +], +[ +"42b1564", +4, +0, +"_walkLayer" +], +[ +"42b15c4", +4, +0, +"_linkage" +], +[ +"42b1624", +4, +0, +"_occupancy" +], +[ +"42b1684", +4, +0, +"_dirMatrix" +], +[ +"42b16e4", +4, +0, +"_cCardinalDistance" +], +[ +"42b1744", +4, +0, +"_result" +], +[ +"42b1384", +4, +1, +"x" +], +[ +"42b1324", +4, +2, +"y" +], +[ +"42b11a4", +4, +1, +"param_6" +], +[ +"42b1384", +4, +2, +"x" +], +[ +"42b1264", +4, +1, +"y2" +], +[ +"42b1384", +4, +3, +"x" +], +[ +"42b12c4", +4, +2, +"x2" +], +[ +"42b11a4", +4, +2, +"param_6" +], +[ +"42b12c4", +4, +3, +"x2" +], +[ +"42b1324", +4, +3, +"y" +], +[ +"42b1744", +4, +1, +"_result" +], +[ +"42b1924", +4, +0, +"_cY" +], +[ +"42b1864", +4, +0, +"_tile" +], +[ +"42b16e4", +4, +1, +"_cCardinalDistance" +], +[ +"42b1684", +4, +1, +"_dirMatrix" +], +[ +"42b1924", +4, +1, +"_cY" +], +[ +"42b16e4", +4, +2, +"_cCardinalDistance" +], +[ +"42b1624", +4, +1, +"_occupancy" +], +[ +"42b13e4", +4, +1, +"_curGen" +], +[ +"42b15c4", +4, +1, +"_linkage" +], +[ +"42b1864", +4, +1, +"_tile" +], +[ +"42b1564", +4, +1, +"_walkLayer" +], +[ +"42b1504", +4, +1, +"_yQueue" +], +[ +"42b14a4", +4, +1, +"_certainPath" +], +[ +"42b1384", +4, +4, +"x" +], +[ +"42b1444", +4, +1, +"_tilesQueue" +], +[ +"42b1204", +4, +2, +"param_5" +], +[ +"42b11a4", +4, +3, +"param_6" +], +[ +"42b1324", +4, +4, +"y" +], +[ +"42b12c4", +4, +4, +"x2" +], +[ +"42b11a4", +4, +4, +"param_6" +], +[ +"42b12c4", +4, +5, +"x2" +], +[ +"42b1324", +4, +5, +"y" +], +[ +"42b1744", +4, +2, +"_result" +], +[ +"42b1924", +4, +2, +"_cY" +], +[ +"42b1864", +4, +2, +"_tile" +], +[ +"42b16e4", +4, +3, +"_cCardinalDistance" +], +[ +"42b1324", +4, +6, +"y" +], +[ +"42b1144", +4, +1, +"this" +], +[ +"42b12c4", +4, +6, +"x2" +], +[ +"42b11a4", +4, +5, +"param_6" +], +[ +"42b1744", +4, +3, +"_result" +] +], +"list": [ +[ +"42b1324", +4, +7, +"y" +], +[ +"42b12c4", +4, +7, +"x2" +], +[ +"42b11a4", +4, +6, +"param_6" +], +[ +"42b1384", +4, +5, +"x" +], +[ +"42b16e4", +4, +4, +"_cCardinalDistance" +], +[ +"42b1744", +4, +4, +"_result" +], +[ +"42b1204", +4, +3, +"param_5" +], +[ +"42b1924", +4, +3, +"_cY" +], +[ +"42b1864", +4, +3, +"_tile" +], +[ +"42b1264", +4, +2, +"y2" +], +[ +"42b1684", +4, +2, +"_dirMatrix" +], +[ +"42b1624", +4, +2, +"_occupancy" +], +[ +"42b13e4", +4, +2, +"_curGen" +], +[ +"42b15c4", +4, +2, +"_linkage" +], +[ +"42b1564", +4, +2, +"_walkLayer" +], +[ +"42b1504", +4, +2, +"_yQueue" +], +[ +"42b14a4", +4, +2, +"_certainPath" +], +[ +"42b1444", +4, +2, +"_tilesQueue" +], +[ +"42b1144", +4, +2, +"this" +] +] +} \ No newline at end of file diff --git a/tools/compiler-internals/tests/fixtures/nameRank-1.json b/tools/compiler-internals/tests/fixtures/nameRank-1.json new file mode 100644 index 0000000000..bb6772f1b6 --- /dev/null +++ b/tools/compiler-internals/tests/fixtures/nameRank-1.json @@ -0,0 +1,1190 @@ +{ +"source": "slots-nameRank-1.log", +"sorted": true, +"adds": [ +[ +"cc8c74", +4, +0, +"p" +], +[ +"cc8cd4", +4, +0, +"_budget" +], +[ +"cc8d34", +4, +0, +"_result" +], +[ +"cc8d94", +4, +0, +"_ppIndex" +], +[ +"cc8df4", +4, +0, +"_direction" +], +[ +"cc8e54", +4, +0, +"_dirn" +], +[ +"cc8eb4", +4, +0, +"_dir" +], +[ +"cc8f14", +4, +0, +"_heading" +], +[ +"cc8f74", +4, +0, +"_moveDir" +], +[ +"cc8fd4", +4, +0, +"_dirIndex" +], +[ +"cc9034", +4, +0, +"_step" +], +[ +"cc9094", +4, +0, +"_facing" +], +[ +"cc90f4", +4, +0, +"_course" +], +[ +"cc9154", +4, +0, +"_octant" +], +[ +"cc91b4", +4, +0, +"_way" +], +[ +"cc9214", +4, +0, +"_turn" +], +[ +"cc9274", +4, +0, +"_tile" +], +[ +"cc92d4", +4, +0, +"_cY" +], +[ +"cc9334", +4, +0, +"_cX" +], +[ +"cc9394", +4, +0, +"_tileIndex" +], +[ +"cc93f4", +4, +0, +"_curTile" +], +[ +"cc9454", +4, +0, +"_cTile" +], +[ +"cc94b4", +4, +0, +"_tileNo" +], +[ +"cc9514", +4, +0, +"_tilePos" +], +[ +"cc9574", +4, +0, +"_distance" +], +[ +"cc95d4", +4, +0, +"_dist" +], +[ +"cc9634", +4, +0, +"_radius" +], +[ +"cc9694", +4, +0, +"_walk" +], +[ +"cc96f4", +4, +0, +"_first" +], +[ +"cc9754", +4, +0, +"_row" +], +[ +"cc97b4", +4, +0, +"_rowEnd" +], +[ +"cd0cdc", +4, +0, +"_readIndex" +], +[ +"cd0d3c", +4, +0, +"_areaID" +], +[ +"cd0d9c", +4, +0, +"_cLink" +], +[ +"cd0dfc", +4, +0, +"_link" +], +[ +"cd0e5c", +4, +0, +"_links" +], +[ +"cd0ebc", +4, +0, +"_best" +], +[ +"cd0f1c", +4, +0, +"_found" +], +[ +"cd0f7c", +4, +0, +"_count" +], +[ +"cd0fdc", +4, +0, +"_n" +], +[ +"cd0ebc", +4, +1, +"_best" +], +[ +"cc8df4", +4, +1, +"_direction" +], +[ +"cd0f7c", +4, +1, +"_count" +], +[ +"cd0f1c", +4, +1, +"_found" +], +[ +"cc9574", +4, +1, +"_distance" +], +[ +"cc8eb4", +4, +1, +"_dir" +], +[ +"cc9454", +4, +1, +"_cTile" +], +[ +"cc9394", +4, +1, +"_tileIndex" +], +[ +"cc9094", +4, +1, +"_facing" +], +[ +"cc96f4", +4, +1, +"_first" +], +[ +"cc9034", +4, +1, +"_step" +], +[ +"cc8d94", +4, +1, +"_ppIndex" +], +[ +"cc8d34", +4, +1, +"_result" +], +[ +"cc8cd4", +4, +1, +"_budget" +], +[ +"cd0e5c", +4, +1, +"_links" +], +[ +"cd0dfc", +4, +1, +"_link" +], +[ +"cc95d4", +4, +1, +"_dist" +], +[ +"cc94b4", +4, +1, +"_tileNo" +], +[ +"cd0d3c", +4, +1, +"_areaID" +], +[ +"cc92d4", +4, +1, +"_cY" +], +[ +"cc9274", +4, +1, +"_tile" +], +[ +"cc91b4", +4, +1, +"_way" +], +[ +"cc90f4", +4, +1, +"_course" +], +[ +"cc9694", +4, +1, +"_walk" +], +[ +"cc9514", +4, +1, +"_tilePos" +], +[ +"cc8fd4", +4, +1, +"_dirIndex" +], +[ +"cd0cdc", +4, +1, +"_readIndex" +], +[ +"cc9154", +4, +1, +"_octant" +], +[ +"cc8e54", +4, +1, +"_dirn" +], +[ +"cc8f14", +4, +1, +"_heading" +], +[ +"cc93f4", +4, +1, +"_curTile" +], +[ +"cc97b4", +4, +1, +"_rowEnd" +], +[ +"cc9214", +4, +1, +"_turn" +], +[ +"cc9634", +4, +1, +"_radius" +], +[ +"cc9334", +4, +1, +"_cX" +], +[ +"cc8f74", +4, +1, +"_moveDir" +], +[ +"cd0fdc", +4, +1, +"_n" +], +[ +"cd0d9c", +4, +1, +"_cLink" +], +[ +"cc9754", +4, +1, +"_row" +], +[ +"cd0ebc", +4, +2, +"_best" +], +[ +"cc8df4", +4, +2, +"_direction" +], +[ +"cd0f7c", +4, +2, +"_count" +], +[ +"cd0f1c", +4, +2, +"_found" +], +[ +"cc9574", +4, +2, +"_distance" +], +[ +"cc8eb4", +4, +2, +"_dir" +], +[ +"cc9454", +4, +2, +"_cTile" +], +[ +"cc9394", +4, +2, +"_tileIndex" +], +[ +"cc9094", +4, +2, +"_facing" +], +[ +"cc96f4", +4, +2, +"_first" +], +[ +"cc9034", +4, +2, +"_step" +], +[ +"cc8d94", +4, +2, +"_ppIndex" +], +[ +"cc8d34", +4, +2, +"_result" +], +[ +"cc8cd4", +4, +2, +"_budget" +], +[ +"cd0e5c", +4, +2, +"_links" +], +[ +"cd0dfc", +4, +2, +"_link" +], +[ +"cc95d4", +4, +2, +"_dist" +], +[ +"cc94b4", +4, +2, +"_tileNo" +], +[ +"cd0d3c", +4, +2, +"_areaID" +], +[ +"cc92d4", +4, +2, +"_cY" +], +[ +"cc9274", +4, +2, +"_tile" +], +[ +"cc91b4", +4, +2, +"_way" +], +[ +"cc90f4", +4, +2, +"_course" +], +[ +"cc9694", +4, +2, +"_walk" +], +[ +"cc9514", +4, +2, +"_tilePos" +], +[ +"cc8fd4", +4, +2, +"_dirIndex" +], +[ +"cd0cdc", +4, +2, +"_readIndex" +], +[ +"cc9154", +4, +2, +"_octant" +], +[ +"cc8e54", +4, +2, +"_dirn" +], +[ +"cc8f14", +4, +2, +"_heading" +], +[ +"cc93f4", +4, +2, +"_curTile" +], +[ +"cc97b4", +4, +2, +"_rowEnd" +], +[ +"cc9214", +4, +2, +"_turn" +], +[ +"cc9634", +4, +2, +"_radius" +], +[ +"cc9334", +4, +2, +"_cX" +], +[ +"cc8f74", +4, +2, +"_moveDir" +], +[ +"cd0fdc", +4, +2, +"_n" +], +[ +"cd0d9c", +4, +2, +"_cLink" +], +[ +"cc9754", +4, +2, +"_row" +], +[ +"cd0f7c", +4, +3, +"_count" +], +[ +"cd0fdc", +4, +3, +"_n" +], +[ +"cd0f1c", +4, +3, +"_found" +], +[ +"cd0ebc", +4, +3, +"_best" +], +[ +"cd0e5c", +4, +3, +"_links" +], +[ +"cd0dfc", +4, +3, +"_link" +], +[ +"cd0d9c", +4, +3, +"_cLink" +], +[ +"cd0d3c", +4, +3, +"_areaID" +], +[ +"cd0cdc", +4, +3, +"_readIndex" +], +[ +"cc97b4", +4, +3, +"_rowEnd" +], +[ +"cc9754", +4, +3, +"_row" +], +[ +"cc96f4", +4, +3, +"_first" +], +[ +"cc9694", +4, +3, +"_walk" +], +[ +"cc9634", +4, +3, +"_radius" +], +[ +"cc95d4", +4, +3, +"_dist" +], +[ +"cc9574", +4, +3, +"_distance" +], +[ +"cc9514", +4, +3, +"_tilePos" +], +[ +"cc94b4", +4, +3, +"_tileNo" +], +[ +"cc9454", +4, +3, +"_cTile" +], +[ +"cc93f4", +4, +3, +"_curTile" +], +[ +"cc9394", +4, +3, +"_tileIndex" +], +[ +"cc9334", +4, +3, +"_cX" +], +[ +"cc92d4", +4, +3, +"_cY" +], +[ +"cc9274", +4, +3, +"_tile" +], +[ +"cc9214", +4, +3, +"_turn" +], +[ +"cc91b4", +4, +3, +"_way" +], +[ +"cc9154", +4, +3, +"_octant" +], +[ +"cc90f4", +4, +3, +"_course" +], +[ +"cc9094", +4, +3, +"_facing" +], +[ +"cc9034", +4, +3, +"_step" +], +[ +"cc8fd4", +4, +3, +"_dirIndex" +], +[ +"cc8f74", +4, +3, +"_moveDir" +], +[ +"cc8f14", +4, +3, +"_heading" +], +[ +"cc8eb4", +4, +3, +"_dir" +], +[ +"cc8e54", +4, +3, +"_dirn" +], +[ +"cc8df4", +4, +3, +"_direction" +], +[ +"cc8d94", +4, +3, +"_ppIndex" +], +[ +"cc8d34", +4, +3, +"_result" +], +[ +"cc8cd4", +4, +3, +"_budget" +] +], +"list": [ +[ +"cd0f7c", +4, +4, +"_count" +], +[ +"cd0fdc", +4, +4, +"_n" +], +[ +"cd0f1c", +4, +4, +"_found" +], +[ +"cd0ebc", +4, +4, +"_best" +], +[ +"cd0e5c", +4, +4, +"_links" +], +[ +"cd0dfc", +4, +4, +"_link" +], +[ +"cd0d9c", +4, +4, +"_cLink" +], +[ +"cd0d3c", +4, +4, +"_areaID" +], +[ +"cd0cdc", +4, +4, +"_readIndex" +], +[ +"cc97b4", +4, +4, +"_rowEnd" +], +[ +"cc9754", +4, +4, +"_row" +], +[ +"cc96f4", +4, +4, +"_first" +], +[ +"cc9694", +4, +4, +"_walk" +], +[ +"cc9634", +4, +4, +"_radius" +], +[ +"cc95d4", +4, +4, +"_dist" +], +[ +"cc9574", +4, +4, +"_distance" +], +[ +"cc9514", +4, +4, +"_tilePos" +], +[ +"cc94b4", +4, +4, +"_tileNo" +], +[ +"cc9454", +4, +4, +"_cTile" +], +[ +"cc93f4", +4, +4, +"_curTile" +], +[ +"cc9394", +4, +4, +"_tileIndex" +], +[ +"cc9334", +4, +4, +"_cX" +], +[ +"cc92d4", +4, +4, +"_cY" +], +[ +"cc9274", +4, +4, +"_tile" +], +[ +"cc9214", +4, +4, +"_turn" +], +[ +"cc91b4", +4, +4, +"_way" +], +[ +"cc9154", +4, +4, +"_octant" +], +[ +"cc90f4", +4, +4, +"_course" +], +[ +"cc9094", +4, +4, +"_facing" +], +[ +"cc9034", +4, +4, +"_step" +], +[ +"cc8fd4", +4, +4, +"_dirIndex" +], +[ +"cc8f74", +4, +4, +"_moveDir" +], +[ +"cc8f14", +4, +4, +"_heading" +], +[ +"cc8eb4", +4, +4, +"_dir" +], +[ +"cc8e54", +4, +4, +"_dirn" +], +[ +"cc8df4", +4, +4, +"_direction" +], +[ +"cc8d94", +4, +4, +"_ppIndex" +], +[ +"cc8d34", +4, +4, +"_result" +], +[ +"cc8cd4", +4, +4, +"_budget" +], +[ +"cc8c74", +4, +1, +"p" +] +] +} \ No newline at end of file diff --git a/tools/compiler-internals/tests/fixtures/renameC-204.json b/tools/compiler-internals/tests/fixtures/renameC-204.json new file mode 100644 index 0000000000..4f3bc3e9ac --- /dev/null +++ b/tools/compiler-internals/tests/fixtures/renameC-204.json @@ -0,0 +1,416 @@ +{ +"source": "slots-renameC-204.log", +"sorted": false, +"adds": [ +[ +"42e822c", +4, +0, +"this" +], +[ +"42e83ac", +4, +0, +"x" +], +[ +"42e834c", +4, +0, +"y" +], +[ +"42e82ec", +4, +0, +"destX" +], +[ +"42e828c", +4, +0, +"destY" +], +[ +"42e82ec", +4, +1, +"destX" +], +[ +"42e83ac", +4, +1, +"x" +], +[ +"42e840c", +4, +0, +"_budget" +], +[ +"42e846c", +4, +0, +"_result" +], +[ +"42e84cc", +4, +0, +"_dirn" +], +[ +"42e852c", +4, +0, +"_gen" +], +[ +"42e858c", +4, +0, +"_fireCount" +], +[ +"42e85ec", +4, +0, +"_tile" +], +[ +"42e864c", +4, +0, +"_destTile" +], +[ +"42e86ac", +4, +0, +"_occupancy" +], +[ +"42e870c", +4, +0, +"_walkLayer" +], +[ +"42e876c", +4, +0, +"_linkage" +], +[ +"42e87cc", +4, +0, +"_dirMatrix" +], +[ +"42e882c", +4, +0, +"_cardinal" +], +[ +"42e888c", +4, +0, +"_pPathPlan" +], +[ +"42e88ec", +4, +0, +"_ppIndex" +], +[ +"42e88ec", +4, +1, +"_ppIndex" +], +[ +"42e846c", +4, +1, +"_result" +], +[ +"42e840c", +4, +1, +"_budget" +], +[ +"42e84cc", +4, +1, +"_dirn" +], +[ +"42e888c", +4, +1, +"_pPathPlan" +], +[ +"42e88ec", +4, +2, +"_ppIndex" +], +[ +"42e882c", +4, +1, +"_cardinal" +], +[ +"42e87cc", +4, +1, +"_dirMatrix" +], +[ +"42e84cc", +4, +2, +"_dirn" +], +[ +"42e82ec", +4, +2, +"destX" +], +[ +"42e828c", +4, +1, +"destY" +], +[ +"42e876c", +4, +1, +"_linkage" +], +[ +"42e86ac", +4, +1, +"_occupancy" +], +[ +"42e858c", +4, +1, +"_fireCount" +], +[ +"42e864c", +4, +1, +"_destTile" +], +[ +"42e852c", +4, +1, +"_gen" +], +[ +"42e840c", +4, +2, +"_budget" +], +[ +"42e85ec", +4, +1, +"_tile" +], +[ +"42e834c", +4, +1, +"y" +], +[ +"42e83ac", +4, +2, +"x" +], +[ +"42e870c", +4, +1, +"_walkLayer" +], +[ +"42e88ec", +4, +3, +"_ppIndex" +], +[ +"42e846c", +4, +2, +"_result" +], +[ +"42e840c", +4, +3, +"_budget" +], +[ +"42e84cc", +4, +3, +"_dirn" +], +[ +"42e88ec", +4, +4, +"_ppIndex" +], +[ +"42e822c", +4, +1, +"this" +], +[ +"42e846c", +4, +3, +"_result" +] +], +"list": [ +[ +"42e88ec", +4, +5, +"_ppIndex" +], +[ +"42e840c", +4, +4, +"_budget" +], +[ +"42e84cc", +4, +4, +"_dirn" +], +[ +"42e846c", +4, +4, +"_result" +], +[ +"42e82ec", +4, +3, +"destX" +], +[ +"42e83ac", +4, +3, +"x" +], +[ +"42e888c", +4, +2, +"_pPathPlan" +], +[ +"42e882c", +4, +2, +"_cardinal" +], +[ +"42e87cc", +4, +2, +"_dirMatrix" +], +[ +"42e828c", +4, +2, +"destY" +], +[ +"42e876c", +4, +2, +"_linkage" +], +[ +"42e86ac", +4, +2, +"_occupancy" +], +[ +"42e858c", +4, +2, +"_fireCount" +], +[ +"42e864c", +4, +2, +"_destTile" +], +[ +"42e852c", +4, +2, +"_gen" +], +[ +"42e85ec", +4, +2, +"_tile" +], +[ +"42e834c", +4, +2, +"y" +], +[ +"42e870c", +4, +2, +"_walkLayer" +], +[ +"42e822c", +4, +2, +"this" +] +] +} \ No newline at end of file diff --git a/tools/compiler-internals/tests/fixtures/splitB-204.json b/tools/compiler-internals/tests/fixtures/splitB-204.json new file mode 100644 index 0000000000..24789f545c --- /dev/null +++ b/tools/compiler-internals/tests/fixtures/splitB-204.json @@ -0,0 +1,416 @@ +{ +"source": "slots-splitB-204.log", +"sorted": false, +"adds": [ +[ +"429822c", +4, +0, +"this" +], +[ +"42983ac", +4, +0, +"x" +], +[ +"429834c", +4, +0, +"y" +], +[ +"42982ec", +4, +0, +"destX" +], +[ +"429828c", +4, +0, +"destY" +], +[ +"42982ec", +4, +1, +"destX" +], +[ +"42983ac", +4, +1, +"x" +], +[ +"429840c", +4, +0, +"_budget" +], +[ +"429846c", +4, +0, +"_result" +], +[ +"42984cc", +4, +0, +"_direction" +], +[ +"429852c", +4, +0, +"_gen" +], +[ +"429858c", +4, +0, +"_fireCount" +], +[ +"42985ec", +4, +0, +"_tile" +], +[ +"429864c", +4, +0, +"_destTile" +], +[ +"42986ac", +4, +0, +"_occupancy" +], +[ +"429870c", +4, +0, +"_walkLayer" +], +[ +"429876c", +4, +0, +"_linkage" +], +[ +"42987cc", +4, +0, +"_dirMatrix" +], +[ +"429882c", +4, +0, +"_cardinal" +], +[ +"429888c", +4, +0, +"_pPathPlan" +], +[ +"42988ec", +4, +0, +"_ppIndex" +], +[ +"42984cc", +4, +1, +"_direction" +], +[ +"42988ec", +4, +1, +"_ppIndex" +], +[ +"429846c", +4, +1, +"_result" +], +[ +"429840c", +4, +1, +"_budget" +], +[ +"429888c", +4, +1, +"_pPathPlan" +], +[ +"42988ec", +4, +2, +"_ppIndex" +], +[ +"429882c", +4, +1, +"_cardinal" +], +[ +"42987cc", +4, +1, +"_dirMatrix" +], +[ +"42984cc", +4, +2, +"_direction" +], +[ +"42982ec", +4, +2, +"destX" +], +[ +"429828c", +4, +1, +"destY" +], +[ +"429876c", +4, +1, +"_linkage" +], +[ +"42986ac", +4, +1, +"_occupancy" +], +[ +"429858c", +4, +1, +"_fireCount" +], +[ +"429864c", +4, +1, +"_destTile" +], +[ +"429852c", +4, +1, +"_gen" +], +[ +"429840c", +4, +2, +"_budget" +], +[ +"42985ec", +4, +1, +"_tile" +], +[ +"429834c", +4, +1, +"y" +], +[ +"42983ac", +4, +2, +"x" +], +[ +"429870c", +4, +1, +"_walkLayer" +], +[ +"42984cc", +4, +3, +"_direction" +], +[ +"42988ec", +4, +3, +"_ppIndex" +], +[ +"429846c", +4, +2, +"_result" +], +[ +"429840c", +4, +3, +"_budget" +], +[ +"42988ec", +4, +4, +"_ppIndex" +], +[ +"429822c", +4, +1, +"this" +], +[ +"429846c", +4, +3, +"_result" +] +], +"list": [ +[ +"42988ec", +4, +5, +"_ppIndex" +], +[ +"42984cc", +4, +4, +"_direction" +], +[ +"429840c", +4, +4, +"_budget" +], +[ +"429846c", +4, +4, +"_result" +], +[ +"42982ec", +4, +3, +"destX" +], +[ +"42983ac", +4, +3, +"x" +], +[ +"429888c", +4, +2, +"_pPathPlan" +], +[ +"429882c", +4, +2, +"_cardinal" +], +[ +"42987cc", +4, +2, +"_dirMatrix" +], +[ +"429828c", +4, +2, +"destY" +], +[ +"429876c", +4, +2, +"_linkage" +], +[ +"42986ac", +4, +2, +"_occupancy" +], +[ +"429858c", +4, +2, +"_fireCount" +], +[ +"429864c", +4, +2, +"_destTile" +], +[ +"429852c", +4, +2, +"_gen" +], +[ +"42985ec", +4, +2, +"_tile" +], +[ +"429834c", +4, +2, +"y" +], +[ +"429870c", +4, +2, +"_walkLayer" +], +[ +"429822c", +4, +2, +"this" +] +] +} \ No newline at end of file diff --git a/tools/compiler-internals/tests/fixtures/tracePathPlan-earlyprobe-204.json b/tools/compiler-internals/tests/fixtures/tracePathPlan-earlyprobe-204.json new file mode 100644 index 0000000000..5beadc1e66 --- /dev/null +++ b/tools/compiler-internals/tests/fixtures/tracePathPlan-earlyprobe-204.json @@ -0,0 +1,416 @@ +{ +"source": "slots-204.log", +"sorted": false, +"adds": [ +[ +"4278204", +4, +0, +"this" +], +[ +"4278384", +4, +0, +"x" +], +[ +"4278324", +4, +0, +"y" +], +[ +"42782c4", +4, +0, +"destX" +], +[ +"4278264", +4, +0, +"destY" +], +[ +"42782c4", +4, +1, +"destX" +], +[ +"4278384", +4, +1, +"x" +], +[ +"42783e4", +4, +0, +"_budget" +], +[ +"4278444", +4, +0, +"_result" +], +[ +"42784a4", +4, +0, +"_direction" +], +[ +"4278504", +4, +0, +"_gen" +], +[ +"4278564", +4, +0, +"_fireCount" +], +[ +"42785c4", +4, +0, +"_tile" +], +[ +"4278624", +4, +0, +"_destTile" +], +[ +"4278684", +4, +0, +"_occupancy" +], +[ +"42786e4", +4, +0, +"_walkLayer" +], +[ +"4278744", +4, +0, +"_linkage" +], +[ +"42787a4", +4, +0, +"_dirMatrix" +], +[ +"4278804", +4, +0, +"_cardinal" +], +[ +"4278864", +4, +0, +"_pPathPlan" +], +[ +"42788c4", +4, +0, +"_ppIndex" +], +[ +"42784a4", +4, +1, +"_direction" +], +[ +"42788c4", +4, +1, +"_ppIndex" +], +[ +"4278444", +4, +1, +"_result" +], +[ +"42783e4", +4, +1, +"_budget" +], +[ +"4278864", +4, +1, +"_pPathPlan" +], +[ +"42788c4", +4, +2, +"_ppIndex" +], +[ +"4278804", +4, +1, +"_cardinal" +], +[ +"42787a4", +4, +1, +"_dirMatrix" +], +[ +"42784a4", +4, +2, +"_direction" +], +[ +"42782c4", +4, +2, +"destX" +], +[ +"4278264", +4, +1, +"destY" +], +[ +"4278744", +4, +1, +"_linkage" +], +[ +"4278684", +4, +1, +"_occupancy" +], +[ +"4278564", +4, +1, +"_fireCount" +], +[ +"4278624", +4, +1, +"_destTile" +], +[ +"4278504", +4, +1, +"_gen" +], +[ +"42783e4", +4, +2, +"_budget" +], +[ +"42785c4", +4, +1, +"_tile" +], +[ +"4278324", +4, +1, +"y" +], +[ +"4278384", +4, +2, +"x" +], +[ +"42786e4", +4, +1, +"_walkLayer" +], +[ +"42784a4", +4, +3, +"_direction" +], +[ +"42788c4", +4, +3, +"_ppIndex" +], +[ +"4278444", +4, +2, +"_result" +], +[ +"42783e4", +4, +3, +"_budget" +], +[ +"42788c4", +4, +4, +"_ppIndex" +], +[ +"4278204", +4, +1, +"this" +], +[ +"4278444", +4, +3, +"_result" +] +], +"list": [ +[ +"42788c4", +4, +5, +"_ppIndex" +], +[ +"42784a4", +4, +4, +"_direction" +], +[ +"42783e4", +4, +4, +"_budget" +], +[ +"4278444", +4, +4, +"_result" +], +[ +"42782c4", +4, +3, +"destX" +], +[ +"4278384", +4, +3, +"x" +], +[ +"4278864", +4, +2, +"_pPathPlan" +], +[ +"4278804", +4, +2, +"_cardinal" +], +[ +"42787a4", +4, +2, +"_dirMatrix" +], +[ +"4278264", +4, +2, +"destY" +], +[ +"4278744", +4, +2, +"_linkage" +], +[ +"4278684", +4, +2, +"_occupancy" +], +[ +"4278564", +4, +2, +"_fireCount" +], +[ +"4278624", +4, +2, +"_destTile" +], +[ +"4278504", +4, +2, +"_gen" +], +[ +"42785c4", +4, +2, +"_tile" +], +[ +"4278324", +4, +2, +"y" +], +[ +"42786e4", +4, +2, +"_walkLayer" +], +[ +"4278204", +4, +2, +"this" +] +] +} \ No newline at end of file diff --git a/tools/compiler-internals/tests/fixtures/tracePathPlanToDestinationViaUnoccupiedTiles-204.json b/tools/compiler-internals/tests/fixtures/tracePathPlanToDestinationViaUnoccupiedTiles-204.json new file mode 100644 index 0000000000..4032d4cdbb --- /dev/null +++ b/tools/compiler-internals/tests/fixtures/tracePathPlanToDestinationViaUnoccupiedTiles-204.json @@ -0,0 +1,416 @@ +{ +"source": "slots-tracePathPlanToDestinationViaUnoccupiedTiles-204.log", +"sorted": false, +"adds": [ +[ +"4368254", +4, +0, +"this" +], +[ +"43683d4", +4, +0, +"x" +], +[ +"4368374", +4, +0, +"y" +], +[ +"4368314", +4, +0, +"destX" +], +[ +"43682b4", +4, +0, +"destY" +], +[ +"4368314", +4, +1, +"destX" +], +[ +"43683d4", +4, +1, +"x" +], +[ +"4368434", +4, +0, +"_budget" +], +[ +"4368494", +4, +0, +"_result" +], +[ +"43684f4", +4, +0, +"_direction" +], +[ +"4368554", +4, +0, +"_gen" +], +[ +"43685b4", +4, +0, +"_fireCount" +], +[ +"4368614", +4, +0, +"_tile" +], +[ +"4368674", +4, +0, +"_destTile" +], +[ +"43686d4", +4, +0, +"_occupancy" +], +[ +"4368734", +4, +0, +"_walkLayer" +], +[ +"4368794", +4, +0, +"_linkage" +], +[ +"43687f4", +4, +0, +"_dirMatrix" +], +[ +"4368854", +4, +0, +"_cardinal" +], +[ +"43688b4", +4, +0, +"_pPathPlan" +], +[ +"4368914", +4, +0, +"_ppIndex" +], +[ +"43684f4", +4, +1, +"_direction" +], +[ +"4368914", +4, +1, +"_ppIndex" +], +[ +"4368494", +4, +1, +"_result" +], +[ +"4368434", +4, +1, +"_budget" +], +[ +"43688b4", +4, +1, +"_pPathPlan" +], +[ +"4368914", +4, +2, +"_ppIndex" +], +[ +"4368854", +4, +1, +"_cardinal" +], +[ +"43687f4", +4, +1, +"_dirMatrix" +], +[ +"43684f4", +4, +2, +"_direction" +], +[ +"4368314", +4, +2, +"destX" +], +[ +"43682b4", +4, +1, +"destY" +], +[ +"4368794", +4, +1, +"_linkage" +], +[ +"43686d4", +4, +1, +"_occupancy" +], +[ +"43685b4", +4, +1, +"_fireCount" +], +[ +"4368674", +4, +1, +"_destTile" +], +[ +"4368554", +4, +1, +"_gen" +], +[ +"4368434", +4, +2, +"_budget" +], +[ +"4368614", +4, +1, +"_tile" +], +[ +"4368374", +4, +1, +"y" +], +[ +"43683d4", +4, +2, +"x" +], +[ +"4368734", +4, +1, +"_walkLayer" +], +[ +"43684f4", +4, +3, +"_direction" +], +[ +"4368914", +4, +3, +"_ppIndex" +], +[ +"4368494", +4, +2, +"_result" +], +[ +"4368434", +4, +3, +"_budget" +], +[ +"4368914", +4, +4, +"_ppIndex" +], +[ +"4368254", +4, +1, +"this" +], +[ +"4368494", +4, +3, +"_result" +] +], +"list": [ +[ +"4368914", +4, +5, +"_ppIndex" +], +[ +"43684f4", +4, +4, +"_direction" +], +[ +"4368434", +4, +4, +"_budget" +], +[ +"4368494", +4, +4, +"_result" +], +[ +"4368314", +4, +3, +"destX" +], +[ +"43683d4", +4, +3, +"x" +], +[ +"43688b4", +4, +2, +"_pPathPlan" +], +[ +"4368854", +4, +2, +"_cardinal" +], +[ +"43687f4", +4, +2, +"_dirMatrix" +], +[ +"43682b4", +4, +2, +"destY" +], +[ +"4368794", +4, +2, +"_linkage" +], +[ +"43686d4", +4, +2, +"_occupancy" +], +[ +"43685b4", +4, +2, +"_fireCount" +], +[ +"4368674", +4, +2, +"_destTile" +], +[ +"4368554", +4, +2, +"_gen" +], +[ +"4368614", +4, +2, +"_tile" +], +[ +"4368374", +4, +2, +"y" +], +[ +"4368734", +4, +2, +"_walkLayer" +], +[ +"4368254", +4, +2, +"this" +] +] +} \ No newline at end of file diff --git a/tools/compiler-internals/tests/fixtures/updateSeparateAreaTileMap-701.json b/tools/compiler-internals/tests/fixtures/updateSeparateAreaTileMap-701.json new file mode 100644 index 0000000000..a5a352a5c9 --- /dev/null +++ b/tools/compiler-internals/tests/fixtures/updateSeparateAreaTileMap-701.json @@ -0,0 +1,416 @@ +{ +"source": "slots-updateSeparateAreaTileMap-701.log", +"sorted": false, +"adds": [ +[ +"2cc6a74", +4, +0, +"this" +], +[ +"2cc6ad4", +4, +0, +"forceUpdate" +], +[ +"2cc6b34", +4, +0, +"_areaLayer" +], +[ +"2ca1c4c", +4, +0, +"nosym" +], +[ +"2cc6b94", +4, +0, +"_tilesQueue" +], +[ +"2ca1cac", +4, +0, +"nosym" +], +[ +"2cc6bf4", +4, +0, +"_yQueue" +], +[ +"2cc6c54", +4, +0, +"_dirMatrix" +], +[ +"2cc6d74", +4, +0, +"_linkage" +], +[ +"2cc6dd4", +4, +0, +"_zoneSizes" +], +[ +"2cc71f4", +4, +0, +"_areaID" +], +[ +"2cc6e94", +4, +0, +"_row" +], +[ +"2cc6ef4", +4, +0, +"_rowEnd" +], +[ +"2cc6f54", +4, +0, +"_readIndex" +], +[ +"2cc6ad4", +4, +1, +"forceUpdate" +], +[ +"2cc6fb4", +4, +0, +"_first" +], +[ +"2cc6ef4", +4, +1, +"_rowEnd" +], +[ +"2cc6e94", +4, +1, +"_row" +], +[ +"2cc6e94", +4, +2, +"_row" +], +[ +"2cc6ef4", +4, +2, +"_rowEnd" +], +[ +"2cc6ef4", +4, +3, +"_rowEnd" +], +[ +"2cc6e94", +4, +3, +"_row" +], +[ +"2cc6ad4", +4, +2, +"forceUpdate" +], +[ +"2cc6f54", +4, +1, +"_readIndex" +], +[ +"2cc6ad4", +4, +3, +"forceUpdate" +], +[ +"2cc7074", +4, +0, +"_tile" +], +[ +"2cc70d4", +4, +0, +"_cY" +], +[ +"2cc6f54", +4, +2, +"_readIndex" +], +[ +"2cc6c54", +4, +1, +"_dirMatrix" +], +[ +"2cc7074", +4, +1, +"_tile" +], +[ +"2cc70d4", +4, +1, +"_cY" +], +[ +"2cc6d74", +4, +1, +"_linkage" +], +[ +"2cc6bf4", +4, +1, +"_yQueue" +], +[ +"2cc6b94", +4, +1, +"_tilesQueue" +], +[ +"2cc71f4", +4, +1, +"_areaID" +], +[ +"2cc6dd4", +4, +1, +"_zoneSizes" +], +[ +"2cc6ad4", +4, +4, +"forceUpdate" +], +[ +"2cc6f54", +4, +3, +"_readIndex" +], +[ +"2cc6b34", +4, +1, +"_areaLayer" +], +[ +"2cc6ad4", +4, +5, +"forceUpdate" +], +[ +"2cc7074", +4, +2, +"_tile" +], +[ +"2cc70d4", +4, +2, +"_cY" +], +[ +"2cc6f54", +4, +4, +"_readIndex" +], +[ +"2cc71f4", +4, +2, +"_areaID" +], +[ +"2cc71f4", +4, +3, +"_areaID" +], +[ +"2cc6fb4", +4, +1, +"_first" +], +[ +"2cc6a74", +4, +1, +"this" +], +[ +"2ca1c4c", +4, +1, +"nosym" +], +[ +"2ca1cac", +4, +1, +"nosym" +], +[ +"2cc6fb4", +4, +2, +"_first" +], +[ +"2cc6a74", +4, +2, +"this" +] +], +"list": [ +[ +"2cc6ad4", +4, +6, +"forceUpdate" +], +[ +"2cc6f54", +4, +5, +"_readIndex" +], +[ +"2cc6ef4", +4, +4, +"_rowEnd" +], +[ +"2cc6e94", +4, +4, +"_row" +], +[ +"2cc71f4", +4, +4, +"_areaID" +], +[ +"2cc7074", +4, +3, +"_tile" +], +[ +"2cc70d4", +4, +3, +"_cY" +], +[ +"2cc6fb4", +4, +3, +"_first" +], +[ +"2cc6a74", +4, +3, +"this" +], +[ +"2cc6c54", +4, +2, +"_dirMatrix" +], +[ +"2cc6d74", +4, +2, +"_linkage" +], +[ +"2cc6bf4", +4, +2, +"_yQueue" +], +[ +"2cc6b94", +4, +2, +"_tilesQueue" +], +[ +"2cc6dd4", +4, +2, +"_zoneSizes" +], +[ +"2cc6b34", +4, +2, +"_areaLayer" +], +[ +"2ca1c4c", +4, +2, +"nosym" +], +[ +"2ca1cac", +4, +2, +"nosym" +] +] +} \ No newline at end of file diff --git a/tools/compiler-internals/tests/test_slot_rules.py b/tools/compiler-internals/tests/test_slot_rules.py new file mode 100644 index 0000000000..9d8026698f --- /dev/null +++ b/tools/compiler-internals/tests/test_slot_rules.py @@ -0,0 +1,116 @@ +"""Regression tests for the decoded stack-slot rules (msvc-backend.md, "The slot tiebreak, decoded"). + + python tools/compiler-internals/tests/test_slot_rules.py (or: python -m pytest tools/compiler-internals/tests) + +No compiler, no cdb: the fixtures are recorded slot_probe.py traces (the AddStackElement call +sequence and the element list c2 built from it), so the tests pin the *rule* - if someone edits +simulate() or regime() and a recorded compile no longer replays, that is the regression. + +Fixtures (tools/compiler-internals/tests/fixtures/*.json), each from one c2.dll run: + tracePathPlanToDestinationViaUnoccupiedTiles-204 the function the rule was decoded on + renameC-204 / declD-204 / splitB-204 its _direction->_dirn rename, declaration swap, + and split-asm variants (only the rename changes the list) + findSuitableSpawnLocationUnk-202, findLinkageBasedPathOrWalkRadius-202, updateSeparateAreaTileMap-701 + nameRank-1 39 asm-written locals, a 0x9c frame: the sort DID run +""" +import glob +import io +import json +import os +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, os.path.join(HERE, '..')) +sys.path.insert(0, os.path.join(HERE, '..', '..', 'reimplementation-helper', 'batch')) +import slot_probe # noqa: E402 + +FIXTURES = sorted(glob.glob(os.path.join(HERE, 'fixtures', '*.json'))) + + +def load(path): + return json.load(io.open(path, encoding='utf-8')) + + +def test_simulate_replays_every_recorded_list(): + assert FIXTURES, 'no fixtures' + for path in FIXTURES: + fx = load(path) + got = [e for e, _s, _w in slot_probe.simulate(fx['adds'])] + want = [e for e, _s, _w, _n in fx['list']] + assert got == want, '%s: simulate() order differs from the recorded list' % os.path.basename(path) + weights = dict((e, w) for e, _s, w in slot_probe.simulate(fx['adds'])) + for e, _s, w, _n in fx['list']: + assert weights[e] == w, '%s: weight of %s is %d, recorded %d' % (os.path.basename(path), e, weights[e], w) + + +def test_list_is_weight_descending_within_size(): + for path in FIXTURES: + fx = load(path) + lst = fx['list'] + for a, b in zip(lst, lst[1:]): + if a[1] == b[1]: + assert a[2] >= b[2], '%s: %s (wt %d) before %s (wt %d)' % (os.path.basename(path), a[3], a[2], b[3], b[2]) + else: + assert a[1] < b[1], '%s: sizes not ascending' % os.path.basename(path) + + +def names(fx): + return [n for _e, _s, _w, n in fx['list']] + + +def test_rename_moves_the_pair_and_declaration_order_does_not(): + base = load(os.path.join(HERE, 'fixtures', 'tracePathPlanToDestinationViaUnoccupiedTiles-204.json')) + assert names(base)[:4] == ['_ppIndex', '_direction', '_budget', '_result'] + decl = load(os.path.join(HERE, 'fixtures', 'declD-204.json')) + assert names(decl)[:4] == names(base)[:4], 'swapping declarations must not move the pair' + split = load(os.path.join(HERE, 'fixtures', 'splitB-204.json')) + assert names(split)[:4] == names(base)[:4], 'splitting the __asm block must not move the pair' + ren = load(os.path.join(HERE, 'fixtures', 'renameC-204.json')) + assert names(ren)[:4] == ['_ppIndex', '_budget', '_dirn', '_result'], 'the rename is what flips the pair' + + +def test_list_order_is_the_frame_when_the_sort_is_skipped(): + # the /FA listing of the committed tracePathPlan... (before the rename): head of the list at + # -4, every further local 4 bytes lower, parameters (kind 5) not in the frame + fx = load(os.path.join(HERE, 'fixtures', 'tracePathPlanToDestinationViaUnoccupiedTiles-204.json')) + assert not fx['sorted'] + params = set(['x', 'y', 'destX', 'destY']) + locals_ = [n for n in names(fx) if n not in params] + listing = {'_ppIndex': -4, '_direction': -8, '_budget': -12, '_result': -16, '_pPathPlan': -20, + '_cardinal': -24, '_dirMatrix': -28, '_linkage': -32, '_occupancy': -36, '_fireCount': -40, + '_destTile': -44, '_gen': -48, '_tile': -52, '_walkLayer': -56, 'this': -60} + assert [listing[n] for n in locals_] == [-4 * (i + 1) for i in range(len(locals_))] + + +def test_regime(): + assert slot_probe.regime(0x3c, False) == 'list-order' # the four asm functions + assert slot_probe.regime(0x80, False) == 'list-order' # the boundary is `jg 0x80` + assert slot_probe.regime(0x84, False) == 'sorted' + assert slot_probe.regime(0x9c, False) == 'sorted' # nameRank-1 recorded SORTED + assert slot_probe.regime(0x10, True) == 'sorted' # a /GS buffer sorts any frame + try: # slot_map's import chain opens the build directory, so only + import slot_map # cross-check the two copies of the rule when one is configured + except (ImportError, OSError): + slot_map = None + if slot_map is not None: + for frame, gs in ((0x3c, False), (0x80, False), (0x84, False), (0x9c, False), (0x10, True)): + assert slot_map.regime(frame, gs) == slot_probe.regime(frame, gs) + for path in FIXTURES: + fx = load(path) + if 'nameRank' in path: + assert fx['sorted'] + else: + assert not fx['sorted'] + + +if __name__ == '__main__': + failed = 0 + for name, fn in sorted(globals().items()): + if name.startswith('test_') and callable(fn): + try: + fn() + print('ok ', name) + except AssertionError as e: + failed += 1 + print('FAIL ', name, '-', e) + sys.exit(1 if failed else 0) diff --git a/tools/reimplementation-helper/batch/README.md b/tools/reimplementation-helper/batch/README.md index ef8e0ab6de..b615fc712f 100644 --- a/tools/reimplementation-helper/batch/README.md +++ b/tools/reimplementation-helper/batch/README.md @@ -28,6 +28,12 @@ Analysis helpers, for deciding *what* to change when a function is stuck below 1 | Script | Purpose | |---|---| | `diff_triage.py [--run] [filter]` | One line per function: match %, raw and register/slot-normalized difflib ratios, `sub esp` frame size per side, byte-sized `[esp]` accesses per side, `ret` placement. Flags `FRAME` / `BYTE` / `RET` / `alloc-only`. | +| `slot_weight.py FILE.cpp [--reorder]` | Which local got which `[esp + N]` slot, read out of MSVC's own `_name$ = N` table in a `/FA` listing, beside each local's reference count and weight per byte - the key `StkColorCmp` actually sorts on. Says whether the layout is monotone in that weight, flags slots shared by several locals and locals placed on a dead parameter's slot, and with a reccmp run names the first position where the original's slot census disagrees with ours. `--reorder` sorts the declarations by descending weight, which only decides ties. | +A walkthrough of these three - frame, then instruction mix, then registers, with two worked +functions and the traps - is in [../../compiler-internals/finding-mismatches.md](../../compiler-internals/finding-mismatches.md). + +| `slot_map.py FILE.cpp [--orig-only] [--refs]` | Compares the function's **stack frame** against the original's, with no reccmp run. Ours is read from MSVC's `/FA` slot table; the original's is inferred from its disassembly by propagating `esp` depth over the control-flow graph, which needs each callee's `ret N` since a `__thiscall` callee pops its own arguments invisibly. Prints both layouts side by side with each slot's offset, size, reference count and the `weight * 1000 / size` key `StkColorCmp` sorts on, and pads the shorter column at the front so a missing local shows up as one `?` row rather than shifting everything. Validated on the AICState build list: 55 of 60 frames exact, including every function already at 100%. A frame that is too small is strong evidence of a missing local - this is what found the two locals `renderDebugDataAiInfo` was missing. Warns instead of guessing when depth tracking cannot be trusted (indirect call, unreached block, conflicting join). | +| `structural_diff.py [NAME] [--all] [--score]` | Scores a function by **structure** instead of match percentage, in three tiers: `kind` (bare mnemonic sequence, every operand stripped), `shape` (operands with registers and constants anonymised) and `skel` (only register NAMES anonymised) and `raw` (reccmp). It also reports WHERE each kind of difference starts: when the first register difference sits at or after the first structural one, the register names are an effect of that mix defect and will follow once it is fixed, and a substitution set that closes into a cycle (`eax -> edx -> ecx -> eax`) is the signature - the allocation is right and the scratch registers were handed out in a different order. It **detects hoisted duplicates**: when MSVC lifts an expression out of both arms of an `if`/`else` and the original did not, the mix delta reads as one missing copy of each mnemonic in the run (`add -1, mov -1, push -1`), which looks like three unrelated missing instructions and is really one missing duplicate of `mov reg, [reg + G]; add reg, 1; push reg`. Counting occurrences with registers anonymised names it outright and says which way to fix it - that signal was worth 67.5% -> 83.6% on `renderDebugDataAiInfo`. It fires on 4 of 60 AICState functions and is silent once fixed. `delta` counts the per-mnemonic mismatches, so it names defects rather than giving a ratio - "we emit two `xor` the original does not". `--all` ranks the build list into three verdicts: `mix` (wrong instruction mix, where source work pays), `order` (right mix, wrong order - statement order, which is what took the sixteen `setAICParameters_NN` to 100%) and `allocation` (same instructions in the same order; the gap is register naming, so leave it alone). Use it before picking a target: the percentage is dominated by allocation and can rank a structurally wrong function above a structurally perfect one. | | `diff_slots.py NAME` | Every `[esp + N]` slot with its access count on each side, and the slots only we use. Run after `diff_triage.py` reports `FRAME`. | | `diff_jcc.py [NAME]` | Comparisons whose operator is one strictness step off the original (`jl`/`jle`, `jg`/`jge`, `ja`/`jae`) or whose `cmp` operands are the other way round, with context. | | `diff_types.py [--run] [NAME]` | The non-comparison counterpart of `diff_jcc.py`: `movsx` against `movzx` for the same field (its signedness is wrong in the generated header), `add` against `sub` with the same magnitude (a sign error in a formula - how `2400/n + 40` was caught being `- 40`). It pairs only equal-length runs inside one reccmp block, so it stays quiet once the streams drift: it reports nothing across the 101 functions of `Game::GameStateStructures` and fires on both bugs when they are reintroduced. | @@ -36,9 +42,9 @@ Analysis helpers, for deciding *what* to change when a function is stuck below 1 | `diff_signed.py [NAME-SUBSTRING]` | Comparisons the original makes signed and we make unsigned (or the reverse): `jg/jge/jl/jle` against our `ja/jae/jb/jbe` over the same `cmp`. Fix it by comparing through `(int)` in the source, not by editing the generated header, and re-run the tool to confirm - one instruction barely moves the percentage. | | `quick_diff.py FILE.cpp... [-q]` | Compile one file with `/FA` and diff it against the original function disassembled straight from `_original/` - about 6 seconds, no link and no reccmp, so a dozen source variants can be tried in the time of one build. Shows the whole original function, which reccmp's diff does not when ours is shorter. A triage loop: confirm a 1.0 with `reccmp_report.py --run`, and do not trust a *rise* of its ratio below about 0.6 - on a badly matching function difflib rewards a different alignment, and one such "gain" was a 10-point loss in reccmp. | | `push_pattern.py FILE.cpp...` | Per call, the argument-push run as `pushes/interleaved` for both sides. A neat run (`0` interleaved) means every argument was already in a register or slot, i.e. the source had a **local**; a mixed run is what writing the expression inline at the call produces. A starred row is a call where the two sides disagree - add the local, or drop it, accordingly. Counts only calls whose set-up it can see, so a differing call *total* means our source has an extra call (a duplicated block), not that the tool failed. | -| `perm_decls.py FILE FIRST LAST` | Permute one short run of declarations and keep the order that matches best, scored by `quick_diff.py -s`. MSVC assigns stack slots in declaration order, so two locals declared the wrong way round sit in each other's slots and usually in each other's registers - `assignSelectionToKey` 95.9% -> 100% from one swapped line. Find the candidates with `diff_slots.py`: same slot count, different use count per slot. | +| `perm_decls.py FILE FIRST LAST` | Permute one short run of declarations and keep the order that matches best, scored by `quick_diff.py -s`. Declaration order does **not** decide stack slots - the colorer sorts by reference weight per byte and leaves only *ties* to the order the symbols arrive in, so this pays exactly when two locals of equal weight are declared the wrong way round (`assignSelectionToKey` 95.9% -> 100% from one swapped line) and does nothing otherwise. Run `slot_weight.py` first to see whether the weights tie at all. For initialised declarations the order does decide which *register* each one gets, which is a separate effect worth permuting for. | | `reorder_search.py FILE NAME [--pairs] [--dry]` | Hill-climb the match % by reordering independent statements: permute runs of >=3 by default, or swap every adjacent independent pair with `--pairs`. Builds once per move and reverts anything that does not improve. | -| `quick_reorder.py FILE START END [START END ...] [-s]` | The same hill-climb scored by `quick_diff.py` instead of a build, so a trial takes seconds: every statement of each named block (from the line containing START up to the line containing END) is tried at every other position, and only strict ratio gains are kept. With `-s` stack offsets count, which makes **declaration order** visible - it took `AICState::selectAttackTarget` 68.8% -> 100% and `smallestTribeOfUnitType` 57% -> 96% with two or three moves each. It does not check data dependencies (a use before a declaration fails to compile and is rejected, a store moved across a read is not), so give it blocks of declarations and pure `return` guards and read `git diff` afterwards. Set `QD_OUT` per run to run several in parallel, and let them finish before a full build (shared PDB, C1033). | +| `quick_reorder.py FILE START END [START END ...] [-s]` | The same hill-climb scored by `quick_diff.py` instead of a build, so a trial takes seconds: every statement of each named block (from the line containing START up to the line containing END) is tried at every other position, and only strict ratio gains are kept. With `-s` stack offsets count, which makes **declaration order** visible - it took `AICState::selectAttackTarget` 68.8% -> 100% and `smallestTribeOfUnitType` 57% -> 96% with two or three moves each. It does not check data dependencies (a use before a declaration fails to compile and is rejected, a store moved across a read is not), so give it blocks of declarations and pure `return` guards and read `git diff` afterwards. Set `QD_OUT` per run to run several in parallel, and let them finish before a full build (shared PDB, C1033). On a branch whose `quick_diff.py` predates `QD_OUT` support the variable is silently ignored, every run shares one `x.asm`, and each trial that reads another run's listing scores -1 and is rejected - the hill-climb then reports `final == start` having measured nothing. Check that `quick_diff.py` mentions `QD_OUT` before running these in parallel. | ### Choosing what to work on diff --git a/tools/reimplementation-helper/batch/asm_streams.py b/tools/reimplementation-helper/batch/asm_streams.py new file mode 100644 index 0000000000..e64dee043b --- /dev/null +++ b/tools/reimplementation-helper/batch/asm_streams.py @@ -0,0 +1,93 @@ +"""The two instruction streams for one function, both complete. + +`reccmp/dll/diff.json` is **not** a source of whole functions, and anything that counts or +compares instructions must not read it. It stores the diff view - changed blocks plus a little +context - so the better a function matches, the less of it is there: + + aiCreateSiegeUnits raw 0.997 diff.json: 22 lines per side real function: 0x4fe bytes + sellExcessGoods raw 0.624 diff.json: 133 lines per side + +A near-perfect function leaves a handful of lines behind, so counts taken from `diff.json` are +wrong in proportion to how good the code is - which is the worst possible direction for a tool +meant to rank functions by how much structural work they need. + +Separately, the original side of `diff.json` is clipped: reccmp reads our function's byte length +from our PDB and dumps that many bytes from the original's address, so when our code is shorter +the original loses its tail. `aiGiveRaidInstructions` is cut 42 bytes early and appears to end +on `push edx; push eax` with no `ret` anywhere, when it plainly ends `004d2dfc ret 4`. + +So both streams come from the code itself, which is what `quick_diff.py` already does: + +* original - disassembled from `_original/Stronghold Crusader.exe` with capstone, the extent + found by walking forward tracking the furthest forward branch target and stopping at the + first `ret`/`jmp` at or past it. Nothing branches over the end of a function. Checked against + Ghidra on three bodies - 004d1320/004d2000/004d2a70 - and exact on all three. +* ours - compiled with `/FA` and read out of the listing between `PROC` and `ENDP`. + +Mnemonics are folded through `orig_body.canon`, because capstone says `jz` where MASM says `je`; +comparing unfolded invents differences that are not there. + +`streams(path)` returns `(original, ours)` as normalised instruction lines, and `mnemonics()` +reduces either to a bare mnemonic list. `raw_entry()` is the one thing still worth taking from +`diff.json`: reccmp's percentage, for reference only. +""" + +import os +import sys + +import common +import orig_body + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +# `common` and `quick_diff` both chdir() to the repo root as they import, so a relative path +# from the command line must be resolved against `common.INVOCATION_DIR` - the cwd as it was - +# and not against os.getcwd(), which is already the root by the time anything here runs +import quick_diff # noqa: E402 - imports cleanly; its CLI body is guarded by __main__ + + +def resolve(path): + """An absolute path, honouring the directory this process started in.""" + if os.path.isabs(path): + return path + for root in (str(common.INVOCATION_DIR), str(common.ROOT)): + cand = os.path.join(root, path) + if os.path.exists(cand): + return os.path.abspath(cand) + return os.path.abspath(os.path.join(str(common.INVOCATION_DIR), path)) + + +def streams(path, keep_slots=False): + """(original, ours) complete normalised instruction lines. `ours` is None if it failed.""" + quick_diff.keep_slots = keep_slots + return quick_diff.streams(resolve(path)) + + +def mnemonics(lines): + """Bare canonical mnemonic sequence.""" + out = [] + for line in lines: + parts = line.split() + if not parts: + continue + m = parts[0].lower() + if not m[0].isalpha() or m in orig_body.PADDING: + continue + out.append(orig_body.canon(m)) + return out + + +def raw_entry(path, byaddr=None): + """reccmp's percentage for this file, or None - reference only, never for counting.""" + byaddr = byaddr if byaddr is not None else common.load_diff() + for _f, e, _a in common.entries(byaddr, [path]): + if e is not None: + return e + return None + + +if __name__ == '__main__': + if len(sys.argv) < 2: + sys.exit(__doc__) + for p in sys.argv[1:]: + o, u = streams(p) + print('%-40s original %4d ours %4d' % (os.path.basename(p), len(o), -1 if u is None else len(u))) diff --git a/tools/reimplementation-helper/batch/orig_body.py b/tools/reimplementation-helper/batch/orig_body.py new file mode 100644 index 0000000000..45a682834b --- /dev/null +++ b/tools/reimplementation-helper/batch/orig_body.py @@ -0,0 +1,180 @@ +"""Disassemble an original function from `_original/` in full, with correct bounds. + +usage: orig_body.py (FILE.cpp | 0xADDRESS) [--count] [--end 0xADDR] [--raw] + +`orig_asm.py` prints the original stream as recorded in `reccmp/dll/diff.json`, and that +stream is **not the whole function**: reccmp reads our function's byte length from our PDB and +dumps that many bytes from the original's address, so whenever our code is shorter in bytes the +original is clipped. `aiGiveRaidInstructions` is clipped 42 bytes early and ends mid-call-setup +on `push edx; push eax`, with no `ret` anywhere - which makes a function that plainly has one +look like it ends on two pushes, and makes any comparison of instruction counts or of +tail-end code meaningless. + +This reads the bytes out of `_original/Stronghold Crusader.exe` instead and finds the end +itself: walk forward tracking the furthest forward branch target, and stop at the first `ret` +or `jmp` that sits at or past that target. Nothing can jump over the end of a function, so a +`ret` with no pending forward target beyond it is the end. On `aiGiveRaidInstructions` that +gives `0x4d2dff`, matching Ghidra's body range `004d2a70 - 004d2dfe` exactly. + +The heuristic can still be fooled - by a function whose last instruction is a conditional +branch, or by one the linker split - so `--end` overrides it, and Ghidra's +`get_function_by_address` is the second opinion worth taking. Ghidra is not always right +either; where the two disagree, read the bytes. + +`--count` prints a mnemonic histogram instead of the listing, which is what structural work +needs. Mnemonics are canonicalised first (`jz` to `je`, `jnz` to `jne`, and so on) because +capstone and reccmp spell the same instruction differently, and an uncanonicalised comparison +invents differences that are not there. + +Other modules import `disassemble`, `histogram` and `find_end` from here. +""" + +import re +import struct +import sys + +import capstone + +import common + +# capstone and reccmp disagree on the spelling of several instructions; comparing without +# folding these invents differences that do not exist +SYNONYMS = { + 'jz': 'je', 'jnz': 'jne', 'jc': 'jb', 'jnc': 'jae', 'jnb': 'jae', 'jnbe': 'ja', + 'jnae': 'jb', 'jna': 'jbe', 'jnge': 'jl', 'jnl': 'jge', 'jng': 'jle', 'jnle': 'jg', + 'jpe': 'jp', 'jpo': 'jnp', 'setz': 'sete', 'setnz': 'setne', 'setc': 'setb', + 'setnc': 'setae', 'setnbe': 'seta', 'setnge': 'setl', 'setnl': 'setge', + 'setng': 'setle', 'setnle': 'setg', 'sal': 'shl', 'cmovz': 'cmove', + 'cmovnz': 'cmovne', 'repe': 'rep', 'repz': 'rep', 'retn': 'ret', +} +PADDING = ('int3', 'nop') +# MSVC pads with nop-equivalents rather than `nop`: `lea esp, [esp]`, `lea ecx, [ecx]` and +# `mov edi, edi`. They are not real memory references, and anything reading stack operands has +# to skip them or `lea esp, [esp]` reads as a reference to the slot at the current esp. +_NOP_LEA = re.compile(r'^(\w+), \[\1(?: \+ 0x0)?\]$') + + +def is_padding(mnemonic, ops): + """True for an instruction that only exists to fill space.""" + if mnemonic in PADDING: + return True + if mnemonic == 'lea' and _NOP_LEA.match(ops): + return True + return mnemonic == 'mov' and ops.replace(' ', '') == 'edi,edi' + + +def canon(mnemonic): + return SYNONYMS.get(mnemonic, mnemonic) + + +class Image(object): + """The original executable, with address-to-file-offset mapping.""" + + def __init__(self, path=None): + path = path or (common.ROOT / '_original' / 'Stronghold Crusader.exe') + self.data = open(str(path), 'rb').read() + d = self.data + pe = struct.unpack_from(' furthest: + furthest = target + if canon(ins.mnemonic) in ('ret', 'jmp') and ins.address >= furthest: + return ins.address + ins.size + return None + + +def disassemble(start, end=None, image=None): + """[(address, mnemonic, operands)] for one original function, padding dropped.""" + image = image or Image() + end = end or find_end(image, start) + if end is None: + raise ValueError('could not find the end of the function at %#x' % start) + md = capstone.Cs(capstone.CS_ARCH_X86, capstone.CS_MODE_32) + off = image.offset(start) + out = [] + for ins in md.disasm(image.data[off:off + (end - start)], start): + if ins.mnemonic in PADDING: + continue + out.append((ins.address, canon(ins.mnemonic), ins.op_str)) + return out + + +def histogram(stream): + counts = {} + for _a, m, _o in stream: + counts[m] = counts.get(m, 0) + 1 + return counts + + +def address_of(arg): + """An explicit 0x address, or the `// FUNCTION:` marker of a source file.""" + if arg.lower().startswith('0x'): + return int(arg, 16) + text = common.read_text(arg) + m = re.search(r'FUNCTION: STRONGHOLDCRUSADER (0x[0-9A-Fa-f]+)', text) + if not m: + sys.exit('%s has no `// FUNCTION:` marker' % arg) + return int(m.group(1), 16) + + +def main(): + args = [a for a in sys.argv[1:] if not a.startswith('--')] + if not args: + sys.exit(__doc__) + end = None + for a in sys.argv[1:]: + if a.startswith('--end'): + end = int(a.split('=', 1)[1], 16) if '=' in a else None + if end is None and '--end' in sys.argv: + i = sys.argv.index('--end') + if i + 1 < len(sys.argv): + end = int(sys.argv[i + 1], 16) + start = address_of(args[0]) + image = Image() + detected = find_end(image, start) + use = end or detected + stream = disassemble(start, use, image) + print('# %#x .. %#x %d instructions %d bytes%s' + % (start, use, len(stream), use - start, + ' (--end override; detected %#x)' % detected if end and end != detected else '')) + if '--count' in sys.argv: + counts = histogram(stream) + for m in sorted(counts, key=lambda k: -counts[k]): + print('%6d %s' % (counts[m], m)) + return + for addr, mnemonic, ops in stream: + print('%08x %-7s %s' % (addr, mnemonic, ops)) + + +if __name__ == '__main__': + main() diff --git a/tools/reimplementation-helper/batch/perm_decls.py b/tools/reimplementation-helper/batch/perm_decls.py index c0da342110..298ad9f8db 100644 --- a/tools/reimplementation-helper/batch/perm_decls.py +++ b/tools/reimplementation-helper/batch/perm_decls.py @@ -1,9 +1,12 @@ """Permute a short run of declarations and keep the order that matches best. -MSVC assigns stack slots in declaration order, so two locals declared the wrong way round sit in -each other's slots and usually end up in each other's registers too - a mismatch that spreads over -every instruction touching either of them while `diff_triage.py` still reports the function as -register-naming only. `assignSelectionToKey` had the original's slot *set* exactly +Declaration order does not decide stack slots: the colorer sorts its elements by reference weight +per byte (`StkColorCmp`, see the cheat sheet) and leaves only the *ties* to the order the symbols +arrive in. So this pays exactly when two locals of equal weight are declared the wrong way round - +they then sit in each other's slots and usually in each other's registers too, a mismatch that +spreads over every instruction touching either of them while `diff_triage.py` still reports the +function as register-naming only. Run `slot_weight.py` first: if the weights differ, no permutation +of the declarations will move those slots. `assignSelectionToKey` had the original's slot *set* exactly (`0x18`, `0x1c`, `0x20`) with `tribeIndex` and `slotID` swapped between `0x18` and `0x20`; declaring `tribeIndex` first took it from 95.9% to 100%. diff --git a/tools/reimplementation-helper/batch/slot_map.py b/tools/reimplementation-helper/batch/slot_map.py new file mode 100644 index 0000000000..7d08cd9c84 --- /dev/null +++ b/tools/reimplementation-helper/batch/slot_map.py @@ -0,0 +1,436 @@ +"""Compare a function's stack frame against the original's, without reccmp. + +usage: slot_map.py FILE.cpp [--orig-only] [--refs] [--end=0xADDR] + +Checking the frame first is worth it because the frame is the one part of the output that is +decided by information the source controls directly - which locals exist, how big they are and +how often each is used - and because reccmp's percentage actively misleads here. Adding a local +shifts every `[esp + N]` displacement in the function, so reccmp scores all of them as +mismatches until the layout happens to line up: on `renderDebugDataAiInfo`, adding the two +locals the original demonstrably has moved the frame from 0x3f8 to exactly 0x400 and the +instruction-mix delta from 13 to 3, while the percentage *fell* from 68.4 to 67.5. Frame first, +then the instruction mix, and reccmp only to confirm at the end. + +What each side gives: + +* ours - MSVC's `/FA` listing states the frame outright, one line per slot with its name, its + offset from the entry `esp` and its size: + + _this$GSCopy$ = -1016 ; size = 4 + _playerID$ = -1012 ; size = 4 + _text$ = -1004 ; size = 1000 + __$ArrayPad$ = -4 ; size = 4 + +* the original - has to be inferred from the disassembly. Every stack reference is + `[esp + displacement]` where the displacement depends on how far `esp` has been pushed down + at that instruction, so the slot it names is `displacement - depth`, with `depth` tracked + across `push`/`pop`/`sub esp`/`add esp` from the function entry. That recovers the same + entry-relative offsets the listing prints, which makes the two directly comparable. + +Slot *sizes* on the original side cannot be read off, only bounded: a 1000-byte buffer is +touched once through `lea` at its base, so its size shows up as the gap to the next slot that +is referenced. The report prints the gap as `span` and the widest observed access as `acc`, +and never pretends the two are the same thing. + +Reference counts matter as much as the offsets, and there are two regimes (decoded from +c2.dll, see tools/compiler-internals/msvc-backend.md "The slot tiebreak, decoded"): + +* frame > 0x80 bytes, or a /GS buffer: `QSortColors` sorts the slots by the class byte, then by + `weight * 1000 / size` descending. The `key` column prints that key. +* otherwise the sort is SKIPPED and the slot order is c2's element list as built: size + ascending, then reference count descending, and among equal counts the local whose LAST + reference comes first in the tuple stream gets the higher address. Declaration order is not in + the rule. An `__asm` block counts each local once per operand set (def-set, use-list, def-set + again), and its def-set is iterated in name-hash order, so for asm-written locals the name + itself is the tiebreak. The `key` column then prints the reference count, and + `tools/compiler-internals/slot_probe.py` shows the exact reference sequence. + +Either way a local in the wrong place usually means our reference count differs from the +original's. `--refs` lists every referencing address per slot. The report names the regime. + +A `?` row on either side is a slot that side does not have. Those are the actionable rows: a +`?` under ours is a local the original has and we do not. + +Needs capstone and a configured build directory; no build and no reccmp run. +""" + +import collections +import os +import struct +import re +import sys + +import asm_streams +import common +import orig_body + +LISTING = common.ROOT / common.TMP / 'quick' / 'x.asm' +SLOT_DECL = re.compile(r'^(\S+) = (-?\d+)\s*;\s*size = (\d+)\s*$') +# `dword ptr [esp + 0x3f8]`, `[esp]`, and the `lea reg, [esp + N]` form with no size prefix +ESP_REF = re.compile(r'(?:(\w+) ptr )?\[esp(?:\s*([+-])\s*(0x[0-9a-f]+|\d+))?\]') +ORIG_ONLY = False +SHOW_REFS = False +WIDTH = {'byte': 1, 'word': 2, 'dword': 4, 'qword': 8, 'tbyte': 10, 'xmmword': 16} + + +def ours(path): + """(frame_size, [(offset, size, refs, name)]) from the /FA listing.""" + text = common.read_text(path) + m = re.search(r'FUNCTION: STRONGHOLDCRUSADER[^\n]*\n[^\n(]*?(\w+)::(\w+)\(', text) + if not m: + raise ValueError('%s has no `// FUNCTION:` marker followed by a definition' % path) + cls, name = m.groups() + asm_streams.streams(path, keep_slots=True) # produces the listing + listing = common.read_text(LISTING).splitlines() + pat = re.compile(r'^\?%s@%s@\S* PROC' % (re.escape(name), re.escape(cls))) + start = next((i for i, l in enumerate(listing) if pat.match(l)), None) + if start is None: + raise ValueError('no code emitted for %s::%s (inlined, or named differently)' % (cls, name)) + # the slot table is the run of `name = N ; size = M` lines just above the PROC line + slots = [] + i = start - 1 + while i >= 0: + d = SLOT_DECL.match(listing[i].replace('\t', ' ').strip()) + if d: + slots.append((int(d.group(2)), int(d.group(3)), d.group(1))) + elif slots: + break + i -= 1 + end = next(i for i in range(start, len(listing)) if ' ENDP' in listing[i]) + body = '\n'.join(listing[start:end]) + frame = 0 + f = re.search(r'\n\tsub\tesp, (\d+)', body) + if f: + frame = int(f.group(1)) + gs = '___security_cookie' in body + out = [] + for off, size, nm in sorted(slots): + if off >= 8: + continue # +8 and up are the incoming arguments, not the frame + refs = len(re.findall(r'\b%s\[e(?:sp|bp)' % re.escape(nm), body)) + if refs == 0: + # MSVC emits slot symbols it then never uses - an unspilled `_this$`, EH + # bookkeeping. The original side can only ever show slots that are actually + # referenced, so keeping these invents a difference: in + # `clearPlayerDataInformationChunk`, which matches 100%, three such symbols + # made a frameless function look three slots too big. + continue + out.append((off, size, refs, nm)) + return frame, out, gs + + +def _imm(text): + text = text.strip() + try: + return int(text, 16) if text.lower().startswith('0x') else int(text) + except ValueError: + return None + + +_POPPED = {} + + +def popped(image, target): + """How many bytes of arguments the function at `target` pops on return. + + A `__thiscall` or `__stdcall` callee cleans up its own arguments, so `esp` rises across + the call by an amount that is nowhere visible at the call site - most of the calls in a + rendering function are of that kind, with nine pushes and no `add esp` afterwards. Reading + the callee's `ret N` is the only way to know, and getting it wrong desynchronises every + stack offset after the first call. + """ + if target in _POPPED: + return _POPPED[target] + _POPPED[target] = 0 # guard against a recursive thunk chain + try: + stream = orig_body.disassemble(target, target + 0x40, image) + except Exception: + return 0 + for _a, mnemonic, ops in stream: + if mnemonic == 'jmp': # an import thunk; follow it once + t = _imm(ops) + _POPPED[target] = popped(image, t) if t is not None else 0 + return _POPPED[target] + if mnemonic == 'ret': + _POPPED[target] = _imm(ops) or 0 + return _POPPED[target] + # no `ret` in the first 0x40 bytes: disassemble the whole body and take its first one + try: + for _a, mnemonic, ops in orig_body.disassemble(target, None, image): + if mnemonic == 'ret': + _POPPED[target] = _imm(ops) or 0 + break + except Exception: + pass + return _POPPED[target] + + +_TABLE = re.compile(r'\[\w+\*4 \+ (0x[0-9a-f]+)\]') + + +def _jump_table(image, ops, lo, hi): + """Targets of a switch's `jmp dword ptr [reg*4 + TABLE]`. + + MSVC emits the table just past the function's last instruction, and for a sparse switch + puts a byte index table after it, so the dword entries are followed only while they point + back inside the body - the first value that does not is the next table's data. On + `addUnitToSmallestBehaviourTypeTribe` that yields exactly five case targets and stops on + `0x04040400`, which is the byte table. + """ + m = _TABLE.search(ops) + if not m: + return [] + base = int(m.group(1), 16) + off = image.offset(base) + if off is None: + return [] + out = [] + for i in range(256): + if off + 4 * i + 4 > len(image.data): + break + t = struct.unpack_from(' 0x80 or gs) else 'list-order' + + +def key(refs, size, mode='sorted'): + """The ordering key per regime: StkColorCmp's weight * 1000 / size when the sort runs, the + bare reference count (the list is weight-descending) when it does not.""" + if mode != 'sorted': + return refs + return (refs * 1000 // size) if size else 0 + + +def main(): + # parsed here, not at import: the other tools import this module for `original()` and + # `ours()`, and a module-level sys.exit turns that into an abrupt exit with a usage dump + global ORIG_ONLY, SHOW_REFS + ORIG_ONLY = '--orig-only' in sys.argv + SHOW_REFS = '--refs' in sys.argv + args = [a for a in sys.argv[1:] if not a.startswith('--')] + if not args: + sys.exit(__doc__) + path = args[0] + address = orig_body.address_of(path) + # --end 0xADDR: the original's true end when orig_body's detection stops early. A function + # whose switch cases sit after the detected end (processHelpRichTextTokens: 703 instructions + # unreached, its jump table at the real end) needs this or its slot counts are a fraction. + end = None + for a in sys.argv[1:]: + if a.startswith('--end='): + end = int(a.split('=', 1)[1], 16) + o_frame, o_slots, trouble = original(address, end) + for addr, had, got in trouble['conflicts'][:3]: + print('WARNING: two paths reach %#x with different esp depths (%+d and %+d), so the' + % (addr, had, got)) + print(' offsets below may be wrong. Usually an indirect call whose argument') + print(' cleanup could not be read, or a tail-merged block.') + if trouble['unknown_calls']: + print('NOTE: %d indirect call(s), e.g. %#x - the callee pops an unknown number of' + % (len(trouble['unknown_calls']), trouble['unknown_calls'][0])) + print(' argument bytes, assumed 0. Offsets after it are suspect.') + if trouble.get('impossible'): + print('WARNING: %d reference(s) landed below the frame (%s), which cannot be a slot -' + % (len(trouble['impossible']), + ', '.join(str(o) for o in trouble['impossible'][:4]))) + print(' the depth tracking drifted. They are excluded, but every offset below') + print(' is suspect; do not read a slot-count difference as a real defect.') + if trouble['unreached']: + print('NOTE: %d instruction(s) were not reached from the entry, so their stack' + % trouble['unreached']) + print(' references are not counted (jump table or a misdetected function end).') + if trouble['conflicts'] or trouble['unknown_calls'] or trouble['unreached']: + print('') + print('frame original %#x ours %s' + % (o_frame, '(not compared)' if ORIG_ONLY else ''), end='') + if ORIG_ONLY: + print('') + for off, span, acc, refs, addrs in o_slots: + print(' %6d span %-5s acc %-2s refs %-3d' % (off, span, acc or '?', refs)) + return + u_frame, u_slots, u_gs = ours(path) + print('%#x (%+d)' % (u_frame, u_frame - o_frame)) + mode = regime(o_frame, trouble.get('gs', False)) + u_mode = regime(u_frame, u_gs) + if mode == 'sorted': + print('regime sorted: frame > 0x80 or /GS buffer, so slots follow weight*1000/size (the `key` column)') + else: + print('regime list-order: frame <= 0x80 and no /GS buffer, so QSortColors is skipped - slots follow') + print(' reference count (the `key` column), ties by whose LAST reference comes first;') + print(' for asm-written locals that is the name-hash order of the def-set (slot_probe.py)') + if u_mode != mode: + print('WARNING: ours is in the %s regime (frame %#x, cookie %s) while the original is %s - the' + % (u_mode, u_frame, 'yes' if u_gs else 'no', mode)) + print(' orders are not comparable until the frame size / GS state agrees') + print('') + # pair the two layouts in offset order; both are entry-relative, so a constant shift means + # the layout is right and only the frame size differs + print('%-28s | %s' % ('ORIGINAL (inferred)', 'OURS (from the /FA listing)')) + print('%6s %6s %4s %5s %6s | %6s %6s %5s %6s %s' + % ('off', 'span', 'acc', 'refs', 'key', 'off', 'size', 'refs', 'key', 'name')) + # both frames end at the same place - the security cookie and the return address anchor the + # high end - so pad the shorter column at the FRONT. Aligning from the top instead would + # shift every row and make one missing local look like every slot being wrong. + n = max(len(o_slots), len(u_slots)) + o_pad, u_pad = n - len(o_slots), n - len(u_slots) + for i in range(n): + if i >= o_pad: + off, span, acc, refs, _a = o_slots[i - o_pad] + size = span if span else (acc or 4) + left = '%6d %6s %4s %5d %6d' % (off, span if span else '-', acc or '?', refs, + key(refs, size, mode)) + else: + left = '%6s %6s %4s %5s %6s' % ('?', '', '', '', '') + if i >= u_pad: + uoff, usize, urefs, nm = u_slots[i - u_pad] + right = '%6d %6d %5d %6d %s' % (uoff, usize, urefs, key(urefs, usize, mode), nm) + else: + right = '%6s %6s %5s %6s %s' % ('?', '', '', '', '<-- a local we do not have') + print('%s | %s' % (left, right)) + print('') + print(' %d slot(s) on the original side, %d on ours.' % (len(o_slots), len(u_slots))) + if u_frame != o_frame: + d = o_frame - u_frame + print(' Frame differs by %+d bytes, so %s %d dword-sized local(s).' + % (-d, 'we are missing about' if d > 0 else 'we have about', abs(d) // 4)) + print(' A frame that is too small is strong evidence of a missing local. Too large can') + print(' be either a surplus local or a spill, so it is weaker evidence.') + else: + print(' Frame sizes agree, so the set of locals is probably right and what is left is') + if mode == 'sorted': + print(' their ORDER - compare the `key` columns, since that is what the sort uses.') + else: + print(' their ORDER - compare the reference counts; equal counts are ordered by the position') + print(' of the LAST reference, which slot_probe.py prints for each local.') + if SHOW_REFS: + print('') + for off, _span, _acc, refs, addrs in o_slots: + print(' original %6d %d refs: %s' + % (off, refs, ' '.join('%08x' % a for a in addrs[:14]))) + + +if __name__ == '__main__': + main() diff --git a/tools/reimplementation-helper/batch/slot_weight.py b/tools/reimplementation-helper/batch/slot_weight.py new file mode 100644 index 0000000000..91d8e7eac4 --- /dev/null +++ b/tools/reimplementation-helper/batch/slot_weight.py @@ -0,0 +1,302 @@ +"""Explain an `[esp + N]` layout through the weight the stack colorer actually sorts on. + +usage: slot_weight.py FILE.cpp [--reorder] [--quiet] + +MSVC 14.00 does not lay spilled locals out in declaration order. `ColorStackElements` +(`P2/stack.c`) builds one 28-byte record per stack element, holding the local's size +rounded up to 4 and its accumulated reference weight, sorts the records with +`StkColorCmp` on `(class, weight * 1000 / size)` and then hands out offsets in that +order. So for a block of same-sized locals the layout is simply **descending reference +weight**, and the lever on it is how often - and at what loop depth - the source reads +each local, not where it is declared. See IMPLEMENTATION_CHEAT_SHEET.md, "Variable Usage". + +This script makes that visible for one function: + +* it compiles the file with `/FA` and reads the `_name$ = N` table MSVC emits, which is + the ground truth for which local got which slot - no guessing from the diff; +* it counts the references to each local in the source, weighting a reference inside a + loop by 10 per nesting level as a proxy for block frequency; +* it reports whether the slots come out monotone in that weight. A violation means the + proxy is wrong for this function (an unequal size, a differing class, or a loop the + brace heuristic misreads), not that the model is wrong - read the report, do not + trust the suggestion blindly; +* with a reccmp run available it prints the original's slot census beside ours and names + the first position where the two disagree, which is the local whose reference count has + to change. + +`--reorder` rewrites the leading run of initialised same-type declarations into descending +weight order. That cannot move a local between weight classes, so it only decides **ties**, +which `StkColorCmp` leaves to the order the symbols arrive in, permuted by an unstable +quicksort (`QSortColors`). Treat it as one cheap variant to measure, not as a repair: +re-run `reccmp_report.py --run` afterwards and revert it if it does not pay. + +Renaming locals does nothing - `allowQuarries` -> `zzz` and `allowWoodcutters` -> +`aaaaaaaaaa` left `planToBuyWhenLowOnResourceAndSnoozeBuildings` byte-identical. The +comparator never looks at a name. +""" + +import json +import os +import re +import subprocess +import sys + +import common + +REORDER = "--reorder" in sys.argv +QUIET = "--quiet" in sys.argv +files = [a for a in sys.argv[1:] if not a.startswith("-")] +if not files: + sys.exit(__doc__) + +# absolute: the compile runs with the build directory as its cwd, so a relative /Fo would miss +OUT = (common.ROOT / common.TMP / "slotweight").resolve() +OUT.mkdir(parents=True, exist_ok=True) + +SLOT_RE = re.compile(r"^(\w+\$\w*|tv\d+) = (-?\d+)", re.M) +DECL_RE = re.compile(r"^(\s*)((?:unsigned |signed |const )*\w[\w:]*)\s+(\w+)\s*=\s*([^;]+);\s*$") +LOOP_RE = re.compile(r"^\s*(for|while|do)\b") + + +def compile_fa(path): + """Compile one file with /FA and return the listing text.""" + cc = json.loads(common.read_text(common.BUILD_DIR / "compile_commands.json")) + ours = [ + e + for e in cc + if "OpenSHC.dll.dir" in e["command"] and "/src/OpenSHC/" in e["file"].replace("\\", "/") + ] + if not ours: + sys.exit("no OpenSHC.dll compile command in compile_commands.json - configure the build first") + cmd = ours[0]["command"] + out = str(OUT).replace("\\", "/") + # lambda replacements: a Windows path in the replacement string would be read as escapes + cmd = re.sub(r"/Fo\S+", lambda _m: "/Fo%s/w.obj /FA /Fa%s/w.asm" % (out, out), cmd) + cmd = re.sub(r"-c \S+$", lambda _m: "-c " + str(path), cmd) + env = common.build_env() + # cl.exe loads its own DLLs from Common7/IDE, as quick_diff.py does for the same reason + env["PATH"] = str(common.ROOT / "MSVC1400-SP1" / "Common7" / "IDE") + os.pathsep + env["PATH"] + r = subprocess.run(cmd, cwd=ours[0]["directory"], env=env, capture_output=True, text=True) + if r.returncode: + out = (r.stdout or "") + (r.stderr or "") + print("compile failed (%d):" % r.returncode) + print("\n".join(l for l in out.splitlines() if "error" in l) or out[-800:] or "(no diagnostics)") + return None + return common.read_text(OUT / "w.asm") + + +def our_slots(listing, cls, name): + """{local: frame offset} for one function, from MSVC's own symbol table.""" + m = re.search(r"^(\?%s@%s@\S+) PROC" % (re.escape(name), re.escape(cls)), listing, re.M) + if not m: + return None + pre = listing[listing.rfind("_TEXT\tSEGMENT", 0, m.start()) : m.start()] + out = {} + for sym, off in SLOT_RE.findall(pre): + if sym.startswith("tv"): + continue + # MSVC prefixes a local with exactly one underscore; sources here often start with + # one of their own (`_textLength` -> `__textLength$`), so strip only the first. + out[re.sub(r"\$\d*$", "", re.sub(r"^_", "", sym))] = int(off) + return out + + +def weights(body, names): + """Reference count per local, a reference inside N *loops* counted 10**N times.""" + raw = dict.fromkeys(names, 0) + weighted = dict.fromkeys(names, 0) + braces, loop_stack = 0, [] + for line in body.split("\n"): + starts_loop = LOOP_RE.match(line) is not None + for n in names: + hits = len(re.findall(r"\b%s\b" % re.escape(n), line)) + if hits: + raw[n] += hits + weighted[n] += hits * 10 ** len(loop_stack) + braces += line.count("{") - line.count("}") + braces = max(braces, 0) + # a loop body is the brace level opened on (or just after) the loop header + if starts_loop: + loop_stack.append(braces) + while loop_stack and braces < loop_stack[-1]: + loop_stack.pop() + return raw, weighted + + +def signature_params(text, cls, name): + """Names of the incoming parameters, which are ABI-placed and never colored.""" + m = re.search( + r"FUNCTION: STRONGHOLDCRUSADER[^\n]*\n\s*[\w:*&<> ]*?%s::%s\s*\(([^{]*?)\)\s*\n?\s*\{" + % (re.escape(cls), re.escape(name)), + text, + re.S, + ) + if not m: + return set() + out = set() + for part in m.group(1).split(","): + ids = re.findall(r"\b(\w+)\b", part) + if ids: + out.add(ids[-1]) + return out + + +def decl_block(text): + """The longest run of consecutive initialised declarations in the body.""" + lines = text.split("\n") + best, cur = [], [] + for i, line in enumerate(lines): + if DECL_RE.match(line): + cur.append(i) + else: + if len(cur) > len(best): + best = cur + cur = [] + return best if len(best) > len(cur) else cur + + +def report(path): + text = common.read_text(path) + m = re.search(r"FUNCTION: STRONGHOLDCRUSADER[^\n]*\n[^\n(]*?(\w+)::(\w+)\(", text) + if not m: + print("%s: no `// FUNCTION:` marker followed by a definition" % path) + return + cls, name = m.groups() + listing = compile_fa(path) + if listing is None: + return + slots = our_slots(listing, cls, name) + if not slots: + print("%s: no code emitted for %s::%s (inlined, or the listing names it differently)" % (path, cls, name)) + return + + names = [mm.group(3) for mm in (DECL_RE.match(l) for l in text.split("\n")) if mm] + names += [n for n in slots if n not in names] + raw, wt = weights(text, names) + + params = signature_params(text, cls, name) + shared = {} + for n, o in slots.items(): + if n not in params: + shared.setdefault(o, []).append(n) + print("### %s::%s" % (cls, name)) + print("%-28s %6s %8s %6s %10s %8s" % ("local", "refs", "wrefs", "size", "wrefs/byte", "slot")) + slot_weight = {o: sum(wt.get(x, 0) for x in ns) for o, ns in shared.items()} + # StkColorCmp divides by the element's size, so recover each size from the gap to the + # next slot: a local wider than 4 bytes legitimately sorts below a lighter narrow one. + negative = sorted(o for o in shared if o <= 0) + size = {} + for i, o in enumerate(negative): + size[o] = (negative[i + 1] - o) if i + 1 < len(negative) else 4 + density = {o: (slot_weight[o] * 1000) // max(size.get(o, 4), 1) for o in negative} + ordered = sorted((o, n) for n, o in slots.items()) + prev, violations, reused = None, [], [] + seen = set() + for off, n in ordered: + if n in params: + # incoming parameters are placed by the calling convention, not by the colorer + print("%-28s %6d %8d %6s %10s %8d (parameter, not colored)" + % (n, raw.get(n, 0), wt.get(n, 0), "-", "-", off)) + continue + mark = "" + if off > 0: + reused.append(n) + mark = " <- positive offset: reuses a dead parameter's slot" + elif off not in seen and prev is not None and density.get(off, 0) > prev: + mark = " <- heavier than the slot below it" + violations.append(n) + if off <= 0 and off not in seen: + prev = density.get(off, 0) + seen.add(off) + print("%-28s %6d %8d %6s %10s %8d%s" + % (n, raw.get(n, 0), wt.get(n, 0), + size.get(off, "-"), density.get(off, "-"), off, mark)) + + multi = {o: ns for o, ns in shared.items() if len(ns) > 1} + if multi: + print("") + for o, ns in sorted(multi.items()): + print("slot %d is shared by %d locals (%s)" % (o, len(ns), ", ".join(sorted(ns)))) + print("Disjoint live ranges share one slot, so a slot count is not a local count.") + if any(o > 0 for o in shared): + print("") + print("A local at a positive offset reuses the slot of a parameter already dead. This is") + print("where Ghidra's \"repurposed parameter\" rendering comes from: the compiler did it,") + print("the original source did not - do not transcribe it back.") + if violations: + print("\nweight per byte is NOT monotone here (%s)." % ", ".join(violations)) + print("Expected when the locals are not one homogeneous class: `StkColorCmp` keys on a") + print("class byte first, and arrays or address-taken locals sort in their own group. It") + print("can also mean the 10x-per-loop proxy misreads this control flow. Use the numbers") + print("as a hint about which local is light or heavy, not as a layout prediction.") + elif not QUIET: + print("\nslots are monotone in weight per byte, as the colorer's comparator implies.") + + compare_original(name, slots, wt) + + if REORDER: + do_reorder(path, text, wt) + + +def compare_original(name, slots, wt): + """Put the original's slot census beside ours and name the first disagreement.""" + try: + byaddr = common.load_diff() + except Exception: + return + esp = re.compile(r"\[esp \+ (0x[0-9a-f]+)\]") + for _f, e, _a in common.entries(byaddr): + if e is None or not e.get("diff") or not e["name"].endswith("::" + name): + continue + orig, ours = common._asm_lines(e) + census = [] + for stream in (orig, ours): + c = {} + for line in stream: + for mm in esp.finditer(line): + v = int(mm.group(1), 16) + c[v] = c.get(v, 0) + 1 + census.append(c) + o, u = census + print("\n original %2d slots: %s" % (len(o), " ".join("%#x:%d" % kv for kv in sorted(o.items())))) + print(" ours %2d slots: %s" % (len(u), " ".join("%#x:%d" % kv for kv in sorted(u.items())))) + if len(o) != len(u): + print(" %+d colored stack elements: a local one side spills and the other enregisters," % (len(u) - len(o))) + print(" which is a live-range difference, not a declaration-order one.") + ol = [c for _, c in sorted(o.items())] + ul = [c for _, c in sorted(u.items())] + for i, (a, b) in enumerate(zip(ol, ul)): + if a != b: + print(" first differing position: slot #%d, original touched %d times, ours %d." % (i, a, b)) + print(" Under the weight model that position holds locals of different weight;") + print(" look for a read the original has and we do not (or the reverse).") + break + return + + +def do_reorder(path, text, wt): + lines = text.split("\n") + idx = decl_block(text) + if len(idx) < 2: + print("\n--reorder: no run of initialised declarations to permute.") + return + block = [lines[i] for i in idx] + decls = [DECL_RE.match(l) for l in block] + types = set(d.group(2) for d in decls) + if len(types) != 1: + print("\n--reorder: the run mixes types (%s); refusing, sizes differ and so does the key." % ", ".join(sorted(types))) + return + order = sorted(range(len(block)), key=lambda k: -wt.get(decls[k].group(3), 0)) + if order == list(range(len(block))): + print("\n--reorder: declarations are already in descending weight order.") + return + for pos, src in zip(idx, order): + lines[pos] = block[src] + common.write_text(path, "\n".join(lines)) + common.format_file(path) + print("\n--reorder: rewrote %d declarations into descending weight order." % len(idx)) + print("This only decides ties. Re-measure with reccmp_report.py --run and revert if it does not pay.") + + +for f in files: + report(os.path.abspath(f)) diff --git a/tools/reimplementation-helper/batch/structural_diff.py b/tools/reimplementation-helper/batch/structural_diff.py new file mode 100644 index 0000000000..a216a2d6e5 --- /dev/null +++ b/tools/reimplementation-helper/batch/structural_diff.py @@ -0,0 +1,468 @@ +"""Score a function by *structure* rather than by match percentage. + +usage: structural_diff.py [NAME-SUBSTRING] one function, full report + structural_diff.py --all every function in the build list, ranked + structural_diff.py NAME --score one number, for hill-climbing + +The reccmp percentage is dominated by register and stack-slot naming, which the source cannot +state directly. On `aiRecruitUnits` the 50% gap decomposes as roughly 7% instruction kind and +order, 4% operand shape, and 39% pure register naming. Optimising the percentage therefore +steers by the one term that is not reachable, and it can select *against* structural +correctness: `recruitCount++` gets the original's `neg`/`sbb`/`jmp` counts right and scores +47.3%, while `recruitCount = interval != 0 ? 2 : 3` gets those counts wrong and scores 50.0%. + +Register allocation is deterministic given the compiler's IR, so the way to reach 100% is to +make the instruction stream identical in kind and order and let the allocation follow - which +is what happened to the sixteen `setAICParameters_NN` functions, where fixing statement order +fixed the register assignment by itself. This script scores the part you can act on. + +Three tiers are reported, each ignoring more than the last: + +* `kind` - bare mnemonic sequence, every operand stripped. Differences here are real: a + different instruction, a missing one, a different order. This is the tier to drive + to 1.000 first. +* `shape` - operands kept but registers anonymised to `r` and immediates to `k`. Differences + here are operand form: a memory access against an immediate, a different + addressing mode, a constant where the original has a register. +* `raw` - what reccmp reports, for reference. + +`delta` sums the per-mnemonic count mismatches, which is the most directly actionable figure: +it says "we emit two `xor` the original does not" rather than giving a ratio. Chase it to zero. + +Both instruction streams are complete, and come from the code itself via `asm_streams.py` - the +original disassembled from `_original/`, ours from a `/FA` compile. They deliberately do NOT +come from `reccmp/dll/diff.json`, which holds only the diff view and therefore keeps *less* of a +function the better it matches: `aiCreateSiegeUnits` at 99.7% leaves 22 lines per side behind, +against a real body of 322 instructions. An earlier version of this script read those lines, and +its figures for well-matching functions were meaningless. Only `raw` still comes from reccmp. + +Each function is compiled on demand, so `--all` takes about a second per file and needs a +configured build directory - but no build and no reccmp run, except for the `raw` column. +""" + +import collections +import difflib +import os +import re +import sys + +import asm_streams +import common + +# parsed in main(), not at import: the other tools import this module for `measure()` and +# `duplication()`, and a module-level sys.exit turns that into an abrupt exit with a usage dump +SCORE_ONLY = False +ALL = False + +REG = re.compile(r'\b(e?(?:ax|bx|cx|dx|si|di|bp|sp)|[a-d][lh])\b') +IMM = re.compile(r'\b(?:0x[0-9a-f]+|\d+)\b') +NAMED = re.compile(r'\b(e(?:ax|bx|cx|dx|si|di|bp))\b') + + +def shape(line): + """Operand form with register names and constants anonymised.""" + return IMM.sub('k', REG.sub('r', line)) + + +def skeleton(line): + """The instruction with only its register NAMES anonymised.""" + return NAMED.sub('r', line) + + +def register_divergence(orig, ours): + """Where each side first diverges, and how the register names map. + + A function can be structurally identical and still score badly purely because the + registers are named differently, and the two cases need opposite responses - so it + matters which one you are looking at. The discriminator is WHERE each kind of difference + starts. Register allocation is deterministic given the IR, so when the first register + difference sits at or after the first structural difference, the registers are an *effect* + of that structural defect and will follow once it is fixed; attacking them directly is + wasted work. There is also no cost-side lever to attack them with: in `ChoosePhysReg` the + per-register cost array came out all zeros for every pick in `renderDebugDataAiInfo`, so + the choice was simply the first free register in class order. + + That function is the worked case: 293 of its 305 instructions align once register names + are stripped, the first 149 have identical registers, and the rotation starts inside the + one block whose instruction stream still differs. + """ + so = [skeleton(l) for l in orig] + su = [skeleton(l) for l in ours] + # the first STRUCTURAL difference is measured on mnemonics alone. Measuring it on whole + # operands instead puts it at instruction 1 of almost every function, because our side + # names `___security_cookie` where the original has a bare address - a symbolisation + # difference, not a structural one, and it would make the comparison below meaningless. + mo = [l.split()[0] for l in orig if l.split()] + mu = [l.split()[0] for l in ours if l.split()] + ops = difflib.SequenceMatcher(None, mo, mu, autojunk=False).get_opcodes() + first_struct = next((i1 for tag, i1, _i2, _j1, _j2 in ops if tag != 'equal'), None) + sm = difflib.SequenceMatcher(None, so, su, autojunk=False) + first_reg = None + aligned = 0 + subs = collections.Counter() + for tag, i1, i2, j1, j2 in sm.get_opcodes(): + if tag != 'equal': + continue + for k in range(i2 - i1): + aligned += 1 + ra = NAMED.findall(orig[i1 + k]) + rb = NAMED.findall(ours[j1 + k]) + if ra != rb: + if first_reg is None: + first_reg = i1 + k + for a, b in zip(ra, rb): + if a != b: + subs[(a, b)] += 1 + return {'skel': sm.ratio(), 'aligned': aligned, 'first_struct': first_struct, + 'first_reg': first_reg, 'subs': subs} + + +def ratio(a, b): + return difflib.SequenceMatcher(None, a, b, autojunk=False).ratio() + + +def measure(path, name, raw): + orig, ours = asm_streams.streams(path) + if ours is None: + return None + mo, mu = asm_streams.mnemonics(orig), asm_streams.mnemonics(ours) + co, cu = collections.Counter(mo), collections.Counter(mu) + deltas = [(k, co.get(k, 0), cu.get(k, 0)) for k in sorted(set(co) | set(cu)) + if co.get(k, 0) != cu.get(k, 0)] + return { + 'name': name, + 'raw': raw, + 'div': register_divergence(orig, ours), + 'dup': duplication(orig, ours), + 'flips': polarity(orig, ours), + 'kind': ratio(mo, mu), + 'shape': ratio([shape(l) for l in orig], [shape(l) for l in ours]), + 'n_orig': len(mo), + 'n_ours': len(mu), + 'delta': sum(abs(a - b) for _k, a, b in deltas), + 'deltas': sorted(deltas, key=lambda t: -abs(t[1] - t[2])), + } + + +NEGATION = {'je': 'jne', 'jne': 'je', 'jl': 'jge', 'jge': 'jl', 'jg': 'jle', 'jle': 'jg', + 'jb': 'jae', 'jae': 'jb', 'ja': 'jbe', 'jbe': 'ja', 'js': 'jns', 'jns': 'js', + 'jp': 'jnp', 'jnp': 'jp'} + + +def polarity(orig, ours): + """Branches where one side jumps on a condition and the other on its negation. + + `diff_jcc.py` covers strictness steps (`jl` against `jle`) and swapped compare operands, + but not a plain polarity flip, which is a different source defect: the `if`/`else` arms are + the other way round, so the original's fallthrough is our jump target. Worth 56.8% -> 60.3% + on `aiGiveRaidInstructions`, whose final navigation test read `== FALSE` where the original + had `!= FALSE`. + + Only pairs inside a differing region are considered, positionally - a flip shows up there + because the inverted branch makes the surrounding block fail to align. + """ + so = [skeleton(l) for l in orig] + su = [skeleton(l) for l in ours] + out = [] + for tag, i1, i2, j1, j2 in difflib.SequenceMatcher(None, so, su, + autojunk=False).get_opcodes(): + if tag != 'replace': + continue + ob = [(i1 + k, orig[i1 + k].split()[0]) for k in range(i2 - i1) + if orig[i1 + k].split() and orig[i1 + k].split()[0] in NEGATION] + ub = [(j1 + k, ours[j1 + k].split()[0]) for k in range(j2 - j1) + if ours[j1 + k].split() and ours[j1 + k].split()[0] in NEGATION] + for (oi, om), (_uj, um) in zip(ob, ub): + if NEGATION[om] == um: + out.append((oi, om, um, ' / '.join(orig[max(0, oi - 2):oi]))) + return out + + +def report_polarity(flips): + """Name an inverted branch and how to fix it.""" + if not flips: + return + print('') + print(' POLARITY: %d branch(es) jump on the opposite condition:' % len(flips)) + for oi, om, um, ctx in flips[:4]: + print(' original instruction %-4d original %-4s ours %-4s' % (oi, om, um)) + if ctx: + print(' preceded by: %s' % ctx[:70]) + print('') + print(' The original falls through where we jump, so its `if`/`else` arms are the other') + print(' way round. Swap them and invert the test - semantically identical, and it is what') + print(' took aiGiveRaidInstructions from 56.8% to 60.3%. Note diff_jcc.py does NOT report') + print(' this: it covers strictness steps and operand order, not a plain polarity flip.') + + +def _count_run(lines, run): + """How many non-overlapping times `run` occurs in `lines`.""" + n, i, total = len(run), 0, 0 + while i + n <= len(lines): + if lines[i:i + n] == run: + total += 1 + i += n + else: + i += 1 + return total + + +def duplication(orig, ours): + """Computations one side repeats and the other shares. + + This is the signal that is easy to misread. When MSVC hoists an expression out of both + arms of an `if`/`else` and the original did not, the mix delta shows up as one missing + copy of each mnemonic in the run - on `renderDebugDataAiInfo` it read `add -1, mov -1, + push -1`, which looks like three unrelated missing instructions and is really one missing + duplicate of `mov reg, [reg + G]; add reg, 1; push reg`. + + Comparing the number of occurrences, with registers anonymised so the two copies match + despite being allocated differently, names it directly. The fix direction follows from + which side repeats: MSVC hoists the *maximal* common expression, so if the original + repeats a run that we share, writing the whole expression out in both arms will not + reproduce it - the source has to share exactly the prefix the original shares, typically + an index in a local, and write the rest out. + """ + so = [skeleton(l) for l in orig] + su = [skeleton(l) for l in ours] + sm = difflib.SequenceMatcher(None, so, su, autojunk=False) + found = [] + for tag, i1, i2, j1, j2 in sm.get_opcodes(): + if tag == 'equal': + continue + # whichever side has the surplus instructions in this region is the side that might + # be repeating something; a `replace` can hold a whole extra copy too + if tag == 'delete' or (tag == 'replace' and (i2 - i1) >= (j2 - j1)): + extra, side, mine, other = so[i1:i2], 'original', so, su + else: + extra, side, mine, other = su[j1:j2], 'ours', su, so + # Search the region's sub-runs, longest first: the repeated computation is often only + # part of a larger differing region. Require the run to occur at least TWICE on its + # own side - otherwise a single instruction that merely MOVED looks like a + # duplication, which is what a first version of this reported (`cmp`/`jne` at 1 vs 0). + best = None + for length in range(len(extra), 1, -1): + for start in range(0, len(extra) - length + 1): + run = extra[start:start + length] + c_mine = _count_run(mine, run) + if c_mine < 2: + continue + c_other = _count_run(other, run) + if c_mine > c_other: + best = (side, run, c_mine, c_other) + break + if best: + break + if best: + found.append(best) + # the longest run is the most informative, and duplicate reports of the same run add + # nothing + seen, out = set(), [] + for side, run, mine_c, other_c in sorted(found, key=lambda f: -len(f[1])): + key = (side, tuple(run)) + if key in seen: + continue + seen.add(key) + out.append((side, run, mine_c, other_c)) + return out + + +def report_duplication(found): + """Name a hoisted-or-not-hoisted duplicate and which way to fix it.""" + for side, run, mine, theirs in found[:3]: + print('') + if side == 'original': + print(' DUPLICATION: the original computes this run %d time(s), we compute it %d:' + % (mine, theirs)) + else: + print(' DUPLICATION: we compute this run %d time(s), the original computes it %d:' + % (mine, theirs)) + for line in run[:8]: + print(' %s' % line) + if len(run) > 8: + print(' ... %d more' % (len(run) - 8)) + print('') + if side == 'original': + print(' So MSVC hoisted out of a branch what the original left duplicated. It hoists') + print(' the MAXIMAL common expression, so writing the whole expression out in both') + print(' arms will NOT reproduce this - that only makes the whole chain common. Share') + print(' in the source exactly the prefix the original shares (an index or a base in a') + print(' local) and write the rest out in each arm. Sharing more than it did costs an') + print(' instruction too: a pointer to the element pays an `add` where the original') + print(' keeps index*size and re-folds the base into each load.') + else: + print(' So we recompute in each arm what the original computes once. Introduce a') + print(' local for it, placed where the original shares it.') + print(' Worked example: tools/compiler-internals/msvc-backend.md, the AIV block of') + print(' renderDebugDataAiInfo - this exact signal was 67.5% -> 83.6%.') + + +def report_registers(div): + """Say whether the register naming is a cause or an effect.""" + if not div['subs']: + return + fs, fr = div['first_struct'], div['first_reg'] + print('') + total = sum(div['subs'].values()) + print(' registers: %d operand(s) differ only in which register was used.' % total) + print(' %-14s %s' % ('substitution', 'count')) + for (a, b), c in div['subs'].most_common(6): + print(' %-4s -> %-6s %5d' % (a, b, c)) + # A permutation that closes into a cycle means the allocation is structurally right and + # the scratch registers were simply handed out in a different order. Take each source + # register's most common target rather than any target, and report what share of the + # substitutions that permutation actually covers - the minority ones are real and + # claiming a clean cycle over them would overstate the result. + best = {} + for (a, b), c in div['subs'].most_common(): + if a not in best: + best[a] = (b, c) + mapping = dict((a, b) for a, (b, _c) in best.items()) + covered = sum(c for _a, (_b, c) in best.items()) + closed = (len(set(mapping.values())) == len(mapping) + and all(b in mapping for b in mapping.values()) + and len(mapping) > 2) + if closed: + # decompose into disjoint cycles and walk each until it returns to its start. Running + # one walk for len(mapping) steps instead printed nonsense whenever the permutation + # had more than one cycle - `eax -> ecx -> edx -> eax -> ecx -> eax` for what is + # really ebp <-> ebx alongside eax -> ecx -> edx -> eax. + cycles, seen_r = [], set() + for start in sorted(mapping): + if start in seen_r: + continue + cycle, a = [], start + while a not in seen_r: + seen_r.add(a) + cycle.append(a) + a = mapping[a] + cycles.append(cycle) + print('') + shown = ', '.join(' -> '.join(c + [c[0]]) if len(c) > 1 else '%s (unchanged)' % c[0] + for c in cycles) + print(' The dominant substitution is a permutation in %d closed cycle(s), covering' + % len(cycles)) + print(' %d of %d (%.0f%%): %s' % (covered, total, 100.0 * covered / total, shown)) + print(' Closed cycles mean the allocation is structurally right and the registers') + print(' were merely handed out in a different order.') + if fs is not None and fr is not None and fr >= fs: + print('') + print(' The first STRUCTURAL difference is at original instruction %d and the first' % fs) + print(' REGISTER difference at %d, so the registers diverge at or after the structure' + % fr) + print(' does. Allocation is deterministic given the IR, so these register names are') + print(' an effect of that structural defect - fix it and they follow. Where this was') + print(' measured (renderDebugDataAiInfo) there was also no cost-side lever to pull:') + print(' ChoosePhysReg picked on an all-zero cost array, so the first free register in') + print(' class order won. Check it per function with reg_scores.py --whyreg.') + elif fr is not None and (fs is None or fr < fs): + print('') + print(' The registers diverge at instruction %d, BEFORE any structural difference' % fr) + print(' (%s), so this is not downstream of a mix defect. The levers are the order' + % ('none at all' if fs is None else 'at %d' % fs)) + print(' values are created in and what is live across each point - statement order -') + print(' not how anything is spelled or typed. See tools/compiler-internals.') + + +def report(m): + print('### %s' % m['name']) + print(' instructions original %d ours %d (%+d)' + % (m['n_orig'], m['n_ours'], m['n_ours'] - m['n_orig'])) + print(' kind %.3f <- drive this to 1.000 first; every difference here is real' + % m['kind']) + print(' shape %.3f <- operand form, once registers and constants are anonymised' + % m['shape']) + print(' skel %.3f <- the same instructions with only REGISTER NAMES anonymised' + % m['div']['skel']) + if m['raw'] is not None: + print(' raw %.3f <- reccmp, mostly register and slot naming' % m['raw']) + print(' delta %d instruction(s) of mismatched kind' % m['delta']) + report_polarity(m['flips']) + report_duplication(m['dup']) + report_registers(m['div']) + if m['deltas']: + print('') + print(' %-8s %-8s %-8s' % ('mnemonic', 'original', 'ours')) + for k, a, b in m['deltas']: + print(' %-8s %-8d %-8d %+d' % (k, a, b, b - a)) + print('') + print(' Each row is a hypothesis: an extra `xor` is a value materialised differently,') + print(' a missing `lea` is an address the original folds, `jmp` with `cmp` one apart is') + print(' a branch shaped differently. Fix these and let the allocation follow.') + elif m['kind'] < 0.99: + print('') + print(' No mnemonic mismatch, but the sequences differ: the two sides emit the same') + print(' instructions in a different ORDER. That is statement order, and it is reachable') + print(' - it is what took the sixteen setAICParameters_NN functions to 100%. Use the') + print(' original stream (orig_body.py FILE) as the skeleton and reorder the source.') + else: + print('') + print(' Same instructions, same order. What is left is operand form and register naming,') + print(' so there is no structural work to do here - see tools/compiler-internals.') + + +def targets(): + """[(path, name, raw)] for the build list; `raw` is None when reccmp has no entry.""" + byaddr = common.load_diff() + for f, e, _a in common.entries(byaddr): + # the build list is git-ignored, so it routinely names files that are not on the + # current branch; both streams need the source, so such an entry has to be skipped + if not os.path.exists(f): + continue + name = e['name'] if e else common.repo_relative(f) + yield f, name, (float(e['matching']) if e else None) + + +def main(): + global SCORE_ONLY, ALL + SCORE_ONLY = '--score' in sys.argv + ALL = '--all' in sys.argv + want = next((a for a in sys.argv[1:] if not a.startswith('--')), None) + if not ALL and want is None: + sys.exit(__doc__) + if ALL: + rows = [] + for f, name, raw in targets(): + if raw is not None and raw >= 0.9999: + continue + try: + m = measure(f, name, raw) + except Exception as exc: + print('# %-44s skipped: %s: %s' % (name, type(exc).__name__, exc)) + continue + if m: + rows.append(m) + if not rows: + sys.exit('every function in the build list matches; nothing to rank') + rows.sort(key=lambda r: (-r['delta'], r['kind'])) + print('%-50s %6s %6s %6s %5s %s' % ('function', 'kind', 'shape', 'raw', 'delta', 'verdict')) + for r in rows: + verdict = 'mix' if r['delta'] > 0 else ('order' if r['kind'] < 0.99 else 'allocation') + print('%-50s %6.3f %6.3f %6s %5d %s' + % (r['name'].split('::')[-1], r['kind'], r['shape'], + '-' if r['raw'] is None else '%.3f' % r['raw'], r['delta'], verdict)) + print('') + print('Three categories, in descending order of how reachable they are from the source:') + print(' delta > 0 wrong instruction mix. Each mismatched mnemonic is a') + print(' concrete defect; this is where source work pays.') + print(' delta 0, kind < 0.99 right mix, wrong order - statement order, reachable.') + print(' delta 0, kind ~ 1.000 same instructions in the same order. The gap is operand') + print(' form and register naming; leave these alone.') + return + + hits = [(f, n, r) for f, n, r in targets() + if want.lower() in n.lower() or want.lower() in f.lower()] + if not hits: + sys.exit('no function matching %r in the build list' % want) + for f, n, r in hits: + m = measure(f, n, r) + if m is None: + continue + if SCORE_ONLY: + print('%.4f %d' % (m['kind'], m['delta'])) + else: + report(m) + + +if __name__ == '__main__': + main()