Repository navigation
Conversation
MSVC 14.00's stack colorer sorts its elements with StkColorCmp on (class byte, weight * 1000 / size) and hands out offsets in that order, so for a homogeneous block of scalars the layout is descending reference weight per byte. Decoded from the bundled c2.dll (14.00.50727.762), whose own assert names P2/stack.c:1372, with symbols recovered from the public c2.pdb. Confirmed on planToBuyWhenLowOnResourceAndSnoozeBuildings: eleven colored int locals come out exactly in weight order with declaration positions scrambled, adding one reference reshuffles the whole table and it is monotone again, and renaming two locals to zzz and aaaaaaaaaa leaves the function byte-identical. Also documents two behaviours that follow from the same pass: locals with disjoint live ranges share a slot (so a slot count is not a local count), and a local can be placed on a dead parameter's slot - which is where Ghidra's repurposed-parameter rendering comes from. Adds slot_weight.py, which reads MSVC's own _name$ = N table out of a /FA listing so the local-to-slot mapping is observed rather than inferred, and corrects the claim in the batch README and perm_decls.py that declaration order assigns slots: it only decides ties. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
The set iterator at c2.dll 0x10b04fa0 takes the lowest set bit of a 64-bit register set and clears it, and .rdata gives the numbering through three parallel 4-byte-stride name tables ending in "noreg": edi=0, esi=1, ebp=2, esp=3, ebx=4, edx=5, ecx=6, eax=7 - the reverse of the x86 encoding order, so callee-saved registers are preferred in that order. Spillcosts carries no cost arithmetic; it decides who gets spilled, not which register a winner gets. So a register mismatch is an interference difference, not a preference to match: at that allocation point a different set was free. Documented as a mechanism, not as a tested source-level recipe - the live-range computation behind those sets is not decoded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
…ervation Tracing one function's allocation under cdb shows the iterator at 0x10b04fa0 is a generic bitset iterator, not a register allocator: during sellExcessGoods it returned members numbered 0..40 across 140 distinct set objects, and ColorStackElements feeds its result into a symbol array (DAT_10d0faa4 + n*4). So "lowest-numbered free register" was never observed - it was inferred from the .rdata name table plus the ascending-order mechanic, and only those two stand. Keeps the numbering and the set mechanics, marks interference as a hypothesis, and records the debugger recipe that works: breakpoints from the PDB's public addresses (c2.dll loads unrebased at 0x10b00000), left disabled and enabled only for the invocation of interest, with the target invocation identified by compiling twice and emptying the function body. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
The MSVC backend material is about the compiler, not about reimplementing a function, so it does not belong in IMPLEMENTATION_CHEAT_SHEET.md. The cheat sheet keeps the part that changes how a diff is read - weight not declaration order, ties, shared slots, dead-parameter reuse, renaming inert - and points at the new document for the mechanism. tools/compiler-internals/msvc-backend.md collects the evidence and the things worth not rediscovering: where c2.pdb comes from and how to use it without a PDB-aware tool, the 66 leaked backend source paths, the symbol addresses, the decoded StkColorCmp rule with its planToBuy evidence and its known unknown (the class byte), what is and is not established about registers, and the cdb harness including the four mistakes that cost time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
…servable Tracing invocation #613 under cdb: Color runs once per function and is the register colourer; among its 29 direct callees @ChoosePhysReg@12 (0x10b14b70) is the one that picks a physical register. Its edx argument is the mask of still-available registers, so the mask progression across calls gives the consumption order directly - for sellExcessGoods esi, ebx, edi, ebp, matching our prologue, against the original's ebp, edi, ebx, esi. ChoosePhysReg does use bsf over a mask, but narrows it first with the set/clear tables, so lowest-free-register holds only within the filtered candidate set - which is why the observed order is not ascending. What does the narrowing is not decoded; OrderGlobregs is the next thing to read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
…nList) OrderGlobregs builds its priority list by insertion, not by sorting, and InsertInList is 128 bytes of readable sorted insertion: descending by a signed benefit score at +0x0c, tie-broken by an unsigned symbol id at +0x44, newest first among equals. Traced over sellExcessGoods the scores run -160..+104 across 40+ insertions, with candidates re-scored between colourer rounds. So the chain is: score and order the candidates, walk that order, serve each from the available mask. The order being a use-derived benefit score means the source lever on register assignment is the same as for stack slots - how often and where a value is read. Records two dead ends: 0x10ce5a08 is not a per-register preference table, and the candidate symbol record does not yield source variable names at offset 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
A candidate's +0x00 is its symbol record, the symbol record's +0x04 points one byte before the variable's name, and the names live in a string pool - so name = poi(poi(poi(candidate)+4))+1. reg_scores.py uses that to dump the scores OrderGlobregs computes, with the candidates resolved to source names. For sellExcessGoods: amount +104, aicIndex +88, j +88, aiType +42, variance -4, resourceOffset -26, resourceType -34, playerResourceBase -52, this and playerID -134, with a second round moving this/playerID to +32 and playerResourceBase to +97. playerID scoring -134 initially is why our build serves it late and leaves it in a volatile register where the original keeps it in ebp. --probe identifies the invocation by candidate count (42 for sellExcessGoods against the template flood at 33), which replaces the compile-twice differential with a single run and no source edits. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
…ount --probe ranks invocations by candidate count, which only singles out our function because the resolver templates all score identically; a function with fewer candidates than the flood cannot be found that way. --find prints every named candidate with its invocation index, so the function can be identified by one of its locals. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
The usage guard rejected --find because it only accepted --probe or --invocation, so the mode printed help instead of running. Scored the four AICState functions that had resisted every source-level hypothesis. They turn out to have three different diagnoses where all four previously looked like one undifferentiated 'allocator noise': recruitHarrassingSiegeEngines is decided by near-ties (four candidates at +36) so no source change can state a preference; aiRecruitUnits carries a loop counter at -279 that the original does not have as a candidate at all, because it reuses the dead playerID parameter; and planToBuy's locals only turn positive in a later colourer round, so its layout is decided by the spill rounds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
… not pay --advise prints the per-candidate score table and names the patterns: near-ties (order falls to the symbol-id tiebreak, so no source change states a preference), strongly disfavoured candidates, and iteration-dependent sign flips. Then tested those suggestions against reccmp on aiRecruitUnits, which shows the strongest structural signal of any function tried. Both lost ground: dropping the -279 loop counter by counting recruitCount down gave 48.7%, and declaring the -335 flag bool as Ghidra renders it gave 49.5%, against a 50.0% baseline. Removing a disfavoured candidate changes the scoring landscape, so 'delete the worst one' is not monotone. Documented accordingly: the scores are triage, not prescription, and the reliably useful output is the negative verdict that a near-tied function cannot be steered from the source and should be abandoned. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
…fixer exists Correcting these notes: +0x0c is a rank, not the benefit. InsertByBenefit orders on +0x40 and copies +0x0c from the list neighbour, and BetterToDefer sets +0x0c to min(components)-1 while gating on +0x40 <= 0. Tracing all three fields for sellExcessGoods confirms it: +0x40 is a small positive weighted use count (amount 70, variance 22, resourceType/j/resourceOffset 20, aicIndex 8, this/aiType 6, playerID 4), so register candidacy runs on the same quantity as stack slot order. That also answers the request for a fixer. The benefit counts uses weighted by block frequency, so it is a property of what the code does, not how it is written - no reordering, renaming or redeclaring can move it, which is why every syntactic hill-climb failed. The numbers instead decide which of two cases a function is in: use counts differing from the original's (restructure the computation) or already matching (tiebreak only, abandon it). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
… register order Registers are consumed in the order they are first needed and MSVC saves a callee-saved register where it first needs it, so the push order is the consumption order - and it comes free from the last reccmp run, with no second trace. --rankdiff pairs our measured benefit ranking against both sides' orders. For sellExcessGoods it establishes the useful half: both sides use the same four registers, so the candidate set matches and neither side carries a value the other does not, which rules out the aiRecruitUnits case. It refuses to over-read the rest - a permutation is either a ranking difference or a difference in how ChoosePhysReg narrows the mask, which is undecoded, so the pairing is labelled a hypothesis. It also detects the exact-reverse case as systematic. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
The pick is not a bit-scan over a narrowed mask. ChoosePhysReg (P2/color.c) builds a per-register cost array at 0x10d14f10: each interfering candidate's preference pairs add to it, a neighbour left with one legal register adds benefit*100 for that register, this candidate's own affinities subtract, and the cheapest legal register from its class list wins with ties going to the earlier list entry. 0x10ce5a08 is the table of per-class candidate lists - class 0 reads eax, ecx, edx, esi, edi, ebx, ebp - which is what it was holding pointers for; the earlier note calling it a dead end is corrected. Also corrects the register numbering: the .rdata name table is indexed backwards from its end, so it is eax=1 .. edi=8 in x86 encoding order, not edi=0 .. eax=7. --whyreg dumps the cost array at each pick. For sellExcessGoods every number is accounted for: ecx costs 2 for everyone because this has a thiscall affinity to it, playerID sees esi at 1200 = playerResourceBase's benefit 12 * 100, and the zeros are ties resolved by class-list order. The counterfactual is now arithmetic, but affinities come from copies between a value and a fixed register - argument slots, return values, this in ecx - and for callee-saved registers those copies are inserted by the allocator rather than stated by the source, so a source-level fixer remains out of reach. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
…harged it Three instructions accumulate the cost array: 0x10b14c2c adds a neighbour's preference count, 0x10b14caa adds benefit*100 when a neighbour has one legal register left (ebx holds it), 0x10b14d59 subtracts this candidate's own affinity. Breaking on them and reading the neighbour's name turns 'ebx costs 1100' into 'ebx is pinned by <name>'. On aiRecruitUnits it gives the first actionable statement of this whole thread: ebp costs 4700 for everyone because behaviourType pins it with 1400 and i with 1000 - so those two do not merely lose their own pick, they tax every other candidate. It also explains why counting recruitCount down instead of i lost ground: the pin just moves to recruitCount. On sellExcessGoods it corrects a guess in these notes: esi's 1200 is charged by a compiler temporary, not by playerResourceBase, whose benefit of 12 times 100 matched by coincidence. Only a named source variable is actionable, and attribution is what tells the two apart. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
Tried every form of the edit --whycost indicates for aiRecruitUnits, where behaviourType (+1400) and i (+1000) pin ebp. All five lose against the 50.0% baseline: bool flag 49.5%, drop i by counting recruitCount down 48.7%, narrow behaviourType 48.9%, narrow behaviourType and unitType 36.6%. Re-running --whycost on the narrowed variant explains it: behaviourType's pin rose from 1400 to 1600 and ebp's total from 4700 to 4900. Benefit is uses per unit of live range, so shortening a range with the same uses raises the benefit, and the pin is benefit*100. Narrowing raises the pin, deleting moves it to the replacement, widening adds interference - so the pin is not reducible from the source, only fewer uses would do it, and the uses are what the function must do. Adds reading-reg-scores.md, a from-scratch explanation of what the tool prints for someone without compiler background, including that trap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
The reccmp percentage is dominated by register and stack-slot naming, which no source change states directly, so optimising it steers by the one term that is not reachable - and it can select against structural correctness. On aiRecruitUnits, recruitCount++ gets the original's neg/sbb/jmp counts right and scores 47.3%, while the ternary gets them wrong and scores 50.0%. structural_diff.py reports three tiers - kind (bare mnemonics), shape (operands with registers and constants anonymised), raw - plus delta, the per-mnemonic count mismatch, which names defects instead of giving a ratio. --all splits the build list into three verdicts, in descending order of how reachable they are: mix (wrong instruction mix), order (right mix, wrong order, which is the setAICParameters case) and allocation (same instructions in the same order; leave alone). On AI::AICState that puts aiGiveRaidInstructions at delta 21 - never looked at before - and marks sellExcessGoods, where most effort went this session, as structurally finished at 62.4%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
…artefacts Three sources of noise, each of which reported an artefact as a source defect: common._asm_lines also yields bare tokens like 0x4 (a normalised operand) and 0xcc (inter-function padding). Counting them inflated every delta - 0xcc showed up as '4 instructions the original has and we do not' on addUnitToSmallestBehaviourTypeTribe, and 0x4/0x2/0x3 only became visible when a variant shifted their counts. Only lines whose mnemonic starts with a letter are counted now. Epilogue instructions are excluded and a function whose original stream holds no ret is flagged truncated, because reccmp's window frequently cuts the original before its epilogue: aiGiveRaidInstructions showed pop 0/4 and ret 0/1 purely from that, 5 of its reported 21. With the noise gone the AICState ranking is aiGiveRaidInstructions 16 (truncated), renderDebugDataAiInfo 13, addUnitToSmallestBehaviourTypeTribe 8, aiRecruitUnits 8, and 7 of 17 comparisons are truncated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
structural_diff was built on `common._asm_lines`, which reads `reccmp/dll/diff.json`.
That file stores the diff VIEW - changed blocks plus a little context - not whole
functions, so the better a function matches the less of it is there:
aiCreateSiegeUnits raw 0.997 diff.json 22 lines/side real body 322 instructions
sellExcessGoods raw 0.624 diff.json 133 lines/side
Counts taken from it are therefore wrong in proportion to how good the code is,
which is the worst possible direction for a tool meant to rank functions by how
much structural work they need. The `--all` table the previous commits produced was
meaningless for every well-matching function - aiCreateSiegeUnits was reported with
a delta of 285 against a true delta of 2.
The original side is clipped as well: reccmp reads our function's byte length from
our PDB and dumps that many bytes from the original's address, so when our code is
shorter the original loses its tail. aiGiveRaidInstructions is cut 42 bytes early and
appears to end on `push edx; push eax` with no `ret` anywhere, when it plainly ends
`004d2dfc ret 4`.
So both streams now come from the code itself, via the new asm_streams.py:
* original - disassembled from `_original/` with capstone, the extent found by
walking forward tracking the furthest forward branch target and stopping at the
first `ret`/`jmp` at or past it. Nothing branches over the end of a function.
Checked against Ghidra on 004d1320, 004d2000 and 004d2a70 - exact on all three.
* ours - compiled with /FA and read between PROC and ENDP.
This is what quick_diff.py already did, so asm_streams wraps it rather than
duplicating it, and orig_body.py exposes the original side on its own for reading a
function's true tail.
Two things this removes:
* the EPILOGUE exclusion. `pop`/`ret`/`leave` were skipped because the clipped stream
kept losing them; with complete streams they are ordinary instructions and a
genuine epilogue difference is now visible instead of silently dropped.
* the non-instruction filter for `0x4`/`0xcc` tokens, which were an artefact of the
diff view's formatting and do not occur in a real disassembly.
The three tiers land where they should now: aiCreateSiegeUnits reports kind 0.997
against raw 0.997 with a delta of 2, where before it claimed 285.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
Three findings from AICState::renderDebugDataAiInfo (0x004d1320): `sub esp, N` states how much stack the original's locals occupy, with no allocation noise in it, so it is worth checking before any reordering work. The original opens `sub esp, 0x400` against our `0x3f8` - two dwords of locals we did not have - and the instruction mix identified both: an array index the original caches (one `imul` 0x2a4 against our three) and the resulting element address it also caches. Adding them hit 0x400 exactly and took the mix delta from 13 to 3. Declaration order is inert where the slot weights are distinct, as the decoded StkColorCmp rule predicts: two placements and both orderings of the new locals gave byte-identical output, 67.5/71.0 every time. Worth trying only on an equal-weight group. And a measured case of the percentage moving against structural similarity across three source forms - 68.4 at mix delta 13, 66.2 at delta 5, 67.5 at delta 3 - because adding a local shifts every [esp + N] displacement and reccmp scores each as a mismatch until the layout lines up. So the percentage is not a safe hill-climbing objective while the frame size is still wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
The frame is the part of the output the source controls most directly - which locals exist, how big they are, how often each is referenced - and it is exactly where the match percentage misleads, because adding a local shifts every [esp + N] displacement and reccmp scores each as a mismatch until the layout lines up. Ours comes from MSVC's /FA slot table, which states each slot's name, offset and size. The original's has to be inferred, and two things make that harder than it looks: * esp depth cannot be tracked by walking the listing top to bottom. Both arms of an if/else push their own arguments before a shared call, so a straight-line walk counts both and drifts: on renderDebugDataAiInfo it was +480 bytes out by the epilogue and invented 27 slots, some below the frame. Depth is propagated over the control-flow graph instead, which also makes a genuine disagreement at a join point detectable rather than silent. * a __thiscall or __stdcall callee pops its own arguments, so esp rises across the call by an amount nothing at the call site records - nine pushes and no `add esp`. The callee's `ret N` is read from the binary to recover it, following import thunks once. Validated against the AICState build list: 55 of 60 frames exact, including all four functions already at 100%, and frame exactness is visibly independent of the percentage - aiGiveRaidInstructions has an exact frame at 56.8%. Two bugs the validation caught: `lea esp, [esp]` is MSVC's alignment filler and was being read as a reference to the slot at the current esp, which gave frameless 100%-matching clearPlayerDataInformationChunk a phantom slot (orig_body gains is_padding for the nop-equivalent forms); and the prologue allocation is not always the first instruction, which reported planToBuyWhenLowOnResourceAndSnoozeBuildings as frameless while its ten slots were plainly there. It found three targets that were not otherwise visible: renderDebugDataAiInfo missing two locals, and planToBuy... and aiRecruitUnits with the right frame but a slot whose reference count differs from the original's. Also documents the order of work - frame, then instruction mix, then statement order, and the percentage only once it is all that is left - since the earlier steps are causes and the later ones effects. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
Adds a `skel` tier to structural_diff - the instruction stream with only register
NAMES anonymised - and makes it report where each kind of difference starts, which is
the question that decides whether register work is worth doing at all.
The motivation is a measurement in c2.dll. Dumping ChoosePhysReg's per-register cost
array at every pick in renderDebugDataAiInfo gave all zeros for every volatile pick, so
the cost minimisation is vacuous there and the winner is just the first free entry in
the class list (eax, ecx, edx, esi, edi, ebx, ebp). Nothing about how the source is
spelled enters into it; the only inputs left are the order candidates are served and
what is live at that point, both set by the instruction stream.
So the report now says where the two sides first diverge structurally (measured on
mnemonics, since measuring it on whole operands puts it at instruction 1 of almost
every function - our side names ___security_cookie where the original has a bare
address) and where they first diverge in register naming:
renderDebugDataAiInfo 293/305 aligned on the skeleton, structure diverges at 139,
registers at 152, dominant substitution the closed cycle
eax -> edx -> ecx -> eax covering 90% of 59
aiCreateSiegeUnits 319/322 aligned, structure at 30, registers at 223
Both diverge in registers only after the structure does, and renderDebugDataAiInfo's
first 149 instructions have identical registers with identical callee-saved assignments
throughout - so those names are an effect of its remaining mix delta of 3, not an
independent defect. A substitution set that closes into a cycle is the signature;
the dominant permutation is reported with the share it actually covers rather than
claiming a clean cycle over the minority substitutions too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
Answers why the original did not merge an expression that appears identically in both arms of an if/else, when our build merged it every way we wrote it. It is not that CSE was disabled. MSVC hoists as much of a common expression as IS common, and the original shares only the front of the chain: one `imul eax, eax, 0x6d98` before the branch, then a separate `mov reg, [eax + 0x1866ac0]` in each arm with the array base folded into the load's displacement. Writing the whole chain out twice makes the whole chain common - member load, scale, element load and the `+ 1` - so all of it is hoisted, together with the value push and the buffer `lea`, leaving only `push OFFSET <string>` per arm. Four source forms all hoisted: inline in both arms, a shared local before the branch, a local declared before and assigned inside each arm, and a ternary on the format string. What reproduces the original is holding the INDEX in a local and writing the array access out in each arm - the index is shared because the source shares it, the two loads are left alone. On renderDebugDataAiInfo that is 305 instructions against the original's 305 with a mix delta of 0, and 67.5% -> 83.6%. A pointer or reference to the element gets to delta 1 but pays an `add` to materialise base + index * size, where the original keeps only index * size and re-folds the base per load - so sharing MORE than the original did costs an instruction too. Also notes a detail that locates the pass: the hoisted `push` moves above the `cmp`, and a post-allocation peephole cannot move a push across a branch, so this is the global optimiser running before register allocation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
The mix delta for renderDebugDataAiInfo read
add 19 18 -1
mov 52 51 -1
push 151 150 -1
which I read as three unrelated missing instructions. It is one missing *duplicate*:
the original emits `mov reg, [reg + G]; add reg, 1; push reg` in both arms of an
if/else and we emitted it once, because MSVC hoisted ours above the branch. Four source
forms were tried before the shape of the signal became clear, and the fix was then
worth 67.5% -> 83.6%.
So count occurrences instead of totals. With register names anonymised the two copies
compare equal despite being allocated to different registers, and the run that one side
repeats and the other shares is named directly, with the fix direction implied by which
side repeats: MSVC hoists the maximal common expression, so share in the source exactly
the prefix the original shares and write the rest out.
Two things the first version got wrong. A run has to occur at least TWICE on its own
side, or an instruction that merely moved looks like a duplication - it reported
`cmp`/`jne` at 1 against 0. And the repeated run is often only part of a larger
differing region, so sub-runs are searched longest-first rather than only whole diff
regions.
Fires on 4 of the 60 AICState functions - aiGiveRaidInstructions at 56.8% repeats a
seven-instruction run twice where we emit it once - and is silent on
renderDebugDataAiInfo once fixed.
Also makes the module import-safe. Its argument parsing ran at import and called
sys.exit, so importing it for `measure()` or `duplication()` exited with a usage dump;
the same flaw slot_map.py had.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
Explains what to measure and in what order - stack frame, then instruction mix, then registers, with reccmp's percentage only at the end - and why that order is the opposite of what feels natural: the early steps are causes and the late ones effects, and the percentage actively misleads while the frame is still wrong. The worked proof is renderDebugDataAiInfo going from a wrong frame at 68.4% to a byte-exact one at 67.5%, then to 95.4% once the instruction mix matched. Written around two real functions rather than in the abstract, including the one that is still stuck, because the dead ends carry most of the information: * renderDebugDataAiInfo, solved. Covers the hoisted-duplicate signal (`add -1, mov -1, push -1` is not three missing instructions, it is one missing copy of a three instruction run), the four source forms that failed, and the rule that came out of it - MSVC hoists the maximal common expression, so share in the source exactly the prefix the original shares and write the rest out. Sharing more costs an instruction too, with the four-way measurement table. * aiGiveRaidInstructions, open. Frame already perfect; delta 9, of which 7 are one missing copy of the tile computation; registers a clean two-cycle swap of `this` and the player-record base between ebx and ebp. Records the rejected attempt - moving `i` into the for-initialiser cut delta from 9 to 3 but dropped the score from 56.8% to 44.1% - and why the register swap resists: ebx precedes ebp in the preference list so whoever is served first takes it, our base temporary scores 82 against `this` at 60, and both sides reference those values an identical 27 and 22 times, so there is no expression to rewrite to change the ranking. Also fixes the cycle report in structural_diff. It walked one chain for as many steps as the permutation had entries, which printed nonsense whenever there was more than one cycle - `eax -> ecx -> edx -> eax -> ecx -> eax` for what is really two disjoint cycles, `eax -> ecx -> edx -> eax` alongside `ebp -> ebx -> ebp`. Now decomposed properly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
An earlier note had InsertInList sorting candidates on +0x40 with +0x0c as a tie-break. It is the other way round, and the wrong version sent a long stretch of work on aiGiveRaidInstructions reasoning from the wrong number. Disassembling @InsertInList@8 at 0x10b11970 shows it compares [eax+0xc] against [edi+0xc] and only falls through to +0x44 on a tie, so the key is +0x0c descending, tie-broken by the symbol id. The old claim came from a second, nearly identical insert function right after it at 0x10b119f0, which does sort on +0x40 - and writes +0x0c by copying a neighbour's. Easy to confuse; the one the harness breakpoints is 0x10b11970. Verified rather than asserted: dump both fields per candidate and check each ordering against the actual register-assignment sequence from --whyreg. +0x0c reproduces it exactly; +0x40 would put `this` and a temporary ahead of `tile`, which is not what happened. +0x0c also reads as a net benefit - the candidates that come out low or negative are the ones slot_map.py shows in stack slots, so the two tools cross-check. reg_scores.py now ranks on +0x0c and prints both fields, since printing +0x40 and calling it the benefit is what made the mistake easy to keep. Two tooling traps recorded. A linear capstone sweep of c2.dll's .text stops at the first undecodable byte - 12106 instructions out of a 1.9 MB section - which made a scan for field writes look empty; resyncing gives 647653. And taking a candidate record's address from one run to arm a write watchpoint in the next does not work: the records are heap-allocated, the addresses are not stable, and the watchpoint fired inside ntdll!RtlpFreeHeap. For aiGiveRaidInstructions the key says our base temporary ranks 128 against `this` at 93, so it is served first and takes ebx where the original gave ebx to `this`. Both sides reference the two values an identical 27 and 22 times with an identical call pattern, and `this` is a real __thiscall parameter in both, so there is no use-count difference to exploit. Records the three source forms measured and rejected. The formula behind +0x0c is still open - only the two places that copy it are located. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
…y balance
+0x0c is accumulated at the tail of FUN_10b10420, which the colourer calls per
register class before building the ordered list. Records the full chain from
dop2_GlobalRegAlloc down to ChoosePhysReg, and the accumulation:
count = elements of list A, plus list B elements flagged 0x10 (referenced here)
esi = count * weight (weight passed in - block frequency)
referenced in this region: key += refs * count * weight
live here but unreferenced: key -= count * weight, or -= weight when the global
at 0x10d0ffdc is 2
So the key is not a use count. It is a balance taken region by region, and a value
that is live across a region without being referenced is charged for holding a
register open. That predicts the measured numbers: playerID at -147 is live across the
whole loop and barely read inside it, i comes out at 1, and the negative-key
candidates are exactly the set slot_map.py independently shows in stack slots.
The consequence matters more than the formula: the key depends on the BLOCK STRUCTURE,
so two versions with identical reference counts can rank differently if their regions
differ. That resolves the aiGiveRaidInstructions question. Our base temporary outranks
`this` 128 to 93 where the original's `this` won ebx, and both sides reference those
values an identical 27 and 22 times - but we still emit one copy of the tile
computation where the original emits two, one region fewer. The register swap is
downstream of that mix defect, exactly as structural_diff reports from the divergence
positions. The lever is the duplication, not anything register-side.
Also records that REA decompiles c2.dll in an ephemeral isolated Ghidra project, so it
does not disturb a shared session's currentProgram, but will not load a caller-supplied
PDB - the P2/*.c assert strings are what anchor a function to its source file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
…ated
Checked the hypothesis from the previous commit - that the original might have been
compiled in the other penalty mode, since FUN_10b10420 switches on
`cmp dword ptr [0x10d0ffdc], 2`. It was not:
* measured during our own compile: MODEFLAG 0, so we take the larger count*weight
penalty.
* `#pragma optimize("s", on)` does not change it, so it is not the per-file size/speed
switch; it is read in ~400 places and set at invocation level.
* and size-favoured code is not what the original used regardless: that pragma takes
aiGiveRaidInstructions from delta 7 to delta 52 and the ratio from 0.728 to 0.372.
Both sides therefore compute the key with the same formula.
Also measured the two pragmas the cheat sheet records as sometimes winning. Both DO
restore the missing tile duplicate - 3 of the original's 4 `imul 0x32c` becomes 4 - and
both lose more than they gain: optimize("y", off) reserves ebp as a frame pointer,
moving `this` to edi and collapsing the ratio to 0.262, and inline_depth(1) adds 22
instructions. The original uses ebp as a general register, so /Oy must stay on, and
with it on the cross-jump happens.
And a correction to the previous commit: the duplicate is NOT upstream of the register
swap. The original's two copies are the same instructions in different registers (edx
against ecx), and cross-jumping runs after allocation - so it was the differing
registers that saved them from being merged. The two are one coupled phenomenon.
Commuting the addition in one arm does not break the tie either; MSVC canonicalises it.
What did pay was an ordinary defect found by subtracting the duplicate from the mix
delta, leaving `je -1, jne +1` - one inverted branch, worth 56.8% -> 60.3%. diff_jcc.py
misses it because it covers strictness steps and operand order, not plain polarity.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
An indirect `jmp dword ptr [reg*4 + TABLE]` left every case body unreachable from the entry, so their stack references went uncounted and the warning was the only sign. On addUnitToSmallestBehaviourTypeTribe that was 52 of 122 instructions - most of the function - and it made the original's two slots look like they had one reference each when they have three, i.e. it reported a difference where there is none. MSVC puts the target table just past the function's last instruction, and for a sparse switch a byte index table after that, so the dword entries are followed only while they point back inside the body; the first value that does not is the next table's data. On that function this yields exactly five case targets and stops on 0x04040400, the byte table. Across the AICState build list: 56 of 60 frames now exact, up from 55, and the functions reporting unreached blocks drop to 7 - the rest are other indirect-jump shapes, which still warn rather than guess. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
…leads slot_map reported 13 slots for aiCreateSiegeUnits against our 12, one of them at -52 inside a 0x30 frame - past the end of what the prologue allocated, so not a slot at all. Its own warning had the cause: two paths reach 0x4d23a7 with esp depths 4 apart and the tracking drifted. References below -frame are now excluded and reported, with the note that a slot-count difference on a function reporting a depth conflict is not evidence of a defect. Both sides then show 12 slots at matching offsets. Third artifact of this shape this session, after unreferenced slot symbols and unreachable switch cases, and they share a cause: our side is stated by the /FA listing while the original's is inferred, so the inferred side needs sanity checks before a difference it reports is believed. Also records planToBuyWhenLowOnResourceAndSnoozeBuildings as a real slot permutation with no reachable lever. 248 instructions both sides, same frame, same slots, same weights, different slot assignment - which with several locals tied on weight looked exactly like the declaration-order tie-break. A quick_reorder -s hill-climb over all fourteen declarations kept no move, start and final both 0.9234. Since that is also what the concurrent-run corruption looks like, it was checked: three hand-made swaps of equal-weight pairs score 0.9194, 0.9153 and 0.9113, all worse, so the scoring is live and our order is already a local optimum. The residue is neither weight nor declaration order, which leaves the undecoded class byte at +0x18. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
…g rule
The class byte at +0x18 was the last undecoded input to StkColorCmp and the primary key
it sorts on. A twelve-instruction classifier at 0x10beb730 computes it, and the
placement pass at 0x10beb760 inlines the same thing:
class(symbol) = 3 when (symbol+0x37 & 0x40000)
and symbol+0x63 <= 2
and symbol+0x1c > 4
= 1 + ((symbol+0x20 & 0x4000000) != 0) otherwise
The only size-dependent term is `symbol+0x1c > 4`, so anything wider than a dword takes
class 3 and sorts as a group apart from the scalars. The byte is written only under a
feature gate and otherwise stays 0, so with that feature off every slot shares one class
and the key degenerates to the weight alone.
Read off the arithmetic, this is /GS buffer ordering: arrays are what can overflow, so
they are classed separately and placed nearest the canary. renderDebugDataAiInfo
corroborates it - char text[1000] sits at -1004, above every int scalar and immediately
below __$ArrayPad$ at -4. The /GS reading is labelled as an inference from the arithmetic
plus that one observation; the arithmetic is read directly off the instructions.
That explains the symptom the old entry recorded as unexplained, arrays and address-taken
locals interleaving with scalars, because the class outranks the weight.
It does NOT unblock planToBuyWhenLowOnResourceAndSnoozeBuildings, which is what prompted
the work, and the note says the prediction was wrong. All ten of that function's locals
are int, so the size term is 4 for each, none reaches class 3, and the class is uniform -
leaving the weight, where both sides agree, and declaration order, already shown to be a
local optimum. The byte discriminates by width and that function has nothing but dwords.
Also records how the file's functions were found: search the image for the UTF-16
P2/stack.c path, then for pushes of its address. The four assert lines are 1372, 2533,
2534 and 2573, and 1372 fixes ColorStackElements at 0x10beaef0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
…arity flips c2.dll with its PDB is now in the Ghidra project, so the whole model in this file could be checked against real symbols rather than left as inference. Every function it names is confirmed: 0x10c07350 dop2_GlobalRegAlloc the allocation driver 0x10b10420 Spillcosts accumulates the +0x0c ordering key 0x10b11970 InsertInList candidate list, sorts on +0x0c 0x10b119f0 InsertByBenefit the other insert, sorts on +0x40 0x10b14b70 ChoosePhysReg picks the cheapest free register 0x10beaef0 ColorStackElements the stack-slot colourer 0x10beb760 colorElemental per-element slot placement 0x10beb730 StkGSBufferLevel the class byte at +0x18 Three matter. `Spillcosts` is the semantic of the +0x0c key derived from the arithmetic before the name was known - gain where referenced, charge for each region live but unreferenced, which is what a spill cost is. `InsertInList` against `InsertByBenefit` explains the error this file used to carry: they are adjacent and nearly identical, and the old note had the fields the wrong way round. And `StkGSBufferLevel` turns the /GS reading of the class byte from an inference into the name itself; its decompilation is identical to the hand-decode. The PDB applies no struct types, so the symbol record's fields stay unnamed - the function names are what it buys. Records the operational detail too: ghidra-mcp's read tools accept a `program` argument, so a non-current program is queryable without touching a shared session's currentProgram. switch_program reports success but does not redirect the query tools - after switching, get_metadata still named the game executable. Also adds polarity-flip detection to structural_diff. diff_jcc covers strictness steps and operand order but not a plain je/jne inversion, which is a different source defect and was worth 56.8% -> 60.3% on aiGiveRaidInstructions. Validated by re-introducing that defect: the detector finds it at original instruction 214 and is silent once fixed. Swept over all 60 AICState functions it reports zero further instances, so that is a genuine negative rather than an unexercised check. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
…ches change Three findings from digging into c2.dll for allocation and scheduling levers, now that the PDB gives real names. The global at 0x10d0ffdc that an earlier section could only call "a mode flag set at invocation level" is _PogoFlag, the per-function copy of _PogoMasterFlag at 0x10d44b4c. It gates three behaviours covering both walls this file keeps hitting: 0 our builds full tail merging; Spillcosts penalty is count * weight 1 instrument FgTryTailMerge returns immediately - no tail merging at all 2 optimise Spillcosts penalty is just weight; PogoOptCodeMotion also runs FgTryTailMerge (0x10b2e1b0) wraps its whole body in `if (_PogoFlag != 1)` - merging blocks would corrupt the profile counters - and that is the pass which merges the duplicate aiGiveRaidInstructions is missing. The twelve dop2_* phase entries give the pass order and settle a question this file had reasoned about twice: GlobalRegAlloc is phase 6, FinalFlowOpts is phase 10, and dop2_FinalFlowOpts calls FgDoLateFlowOpts, FgCodeMotion and - only at flag 2 - PogoOptCodeMotion. Tail merging and code motion therefore run AFTER register allocation, confirming that two arms identical except for their registers are not mergeable, so the allocation can be what preserves a duplicate. FgCodeMotion is also the sink no source form of playNervous2BikFromPlayer could prevent. pogo_probe.py tests the allocation lever by forcing the flag across Spillcosts only, bracketed on its entry and its ret. It works: on addUnitToSmallestBehaviourTypeTribe the frame drops from 12 to the original's 8 and a fourth value is enregistered, matching the original - exactly what the smaller penalty predicts and the direction every pressure difference pointed. But the overall match gets worse (kind 0.893 -> 0.624, delta 10 -> 15) because other flag-2 paths strip seven instructions, and on aiGiveRaidInstructions it moves the contested pair away from the original. A confirmed lever, not the explanation. Records the mistake too: the first bracket used the last byte of Spillcosts rather than its ret, so the restore never fired and the compile died on a null dereference in FgFindDominatorsGraph. And the measurement correction. Both resolver switches are off in a fresh checkout, which is how every percentage here had been taken. Turning them on moves the score a long way - computeAttackPossibilities 72.4 to 89.7, playNervous2BikFromPlayer 64.0 to 76.0, renderDebugDataAiInfo 95.4 to 99.0, with the build list averaging 94.20 over 60 - but leaves slot_map, structural_diff and quick_diff byte-identical, because a resolver changes an operand and not an instruction. So percentages from a default checkout are understated while the structural conclusions stand; the two need stating in the same configuration to be comparable. Finally, /GL does not transfer to this namespace. enable_local_functions.py credits it with getSalesPrice 68.4% to 100% once its callee's resolver was flipped, which is exactly the symptom class behind every register-pressure wall here, and no AICState file had ever carried /GL. Adding all sixty with the resolvers on, and verifying the flag reached the compile command, gained nothing at all and cost planToBuy 0.8. The callees that matter - checkTribeActivityPercentages, selectRandomBuildingIDFromListOfBuildingTypes, the TribesState and UnitsState methods - are not in the build list, so those calls leave the module regardless. Bring them in first, then retest. pdbnames.py recovers 9687 public symbols from c2.pdb by scanning S_PUB32 records, which is the practical name-to-address route because ghidra-mcp's name searches ignore the `program` argument while get_function_by_address honours it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack slot assignment is not declaration order. It is reference weight per byte, and this PR documents the mechanism, the evidence, and a tool to read it.
Where this comes from
The bundled
MSVC1400-SP1toolchain is the compiler that reproduces the game's codegen (c2.dll,Microsoft (R) Optimizing Compiler Version 14.00.50727.762). Microsoft still publishesc2.pdbon its symbol server, which names every function in the optimizer, andc2.dll's own fatal-assert path leaks its source paths (F:\SP\vctools\compiler\utc\src\P2\stack.c,P2\color.c,P2\regasg.c, 66 files in all). So the backend's stack colorer could be read directly rather than guessed at.ColorStackElements(P2/stack.c, named by the assert at line 1372) builds one 28-byte record per stack element holding the local's size rounded up to 4 and its accumulated reference weight, sorts the records withStkColorCmp, and then hands out offsets in that order:Evidence
planToBuyWhenLowOnResourceAndSnoozeBuildingshas fourteenint allowX = 1;locals. Eleven get colored, and they come out exactly in descending reference count from the lowestespoffset upward —5 5 4 4 4 4 4 3 3 3— with their declaration positions scrambled (the 4th declared sits between the 8th and the 9th, the 1st sits last).Three checks on top of that correlation:
5 5 5 4 4 4 4 3 3 3).allowQuarries→zzzandallowWoodcutters→aaaaaaaaaa, inverting both the alphabetical order and the lengths of an adjacent equal-weight pair, left the function byte-identical. The comparator never looks at a name.What changes for day-to-day work
Map::Units::UnitsStateframe mismatches came back a tie.QSortColors, an unstable quicksort. That is exactly whyperm_decls.pytookassignSelectionToKey95.9% → 100% once and does nothing in most functions. The README andperm_decls.py's docstring claimed declaration order assigns slots; both are corrected to say it only decides ties.writeMapOrSaveFileputs eight locals on the first parameter's slot), so a slot count is not a local count —diff_slots.py's note is qualified accordingly.playerID = 0used as a loop counter inaiRecruitUnits. It is the compiler, not the original source; transcribing it back cost 7 points.slot_weight.pyCompiles one file with
/FAand reads MSVC's own_name$ = Nsymbol table, so the local-to-slot mapping is observed, not inferred from the diff. It prints each local's reference count, weight, inferred size and weight per byte against its slot, flags shared slots and dead-parameter reuse, and with a reccmp run available names the first position where the original's slot census disagrees with ours.Honest limits, which the docstring states: the primary key — the class byte at
+0x18— is not decoded, so arrays and address-taken locals interleave for reasons the tool cannot explain (wrapTextIntoLinesis a worked example of a non-monotone report). The weight itself is approximated as references × 10 per loop nesting level, where the compiler uses real block frequencies fromCgAnnotateEdgeWeight. Treat a non-monotone report as a hint about which local is light or heavy, not as a layout prediction.--reordersorts declarations by descending weight and can therefore only decide ties — one cheap variant to measure, not a repair.Register assignment — which callee-saved register receives which value, and the
pushorder — lives inP2/regasg.candP2/color.cand is untouched by this; that is the other half ofalloc-onlyand remains open.Testing
slot_weight.pywas exercised onplaySFXAtLocation,setupWorkshopProductionType,writeMapOrSaveFile,wrapTextIntoLinesandplanToBuyWhenLowOnResourceAndSnoozeBuildings. Documentation and one new tool only — no reimplementation source, headers or build files are touched, so no function's match percentage can move.🤖 Generated with Claude Code
https://claude.ai/code/session_012yZukvLckY9Pd4jxzqcLRq