Integrate community PRs #1–#26: 100% real-world corpus recompiles

CHANGELOG.md

@@ -6,6 +6,22 @@ cargo tests, on 13 interpreters (2.6–3.14).
 
 ## Unreleased
 
+### Community fixes (PRs #1–#26)
+- merged 26 targeted fixes from @EiNSTeiN- covering 3.11–3.14 bytecode
+  families: chained assignments and comparison chains; COPY/SWAP and
+  walrus in the bounded expression simulator; keyword calls and ordered
+  `**` expansions; comprehension filter polarity, Boolean elements,
+  nested conditional elements and nested comprehensions inside generator
+  expressions; `for x in await f()` iteration protocol; padded handler
+  returns; EXTENDED_ARG loop back edges; guarded `else` suites; split
+  try/except/else/finally printing; marshalled slice constants vs runtime
+  `slice()` calls; and PEP 649 class-annotation registration gates.
+- the 26 branches and their regression tests were merged and verified
+  together. Three textual conflicts were resolved by keeping both sides:
+  PR #6 stacks on #5, all PRs append tests at the same point in
+  `tests/integration.rs`, and #4/#5/#6/#24 append test-module
+  declarations at the end of `decompiler.rs`.
+
 ### Fixed
 - value-position and-chains in a ternary: `try_dup_tail_ternary` now
   absorbs leading same-target cond jumps into the condition, so
@@ -16,9 +32,9 @@ cargo tests, on 13 interpreters (2.6–3.14).
   merge, so `d[a if c and e else b]` keeps the shared lookup.
 
 ### Benchmarks
-- real-world corpus: 6408/6411 modules recompile (99.95%); pandas is at
-  1419/1420 with only the nested-comprehension family left (sklearn ×2,
-  pandas to_dict)
+- real-world corpus: 6411/6411 modules recompile (100%); the former
+  nested-comprehension family (pandas `to_dict`, sklearn `_logistic`,
+  `pairwise`) is closed, and strict AST equality rises to 2493/6411
 
 
 ### Fixed

README.md

@@ -103,25 +103,26 @@ seven projects **at its latest release tag**, compiles every `.py` to
 
 | Project (tag) | modules | serial `-j 1` | parallel (default) | recompiles | AST-identical¹ |
 |---|---|---|---|---|---|
-| yt-dlp (2026.08.19) | 1045 | 0.93 s | 0.12 s | **1045/1045** | 500/1045 |
-| matplotlib (v3.11.2) | 253 | 0.47 s | 0.09 s | **253/253** | 59/253 |
-| pandas (v3.0.5) | 1420 | 1.35 s | 0.21 s | 1419/1420 | 582/1420 |
-| django (6.1.1) | 907 | 0.67 s | 0.16 s | **907/907** | 467/907 |
-| sympy (1.14.0) | 1532 | 3.68 s | 1.85 s | **1532/1532** | 479/1532 |
-| scikit-learn (1.9.1) | 671 | 0.71 s | 0.10 s | 669/671 | 216/671 |
-| ansible (v2.21.4) | 583 | 0.35 s | 0.06 s | **583/583** | 185/583 |
-| **total** | **6411** | **8.16 s** | **2.59 s** | **6408/6411 (99.95 %)** | 2488/6411 |
+| yt-dlp (2026.08.19) | 1045 | 0.46 s | 0.08 s | **1045/1045** | 501/1045 |
+| matplotlib (v3.11.2) | 253 | 0.33 s | 0.06 s | **253/253** | 60/253 |
+| pandas (v3.0.5) | 1420 | 0.90 s | 0.14 s | **1420/1420** | 584/1420 |
+| django (6.1.1) | 907 | 0.38 s | 0.10 s | **907/907** | 468/907 |
+| sympy (1.14.0) | 1532 | 3.03 s | 1.63 s | **1532/1532** | 479/1532 |
+| scikit-learn (1.9.1) | 671 | 0.48 s | 0.07 s | **671/671** | 216/671 |
+| ansible (v2.21.4) | 583 | 0.27 s | 0.05 s | **583/583** | 185/583 |
+| **total** | **6411** | **5.85 s** | **2.13 s** | **6411/6411 (100 %)** | 2493/6411 |
 
 ¹ strict: docstring-stripped AST equal to the original source —
 equivalent normalizations (an if/else instead of a ternary, a redundant
 `*()`/`**{}` around an unpack, reordered keyword arguments) count as
 differences, so treat it as a shape-fidelity floor rather than an error
 rate. The parse/recompile column is the acceptance bar.
 
-One pycdc process per project: ~1.0 ms per module serial, peak RSS
-≈99 MB for the whole 6.4 k-module batch. The three remaining modules are one shape: a dict comprehension
-nested inside a generator expression's element (call arguments
-included) — marked incomplete, never a crash. Raw data: [`benchmarks/realworld.json`](benchmarks/realworld.json),
+One pycdc process per project: ~0.9 ms per module serial, peak RSS
+≈81 MB for the whole 6.4 k-module batch. **All 6411 modules recompile**;
+the last family (a dict comprehension nested inside a generator
+expression's element, call arguments included) was closed by the
+community PR batch. Raw data: [`benchmarks/realworld.json`](benchmarks/realworld.json),
 harness: [`tools/bench_realworld.py`](tools/bench_realworld.py).
 
 

README.zh-CN.md

@@ -99,23 +99,23 @@ Apple Silicon 实测，526 个真实 `.pyc`（约 9.5MB，2.6–3.14 标准库
 
 | 项目（tag） | 模块数 | 串行 `-j 1` | 并行（默认） | 可重编译 | AST 完全一致¹ |
 |---|---|---|---|---|---|
-| yt-dlp (2026.08.19) | 1045 | 0.93 s | 0.12 s | **1045/1045** | 500/1045 |
-| matplotlib (v3.11.2) | 253 | 0.47 s | 0.09 s | **253/253** | 59/253 |
-| pandas (v3.0.5) | 1420 | 1.35 s | 0.21 s | 1419/1420 | 582/1420 |
-| django (6.1.1) | 907 | 0.67 s | 0.16 s | **907/907** | 467/907 |
-| sympy (1.14.0) | 1532 | 3.68 s | 1.85 s | **1532/1532** | 479/1532 |
-| scikit-learn (1.9.1) | 671 | 0.71 s | 0.10 s | 669/671 | 216/671 |
-| ansible (v2.21.4) | 583 | 0.35 s | 0.06 s | **583/583** | 185/583 |
-| **合计** | **6411** | **8.16 s** | **2.59 s** | **6408/6411（99.95%）** | 2488/6411 |
+| yt-dlp (2026.08.19) | 1045 | 0.46 s | 0.08 s | **1045/1045** | 501/1045 |
+| matplotlib (v3.11.2) | 253 | 0.33 s | 0.06 s | **253/253** | 60/253 |
+| pandas (v3.0.5) | 1420 | 0.90 s | 0.14 s | **1420/1420** | 584/1420 |
+| django (6.1.1) | 907 | 0.38 s | 0.10 s | **907/907** | 468/907 |
+| sympy (1.14.0) | 1532 | 3.03 s | 1.63 s | **1532/1532** | 479/1532 |
+| scikit-learn (1.9.1) | 671 | 0.48 s | 0.07 s | **671/671** | 216/671 |
+| ansible (v2.21.4) | 583 | 0.27 s | 0.05 s | **583/583** | 185/583 |
+| **合计** | **6411** | **5.85 s** | **2.13 s** | **6411/6411（100%）** | 2493/6411 |
 
 ¹ 严格口径：去掉 docstring 后 AST 与原源码完全一致。等价归一化
 （if/else 代替三元、解包外多余的 `*()`/`**{}`、关键字参数顺序调整）也
 会被计为差异，因此它是「形状保真度下限」而非错误率；验收口径是左侧的
 可重编译列。
 
-每个项目一个 pycdc 进程：串行约 1.0 ms/模块，整批 6.4k 模块峰值
-RSS ≈98 MB。剩余 3 个模块属一种形态：生成器表达式元素（含调用参数）内嵌
-dict 推导式——标记为不完整，不崩溃。
+每个项目一个 pycdc 进程：串行约 0.9 ms/模块，整批 6.4k 模块峰值
+RSS ≈81 MB。**6411 个模块全部可重编译**；最后一族（生成器表达式元素 /
+调用参数里的 dict 推导式）已由社区 PR 批次修复。
 [`benchmarks/realworld.json`](benchmarks/realworld.json)，
 脚本见 [`tools/bench_realworld.py`](tools/bench_realworld.py)。
 

benchmarks/realworld.json

@@ -3,17 +3,17 @@
  "pycdc": "target/release/pycdc",
  "interp": "~/.local/share/uv/python/cpython-3.12.14-macos-aarch64-none/bin/python3.12",
  "host": "Darwin 25.6.0 arm64",
- "generated": "2026-09-15 13:38:23",
+ "generated": "2026-10-23 14:52:30",
  "note": "one pycdc process per project (batch directory mode); serial = -j 1, parallel = default worker count; recompiles = decompiled output parses under the corpus interpreter",
  "projects": [
   {
    "project": "ansible",
    "files": 583,
    "source_bytes": 5372361,
-   "serial_wall_s": 0.35,
-   "serial_rss_mb": 9.0,
-   "parallel_wall_s": 0.06,
-   "parallel_rss_mb": 18.3,
+   "serial_wall_s": 0.27,
+   "serial_rss_mb": 8.5,
+   "parallel_wall_s": 0.05,
+   "parallel_rss_mb": 18.6,
    "decompiled": 583,
    "recompiles": 583,
    "ast_equal": 185,
@@ -23,62 +23,62 @@
    "project": "django",
    "files": 907,
    "source_bytes": 5914265,
-   "serial_wall_s": 0.67,
-   "serial_rss_mb": 8.8,
-   "parallel_wall_s": 0.16,
-   "parallel_rss_mb": 23.5,
+   "serial_wall_s": 0.38,
+   "serial_rss_mb": 8.3,
+   "parallel_wall_s": 0.1,
+   "parallel_rss_mb": 23.9,
    "decompiled": 907,
    "recompiles": 907,
-   "ast_equal": 467,
+   "ast_equal": 468,
    "ast_checked": 907
   },
   {
    "project": "matplotlib",
    "files": 253,
    "source_bytes": 7415404,
-   "serial_wall_s": 0.47,
-   "serial_rss_mb": 20.9,
-   "parallel_wall_s": 0.09,
-   "parallel_rss_mb": 38.0,
+   "serial_wall_s": 0.33,
+   "serial_rss_mb": 18.5,
+   "parallel_wall_s": 0.06,
+   "parallel_rss_mb": 35.9,
    "decompiled": 253,
    "recompiles": 253,
-   "ast_equal": 59,
+   "ast_equal": 60,
    "ast_checked": 253
   },
   {
    "project": "pandas",
    "files": 1420,
    "source_bytes": 21287834,
-   "serial_wall_s": 1.35,
-   "serial_rss_mb": 13.5,
-   "parallel_wall_s": 0.21,
-   "parallel_rss_mb": 39.1,
+   "serial_wall_s": 0.9,
+   "serial_rss_mb": 12.8,
+   "parallel_wall_s": 0.14,
+   "parallel_rss_mb": 38.9,
    "decompiled": 1420,
-   "recompiles": 1419,
-   "ast_equal": 582,
+   "recompiles": 1420,
+   "ast_equal": 584,
    "ast_checked": 1420
   },
   {
    "project": "sklearn",
    "files": 671,
    "source_bytes": 13444062,
-   "serial_wall_s": 0.71,
-   "serial_rss_mb": 10.8,
-   "parallel_wall_s": 0.1,
-   "parallel_rss_mb": 25.0,
+   "serial_wall_s": 0.48,
+   "serial_rss_mb": 10.6,
+   "parallel_wall_s": 0.07,
+   "parallel_rss_mb": 30.1,
    "decompiled": 671,
-   "recompiles": 669,
+   "recompiles": 671,
    "ast_equal": 216,
    "ast_checked": 671
   },
   {
    "project": "sympy",
    "files": 1532,
    "source_bytes": 26169877,
-   "serial_wall_s": 3.68,
-   "serial_rss_mb": 49.6,
-   "parallel_wall_s": 1.85,
-   "parallel_rss_mb": 78.8,
+   "serial_wall_s": 3.03,
+   "serial_rss_mb": 48.6,
+   "parallel_wall_s": 1.63,
+   "parallel_rss_mb": 80.6,
    "decompiled": 1532,
    "recompiles": 1532,
    "ast_equal": 479,
@@ -88,27 +88,27 @@
    "project": "yt_dlp",
    "files": 1045,
    "source_bytes": 9208581,
-   "serial_wall_s": 0.93,
-   "serial_rss_mb": 15.5,
-   "parallel_wall_s": 0.12,
-   "parallel_rss_mb": 23.2,
+   "serial_wall_s": 0.46,
+   "serial_rss_mb": 11.8,
+   "parallel_wall_s": 0.08,
+   "parallel_rss_mb": 21.0,
    "decompiled": 1045,
    "recompiles": 1045,
-   "ast_equal": 500,
+   "ast_equal": 501,
    "ast_checked": 1045
   }
  ],
  "total": {
   "project": "TOTAL",
   "files": 6411,
   "source_bytes": 88812384,
-  "serial_wall_s": 8.16,
-  "parallel_wall_s": 2.59,
-  "serial_rss_mb": 49.6,
-  "parallel_rss_mb": 78.8,
+  "serial_wall_s": 5.85,
+  "parallel_wall_s": 2.13,
+  "serial_rss_mb": 48.6,
+  "parallel_rss_mb": 80.6,
   "decompiled": 6411,
-  "recompiles": 6408,
-  "ast_equal": 2488,
+  "recompiles": 6411,
+  "ast_equal": 2493,
   "ast_checked": 6411
  }
 }

docs/LIMITATIONS.md

@@ -143,57 +143,22 @@ rendering bugs:
   (300+ regression-driven special cases), so it is left documented
   rather than patched blind.
 
-### Nested comprehension inside a genexp (3 modules, deliberately not patched)
-
-`({c: v for c, v in zip(cols, row) if c} for row in rows)` — a genexp whose
-*element* (or a call argument inside it) is an inlined (PEP 709) dict
-comprehension. The genexp keeps its own code object and is decompiled by
-the comprehension walker, which has no notion of a nested level: the
-inner `BUILD_MAP 0` accumulator lands on the walker's stack, `SWAP` is
-not modeled, and the following `FOR_ITER` treats it as a second `for`
-clause (`... for c, v in {}`), so the output does not parse.
-
-A frame-stack prototype (one frame per comprehension level, `SWAP`
-modeled, the accumulator placeholder replaced by Rc identity when a level
-ends, `pending_or_filter` flushed per level) was implemented **twice** and
-reverted both times. What it achieved, verified by runtime-equivalence
-checks against the original sources: the whole-element shape and the
-call-argument shapes *as minimal repros* became semantically correct
-(`({c: v for c, v in zip(cols, t) if c} for t in rows)`,
-`(f(**d) for x in xs)`, `(f(x, y=1) for x in xs)`; the last two were
-fixed for real and are committed).
-
-It still produced *plausible but semantically wrong* output for the three
-real modules (`pandas/core/methods/to_dict.py`,
-`sklearn/linear_model/_logistic.py`, `metrics/pairwise.py`), e.g.
-
-    fd(func, ret, s, X, Y[s, ...], **{k: (v[s] if k == "Y" else v) for k, v in kwds.items()})
-    ->  (fd if k == 'Y' else (func, ret, s, X, Y[s, ...])(*{}, **{**k}) for s in ...)
-
-The blocker is not the nesting itself: the walker's stack model is a
-heuristic that happens to agree with the VM on the shapes the corpus
-exercises, and a nested level exposes several concrete divergences at
-once (all observed with probes during the third attempt, 2026-09-15):
-
-  * `GET_ITER` is a no-op, so the 3.12 prologue's `SWAP`s reorder the
-    wrong slots (the VM's is `[saved…, iter]` -> `BUILD_* 0` -> `SWAP 2`;
-    the walker has no `iter` on the stack there);
-  * the `LOAD_FAST_AND_CLEAR` save slots are pushed but never popped —
-    the epilogue's `STORE_FAST`s restore them in the VM, while the walker
-    treats stores as target assignments, so they linger on the stack;
-  * `PUSH_NULL` markers then sit between the level's value and the call
-    slot, and the `CALL_FUNCTION_EX` pops pick up the wrong values
-    (`fd(...)` came out as `fd if k == 'Y' else (func, ret, X, Y)(*{}, **{**k})`).
-
-Fixing one of these shifts the divergence to the next (truncating the
-level's stack to its base fixed the kwargs slot but stripped the outer
-operands, and excluding the save slots then leaked the `PUSH_NULL`
-marker into the output), which is exactly why this needs the call model
-reworked rather than another local patch. A correct fix
-therefore starts with making that call model faithful (or by delegating
-the enclosing expression to the main engine's `exec`), and only then
-re-adding the nesting. Emitting a visible syntax error stays preferable
-to silently wrong code, so the modules remain in the failing set.
+### ~~Nested comprehension inside a genexp~~ (fixed 2026-10-23)
+
+PR #24 recovers the inlined comprehension by decompiling the generator's
+code object with the main statement parser and extracting its
+For/If/Yield tree; the outer iterable is restored at the call site. This
+closes the whole-element and call-argument shapes, including
+`({c: v for c, v in zip(cols, row) if c} for row in rows)`.
+
+The three full-module cases also needed the keyword/dictionary handling
+from #10 and #22: a large `MAP_ADD 1` kwargs dictionary must not be
+mistaken for the comprehension's own key/value, and `DICT_MERGE`
+expansions must keep their boundary. With the consolidated merge all
+6411 real-world modules recompile. The earlier frame-stack analysis
+(kept in git history) confirmed the right fix was to delegate the
+enclosing expression to the statement engine rather than to model every
+nested level in the comprehension walker.
 
 ### ~~Multi-link ternary on the value path~~ (fixed 2026-09-15)
 

src/codegen.rs

@@ -535,6 +535,28 @@ impl Printer {
                 orelse,
                 finalbody,
             } => {
+                // Exception-region reconstruction can split try/except from
+                // its success and finally suites. An else without handlers
+                // cannot belong to the outer try. Rejoin the single inner
+                // handler-bearing try, retaining every recovered statement.
+                if self.version.at_least(2, 5)
+                    && handlers.is_empty() && !orelse.is_empty() && !finalbody.is_empty()
+                    && body.len() == 1
+                {
+                    if let Stmt::Try {
+                        body: inner_body, handlers: inner_handlers,
+                        orelse: inner_else, finalbody: inner_finally,
+                    } = &body[0]
+                    {
+                        if !inner_handlers.is_empty() && inner_else.is_empty() && inner_finally.is_empty() {
+                            self.stmt(&Stmt::Try {
+                                body: inner_body.clone(), handlers: inner_handlers.clone(),
+                                orelse: orelse.clone(), finalbody: finalbody.clone(),
+                            });
+                            return;
+                        }
+                    }
+                }
                 // a bare `except:` must be last (and unique); drop extras
                 let mut handlers = handlers.clone();
                 handlers.sort_by_key(|h| if h.type_.is_none() { 1 } else { 0 });
@@ -1134,6 +1156,8 @@ impl Printer {
                     first = false;
                     if let Some(kn) = k {
                         self.write(&format!("{kn}="));
+                    } else {
+                        self.write("**");
                     }
                     self.expr(v, prec::TERNARY);
                 }