From 70c431c53da9aa8d86ced8c95856ad7070c30120 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Wed, 9 Sep 2026 14:22:53 +0200 Subject: [PATCH 1/8] docs: plan PPR compatibility completion Record the known PPR 0.001010 failures and the staged acceptance plan for issue #1318 before implementation begins. Generated with Codex (https://openai.com/codex) Co-Authored-By: Codex --- dev/design/ppr-compatibility.md | 107 ++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) create mode 100644 dev/design/ppr-compatibility.md diff --git a/dev/design/ppr-compatibility.md b/dev/design/ppr-compatibility.md new file mode 100644 index 000000000..3ad520b80 --- /dev/null +++ b/dev/design/ppr-compatibility.md @@ -0,0 +1,107 @@ +# PPR 0.001010 Compatibility + +Issue: #1318 + +## Objective + +Make the complete non-optional test suite for CPAN distribution +`DCONWAY/PPR-0.001010` pass under PerlOnJava. The final integration gate is a +bounded `./jcpan -t PPR` run with no failed test files or stuck harness +processes. + +This work follows #1299, which fixed the Joni analysis hang while loading +PPR's large dynamic named-subexpression grammar. Loading the module is no +longer the acceptance boundary: PPR grammar matching must now be compatible. + +## Known Baseline + +On the merged #1317 source, `timeout 1800 ./jcpan -t PPR` configured and built +PPR successfully, then exposed these failures: + +| Test | Observed symptom | Initial ownership hypothesis | +| --- | --- | --- | +| `t/blocks.t` | `Range [4, 0) out of bounds for length 61` | Joni matcher offset/capture state | +| `t/control.t` | `Range [7, 0) out of bounds for length 49` | Joni matcher offset/capture state | +| `t/for_ref_iterator.t` | `Range [1, 0) out of bounds for length 43` | Joni matcher offset/capture state | +| `t/format.t` | `Range [1, 0) out of bounds for length 427` | Joni matcher offset/capture state | +| `t/heredoc.t` | second heredoc block does not match; harness then stops advancing | grammar matching / progression state | + +The current baseline also proves that `t/00.load.t`, `t/decomment.t`, +`t/decomment_heredoc_large.t`, `t/document_self.t`, `t/erudil.t`, and +`t/eyedrops.t` passed before the stalled test. `t/disapproval.t` is an +explicit optional-dependency skip. + +## Design Constraints + +- Keep native regex semantics in the Joni fork. Do not special-case PPR source + text, rewrite its patterns, or introduce a Java-regex fallback. +- Reduce each failure into a project-owned test under + `src/test/resources/unit` before its fix is considered complete. +- Run each new Perl-level test on system Perl first, then on JVM and + interpreter backends. Preserve positive and negative controls. +- Treat a timeout or a `0/0` test record as a failure, not a skip. +- Do not modify imported CPAN test files. + +## Work Plan + +### Phase 1: Establish the range-error owner + +1. Reduce the first `t/blocks.t` grammar fragment to a small named-subpattern + match. +2. Capture JVM debug stack traces and compare JVM/interpreter outcomes. +3. Locate the earliest invalid begin/end offset in Joni matcher, capture + publication, or PerlOnJava's byte-to-character conversion. +4. Add the permanent reducer and correct the shared root cause. + +### Phase 2: Complete adjacent grammar failures + +1. Verify whether `control`, `for_ref_iterator`, and `format` share the Phase + 1 root cause. +2. Add distinct reducers for any non-shared behavior. +3. Test both matching and non-matching paths, including nested and empty + recursive paths where relevant. + +### Phase 3: Heredoc correctness and progress + +1. Reduce PPR's second `t/heredoc.t` fixture unchanged in semantic form. +2. Establish the system Perl oracle and locate the first divergent regex stage. +3. Fix the mismatch and any subsequent non-progress condition separately if + they have different causes. + +### Phase 4: Distribution acceptance + +1. Run focused PPR tests on both PerlOnJava backends. +2. Run the complete bounded `./jcpan -t PPR` suite and inspect its full log. +3. Run `make`, scan for warnings, and validate the exact clean candidate head. + +## Progress Tracking + +### Current Status: Phase 1 in progress + +### Completed Phases + +- [x] Phase 0: #1299 load-time analysis hang (2026-09-09) + - Replaced exponential named-subexpression recursion analysis with graph + analysis and retained dynamic-callout safety handling. + - Added `src/test/resources/unit/regex_large_named_grammar.t`. + - Merged in PR #1317. + +### Next Steps + +1. Capture the `t/blocks.t` JVM stack trace and create its smallest valid + project-owned reproducer. +2. Compare the reducer on system Perl, JVM, and interpreter backends. +3. Identify and repair the narrowest shared owner before proceeding to the + remaining PPR test files. + +### Open Questions + +- Are the four range errors one invalid region/capture-publication defect or + multiple matcher defects? +- Does the `t/heredoc.t` stall remain after fixing the range-error family? + +## Related Work + +- Issue #1299: PPR grammar load-time Joni analysis hang. +- Issue #1318: PPR compatibility follow-up and acceptance tracking. +- `.agents/skills/debug-regex-engine/SKILL.md`. From dbfc1ee14a51b04f3dba06db80d58ab3d8ee8925 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Wed, 9 Sep 2026 14:51:31 +0200 Subject: [PATCH 2/8] fix(regex): safely publish recursive unmatched captures Treat unordered native capture offsets as nonparticipating captures and keep recursive-call detection on the active call-frame chain. This clears PPR's initial range-error family while preserving the remaining heredoc work for issue #1318. Generated with Codex (https://openai.com/codex) Co-Authored-By: Codex --- dev/design/ppr-compatibility.md | 28 +++++++++++++------ .../runtime/regex/JoniRegexPattern.java | 15 ++++++++-- .../runtime/regex/JoniRegexPatternTest.java | 7 +++++ .../joni/src/org/joni/ByteCodeMachine.java | 2 +- third_party/joni/src/org/joni/StackEntry.java | 18 ++++++++++++ .../joni/src/org/joni/StackMachine.java | 26 ++++++++--------- 6 files changed, 70 insertions(+), 26 deletions(-) diff --git a/dev/design/ppr-compatibility.md b/dev/design/ppr-compatibility.md index 3ad520b80..ef938b2eb 100644 --- a/dev/design/ppr-compatibility.md +++ b/dev/design/ppr-compatibility.md @@ -76,7 +76,7 @@ explicit optional-dependency skip. ## Progress Tracking -### Current Status: Phase 1 in progress +### Current Status: Phase 2 in progress ### Completed Phases @@ -85,20 +85,30 @@ explicit optional-dependency skip. analysis and retained dynamic-callout safety handling. - Added `src/test/resources/unit/regex_large_named_grammar.t`. - Merged in PR #1317. +- [x] Phase 1: Capture-range publication (2026-09-09) + - Identified Joni's stale nonnegative begin / zero end sentinel as an + unmatched capture in PerlOnJava's adapter. + - Added direct adapter coverage and published it as `undef` rather than an + invalid Java substring range. + - This clears PPR's `blocks`, `control`, `for_ref_iterator`, and `format` + range-error family. ### Next Steps -1. Capture the `t/blocks.t` JVM stack trace and create its smallest valid - project-owned reproducer. -2. Compare the reducer on system Perl, JVM, and interpreter backends. -3. Identify and repair the narrowest shared owner before proceeding to the - remaining PPR test files. +1. Complete Phase 2 by running the remaining range-family PPR tests and + separating any non-shared failures. +2. Reduce the second `t/heredoc.t` fixture, which now reports its expected + assertion failure and then times out in `ByteCodeMachine.opCall`. +3. Compare that reducer on system Perl, JVM, and interpreter backends. +4. Fix the remaining recursive-call execution path without weakening PPR's + grammar or introducing source-specific handling. ### Open Questions -- Are the four range errors one invalid region/capture-publication defect or - multiple matcher defects? -- Does the `t/heredoc.t` stall remain after fixing the range-error family? +- Why does the second heredoc fixture fail before the subsequent recursive-call + execution timeout? +- Does the heredoc execution path need an additional semantic guard distinct + from call-frame lookup performance? ## Related Work diff --git a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java index c3e040c17..08169ac97 100644 --- a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java +++ b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java @@ -53,6 +53,16 @@ /** Sole production adapter from Perl regex operations to the vendored Joni fork. */ final class JoniRegexPattern { + /** + * Joni records an unmatched capture with negative offsets in ordinary + * patterns. Deep recursive patterns can also leave a stale begin offset + * paired with the region's zero end sentinel. Perl exposes both forms as + * an unmatched capture, never as an invalid substring range. + */ + static boolean isParticipatingCapture(int begin, int end) { + return begin >= 0 && end >= begin; + } + record DeferredPropertyFact(String name, String displayName, CharacterPropertyResolver.Context context, int option, int position, boolean negated) {} @@ -1199,8 +1209,7 @@ public String group(int index) { requireMatch(); int begin = index == 0 ? matcher.getBegin() : captures.getBeg(index); int end = index == 0 ? matcher.getEnd() : captures.getEnd(index); - if (begin < 0 || end < 0) return null; - if (index == 0 && begin > end) return null; + if (!JoniRegexPattern.isParticipatingCapture(begin, end)) return null; return input.substring(toCharOffset(begin), toCharOffset(end)); } @@ -1211,7 +1220,7 @@ public String group(String name) { if (physical == null) return group(namedGroupNumber(name)); int begin = matcher.physicalNamedCaptureBegin(physical); int end = matcher.physicalNamedCaptureEnd(physical); - if (begin < 0 || end < 0) return null; + if (!JoniRegexPattern.isParticipatingCapture(begin, end)) return null; return input.substring(toCharOffset(begin), toCharOffset(end)); } diff --git a/src/test/java/org/perlonjava/runtime/regex/JoniRegexPatternTest.java b/src/test/java/org/perlonjava/runtime/regex/JoniRegexPatternTest.java index 26d8bdc13..0bb7552e5 100644 --- a/src/test/java/org/perlonjava/runtime/regex/JoniRegexPatternTest.java +++ b/src/test/java/org/perlonjava/runtime/regex/JoniRegexPatternTest.java @@ -16,6 +16,13 @@ class JoniRegexPatternTest { private static final RegexFlags FLAGS = RegexFlags.fromModifiers("", ""); + @Test + void malformedNativeCaptureRangeIsPublishedAsUnmatched() { + assertTrue(JoniRegexPattern.isParticipatingCapture(4, 4)); + assertFalse(JoniRegexPattern.isParticipatingCapture(-1, -1)); + assertFalse(JoniRegexPattern.isParticipatingCapture(4, 0)); + } + @Test void nativeDynamicCalloutSourceAndMetadataRemainVisible() { String source = "(?{=DYNAMIC:0})"; diff --git a/third_party/joni/src/org/joni/ByteCodeMachine.java b/third_party/joni/src/org/joni/ByteCodeMachine.java index a4c66d0c0..4626418d7 100644 --- a/third_party/joni/src/org/joni/ByteCodeMachine.java +++ b/third_party/joni/src/org/joni/ByteCodeMachine.java @@ -3943,7 +3943,7 @@ private void opReturn() { StackEntry frame = returnFrame(); restoreCallFrameCaptureSnapshot(frame); ip = frame.getCallFrameRetAddr(); - pushReturn(); + pushReturn(frame); } private void opFail() { diff --git a/third_party/joni/src/org/joni/StackEntry.java b/third_party/joni/src/org/joni/StackEntry.java index c53c0afb7..786af5fbb 100644 --- a/third_party/joni/src/org/joni/StackEntry.java +++ b/third_party/joni/src/org/joni/StackEntry.java @@ -21,6 +21,8 @@ class StackEntry { int type; + private int activeCallFrameHead = -1; + private int callFramePreviousHead = -1; private int E1, E2, E3, E4; private Object calloutToken; private ByteCodeMachine.DynamicContinuation dynamicContinuation; @@ -205,6 +207,22 @@ boolean getCallFrameRecursiveVisibility() { return callFrameRecursiveVisibility; } + void setActiveCallFrameHead(int head) { + activeCallFrameHead = head; + } + + int getActiveCallFrameHead() { + return activeCallFrameHead; + } + + void setCallFramePreviousHead(int head) { + callFramePreviousHead = head; + } + + int getCallFramePreviousHead() { + return callFramePreviousHead; + } + /* absent position */ void setAbsentStr(int pos) { E1 = pos; diff --git a/third_party/joni/src/org/joni/StackMachine.java b/third_party/joni/src/org/joni/StackMachine.java index b534095df..94cb567d6 100644 --- a/third_party/joni/src/org/joni/StackMachine.java +++ b/third_party/joni/src/org/joni/StackMachine.java @@ -162,6 +162,7 @@ private final StackEntry ensure1() { if (stk >= stack.length) doubleStack(); StackEntry e = stack[stk]; if (e == null) stack[stk] = e = USE_CEC ? new SCStackEntry() : new StackEntry(); + e.setActiveCallFrameHead(stk == 0 ? -1 : stack[stk - 1].getActiveCallFrameHead()); return e; } @@ -239,6 +240,7 @@ private void push(int type, int pat, int s, int prev, int pkeep) { private final void pushEnsured(int type, int pat) { StackEntry e = stack[stk]; + e.setActiveCallFrameHead(stk == 0 ? -1 : stack[stk - 1].getActiveCallFrameHead()); e.type = type; e.setStatePCode(pat); if (USE_CEC) ((SCStackEntry)e).setStateCheck(0); @@ -436,6 +438,8 @@ protected final void pushNullCheckEnd(int cnum) { protected final void pushCallFrame(int pat, int groupNum, boolean snapshotCaptures, boolean restoreCallerCaptures, boolean recursiveVisibility) { StackEntry e = ensure1(); + e.setCallFramePreviousHead(e.getActiveCallFrameHead()); + e.setActiveCallFrameHead(stk); e.type = CALL_FRAME; e.setCallFrameRetAddr(pat); e.setCallFrameNum(groupNum); @@ -491,26 +495,22 @@ protected final void restoreCallFrameCaptureSnapshot(StackEntry frame) { } protected final boolean isInsideSubexpCall(int groupNum) { - int returned = 0; - for (int i = stk - 1; i >= 0; i--) { - StackEntry e = stack[i]; - if (e.type == RETURN) { - returned++; - } else if (e.type == CALL_FRAME) { - if (returned > 0) { - returned--; - } else if (e.getCallFrameNum() >= 0 - && (groupNum == 0 || e.getCallFrameNum() == groupNum)) { - return true; - } + int callFrame = stk == 0 ? -1 : stack[stk - 1].getActiveCallFrameHead(); + while (callFrame >= 0) { + StackEntry e = stack[callFrame]; + if (e.getCallFrameNum() >= 0 + && (groupNum == 0 || e.getCallFrameNum() == groupNum)) { + return true; } + callFrame = e.getCallFramePreviousHead(); } return false; } - protected final void pushReturn() { + protected final void pushReturn(StackEntry frame) { StackEntry e = ensure1(); e.type = RETURN; + e.setActiveCallFrameHead(frame.getCallFramePreviousHead()); stk++; } From 697b80e20c7727a8c56f0471376e832bdfd528e1 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Wed, 9 Sep 2026 19:24:27 +0200 Subject: [PATCH 3/8] fix(regex): bound pathological backtracking memory Retain PPR's recursive grammar callbacks and capture state while bounding Joni's heap-backed backtracking stack. Exhaustion now follows the existing Perl recursion-exhaustion behavior rather than consuming the JVM heap. Generated with Codex (https://openai.com/codex/) Co-Authored-By: Codex --- .../runtime/regex/JoniRegexPattern.java | 176 +++++++++++++----- .../runtime/regex/RuntimeRegex.java | 22 +++ .../runtimetypes/HashSpecialVariable.java | 44 ++++- .../runtime/runtimetypes/RegexState.java | 6 + .../runtimetypes/RuntimeRegexState.java | 16 ++ .../regex_callout_provisional_capture_state.t | 19 ++ .../joni/src/org/joni/ByteCodeMachine.java | 131 ++++++++++++- third_party/joni/src/org/joni/Config.java | 4 + third_party/joni/src/org/joni/MatchView.java | 12 +- third_party/joni/src/org/joni/StackEntry.java | 17 +- .../joni/src/org/joni/StackMachine.java | 32 ++-- 11 files changed, 391 insertions(+), 88 deletions(-) create mode 100644 src/test/resources/unit/regex_callout_provisional_capture_state.t diff --git a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java index 08169ac97..641025537 100644 --- a/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java +++ b/src/main/java/org/perlonjava/runtime/regex/JoniRegexPattern.java @@ -38,6 +38,7 @@ import java.util.ArrayDeque; import java.util.Iterator; import java.util.ArrayList; +import java.util.Arrays; import java.util.Collections; import java.util.LinkedHashMap; import java.util.Map; @@ -72,6 +73,7 @@ record DeferredPropertyFact(String name, String displayName, "Both or neither range ends should be Unicode"; private static final int INPUT_ENCODING_CACHE_ENTRIES = 512; private static final int INPUT_ENCODING_CACHE_MAX_LENGTH = 8_192; + private static final int DYNAMIC_PATTERN_CACHE_ENTRIES = 128; private static final Map INPUT_ENCODINGS = inputEncodingCache(); private static final Map BYTE_INPUT_ENCODINGS = inputEncodingCache(); private static final Map SUBJECT_INPUT_ENCODINGS = @@ -953,7 +955,8 @@ private boolean find(int option, boolean anchored) { } if (!callbacks.isEmpty()) { calloutHandler = new PerlCalloutHandler( - input, byteToChar, callbacks, flags, hasControlVerbState, byteMode, subject); + input, byteToChar, callbacks, namedGroups, flags, + hasControlVerbState, byteMode, subject); matcher.setCalloutHandler(calloutHandler); } int result; @@ -976,7 +979,7 @@ private boolean find(int option, boolean anchored) { } if (!callbacks.isEmpty()) { calloutHandler = new PerlCalloutHandler( - input, byteToChar, callbacks, flags, + input, byteToChar, callbacks, namedGroups, flags, hasControlVerbState, byteMode, subject); matcher.setCalloutHandler(calloutHandler); } @@ -1319,34 +1322,81 @@ private static int[] buildByteToChar(String input, int byteLength, int[] charToB } private static final class PerlCalloutHandler implements CalloutHandler { + private record DynamicPatternCacheKey(String source, RegexFlags flags, + boolean compileAsBytes, + String userPropertyPackage) {} + private record Token(int localLevel, RegexState regexState, RuntimeScalar previousR, RuntimeScalar result, boolean block, boolean dynamic, CaptureSnapshot previousDynamicView) {} - private record CaptureSnapshot(int position, int[] begins, int[] ends, - int lastClosed, String controlMark) implements MatchView { + /** + * Dynamic callbacks only need to remember captures that were open at + * the callback position. Retaining every offset made large recursive + * grammars copy and resolve all captures twice per callback. + */ + private record CaptureSnapshot(int position, int[] openAtPosition) { + boolean hasOpenCapture(int capture) { + for (int openCapture : openAtPosition) { + if (openCapture == capture) return true; + } + return false; + } + static CaptureSnapshot of(MatchView match) { int count = match.captureCount(); - int[] begins = new int[count + 1]; - int[] ends = new int[count + 1]; - for (int capture = 0; capture <= count; capture++) { - begins[capture] = match.captureBegin(capture); - ends[capture] = match.captureEnd(capture); + int position = match.currentBytePosition(); + int[] directOpenCaptures = match.openCapturesAtCurrentPosition(); + if (directOpenCaptures != null) { + return new CaptureSnapshot(position, directOpenCaptures); + } + int[] openAtPosition = new int[Math.min(count, 8)]; + int openCount = 0; + MatchView.CaptureOffsets[] allOffsets = match.captureOffsets(); + for (int capture = 1; capture <= count; capture++) { + MatchView.CaptureOffsets offsets = allOffsets == null + ? match.captureOffsets(capture) : allOffsets[capture]; + int begin = offsets.begin(); + int end = offsets.end(); + if (begin == position && (end < 0 || end == position)) { + if (openCount == openAtPosition.length) { + openAtPosition = Arrays.copyOf(openAtPosition, + openAtPosition.length << 1); + } + openAtPosition[openCount++] = capture; + } } - return new CaptureSnapshot(match.currentBytePosition(), begins, ends, - match.lastClosedCapture(), match.controlMark()); + return new CaptureSnapshot(position, + openCount == openAtPosition.length ? openAtPosition + : Arrays.copyOf(openAtPosition, openCount)); + } + } + + private record DynamicCaptureView(MatchView current, CaptureSnapshot previous) + implements MatchView { + private boolean restoresPreviousStart(int capture) { + int position = current.currentBytePosition(); + return current.captureBegin(capture) == position + && current.captureEnd(capture) == position + && previous.hasOpenCapture(capture); } - @Override public int currentBytePosition() { return position; } - @Override public int captureCount() { return begins.length - 1; } - @Override public int captureBegin(int capture) { return begins[capture]; } - @Override public int captureEnd(int capture) { return ends[capture]; } - @Override public int lastClosedCapture() { return lastClosed; } + @Override public int currentBytePosition() { return current.currentBytePosition(); } + @Override public int captureCount() { return current.captureCount(); } + @Override public int captureBegin(int capture) { + return restoresPreviousStart(capture) ? previous.position() + : current.captureBegin(capture); + } + @Override public int captureEnd(int capture) { return current.captureEnd(capture); } + @Override public int lastClosedCapture() { return current.lastClosedCapture(); } + @Override public String controlMark() { return current.controlMark(); } } private final String input; private final int[] byteToChar; private final List callbacks; + private final Map namedGroups; + private final Map> provisionalNamedCaptureGroups; private final RegexFlags outerFlags; private final boolean publishesControlVerbState; private final boolean byteMode; @@ -1364,21 +1414,33 @@ static CaptureSnapshot of(MatchView match) { private final ArrayDeque callbackMutations = new ArrayDeque<>(); private boolean preserveCallbackMutations; + private final Map dynamicPatternCache = + new LinkedHashMap<>(16, 0.75f, true) { + @Override + protected boolean removeEldestEntry( + Map.Entry eldest) { + return size() > DYNAMIC_PATTERN_CACHE_ENTRIES; + } + }; PerlCalloutHandler(String input, int[] byteToChar, List callbacks, + Map namedGroups, RegexFlags outerFlags, boolean publishesControlVerbState, boolean byteMode, RuntimeScalar subject) { - this(input, byteToChar, callbacks, outerFlags, publishesControlVerbState, + this(input, byteToChar, callbacks, namedGroups, outerFlags, publishesControlVerbState, byteMode, subject, null); } private PerlCalloutHandler( String input, int[] byteToChar, List callbacks, + Map namedGroups, RegexFlags outerFlags, boolean publishesControlVerbState, boolean byteMode, RuntimeScalar subject, PerlCalloutHandler parent) { this.input = input; this.byteToChar = byteToChar; this.callbacks = callbacks; + this.namedGroups = namedGroups; + this.provisionalNamedCaptureGroups = buildProvisionalNamedCaptureGroups(namedGroups); this.outerFlags = outerFlags; this.publishesControlVerbState = publishesControlVerbState; this.byteMode = byteMode; @@ -1470,11 +1532,17 @@ public DynamicPatternResult executeDynamic( boolean compileAsBytes = byteMode && (byteBackedDynamic || latin1Dynamic); inputEncodingCompatible = !byteMode || latin1Dynamic; - nestedPattern = UnicodeResolver.withUserPropertyPackage( - dynamicPackage, - () -> new JoniRegexPattern(dynamicSource, - scopedFlags, 0, compileAsBytes, - compileAsBytes, compileAsBytes)); + DynamicPatternCacheKey cacheKey = new DynamicPatternCacheKey( + dynamicSource, scopedFlags, compileAsBytes, dynamicPackage); + nestedPattern = dynamicPatternCache.get(cacheKey); + if (nestedPattern == null) { + nestedPattern = UnicodeResolver.withUserPropertyPackage( + dynamicPackage, + () -> new JoniRegexPattern(dynamicSource, + scopedFlags, 0, compileAsBytes, + compileAsBytes, compileAsBytes)); + dynamicPatternCache.put(cacheKey, nestedPattern); + } } catch (SyntaxException exception) { String message = exception.getMessage(); if (message != null && (message.contains("premature end of char-class") @@ -1490,6 +1558,7 @@ public DynamicPatternResult executeDynamic( nestedPattern.materializeDefinedDeferredProperties(); CalloutHandler nestedHandler = nestedCallbacks.isEmpty() ? null : new PerlCalloutHandler(input, byteToChar, nestedCallbacks, + nestedPattern.namedGroups, value.value instanceof RuntimeRegex runtimeRegex && runtimeRegex.getRegexFlags() != null ? runtimeRegex.getRegexFlags() : scopedFlags, @@ -1545,14 +1614,18 @@ private Evaluation evaluate(RuntimeRegexCallback callback, MatchView match) { } } CaptureSnapshot priorDynamicView = previousDynamicView; + boolean dynamicReadsCaptures = callback.kind == RuntimeRegexCallback.Kind.DYNAMIC + && callbackReadsCaptures(callback); if (callback.kind.isBlock() && parent == null) { callbackMutations.addLast(RegexCallbackMutationSnapshot.capture(callback.code)); } - MatchView provisional = callback.kind == RuntimeRegexCallback.Kind.DYNAMIC + MatchView provisional = dynamicReadsCaptures ? dynamicCaptureView(match, priorDynamicView) : match; publishProvisional(provisional); - if (callback.kind == RuntimeRegexCallback.Kind.DYNAMIC) { + if (dynamicReadsCaptures) { previousDynamicView = CaptureSnapshot.of(match); + } else if (callback.kind == RuntimeRegexCallback.Kind.DYNAMIC) { + previousDynamicView = null; } var callbackLocations = PerlRuntime.current().executionState() .activeRegexCallbackLocations; @@ -1619,6 +1692,27 @@ private Evaluation evaluate(RuntimeRegexCallback callback, MatchView match) { } } + private static boolean callbackReadsCaptures(RuntimeRegexCallback callback) { + String source = callback.source; + if (source == null || source.isEmpty()) return false; + return source.matches("(?s).*\\$(?:[0-9]|\\^N|[&`']|[+\\-]\\{|[+\\-](?![A-Za-z_])).*") + || source.contains("@-") || source.contains("@+") + || source.contains("%-") || source.contains("%+"); + } + + private static Map> buildProvisionalNamedCaptureGroups( + Map namedGroups) { + Map> groups = new LinkedHashMap<>(); + for (Map.Entry entry : namedGroups.entrySet()) { + String encodedName = entry.getKey(); + if (CaptureNameEncoder.isInternalCapture(encodedName)) continue; + groups.computeIfAbsent(CaptureNameEncoder.decodeGroupName(encodedName), + ignored -> new ArrayList<>()).add(entry.getValue()); + } + return groups; + } + + @Override public void unwind(Object value) { restore((Token) value, false); @@ -1708,19 +1802,8 @@ private void restoreCallbackMutations() { private static MatchView dynamicCaptureView( MatchView current, CaptureSnapshot previous) { - CaptureSnapshot adjusted = CaptureSnapshot.of(current); - if (previous == null || previous.position() >= adjusted.position()) return adjusted; - for (int capture = 1; capture <= adjusted.captureCount(); capture++) { - if (adjusted.begins()[capture] == adjusted.position() - && adjusted.ends()[capture] == adjusted.position() - && previous.begins()[capture] == previous.position() - && (previous.ends()[capture] < 0 - || previous.ends()[capture] == previous.position())) { - adjusted.begins()[capture] = previous.position(); - adjusted.ends()[capture] = adjusted.position(); - } - } - return adjusted; + if (previous == null || previous.position() >= current.currentBytePosition()) return current; + return new DynamicCaptureView(current, previous); } private static void restoreCallbackScope(int localLevel, RegexState regexState, @@ -1770,22 +1853,21 @@ private void publishProvisional(MatchView match) { state.lastCaptureGroups = new String[count]; state.manualCaptureStarts = new int[count]; state.manualCaptureEnds = new int[count]; - for (int group = 1; group <= count; group++) { + Arrays.fill(state.manualCaptureStarts, Integer.MIN_VALUE); + Arrays.fill(state.manualCaptureEnds, Integer.MIN_VALUE); + state.provisionalCaptureResolver = group -> { int begin = charOffset(match.captureBegin(group)); int end = charOffset(match.captureEnd(group)); if (begin < 0 || end < begin) { - state.manualCaptureStarts[group - 1] = -1; - state.manualCaptureEnds[group - 1] = -1; - state.lastCaptureGroups[group - 1] = null; - } else { - state.manualCaptureStarts[group - 1] = begin; - state.manualCaptureEnds[group - 1] = end; - state.lastCaptureGroups[group - 1] = input.substring(begin, end); + return new RuntimeRegexState.ProvisionalCapture(null, -1, -1); } - } + return new RuntimeRegexState.ProvisionalCapture(input.substring(begin, end), begin, end); + }; + state.lastNamedCaptureGroups = new LinkedHashMap<>(); + state.provisionalNamedCaptureGroups = provisionalNamedCaptureGroups; int lastClosed = match.lastClosedCapture(); state.lastClosedCapture = lastClosed > 0 && lastClosed <= count - ? state.lastCaptureGroups[lastClosed - 1] : null; + ? RuntimeRegex.captureString(lastClosed) : null; if (publishesControlVerbState || match.controlMark() != null) { RuntimeRegex.updateControlVerbVariables(match.controlMark(), null); } diff --git a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java index 9b0a2b14c..675d68e51 100644 --- a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java +++ b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java @@ -3122,6 +3122,7 @@ public static RuntimeBase matchRegexBytes(RuntimeScalar quotedRegex, RuntimeScal private static void updateLastNamedCaptureGroups(RegexMatcher matcher) { RuntimeRegexState regexState = state(); + regexState.provisionalNamedCaptureGroups = null; Map namedGroups = matcher.namedGroups(); Map> byPerlName = new LinkedHashMap<>(); if (namedGroups == null || namedGroups.isEmpty()) { @@ -3159,6 +3160,8 @@ private static void updateNumberedCaptureGroups(RegexMatcher matcher) { regexState.lastParenMatchOverride = null; regexState.manualCaptureStarts = null; regexState.manualCaptureEnds = null; + regexState.provisionalCaptureResolver = null; + regexState.provisionalNamedCaptureGroups = null; int captureCount = matcher.groupCount(); int lastClosedCapture = matcher.lastClosedCapture(); regexState.lastClosedCapture = lastClosedCapture > 0 @@ -4094,6 +4097,7 @@ public static String captureString(int group) { if (state().lastCaptureGroups == null || group > state().lastCaptureGroups.length) { return null; } + materializeProvisionalCapture(group); return state().lastCaptureGroups[group - 1]; } @@ -4105,6 +4109,7 @@ public static String lastCaptureString() { // in the match (i.e., is non-null). Non-participating groups in alternations // have null values from Java's Matcher.group(). for (int i = state().lastCaptureGroups.length - 1; i >= 0; i--) { + materializeProvisionalCapture(i + 1); if (state().lastCaptureGroups[i] != null) { return state().lastCaptureGroups[i]; } @@ -4133,6 +4138,7 @@ public static RuntimeScalar matcherStart(int group) { return publicMatcherOffset(state().lastMatchStart); } if (state().manualCaptureStarts != null && group > 0 && group <= state().manualCaptureStarts.length) { + materializeProvisionalCapture(group); return publicMatcherOffset(state().manualCaptureStarts[group - 1]); } if (state().globalMatcher == null) { @@ -4157,6 +4163,7 @@ public static RuntimeScalar matcherEnd(int group) { return publicMatcherOffset(state().lastMatchEnd); } if (state().manualCaptureEnds != null && group > 0 && group <= state().manualCaptureEnds.length) { + materializeProvisionalCapture(group); return publicMatcherOffset(state().manualCaptureEnds[group - 1]); } if (state().globalMatcher == null) { @@ -4200,6 +4207,21 @@ public static int matcherSize() { return size + 1; } + private static void materializeProvisionalCapture(int group) { + RuntimeRegexState regexState = state(); + RuntimeRegexState.ProvisionalCaptureResolver resolver = + regexState.provisionalCaptureResolver; + if (resolver == null || regexState.manualCaptureStarts == null + || group <= 0 || group > regexState.manualCaptureStarts.length + || regexState.manualCaptureStarts[group - 1] != Integer.MIN_VALUE) { + return; + } + RuntimeRegexState.ProvisionalCapture capture = resolver.resolve(group); + regexState.lastCaptureGroups[group - 1] = capture.value(); + regexState.manualCaptureStarts[group - 1] = capture.start(); + regexState.manualCaptureEnds[group - 1] = capture.end(); + } + /** Perl trims trailing non-participating captures from {@code @-}, but not {@code @+}. */ public static int matcherStartSize() { int size = matcherSize(); diff --git a/src/main/java/org/perlonjava/runtime/runtimetypes/HashSpecialVariable.java b/src/main/java/org/perlonjava/runtime/runtimetypes/HashSpecialVariable.java index 7598ae3ae..39b5485d3 100644 --- a/src/main/java/org/perlonjava/runtime/runtimetypes/HashSpecialVariable.java +++ b/src/main/java/org/perlonjava/runtime/runtimetypes/HashSpecialVariable.java @@ -83,7 +83,7 @@ public static RuntimeHash getStash(String namespace) { public Set> entrySet() { Set> entries = new HashSet<>(); if (this.mode == Id.CAPTURE_ALL || this.mode == Id.CAPTURE) { - Map> namedCaptures = PerlRuntime.current().regexState.lastNamedCaptureGroups; + Map> namedCaptures = namedCaptures(); if (namedCaptures != null) { for (Map.Entry> e : namedCaptures.entrySet()) { if (this.mode == Id.CAPTURE_ALL) { @@ -119,9 +119,8 @@ public Set> entrySet() { @Override public RuntimeScalar get(Object key) { if (this.mode == Id.CAPTURE_ALL || this.mode == Id.CAPTURE) { - Map> namedCaptures = PerlRuntime.current().regexState.lastNamedCaptureGroups; - if (namedCaptures != null && key instanceof String name) { - List captures = namedCaptures.get(name); + if (key instanceof String name) { + List captures = namedCapturesFor(name); if (captures == null) return scalarUndef; if (this.mode == Id.CAPTURE_ALL) { return captureAllArrayRef(captures); @@ -148,15 +147,17 @@ public RuntimeScalar get(Object key) { @Override public boolean containsKey(Object key) { if (this.mode == Id.CAPTURE_ALL) { - // For %-, all named groups exist (even non-participating ones) - Map> namedCaptures = PerlRuntime.current().regexState.lastNamedCaptureGroups; - return namedCaptures != null && key instanceof String name && namedCaptures.containsKey(name); + // For %-, all named groups exist (even non-participating ones). + if (!(key instanceof String name)) return false; + RuntimeRegexState state = PerlRuntime.current().regexState; + return namedCapturesFor(name) != null + || state.provisionalNamedCaptureGroups != null + && state.provisionalNamedCaptureGroups.containsKey(name); } if (this.mode == Id.CAPTURE) { // For %+, only groups that actually captured - Map> namedCaptures = PerlRuntime.current().regexState.lastNamedCaptureGroups; - if (namedCaptures != null && key instanceof String name) { - List captures = namedCaptures.get(name); + if (key instanceof String name) { + List captures = namedCapturesFor(name); return captures != null && captures.stream().anyMatch(v -> v != null); } return false; @@ -168,6 +169,29 @@ public boolean containsKey(Object key) { return super.containsKey(key); } + private Map> namedCaptures() { + RuntimeRegexState state = PerlRuntime.current().regexState; + if (state.provisionalNamedCaptureGroups != null) { + for (String name : state.provisionalNamedCaptureGroups.keySet()) { + namedCapturesFor(name); + } + } + return state.lastNamedCaptureGroups; + } + + private List namedCapturesFor(String name) { + RuntimeRegexState state = PerlRuntime.current().regexState; + List captures = state.lastNamedCaptureGroups == null + ? null : state.lastNamedCaptureGroups.get(name); + if (captures != null || state.provisionalNamedCaptureGroups == null) return captures; + List groups = state.provisionalNamedCaptureGroups.get(name); + if (groups == null) return null; + captures = new ArrayList<>(groups.size()); + for (int group : groups) captures.add(RuntimeRegex.captureString(group)); + state.lastNamedCaptureGroups.put(name, captures); + return captures; + } + @Override public Set keySet() { if (this.mode != Id.STASH) { diff --git a/src/main/java/org/perlonjava/runtime/runtimetypes/RegexState.java b/src/main/java/org/perlonjava/runtime/runtimetypes/RegexState.java index 8aa93d9ff..9c85df25e 100644 --- a/src/main/java/org/perlonjava/runtime/runtimetypes/RegexState.java +++ b/src/main/java/org/perlonjava/runtime/runtimetypes/RegexState.java @@ -26,10 +26,12 @@ public class RegexState implements DynamicState { private final boolean lastParenMatchOverrideActive; private final String lastParenMatchOverride; private final Map> lastNamedCaptureGroups; + private final Map> provisionalNamedCaptureGroups; private final boolean lastMatchWasByteString; private final boolean lastMatchResultsTainted; private final int[] manualCaptureStarts; private final int[] manualCaptureEnds; + private final RuntimeRegexState.ProvisionalCaptureResolver provisionalCaptureResolver; public RegexState() { owner = PerlRuntime.current(); @@ -51,10 +53,12 @@ public RegexState() { lastParenMatchOverrideActive = state.lastParenMatchOverrideActive; lastParenMatchOverride = state.lastParenMatchOverride; lastNamedCaptureGroups = state.lastNamedCaptureGroups; + provisionalNamedCaptureGroups = state.provisionalNamedCaptureGroups; lastMatchWasByteString = state.lastMatchWasByteString; lastMatchResultsTainted = state.lastMatchResultsTainted; manualCaptureStarts = state.manualCaptureStarts; manualCaptureEnds = state.manualCaptureEnds; + provisionalCaptureResolver = state.provisionalCaptureResolver; } public static void save() { @@ -98,9 +102,11 @@ public void dynamicRestoreState() { state.lastParenMatchOverrideActive = lastParenMatchOverrideActive; state.lastParenMatchOverride = lastParenMatchOverride; state.lastNamedCaptureGroups = lastNamedCaptureGroups; + state.provisionalNamedCaptureGroups = provisionalNamedCaptureGroups; state.lastMatchWasByteString = lastMatchWasByteString; state.lastMatchResultsTainted = lastMatchResultsTainted; state.manualCaptureStarts = manualCaptureStarts; state.manualCaptureEnds = manualCaptureEnds; + state.provisionalCaptureResolver = provisionalCaptureResolver; } } diff --git a/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeRegexState.java b/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeRegexState.java index 357667501..9407d7575 100644 --- a/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeRegexState.java +++ b/src/main/java/org/perlonjava/runtime/runtimetypes/RuntimeRegexState.java @@ -18,6 +18,17 @@ * or {@code /o} state.

*/ public final class RuntimeRegexState { + /** + * Resolves a capture while a regex callback is executing. The matcher is + * paused during that callback, so individual captures can be materialized + * only when Perl code actually reads them. + */ + public interface ProvisionalCaptureResolver { + ProvisionalCapture resolve(int group); + } + + public record ProvisionalCapture(String value, int start, int end) {} + public static final int MAX_REGEX_CACHE_SIZE = 1000; static final int MAX_POSITION_CACHE_SIZE = 1000; @@ -38,10 +49,13 @@ public final class RuntimeRegexState { public boolean lastParenMatchOverrideActive; public String lastParenMatchOverride; public Map> lastNamedCaptureGroups; + /** Named groups available from a paused callback matcher, by Perl name. */ + public Map> provisionalNamedCaptureGroups; public boolean lastMatchWasByteString; public boolean lastMatchResultsTainted; public int[] manualCaptureStarts; public int[] manualCaptureEnds; + public ProvisionalCaptureResolver provisionalCaptureResolver; /** Per-runtime locale publication used by matcher-time /l resolution. */ public final RuntimeLocaleState localeState = new RuntimeLocaleState(); @@ -109,10 +123,12 @@ public void clearMatchState() { lastParenMatchOverrideActive = false; lastParenMatchOverride = null; lastNamedCaptureGroups = null; + provisionalNamedCaptureGroups = null; lastMatchWasByteString = false; lastMatchResultsTainted = false; manualCaptureStarts = null; manualCaptureEnds = null; + provisionalCaptureResolver = null; } /** diff --git a/src/test/resources/unit/regex_callout_provisional_capture_state.t b/src/test/resources/unit/regex_callout_provisional_capture_state.t new file mode 100644 index 000000000..9e5c9edb7 --- /dev/null +++ b/src/test/resources/unit/regex_callout_provisional_capture_state.t @@ -0,0 +1,19 @@ +use strict; +use warnings; +use Test::More; +use re 'eval'; + +my ($named, $numbered, $start, $end); +ok 'ok' =~ /(?ok)(?{ + $named = $+{word}; + $numbered = $1; + $start = $-[1]; + $end = $+[1]; +})/, 'dynamic callback matches'; + +is $named, 'ok', 'dynamic callback sees its named capture'; +is $numbered, 'ok', 'dynamic callback sees its numbered capture'; +is $start, 0, 'dynamic callback sees capture start'; +is $end, 2, 'dynamic callback sees capture end'; + +done_testing; diff --git a/third_party/joni/src/org/joni/ByteCodeMachine.java b/third_party/joni/src/org/joni/ByteCodeMachine.java index 4626418d7..5ab2ddc6e 100644 --- a/third_party/joni/src/org/joni/ByteCodeMachine.java +++ b/third_party/joni/src/org/joni/ByteCodeMachine.java @@ -3458,12 +3458,7 @@ private void opCall() { } private int activeSubexpCallDepth() { - int depth = 0; - for (int i = 0; i < stk; i++) { - if (stack[i].type == CALL_FRAME) depth++; - else if (stack[i].type == RETURN) depth--; - } - return depth; + return stk == 0 ? 0 : stack[stk - 1].getActiveCallDepth(); } private void opCallout() { @@ -3756,6 +3751,113 @@ public int captureEnd(int capture) { return (bsAt(regex.btMemEnd, capture) ? stack[value].getMemPStr() : value) - str; } + @Override + public MatchView.CaptureOffsets captureOffsets(int capture) { + checkCapture(capture); + if (capture == 0) return new MatchView.CaptureOffsets(sstart - str, s - str); + // A completed recursive call has caller-snapshot visibility rules; + // retain the established individual resolver in that representation. + if (completedRecursiveCall() != null) { + return new MatchView.CaptureOffsets(captureBegin(capture), captureEnd(capture)); + } + int begin = repeatStk[memStartStk + capture]; + int end = repeatStk[memEndStk + capture]; + if (end == INVALID_INDEX && begin != INVALID_INDEX) { + CapturePointers previous = previousClosedCapturePointers(capture); + if (previous.begin != INVALID_INDEX) begin = previous.begin; + if (previous.end != INVALID_INDEX) end = previous.end; + } + int beginOffset = begin == INVALID_INDEX ? INVALID_INDEX + : (bsAt(regex.btMemStart, capture) ? stack[begin].getMemPStr() : begin) - str; + int endOffset = end == INVALID_INDEX ? INVALID_INDEX + : (bsAt(regex.btMemEnd, capture) ? stack[end].getMemPStr() : end) - str; + return new MatchView.CaptureOffsets(beginOffset, endOffset); + } + + @Override + public MatchView.CaptureOffsets[] captureOffsets() { + if (completedRecursiveCall() != null) return null; + int count = regex.numMem; + int[] previousStarts = new int[count + 1]; + int[] previousEnds = new int[count + 1]; + java.util.Arrays.fill(previousStarts, INVALID_INDEX); + java.util.Arrays.fill(previousEnds, INVALID_INDEX); + boolean[] needsPrevious = new boolean[count + 1]; + int unresolvedPrevious = 0; + for (int capture = 1; capture <= count; capture++) { + needsPrevious[capture] = repeatStk[memEndStk + capture] == INVALID_INDEX + && repeatStk[memStartStk + capture] != INVALID_INDEX; + if (needsPrevious[capture]) unresolvedPrevious++; + } + for (int i = stk - 1; i >= 0; i--) { + StackEntry entry = stack[i]; + int capture = entry.type == MEM_END || entry.type == MEM_START + ? entry.getMemNum() : -1; + if (capture <= 0 || capture > count || !needsPrevious[capture]) continue; + if (entry.type == MEM_END && previousEnds[capture] == INVALID_INDEX) { + previousEnds[capture] = i; + } else if (entry.type == MEM_START && previousEnds[capture] != INVALID_INDEX + && previousStarts[capture] == INVALID_INDEX) { + previousStarts[capture] = i; + if (--unresolvedPrevious == 0) break; + } + } + MatchView.CaptureOffsets[] offsets = new MatchView.CaptureOffsets[count + 1]; + offsets[0] = new MatchView.CaptureOffsets(sstart - str, s - str); + for (int capture = 1; capture <= count; capture++) { + int begin = repeatStk[memStartStk + capture]; + int end = repeatStk[memEndStk + capture]; + if (previousStarts[capture] != INVALID_INDEX) begin = previousStarts[capture]; + if (previousEnds[capture] != INVALID_INDEX) end = previousEnds[capture]; + int beginOffset = begin == INVALID_INDEX ? INVALID_INDEX + : (bsAt(regex.btMemStart, capture) ? stack[begin].getMemPStr() : begin) - str; + int endOffset = end == INVALID_INDEX ? INVALID_INDEX + : (bsAt(regex.btMemEnd, capture) ? stack[end].getMemPStr() : end) - str; + offsets[capture] = new MatchView.CaptureOffsets(beginOffset, endOffset); + } + return offsets; + } + + @Override + public int[] openCapturesAtCurrentPosition() { + if (completedRecursiveCall() != null) return null; + int position = s - str; + int[] open = new int[Math.min(regex.numMem, 8)]; + int openCount = 0; + for (int capture = 1; capture <= regex.numMem; capture++) { + int begin = repeatStk[memStartStk + capture]; + int end = repeatStk[memEndStk + capture]; + if (end == INVALID_INDEX && begin != INVALID_INDEX + && bsAt(regex.btMemStart, capture)) { + // The active MEM_START records the visible capture state that + // preceded this open iteration. It is the same state the + // general resolver finds by walking back to the prior matched + // MEM_END/MEM_START pair, but is available in constant time. + StackEntry activeStart = stack[begin]; + if (activeStart.type == MEM_START + && activeStart.getMemEnd() != INVALID_INDEX) { + begin = activeStart.getMemStart(); + end = activeStart.getMemEnd(); + } + } + int beginOffset = capturePointerOffset(capture, begin, true); + int endOffset = capturePointerOffset(capture, end, false); + if (beginOffset == position && (endOffset < 0 || endOffset == position)) { + if (openCount == open.length) { + open = java.util.Arrays.copyOf(open, open.length << 1); + } + open[openCount++] = capture; + } + } + return openCount == open.length ? open : java.util.Arrays.copyOf(open, openCount); + } + + private int capturePointerOffset(int capture, int pointer, boolean begin) { + if (pointer == INVALID_INDEX) return INVALID_INDEX; + return (bsAt(begin ? regex.btMemStart : regex.btMemEnd, capture) + ? stack[pointer].getMemPStr() : pointer) - str; + } + private int committedCaptureOffset(int capture, boolean begin) { CompletedRecursiveCall completed = completedRecursiveCall(); if (completed == null || msaRegion == null || isFindLongest(regex.options | msaOptions) @@ -3889,6 +3991,23 @@ private int previousClosedCapturePointer(int capture, boolean begin) { return INVALID_INDEX; } + private record CapturePointers(int begin, int end) {} + + private CapturePointers previousClosedCapturePointers(int capture) { + int endPointer = INVALID_INDEX; + for (int i = stk - 1; i >= 0; i--) { + StackEntry entry = stack[i]; + if (endPointer == INVALID_INDEX) { + if (entry.type == MEM_END && entry.getMemNum() == capture) { + endPointer = i; + } + } else if (entry.type == MEM_START && entry.getMemNum() == capture) { + return new CapturePointers(i, endPointer); + } + } + return new CapturePointers(INVALID_INDEX, endPointer); + } + private boolean captureClosedAfterReturn(int capture, int returnIndex) { for (int i = stk - 1; i > returnIndex; i--) { StackEntry entry = stack[i]; diff --git a/third_party/joni/src/org/joni/Config.java b/third_party/joni/src/org/joni/Config.java index 849e4fe08..1a58c7395 100644 --- a/third_party/joni/src/org/joni/Config.java +++ b/third_party/joni/src/org/joni/Config.java @@ -60,6 +60,10 @@ public interface Config extends org.jcodings.Config { boolean USE_QTFR_PEEK_NEXT = ConfigSupport.getBoolean("joni.use_qtfr_peek_next", true); int INIT_MATCH_STACK_SIZE = ConfigSupport.getInt("joni.init_match_stack_size", 64); + // The backtracking stack is heap-backed. Keep an explicit ceiling so a + // pathological failed match is reported through Perl's existing recursion + // exhaustion path instead of exhausting the whole JVM heap. + int MAX_MATCH_STACK_SIZE = ConfigSupport.getInt("joni.max_match_stack_size", 1 << 20); boolean OPTIMIZE = ConfigSupport.getBoolean("joni.optimize", true); @Deprecated boolean DONT_OPTIMIZE = !OPTIMIZE; diff --git a/third_party/joni/src/org/joni/MatchView.java b/third_party/joni/src/org/joni/MatchView.java index 9a1f6a143..da7024dc1 100644 --- a/third_party/joni/src/org/joni/MatchView.java +++ b/third_party/joni/src/org/joni/MatchView.java @@ -18,9 +18,9 @@ * SOFTWARE. */ package org.joni; - /** Read-only provisional matcher state, valid only during a callout. */ public interface MatchView { + record CaptureOffsets(int begin, int end) {} int currentBytePosition(); int captureCount(); @@ -29,6 +29,16 @@ public interface MatchView { int captureEnd(int capture); + default CaptureOffsets captureOffsets(int capture) { + return new CaptureOffsets(captureBegin(capture), captureEnd(capture)); + } + + /** All capture offsets, or {@code null} when no bulk view is available. */ + default CaptureOffsets[] captureOffsets() { return null; } + + /** Capture IDs visibly open at the current byte position, or {@code null}. */ + default int[] openCapturesAtCurrentPosition() { return null; } + /** Number of the most recently closed active capture, or -1 if none. */ int lastClosedCapture(); diff --git a/third_party/joni/src/org/joni/StackEntry.java b/third_party/joni/src/org/joni/StackEntry.java index 786af5fbb..757d072c1 100644 --- a/third_party/joni/src/org/joni/StackEntry.java +++ b/third_party/joni/src/org/joni/StackEntry.java @@ -22,6 +22,7 @@ class StackEntry { int type; private int activeCallFrameHead = -1; + private int activeCallDepth; private int callFramePreviousHead = -1; private int E1, E2, E3, E4; private Object calloutToken; @@ -181,13 +182,6 @@ void setCallFrameNum(int num) { int getCallFrameNum() { return E2; } - /* string position */ - void setCallFramePStr(int pstr) { - E3 = pstr; - } - int getCallFramePStr() { - return E3; - } void setCallFrameCaptureSnapshot(int[] snapshot) { callFrameCaptureSnapshot = snapshot; } @@ -215,6 +209,15 @@ int getActiveCallFrameHead() { return activeCallFrameHead; } + void setActiveCallDepth(int depth) { + activeCallDepth = depth; + } + + int getActiveCallDepth() { + return activeCallDepth; + } + + void setCallFramePreviousHead(int head) { callFramePreviousHead = head; } diff --git a/third_party/joni/src/org/joni/StackMachine.java b/third_party/joni/src/org/joni/StackMachine.java index 94cb567d6..72e3e491a 100644 --- a/third_party/joni/src/org/joni/StackMachine.java +++ b/third_party/joni/src/org/joni/StackMachine.java @@ -135,7 +135,11 @@ protected final void leaveMatcherExecution() { } private void doubleStack() { - StackEntry[] newStack = new StackEntry[stack.length << 1]; + if (stack.length >= Config.MAX_MATCH_STACK_SIZE) { + throw new StackOverflowError("regex match stack limit exceeded"); + } + int newLength = Math.min(stack.length << 1, Config.MAX_MATCH_STACK_SIZE); + StackEntry[] newStack = new StackEntry[newLength]; System.arraycopy(stack, 0, newStack, 0, stack.length); stack = newStack; } @@ -163,6 +167,7 @@ private final StackEntry ensure1() { StackEntry e = stack[stk]; if (e == null) stack[stk] = e = USE_CEC ? new SCStackEntry() : new StackEntry(); e.setActiveCallFrameHead(stk == 0 ? -1 : stack[stk - 1].getActiveCallFrameHead()); + e.setActiveCallDepth(stk == 0 ? 0 : stack[stk - 1].getActiveCallDepth()); return e; } @@ -241,6 +246,7 @@ private void push(int type, int pat, int s, int prev, int pkeep) { private final void pushEnsured(int type, int pat) { StackEntry e = stack[stk]; e.setActiveCallFrameHead(stk == 0 ? -1 : stack[stk - 1].getActiveCallFrameHead()); + e.setActiveCallDepth(stk == 0 ? 0 : stack[stk - 1].getActiveCallDepth()); e.type = type; e.setStatePCode(pat); if (USE_CEC) ((SCStackEntry)e).setStateCheck(0); @@ -440,6 +446,7 @@ protected final void pushCallFrame(int pat, int groupNum, boolean snapshotCaptur StackEntry e = ensure1(); e.setCallFramePreviousHead(e.getActiveCallFrameHead()); e.setActiveCallFrameHead(stk); + e.setActiveCallDepth(e.getActiveCallDepth() + 1); e.type = CALL_FRAME; e.setCallFrameRetAddr(pat); e.setCallFrameNum(groupNum); @@ -511,6 +518,7 @@ protected final void pushReturn(StackEntry frame) { StackEntry e = ensure1(); e.type = RETURN; e.setActiveCallFrameHead(frame.getCallFramePreviousHead()); + e.setActiveCallDepth(frame.getActiveCallDepth() - 1); stk++; } @@ -1027,21 +1035,11 @@ protected final int sreturn() { } protected final StackEntry returnFrame() { - int level = 0; - int k = stk; - while (true) { - k--; - StackEntry e = stack[k]; - - if (e.type == CALL_FRAME) { - if (level == 0) { - return e; - } else { - level--; - } - } else if (e.type == RETURN) { - level++; - } - } + // Every stack entry inherits the active call-frame head when pushed; + // RETURN entries explicitly advance it to the caller. Looking it up + // directly avoids rescanning deep recursive grammar stacks at every + // subexpression return. + int callFrame = stack[stk - 1].getActiveCallFrameHead(); + return stack[callFrame]; } } From d034f04253fe8fa2a1c0a1638192d130c86e8ded Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Thu, 10 Sep 2026 02:42:49 +0200 Subject: [PATCH 4/8] fix(regex): complete PPR compatibility Correct recursive capture and nullable-call behavior in Joni, preserve callback-bearing regexes during empty-pattern reuse, and cover the resulting PPR grammar paths with focused regressions. Generated with Codex (https://openai.com/codex/) Co-Authored-By: Codex --- dev/design/ppr-compatibility.md | 25 ++--- docs/about/changelog.md | 3 + .../perlonjava/frontend/parser/Parser.java | 14 +++ .../runtime/regex/RuntimeRegex.java | 17 +++- .../runtimetypes/ErrorMessageUtil.java | 25 ++++- .../regex/duplicate_named_backref_selection.t | 18 ++++ .../unit/regex/empty_pattern_callback_reuse.t | 13 +++ .../recursive_false_condition_empty_branch.t | 21 ++++ .../unit/regex/recursive_nullable_repeat.t | 16 ++++ ...egex_callback_immediate_fail_side_effect.t | 11 +++ .../unit/syntax_error_colon_context.t | 9 ++ .../unit/unmatched_right_curly_eval.t | 10 ++ .../joni/src/org/joni/ArrayCompiler.java | 8 ++ .../joni/src/org/joni/ByteCodeMachine.java | 96 ++++++++----------- .../joni/src/org/joni/ByteCodePrinter.java | 14 +++ third_party/joni/src/org/joni/Regex.java | 14 ++- .../joni/src/org/joni/StackMachine.java | 30 +++++- 17 files changed, 270 insertions(+), 74 deletions(-) create mode 100644 src/test/resources/unit/regex/duplicate_named_backref_selection.t create mode 100644 src/test/resources/unit/regex/empty_pattern_callback_reuse.t create mode 100644 src/test/resources/unit/regex/recursive_false_condition_empty_branch.t create mode 100644 src/test/resources/unit/regex/recursive_nullable_repeat.t create mode 100644 src/test/resources/unit/regex_callback_immediate_fail_side_effect.t create mode 100644 src/test/resources/unit/syntax_error_colon_context.t create mode 100644 src/test/resources/unit/unmatched_right_curly_eval.t diff --git a/dev/design/ppr-compatibility.md b/dev/design/ppr-compatibility.md index ef938b2eb..4a409f685 100644 --- a/dev/design/ppr-compatibility.md +++ b/dev/design/ppr-compatibility.md @@ -76,7 +76,7 @@ explicit optional-dependency skip. ## Progress Tracking -### Current Status: Phase 2 in progress +### Current Status: Complete (2026-09-10) ### Completed Phases @@ -92,23 +92,24 @@ explicit optional-dependency skip. invalid Java substring range. - This clears PPR's `blocks`, `control`, `for_ref_iterator`, and `format` range-error family. +- [x] Phases 2–4: Recursive grammar execution and distribution acceptance (2026-09-10) + - Corrected nullable recursive-call empty checks, recursive capture + restoration for duplicate named groups, and duplicate-name backreference + selection in Joni. + - Kept callback-bearing regex programs intact for empty-pattern reuse with + changed modifiers, allowing PPR's unpunctuated JAPH to execute. + - Added focused coverage in `src/test/resources/unit/regex/` and validated + it on system Perl plus both PerlOnJava backends. + - `timeout 1800 ./jcpan -t PPR` passes: 75 files, 1,255 tests. ### Next Steps -1. Complete Phase 2 by running the remaining range-family PPR tests and - separating any non-shared failures. -2. Reduce the second `t/heredoc.t` fixture, which now reports its expected - assertion failure and then times out in `ByteCodeMachine.opCall`. -3. Compare that reducer on system Perl, JVM, and interpreter backends. -4. Fix the remaining recursive-call execution path without weakening PPR's - grammar or introducing source-specific handling. +1. Monitor the PR checks and review feedback. +2. Keep PPR in the CPAN compatibility acceptance rotation. ### Open Questions -- Why does the second heredoc fixture fail before the subsequent recursive-call - execution timeout? -- Does the heredoc execution path need an additional semantic guard distinct - from call-frame lookup performance? +- None for the current PPR acceptance scope. ## Related Work diff --git a/docs/about/changelog.md b/docs/about/changelog.md index 1a7162550..d8432b09c 100644 --- a/docs/about/changelog.md +++ b/docs/about/changelog.md @@ -26,6 +26,9 @@ priorities and future plans. - Fixed large dynamic named-subexpression grammars hanging during regex compilation. +- Restore PPR's complete suite by correcting recursive duplicate-name captures, + nullable recursion checks, and callback regex reuse. + - Preserve Data::Dumper's pure-Perl numeric-string behavior for Test::Differences, including copied `qw` values and numeric zero fixtures. diff --git a/src/main/java/org/perlonjava/frontend/parser/Parser.java b/src/main/java/org/perlonjava/frontend/parser/Parser.java index 5f3fafc75..44864ffa3 100644 --- a/src/main/java/org/perlonjava/frontend/parser/Parser.java +++ b/src/main/java/org/perlonjava/frontend/parser/Parser.java @@ -223,6 +223,20 @@ public Node parse() { } finally { compilationState.unitcheckQueueStack.get().pop(); } + // ParseBlock stops before a closing brace so callers parsing a nested + // block can consume it. At file/eval scope there is no such caller: + // leaving it accepted makes `eval 'sub {} }'` silently compile. + LexerToken remaining = TokenUtils.peek(this); + if (remaining.type == LexerTokenType.OPERATOR && "}".equals(remaining.text)) { + ErrorMessageUtil.SourceLocation loc = ctx.errorUtil + .getSourceLocationAccurate(tokenIndex); + String message = "Unmatched right curly bracket at " + loc.fileName() + + " line " + loc.lineNumber() + ", at end of line\n" + + ctx.errorUtil.errorMessage(tokenIndex, "syntax error") + + "Execution of " + loc.fileName() + + " aborted due to compilation errors.\n"; + throw new PerlCompilerException(message); + } // Mark the AST as a top-level file block for proper bare block return value handling // This annotation is checked in EmitBlock to handle RUNTIME context bare blocks if (!isTopLevelScript && ast instanceof AbstractNode) { diff --git a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java index 675d68e51..19b77736d 100644 --- a/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java +++ b/src/main/java/org/perlonjava/runtime/regex/RuntimeRegex.java @@ -2499,7 +2499,8 @@ && scanExecutableSource(sourcePattern, extendedSource, true, true).executable(); if (executableSource || unterminatedClassExecutableCandidate) { if (RuntimeRegexSourceCompiler.isCompilingRuntimeSource() - && unterminatedClassExecutableCandidate) { + && unterminatedClassExecutableCandidate + && hasInitiallyClosedCharacterClass(sourcePattern)) { String recursionDiagnostic = unterminatedExecutableSequence( sourcePattern); if (recursionDiagnostic != null) { @@ -2514,6 +2515,7 @@ && scanExecutableSource(sourcePattern, extendedSource, return RuntimeRegexSourceCompiler.compile( patternString, rawModifierStr, executableSource && eagerInitialClassExecutableCandidate + && hasInitiallyClosedCharacterClass(sourcePattern) ? unterminatedExecutableSequence(sourcePattern) : null, !executableSource || sourcePolicy.admitRuntimeEval()) @@ -2547,6 +2549,12 @@ static boolean containsExecutableSource(String pattern, boolean extended) { false, false).executable(); } + /** Diagnostic used when malformed synthetic source attempts to re-enter itself. */ + private static boolean hasInitiallyClosedCharacterClass(String pattern) { + return pattern != null + && (pattern.contains("[](?{") || pattern.contains("[^](?{")); + } + /** Diagnostic used when malformed synthetic source attempts to re-enter itself. */ private static String unterminatedExecutableSequence(String pattern) { int dynamic = pattern == null ? -1 : pattern.lastIndexOf("(??{"); @@ -3191,7 +3199,12 @@ private static RuntimeRegex emptyPatternReuse(RuntimeRegex previous, boolean lexicalReStrict, RuntimeScalar replacement) { RuntimeRegex reused; - if (previous != null && (flags == null || flags.equals(previous.regexFlags))) { + if (previous != null && (flags == null || flags.equals(previous.regexFlags) + // Runtime executable callbacks are compiled into the native + // program, not recoverable from the marker-bearing source. + // Empty-pattern reuse must therefore retain that program even + // when the new operation supplies modifiers. + || !previous.executableCallbacks.isEmpty())) { reused = previous.cloneTracked(); } else { String source = previous == null ? "" : previous.patternString; diff --git a/src/main/java/org/perlonjava/runtime/runtimetypes/ErrorMessageUtil.java b/src/main/java/org/perlonjava/runtime/runtimetypes/ErrorMessageUtil.java index 7f1206c0a..7a71b1f2e 100644 --- a/src/main/java/org/perlonjava/runtime/runtimetypes/ErrorMessageUtil.java +++ b/src/main/java/org/perlonjava/runtime/runtimetypes/ErrorMessageUtil.java @@ -362,10 +362,30 @@ private String buildNearString(int index, String message) { } } + // Perl includes the completed operand immediately before a stray + // colon in its syntax context (for example, `near "2:"`), rather + // than starting the excerpt at the unexpected punctuation. + int start = index; + boolean trimTrailingWhitespace = false; + if ("syntax error".equals(message) + && index > 0 + && ":".equals(tokens.get(index).text)) { + int previous = index - 1; + while (previous >= 0 + && tokens.get(previous).type == LexerTokenType.WHITESPACE) { + previous--; + } + if (previous >= 0 + && tokens.get(previous).type != LexerTokenType.NEWLINE) { + start = previous; + trimTrailingWhitespace = true; + } + } + int end = Math.min(tokens.size() - 1, index + 5); StringBuilder sb = new StringBuilder(); int nonWsCount = 0; - for (int i = index; i <= end; i++) { + for (int i = start; i <= end; i++) { LexerToken tok = tokens.get(i); if (tok.type == LexerTokenType.EOF || tok.type == LexerTokenType.NEWLINE) break; if (tok.text.equals("{") || tok.text.equals("}")) break; @@ -377,6 +397,9 @@ private String buildNearString(int index, String message) { } String near = sb.toString(); near = near.replaceAll("^\\s+", ""); + if (trimTrailingWhitespace) { + near = near.replaceAll("\\s+$", ""); + } return near; } diff --git a/src/test/resources/unit/regex/duplicate_named_backref_selection.t b/src/test/resources/unit/regex/duplicate_named_backref_selection.t new file mode 100644 index 000000000..e734ad6dd --- /dev/null +++ b/src/test/resources/unit/regex/duplicate_named_backref_selection.t @@ -0,0 +1,18 @@ +use strict; +use warnings; +use Test::More; + +my $pattern = qr/\A(?a)(?b)\k\z/; + +ok('aba' =~ $pattern, + 'duplicate named backreference uses the oldest participating capture'); +ok('abb' !~ $pattern, + 'duplicate named backreference does not retry a newer capture'); + +my $case_insensitive = qr/\A(?a)(?b)\k\z/i; +ok('aBa' =~ $case_insensitive, + 'case-insensitive duplicate named backreference uses the oldest capture'); +ok('aBB' !~ $case_insensitive, + 'case-insensitive duplicate named backreference does not retry a newer capture'); + +done_testing; diff --git a/src/test/resources/unit/regex/empty_pattern_callback_reuse.t b/src/test/resources/unit/regex/empty_pattern_callback_reuse.t new file mode 100644 index 000000000..aa2699a8d --- /dev/null +++ b/src/test/resources/unit/regex/empty_pattern_callback_reuse.t @@ -0,0 +1,13 @@ +use strict; +use warnings; +use re 'eval'; +use Test::More; + +my $callback_pattern = qr/(?{ 1 })x/; +ok 'x' =~ $callback_pattern, 'callback-bearing pattern matches'; + +my $result = eval q{'x' =~ s//y/r}; +is $@, '', 'empty-pattern reuse recompiles without an error'; +is $result, 'y', 'empty-pattern reuse retains the prior callback regex'; + +done_testing; diff --git a/src/test/resources/unit/regex/recursive_false_condition_empty_branch.t b/src/test/resources/unit/regex/recursive_false_condition_empty_branch.t new file mode 100644 index 000000000..2aac957bf --- /dev/null +++ b/src/test/resources/unit/regex/recursive_false_condition_empty_branch.t @@ -0,0 +1,21 @@ +use strict; +use warnings; +use Test::More tests => 4; + +our $error = 'previous parse error'; + +my $grammar = qr{ + (?(DEFINE) + (? (?&statement)+ ) + (? + a + | (?(?{ !defined $error }) b (?!)) + ) + ) + \A (?&document) \z +}x; + +ok('a' =~ $grammar, 'one non-empty statement matches after failed condition'); +ok('aa' =~ $grammar, 'two non-empty statements match after failed condition'); +ok('aaa' =~ $grammar, 'three non-empty statements match after failed condition'); +ok('b' !~ $grammar, 'failed condition does not admit its branch'); diff --git a/src/test/resources/unit/regex/recursive_nullable_repeat.t b/src/test/resources/unit/regex/recursive_nullable_repeat.t new file mode 100644 index 000000000..c3df15825 --- /dev/null +++ b/src/test/resources/unit/regex/recursive_nullable_repeat.t @@ -0,0 +1,16 @@ +use strict; +use warnings; + +my $grammar = qr{ + (?(DEFINE) + (? (?&statement)+ ) + (? a | ) + ) + \A (?&document) \z +}x; + +for my $source ('a', 'aa', 'aaa') { + print $source =~ $grammar + ? "ok - nullable recursive repeat matches $source\n" + : "not ok - nullable recursive repeat rejects $source\n"; +} diff --git a/src/test/resources/unit/regex_callback_immediate_fail_side_effect.t b/src/test/resources/unit/regex_callback_immediate_fail_side_effect.t new file mode 100644 index 000000000..c3f91382f --- /dev/null +++ b/src/test/resources/unit/regex_callback_immediate_fail_side_effect.t @@ -0,0 +1,11 @@ +use strict; +use warnings; +use Test::More; + +my $value = 1; +ok !('a' =~ /a(?{ $value = 3 })(?!)/), + 'an immediate negative assertion fails the match'; +is $value, 3, + 'callback assignment survives an immediate zero-width failure'; + +done_testing; diff --git a/src/test/resources/unit/syntax_error_colon_context.t b/src/test/resources/unit/syntax_error_colon_context.t new file mode 100644 index 000000000..c0cbd53cd --- /dev/null +++ b/src/test/resources/unit/syntax_error_colon_context.t @@ -0,0 +1,9 @@ +use strict; +use warnings; +use Test::More; + +my $error = eval q{sub example { my $value = 2: }}; +ok !defined $error, 'malformed source does not compile'; +like $@, qr/syntax error.*near "2:"/, 'syntax diagnostic retains operand before colon'; + +done_testing; diff --git a/src/test/resources/unit/unmatched_right_curly_eval.t b/src/test/resources/unit/unmatched_right_curly_eval.t new file mode 100644 index 000000000..a7be5f3ac --- /dev/null +++ b/src/test/resources/unit/unmatched_right_curly_eval.t @@ -0,0 +1,10 @@ +use strict; +use warnings; +use Test::More; + +my $compiled = eval q!sub example { } }!; +ok !defined $compiled, 'unmatched right curly does not compile'; +like $@, qr/^Unmatched right curly bracket at /, + 'eval reports an unmatched right curly diagnostic'; + +done_testing; diff --git a/third_party/joni/src/org/joni/ArrayCompiler.java b/third_party/joni/src/org/joni/ArrayCompiler.java index e5a15a291..8c847877b 100644 --- a/third_party/joni/src/org/joni/ArrayCompiler.java +++ b/third_party/joni/src/org/joni/ArrayCompiler.java @@ -255,6 +255,14 @@ private int selectStrOpcode(int mbLength, int byteLength, boolean ignoreCase) { private void compileTreeEmptyCheck(Node node, int emptyInfo) { int savedNumNullCheck = regex.numNullCheck; + // A subexpression call can update its captures through an empty + // alternative without consuming input. Repeating that call must stop + // on input position alone; the capture-sensitive guard would otherwise + // keep revisiting the same empty call indefinitely. + if (node.getType() == NodeType.CALL && emptyInfo == TargetInfo.IS_EMPTY_MEM) { + emptyInfo = TargetInfo.IS_EMPTY; + } + if (emptyInfo != 0) { regex.requireStack = true; addOpcode(OPCode.NULL_CHECK_START); diff --git a/third_party/joni/src/org/joni/ByteCodeMachine.java b/third_party/joni/src/org/joni/ByteCodeMachine.java index 5ab2ddc6e..14467211b 100644 --- a/third_party/joni/src/org/joni/ByteCodeMachine.java +++ b/third_party/joni/src/org/joni/ByteCodeMachine.java @@ -2900,70 +2900,50 @@ private void opBackRefPreviousIC() { } private void opBackRefMulti() { - int tlen = code[ip++]; - - int i; - loop:for (i=0; i range) continue; + int mem = selectBackrefMultiCapture(); + if (mem < 0) {opFail(); return;} + backref(mem); + } - sprev = s; - int swork = s; + private void opBackRefMultiIC() { + int mem = selectBackrefMultiCapture(); + if (mem < 0) {opFail(); return;} - while (n-- > 0) { - if (bytes[pstart++] != bytes[swork++]) continue loop; - } + int pstart = backrefStart(mem); + int pend = backrefEnd(mem); + int n = pend - pstart; + if (s + n > range) {opFail(); return;} - s = swork; + sprev = s; + value = s; + if (!backrefStringCmpIC(currentCaseFoldFlag(), pstart, this, n, end)) { + opFail(); + return; + } + s = value; + if (sprev < range) { int len; - - // beyond string check - if (sprev < range) { - while (sprev + (len = enc.length(bytes, sprev, end)) < s) sprev += len; - } - - ip += tlen - i - 1; // * SIZE_MEMNUM (1) - break; /* success */ + while (sprev + (len = enc.length(bytes, sprev, end)) < s) sprev += len; } - if (i == tlen) {opFail(); return;} } - private void opBackRefMultiIC() { - int tlen = code[ip++]; - - int i; - loop:for (i=0; i range) continue; - - sprev = s; - - value = s; - if (!backrefStringCmpIC(currentCaseFoldFlag(), pstart, this, n, end)) continue loop; // STRING_CMP_VALUE_IC - s = value; - - int len; - if (sprev < range) { - while (sprev + (len = enc.length(bytes, sprev, end)) < s) sprev += len; + if (!backrefInvalid(mem)) { + selected = mem; } - - ip += tlen - i - 1; // * SIZE_MEMNUM (1) - break; /* success */ } - if (i == tlen) {opFail(); return;} + return selected; } private boolean memIsInMemp(int mem, int num, int memp) { @@ -3116,7 +3096,8 @@ private void opNullCheckEndMemSTPush() { int isNull; if (Config.USE_MONOMANIAC_CHECK_CAPTURES_IN_ENDLESS_REPEAT) { - isNull = nullCheckMemStRec(mem, s); + int positional = nullCheckRecIfPresent(mem, s); + isNull = positional == Integer.MIN_VALUE ? nullCheckMemStRec(mem, s) : positional; } else { isNull = nullCheckRec(mem, s); } @@ -3680,6 +3661,13 @@ protected boolean calloutSamePositionFailureCommits() { case OPCode.NULL_CHECK_START: cursor += OPSize.NULL_CHECK_START; break; + case OPCode.PUSH_POS_NOT: + // `(?! )` is compiled as an empty negative assertion: + // PUSH_POS_NOT followed immediately by FAIL_POS. Perl + // retains a preceding callback's assignment through that + // explicit, zero-width failure. + return cursor + OPSize.PUSH_POS_NOT < code.length + && code[cursor + OPSize.PUSH_POS_NOT] == OPCode.FAIL_POS; default: return false; } diff --git a/third_party/joni/src/org/joni/ByteCodePrinter.java b/third_party/joni/src/org/joni/ByteCodePrinter.java index 651b95d63..2825882d6 100644 --- a/third_party/joni/src/org/joni/ByteCodePrinter.java +++ b/third_party/joni/src/org/joni/ByteCodePrinter.java @@ -409,6 +409,20 @@ public int compiledByteCodeToString(StringBuilder sb, int bp) { sb.append(':').append(addr).append(':').append(mem); break; + case OPCode.CALLOUT: + mem = code[bp]; + bp += OPSize.MEMNUM; + sb.append(':').append(mem); + break; + + case OPCode.CALLOUT_CONDITION: + mem = code[bp]; + bp += OPSize.MEMNUM; + addr = code[bp]; + bp += OPSize.RELADDR; + sb.append(':').append(mem).append(":(").append(addr).append(')'); + break; + case OPCode.DYNAMIC_CALLOUT: mem = code[bp]; bp += OPSize.MEMNUM; diff --git a/third_party/joni/src/org/joni/Regex.java b/third_party/joni/src/org/joni/Regex.java index 3ed01b8c9..0a22d29d1 100644 --- a/third_party/joni/src/org/joni/Regex.java +++ b/third_party/joni/src/org/joni/Regex.java @@ -30,6 +30,7 @@ import java.util.Arrays; import java.util.Collections; import java.util.EnumSet; +import java.util.HashSet; import java.util.Iterator; import java.util.List; import java.util.Map; @@ -111,6 +112,7 @@ static ParsedProgramMetadata copyOf( int numMem; /* used memory(...) num counted from 1 */ int numPhysicalNamedCaptures; + private final Set multiplexNamedGroups = new HashSet<>(); int numRepeat; /* OP_REPEAT/OP_REPEAT_NG id-counter */ int numNullCheck; /* OP_NULL_CHECK_START/END id counter */ int numCombExpCheck; /* combination explosion check */ @@ -450,8 +452,12 @@ int nameAdd(byte[]name, int nameP, int nameEnd, int backRef, Syntax syntax) { // dup the name here as oni does ?, what for ? (it has to manage it, we don't) e = new NameEntry(name, nameP, nameEnd); nameTable.putDirect(name, nameP, nameEnd, e); - } else if (e.backNum >= 1 && !syntax.allowMultiplexDefinitionName()) { - throw new ValueException(ErrorMessages.MULTIPLEX_DEFINED_NAME, new String(name, nameP, nameEnd - nameP)); + } else if (e.backNum >= 1) { + if (!syntax.allowMultiplexDefinitionName()) { + throw new ValueException(ErrorMessages.MULTIPLEX_DEFINED_NAME, new String(name, nameP, nameEnd - nameP)); + } + multiplexNamedGroups.add(backRef); + for (int existing : e.getBackRefs()) multiplexNamedGroups.add(existing); } int physicalRef = ++numPhysicalNamedCaptures; @@ -463,6 +469,10 @@ public int numberOfPhysicalNamedCaptures() { return numPhysicalNamedCaptures; } + boolean isMultiplexNamedGroup(int group) { + return multiplexNamedGroups.contains(group); + } + NameEntry nameToGroupNumbers(byte[]name, int nameP, int nameEnd) { return nameFind(name, nameP, nameEnd); } diff --git a/third_party/joni/src/org/joni/StackMachine.java b/third_party/joni/src/org/joni/StackMachine.java index 72e3e491a..f1ef33b46 100644 --- a/third_party/joni/src/org/joni/StackMachine.java +++ b/third_party/joni/src/org/joni/StackMachine.java @@ -485,13 +485,19 @@ protected final void restoreCallFrameCaptureSnapshot(StackEntry frame) { // A nested call reuses physical capture slots. Closed values belonged // to the caller before the call and must be visible again after return // (for example, a palindrome's enclosing character backreference). - // Deliberately do not restore open or unset slots: their final state - // belongs to the successful nested path. + // Deliberately do not restore ordinary open or unset slots: their + // final state belongs to the successful nested path. Multiplex named + // definitions are different: a value introduced only by the nested + // call must not remain a candidate for a caller's same-name + // backreference after that call returns. for (int mem = 1; mem < count; mem++) { if (snapshot[mem] != INVALID_INDEX && snapshot[count + mem] != INVALID_INDEX) { repeatStk[memStartStk + mem] = snapshot[mem]; repeatStk[memEndStk + mem] = snapshot[count + mem]; + } else if (regex.isMultiplexNamedGroup(mem)) { + repeatStk[memStartStk + mem] = INVALID_INDEX; + repeatStk[memEndStk + mem] = INVALID_INDEX; } } @@ -917,6 +923,24 @@ protected final int nullCheckRec(int id, int s) { } } + /** + * Recursive null check that reports {@link Integer#MIN_VALUE} when an + * older program has no matching start marker on the stack. + */ + protected final int nullCheckRecIfPresent(int id, int s) { + int level = 0; + for (int k = stk - 1; k >= 0; k--) { + StackEntry e = stack[k]; + if (e.type == NULL_CHECK_START && e.getNullCheckNum() == id) { + if (level == 0) return e.getNullCheckPStr() == s ? 1 : 0; + level--; + } else if (e.type == NULL_CHECK_END && e.getNullCheckNum() == id) { + level++; + } + } + return Integer.MIN_VALUE; + } + protected final int nullCheckMemSt(int id, int s) { int k = stk; int isNull; @@ -1036,7 +1060,7 @@ protected final int sreturn() { protected final StackEntry returnFrame() { // Every stack entry inherits the active call-frame head when pushed; - // RETURN entries explicitly advance it to the caller. Looking it up + // RETURN entries explicitly advance it to the caller. Looking it up // directly avoids rescanning deep recursive grammar stacks at every // subexpression return. int callFrame = stack[stk - 1].getActiveCallFrameHead(); From 5f33ca2a006cf48ce18d383e3572c864c04a22ab Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Thu, 10 Sep 2026 10:37:04 +0200 Subject: [PATCH 5/8] fix(io): route argumentless readline through ARGV Preserve Perl's localized diamond-reader behavior for argumentless readline, preventing PPR document_self warnings on the JVM backend. Generated with Codex (https://openai.com/codex/) Co-Authored-By: Codex --- docs/about/changelog.md | 3 +++ .../perlonjava/backend/jvm/EmitOperator.java | 14 ++++++++++++++ .../frontend/parser/OperatorParser.java | 8 +++++++- .../resources/unit/implicit_argv_readline.t | 18 ++++++++++++++++++ 4 files changed, 42 insertions(+), 1 deletion(-) create mode 100644 src/test/resources/unit/implicit_argv_readline.t diff --git a/docs/about/changelog.md b/docs/about/changelog.md index d8432b09c..5c4bc6b4b 100644 --- a/docs/about/changelog.md +++ b/docs/about/changelog.md @@ -29,6 +29,9 @@ priorities and future plans. - Restore PPR's complete suite by correcting recursive duplicate-name captures, nullable recursion checks, and callback regex reuse. +- Route argumentless `readline` through localized `@ARGV`, matching Perl's + diamond-reader behavior and keeping PPR's self-document test warning-free. + - Preserve Data::Dumper's pure-Perl numeric-string behavior for Test::Differences, including copied `qw` values and numeric zero fixtures. diff --git a/src/main/java/org/perlonjava/backend/jvm/EmitOperator.java b/src/main/java/org/perlonjava/backend/jvm/EmitOperator.java index 1e820620e..1d4476b8e 100644 --- a/src/main/java/org/perlonjava/backend/jvm/EmitOperator.java +++ b/src/main/java/org/perlonjava/backend/jvm/EmitOperator.java @@ -182,6 +182,20 @@ static void handleReadlineOperator(EmitterVisitor emitterVisitor, BinaryOperator if (operator.equals("readline")) { emitterVisitor.pushCallContext(); + if (Boolean.TRUE.equals(node.getAnnotation("implicitArgvReadline"))) { + emitterVisitor.ctx.mv.visitMethodInsn(Opcodes.INVOKESTATIC, + "org/perlonjava/runtime/runtimetypes/DiamondIO", + "readline", + "(Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;I)" + + "Lorg/perlonjava/runtime/runtimetypes/RuntimeBase;", + false); + if (emitterVisitor.ctx.contextType == RuntimeContextType.VOID) { + handleVoidContext(emitterVisitor); + } else if (emitterVisitor.ctx.contextType == RuntimeContextType.SCALAR) { + handleScalarContext(emitterVisitor, node); + } + return; + } } emitOperator(node, emitterVisitor); } diff --git a/src/main/java/org/perlonjava/frontend/parser/OperatorParser.java b/src/main/java/org/perlonjava/frontend/parser/OperatorParser.java index 40644a97b..9867836b8 100644 --- a/src/main/java/org/perlonjava/frontend/parser/OperatorParser.java +++ b/src/main/java/org/perlonjava/frontend/parser/OperatorParser.java @@ -1145,6 +1145,7 @@ static BinaryOperatorNode parseReadline(Parser parser, LexerToken token, int cur // Handle file-related operators with special handling for default handles ListNode operand = ListParser.parseZeroOrMoreList(parser, 0, false, true, false, false); Node handle; + boolean implicitArgvReadline = false; if (operand.elements.isEmpty()) { String defaultHandle = switch (operator) { case "readline" -> "main::ARGV"; @@ -1158,6 +1159,7 @@ static BinaryOperatorNode parseReadline(Parser parser, LexerToken token, int cur handle = new OperatorNode("undef", null, currentIndex); } else { handle = new IdentifierNode(defaultHandle, currentIndex); + implicitArgvReadline = operator.equals("readline"); } } else { handle = operand.elements.removeFirst(); @@ -1173,7 +1175,11 @@ static BinaryOperatorNode parseReadline(Parser parser, LexerToken token, int cur } } } - return new BinaryOperatorNode(operator, handle, operand, currentIndex); + BinaryOperatorNode result = new BinaryOperatorNode(operator, handle, operand, currentIndex); + if (implicitArgvReadline) { + result.setAnnotation("implicitArgvReadline", true); + } + return result; } static BinaryOperatorNode parseSplit(Parser parser, LexerToken token, int currentIndex) { diff --git a/src/test/resources/unit/implicit_argv_readline.t b/src/test/resources/unit/implicit_argv_readline.t new file mode 100644 index 000000000..e266abd14 --- /dev/null +++ b/src/test/resources/unit/implicit_argv_readline.t @@ -0,0 +1,18 @@ +use strict; +use warnings; +use File::Temp qw(tempfile); +use Test::More; + +my ($fh, $filename) = tempfile(); +print {$fh} "first\nsecond\n"; +close $fh; + +my $source = do { + local (@ARGV, $/) = $filename; + readline; +}; + +is $source, "first\nsecond\n", + 'argumentless readline slurps the localized @ARGV file'; + +done_testing; From 55e666e4227b211f7fa9a0254e39c12ad244a0b8 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Thu, 10 Sep 2026 13:17:38 +0200 Subject: [PATCH 6/8] fix(io): restore diamond ARGV compatibility Preserve diamond and double-diamond semantics across both backends, including ARGV lifecycle, EOF handling, diagnostics, and ordinary-diamond fork warnings. Generated with Codex (https://openai.com/codex/) Co-Authored-By: Codex --- docs/about/changelog.md | 3 + .../backend/bytecode/CompileOperator.java | 10 +- .../bytecode/OpcodeHandlerExtended.java | 12 +- .../perlonjava/backend/jvm/EmitOperator.java | 4 +- .../frontend/parser/OperatorParser.java | 16 ++ .../frontend/parser/StringParser.java | 9 +- .../runtime/io/CustomFileChannel.java | 13 +- .../runtime/operators/IOOperator.java | 8 + .../perlonjava/runtime/operators/WarnDie.java | 3 + .../runtime/runtimetypes/DiamondIO.java | 172 +++++++++++++++++- .../resources/unit/diamond_argv_lifecycle.t | 49 +++++ 11 files changed, 280 insertions(+), 19 deletions(-) create mode 100644 src/test/resources/unit/diamond_argv_lifecycle.t diff --git a/docs/about/changelog.md b/docs/about/changelog.md index 5c4bc6b4b..2cf1e35cc 100644 --- a/docs/about/changelog.md +++ b/docs/about/changelog.md @@ -12,6 +12,9 @@ priorities and future plans. same-named constant calls, restoring `Types::Numbers` loading through `Data::Float`. +- Restore Perl-compatible `<>` and `<<>>` ARGV traversal, `eof()` behavior, + diagnostics, and warning handling on both execution backends. + - Implement undef-aware experimental equality operators (`===`, `!==`, `equ`, and `neu`) with lexical warnings and single-evaluation chained comparisons. diff --git a/src/main/java/org/perlonjava/backend/bytecode/CompileOperator.java b/src/main/java/org/perlonjava/backend/bytecode/CompileOperator.java index 6c03d26a5..ea371d42b 100644 --- a/src/main/java/org/perlonjava/backend/bytecode/CompileOperator.java +++ b/src/main/java/org/perlonjava/backend/bytecode/CompileOperator.java @@ -1794,7 +1794,15 @@ private static void visitDiamond(BytecodeCompiler bc, OperatorNode node) { bc.compileNode(node.operand, -1, RuntimeContextType.SCALAR); int fhReg = bc.lastResultReg; int rd = bc.allocateOutputRegister(); - bc.emit(Opcodes.READLINE); bc.emitReg(rd); bc.emitReg(fhReg); bc.emit(bc.currentCallContext); + // Preserve that this READLINE originated from <>/<<>>. The runtime + // value can be a glob (not merely the empty-string marker), so it + // cannot reliably infer diamond semantics from the filehandle. + bc.emit(Opcodes.READLINE); bc.emitReg(rd); bc.emitReg(fhReg); + int diamondFlags = 0x100; + if (Boolean.TRUE.equals(node.getAnnotation("doubleDiamond"))) { + diamondFlags |= 0x200; + } + bc.emit(bc.currentCallContext | diamondFlags); bc.lastResultReg = rd; } else { OperatorNode globNode = new OperatorNode("glob", node.operand, node.tokenIndex); diff --git a/src/main/java/org/perlonjava/backend/bytecode/OpcodeHandlerExtended.java b/src/main/java/org/perlonjava/backend/bytecode/OpcodeHandlerExtended.java index e4d85d783..f9a901ec8 100644 --- a/src/main/java/org/perlonjava/backend/bytecode/OpcodeHandlerExtended.java +++ b/src/main/java/org/perlonjava/backend/bytecode/OpcodeHandlerExtended.java @@ -919,18 +919,18 @@ public static int executeReadline(int[] bytecode, int pc, RuntimeBase[] register int rd = bytecode[pc++]; int fhReg = bytecode[pc++]; int ctx = bytecode[pc++]; + boolean diamond = (ctx & 0x100) != 0; + boolean doubleDiamond = (ctx & 0x200) != 0; + ctx &= ~(0x100 | 0x200); if (ctx == RuntimeContextType.RUNTIME) ctx = ((RuntimeScalar) registers[2]).getInt(); RuntimeScalar fh = (RuntimeScalar) registers[fhReg]; // Diamond operator <> passes a plain string scalar (not a glob/IO). // Route to DiamondIO.readline which manages @ARGV / STDIN iteration. // But blessed objects may have <> overload, so route those to Readline. - if (fh.getRuntimeIO() == null) { - if (RuntimeScalarType.blessedId(fh) < 0) { - registers[rd] = Readline.readline(fh, ctx); - } else { - registers[rd] = DiamondIO.readline(fh, ctx); - } + if (diamond || (RuntimeScalarType.blessedId(fh) < 0 + && (fh.toString().isEmpty() || "<>".equals(fh.toString())))) { + registers[rd] = DiamondIO.readline(fh, ctx, doubleDiamond); } else { registers[rd] = Readline.readline(fh, ctx); } diff --git a/src/main/java/org/perlonjava/backend/jvm/EmitOperator.java b/src/main/java/org/perlonjava/backend/jvm/EmitOperator.java index 1d4476b8e..631282b54 100644 --- a/src/main/java/org/perlonjava/backend/jvm/EmitOperator.java +++ b/src/main/java/org/perlonjava/backend/jvm/EmitOperator.java @@ -693,11 +693,13 @@ static void handleDiamondBuiltin(EmitterVisitor emitterVisitor, OperatorNode nod // Handle null filehandle: <> <<>> node.operand.accept(emitterVisitor.with(RuntimeContextType.SCALAR)); emitterVisitor.pushCallContext(); + mv.visitInsn(Boolean.TRUE.equals(node.getAnnotation("doubleDiamond")) + ? Opcodes.ICONST_1 : Opcodes.ICONST_0); // Invoke the static method for reading lines. mv.visitMethodInsn(Opcodes.INVOKESTATIC, "org/perlonjava/runtime/runtimetypes/DiamondIO", "readline", - "(Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;I)Lorg/perlonjava/runtime/runtimetypes/RuntimeBase;", false); + "(Lorg/perlonjava/runtime/runtimetypes/RuntimeScalar;IZ)Lorg/perlonjava/runtime/runtimetypes/RuntimeBase;", false); // Handle context if (emitterVisitor.ctx.contextType == RuntimeContextType.VOID) { diff --git a/src/main/java/org/perlonjava/frontend/parser/OperatorParser.java b/src/main/java/org/perlonjava/frontend/parser/OperatorParser.java index 9867836b8..e8a4659ee 100644 --- a/src/main/java/org/perlonjava/frontend/parser/OperatorParser.java +++ b/src/main/java/org/perlonjava/frontend/parser/OperatorParser.java @@ -326,6 +326,15 @@ static BinaryOperatorNode parsePrint(Parser parser, LexerToken token, int curren handle = operand.handle; operand.handle = null; + // `$.' immediately followed by concatenation, as in + // `print $..$ARGV.$_`, is a print argument rather than a filehandle. + // The lexer represents the second dot as the following expression, + // which otherwise makes the filehandle probe consume `$.' and discard + // the leading output value. + if (isInputLineNumber(handle)) { + operand.elements.addFirst(handle); + handle = null; + } if (handle == null) { // `print` without arguments means `print to last selected filehandle` handle = new OperatorNode("select", new ListNode(currentIndex), currentIndex); @@ -339,6 +348,13 @@ static BinaryOperatorNode parsePrint(Parser parser, LexerToken token, int curren return new BinaryOperatorNode(token.text, handle, operand, currentIndex); } + private static boolean isInputLineNumber(Node node) { + return node instanceof OperatorNode sigil + && "$".equals(sigil.operator) + && sigil.operand instanceof IdentifierNode identifier + && ".".equals(identifier.name); + } + /** True for {@code print(foo(...), ...)}, but not {@code print(FH (...))}. */ private static boolean isParenthesizedBarewordCall(Parser parser) { if (!peek(parser).text.equals("(")) { diff --git a/src/main/java/org/perlonjava/frontend/parser/StringParser.java b/src/main/java/org/perlonjava/frontend/parser/StringParser.java index cf7d91372..eb3d7136e 100644 --- a/src/main/java/org/perlonjava/frontend/parser/StringParser.java +++ b/src/main/java/org/perlonjava/frontend/parser/StringParser.java @@ -1097,7 +1097,14 @@ public static Node parseRawString(Parser parser, String operator) { } ListNode diamondList = new ListNode(rawStr.index); diamondList.elements.add(interpolated); - return new OperatorNode("<>", diamondList, rawStr.index); + OperatorNode diamond = new OperatorNode("<>", diamondList, rawStr.index); + // <> interpolates to an empty string, while <<>> preserves a + // literal "<>" marker. Keep that syntactic distinction after + // the operand later resolves to the ARGV glob. + if (interpolated instanceof StringNode stringNode && "<>".equals(stringNode.value)) { + diamond.setAnnotation("doubleDiamond", true); + } + return diamond; } } diff --git a/src/main/java/org/perlonjava/runtime/io/CustomFileChannel.java b/src/main/java/org/perlonjava/runtime/io/CustomFileChannel.java index 668f86f50..c9aa187fb 100644 --- a/src/main/java/org/perlonjava/runtime/io/CustomFileChannel.java +++ b/src/main/java/org/perlonjava/runtime/io/CustomFileChannel.java @@ -441,7 +441,18 @@ public RuntimeScalar close() { */ @Override public RuntimeScalar eof() { - return new RuntimeScalar(isEOF); + if (isEOF) { + return new RuntimeScalar(true); + } + // Perl's eof() probes a regular file even before readline has tried + // to consume it. In particular, a freshly opened /dev/null is EOF; + // waiting for a read first makes argumentless eof() disagree with + // standard Perl after STDIN is reopened. + try { + return new RuntimeScalar(fileChannel.position() >= fileChannel.size()); + } catch (IOException e) { + return new RuntimeScalar(false); + } } /** diff --git a/src/main/java/org/perlonjava/runtime/operators/IOOperator.java b/src/main/java/org/perlonjava/runtime/operators/IOOperator.java index cedf73690..6a4f17028 100644 --- a/src/main/java/org/perlonjava/runtime/operators/IOOperator.java +++ b/src/main/java/org/perlonjava/runtime/operators/IOOperator.java @@ -544,6 +544,10 @@ public static RuntimeScalar tell(RuntimeScalar fileHandle) { boolean argless = !fileHandle.getDefinedBoolean(); RuntimeIO fh = fileHandle.getRuntimeIO(); + if (argless && DiamondIO.hasActiveTraversal()) { + return DiamondIO.eof(); + } + // If no explicit filehandle was provided (tell with no args), // fall back to the last accessed handle like Perl does. if (fh == null) { @@ -1159,6 +1163,10 @@ public static RuntimeScalar eof(RuntimeScalar fileHandle) { boolean argless = !fileHandle.getDefinedBoolean(); RuntimeIO fh = fileHandle.getRuntimeIO(); + if (argless && DiamondIO.hasActiveTraversal()) { + return DiamondIO.eof(); + } + // Handle undefined or invalid filehandle if (fh == null) { if (argless) { diff --git a/src/main/java/org/perlonjava/runtime/operators/WarnDie.java b/src/main/java/org/perlonjava/runtime/operators/WarnDie.java index 3b9918d8f..05c102b82 100644 --- a/src/main/java/org/perlonjava/runtime/operators/WarnDie.java +++ b/src/main/java/org/perlonjava/runtime/operators/WarnDie.java @@ -801,6 +801,9 @@ public static String getFilehandleContext() { * @return String with the bare handle name (e.g., "DATA", "STDIN"), or null if not found */ private static String findFilehandleName(RuntimeIO handle) { + if (DiamondIO.isDiamondReader(handle)) { + return ""; // caller adds angle brackets: <> line N + } if (handle.globName != null && !handle.globName.isEmpty()) { // Strip package prefix (e.g., "main::DATA" -> "DATA") String name = handle.globName; diff --git a/src/main/java/org/perlonjava/runtime/runtimetypes/DiamondIO.java b/src/main/java/org/perlonjava/runtime/runtimetypes/DiamondIO.java index 26a16b05b..b1febadef 100644 --- a/src/main/java/org/perlonjava/runtime/runtimetypes/DiamondIO.java +++ b/src/main/java/org/perlonjava/runtime/runtimetypes/DiamondIO.java @@ -3,6 +3,7 @@ import org.perlonjava.app.cli.CompilerOptions; import org.perlonjava.runtime.io.ClosedIOHandle; import org.perlonjava.runtime.operators.Readline; +import org.perlonjava.runtime.operators.WarnDie; import java.io.IOException; import java.nio.file.Files; @@ -25,6 +26,9 @@ public static final class State { boolean eofReached; boolean readingStarted; boolean argvWasInitiallyEmpty; + boolean doubleDiamond; + RuntimeIO stdinReader; + RuntimeIO lastDiamondReader; int accumulatedLineNumber; String inPlaceExtension; boolean inPlaceEdit; @@ -37,6 +41,9 @@ void clear() { eofReached = false; readingStarted = false; argvWasInitiallyEmpty = false; + doubleDiamond = false; + stdinReader = null; + lastDiamondReader = null; accumulatedLineNumber = 0; inPlaceExtension = null; inPlaceEdit = false; @@ -76,35 +83,52 @@ public static void reset() { * undefined scalar if EOF is reached for all files. */ public static RuntimeBase readline(RuntimeScalar arg, int ctx) { + return readline(arg, ctx, arg != null && "<>".equals(arg.toString())); + } + + /** Reads diamond input while preserving whether the source used <<>>. */ + public static RuntimeBase readline(RuntimeScalar arg, int ctx, boolean doubleDiamond) { State state = state(); if (ctx == RuntimeContextType.LIST) { // Handle LIST context RuntimeList lines = new RuntimeList(); RuntimeScalar line; - while ((line = (RuntimeScalar) readline(arg, RuntimeContextType.SCALAR)).type != RuntimeScalarType.UNDEF) { + while ((line = (RuntimeScalar) readline(arg, RuntimeContextType.SCALAR, doubleDiamond)).type != RuntimeScalarType.UNDEF) { lines.elements.add(line); } return lines; } else { + RuntimeArray argv = getGlobalArray("main::ARGV"); + // A completed diamond loop is reusable after Perl code refills + // @ARGV. In particular, `@ARGV = (...)` between separate `<>` + // loops must begin a new traversal rather than retain EOF forever. + RuntimeIO stdin = getGlobalIO("main::STDIN").getRuntimeIO(); + // A new non-empty @ARGV always starts a new traversal. An empty + // @ARGV can also be intentional: reopening STDIN between diamond + // loops must make the later loop consume the replacement handle. + if (state.eofReached && (!argv.isEmpty() + || (stdin != null && stdin != state.stdinReader))) { + resetTraversalState(state); + } // Handle SCALAR context // Initialize the reading process if it hasn't started yet if (!state.readingStarted) { state.readingStarted = true; + state.doubleDiamond = doubleDiamond; // Check if @ARGV was initially empty to determine STDIN fallback behavior - state.argvWasInitiallyEmpty = getGlobalArray("main::ARGV").isEmpty(); + state.argvWasInitiallyEmpty = argv.isEmpty(); - RuntimeIO argv = getGlobalIO("main::ARGV").getRuntimeIO(); + RuntimeIO argvHandle = getGlobalIO("main::ARGV").getRuntimeIO(); // Only use ARGV filehandle directly if @ARGV is empty (handles aliased filehandles like *ARGV = *DATA) - if (argv != null && !(argv.ioHandle instanceof ClosedIOHandle) && getGlobalArray("main::ARGV").isEmpty()) { - state.currentReader = argv; + if (argvHandle != null && !(argvHandle.ioHandle instanceof ClosedIOHandle) && getGlobalArray("main::ARGV").isEmpty()) { + state.currentReader = argvHandle; } else if (state.argvWasInitiallyEmpty) { - RuntimeIO stdin = getGlobalIO("main::STDIN").getRuntimeIO(); if (stdin == null || stdin.ioHandle instanceof ClosedIOHandle) { state.eofReached = true; return scalarUndef; } // Only use STDIN if @ARGV was initially empty, not if it became empty after processing files - RuntimeArray.push(getGlobalArray("main::ARGV"), new RuntimeScalar("-")); + RuntimeArray.push(argv, new RuntimeScalar("-")); } } @@ -124,11 +148,18 @@ public static RuntimeBase readline(RuntimeScalar arg, int ctx) { RuntimeScalar line = Readline.readline(state.currentReader); if (line.type != RuntimeScalarType.UNDEF) { state.accumulatedLineNumber = state.currentReader.currentLineNumber; + state.lastDiamondReader = state.currentReader; return line; } - // EOF for current file — save accumulated line count before discarding reader + // EOF for current file — close it before discarding it. The + // ARGV glob otherwise retains an exhausted but still-open + // handle, which incorrectly wins over a later reopened STDIN + // when a new empty-@ARGV diamond loop begins. state.accumulatedLineNumber = state.currentReader.currentLineNumber; + state.lastDiamondReader = state.currentReader; + state.currentReader.close(); + state.lastDiamondReader.currentLineNumber = state.accumulatedLineNumber; state.currentReader = null; } } @@ -165,6 +196,33 @@ private static boolean openNextFile() { String originalFileName = fileName.toString(); String backupFileName = null; + // Unlike the zero-argument diamond fallback, an explicit empty @ARGV + // entry is a filename and must fail immediately. Passing it to the + // generic path opener resolves it as a directory on some platforms, + // which turns `while (<>)` into a non-terminating read attempt. + if (originalFileName.isEmpty()) { + GlobalVariable.getGlobalVariable("main::!").set("No such file or directory"); + return dieCannotOpen(originalFileName); + } + + if (!state.doubleDiamond && isForkLikeOpen(originalFileName)) { + WarnDie.warn(new RuntimeScalar("Forked open '" + originalFileName + + "' not meaningful in <>"), new RuntimeScalar("\n")); + return false; + } + + // $ARGV shares its spelling with the ARGV typeglob, but it must remain + // a scalar slot. Some script entry paths leave the scalar map pointing + // at that glob; assigning through it performs a typeglob assignment + // and corrupts subsequent diamond reads. Replace that placeholder with + // the actual scalar value instead. + RuntimeScalar argvName = GlobalVariable.getGlobalVariable("main::ARGV"); + if (argvName instanceof RuntimeGlob) { + GlobalVariable.globalVariables.put("main::ARGV", new RuntimeScalar(originalFileName)); + } else { + argvName.set(originalFileName); + } + // Check if in-place editing is enabled (either via -i switch or $^I variable) boolean isInPlaceEnabled = state.inPlaceEdit; String extension = state.inPlaceExtension; @@ -252,12 +310,40 @@ private static boolean openNextFile() { // Open the renamed file for reading String readerPath = state.tempFilePath != null ? state.tempFilePath.toString() : (backupFileName != null ? backupFileName : originalFileName); - state.currentReader = RuntimeIO.open(readerPath); + if ("-".equals(readerPath) && (!state.doubleDiamond || state.argvWasInitiallyEmpty)) { + // A diamond '-' is the current Perl STDIN handle, which can have + // been reopened by the program; it is not necessarily System.in. + // With <<>>, a synthetic '-' created solely because @ARGV was + // empty retains that fallback; an explicit '-' argument remains a + // literal filename as Perl requires. + state.currentReader = getGlobalIO("main::STDIN").getRuntimeIO(); + state.stdinReader = state.currentReader; + } else { + // The explicit two-argument form intentionally does not apply + // RuntimeIO.open(String)'s special empty-string and "-" handling: + // double diamond treats both as ordinary filenames. + state.currentReader = RuntimeIO.open(readerPath, "<"); + } + if (state.currentReader == null) { + return dieCannotOpen(originalFileName); + } getGlobalIO("main::ARGV").set(state.currentReader); return state.currentReader != null; } + private static boolean dieCannotOpen(String fileName) { + RuntimeScalar error = GlobalVariable.getGlobalVariable("main::!"); + WarnDie.die(new RuntimeScalar("Can't open " + fileName + ": " + error), + new RuntimeScalar(WarnDie.getPerlLocationFromStack())); + return false; // unreachable; keeps the compiler's flow analysis explicit + } + + private static boolean isForkLikeOpen(String fileName) { + String normalized = fileName.replaceAll("\\s+", ""); + return "|-".equals(normalized) || "-|".equals(normalized); + } + /** Restore the handle selected before diamond in-place editing began. */ private static void finishInPlaceEditing() { State state = state(); @@ -266,4 +352,72 @@ private static void finishInPlaceEditing() { state.selectedHandleBeforeInPlace = null; } } + + /** Reset only per-traversal state while retaining command-line -i settings. */ + private static void resetTraversalState(State state) { + if (state.currentReader != null) { + state.currentReader.close(); + } + if (state.currentWriter != null) { + state.currentWriter.close(); + } + state.currentReader = null; + state.currentWriter = null; + state.eofReached = false; + state.readingStarted = false; + state.argvWasInitiallyEmpty = false; + state.stdinReader = null; + state.lastDiamondReader = null; + state.accumulatedLineNumber = 0; + state.tempFilePath = null; + finishInPlaceEditing(); + } + + /** True when {@code handle} is the active diamond reader. */ + public static boolean isCurrentReader(RuntimeIO handle) { + State state = state(); + return state.readingStarted && handle != null && handle == state.currentReader; + } + + /** True once <> or <<>> has established its per-runtime traversal state. */ + public static boolean hasActiveTraversal() { + return state().readingStarted; + } + + /** True for a reader used by <> or <<>>, including its final EOF reader. */ + public static boolean isDiamondReader(RuntimeIO handle) { + State state = state(); + return handle != null && (handle == state.currentReader || handle == state.lastDiamondReader); + } + + /** + * Implements argumentless eof() for the diamond handle. Perl considers it + * true only when the current source is exhausted and no later @ARGV entry + * remains; individual file boundaries are not final EOF. + */ + public static RuntimeScalar eof() { + State state = state(); + RuntimeArray argv = getGlobalArray("main::ARGV"); + RuntimeIO argvHandle = getGlobalIO("main::ARGV").getRuntimeIO(); + // A closed ARGV handle with queued entries is an explicit `close ARGV` + // request. With an empty @ARGV it is commonly just the exhausted + // reader that diamond itself closed, so STDIN must take precedence. + if (!argv.isEmpty() && argvHandle != null && argvHandle.ioHandle instanceof ClosedIOHandle) { + return RuntimeScalarCache.scalarTrue; + } + + // Pending @ARGV entries guarantee that argumentless eof() is false; + // this is what lets eof() look ahead across diamond file boundaries. + if (!argv.isEmpty()) { + return RuntimeScalarCache.scalarFalse; + } + + // With an empty @ARGV, diamond is defined in terms of the current + // STDIN handle. It may have been reopened since a previous traversal. + RuntimeIO stdin = getGlobalIO("main::STDIN").getRuntimeIO(); + if (stdin != null && !(stdin.ioHandle instanceof ClosedIOHandle)) { + return stdin.eof(); + } + return state.eofReached ? RuntimeScalarCache.scalarTrue : RuntimeScalarCache.scalarFalse; + } } diff --git a/src/test/resources/unit/diamond_argv_lifecycle.t b/src/test/resources/unit/diamond_argv_lifecycle.t new file mode 100644 index 000000000..e3bfe675e --- /dev/null +++ b/src/test/resources/unit/diamond_argv_lifecycle.t @@ -0,0 +1,49 @@ +use strict; +use warnings; +use File::Temp qw(tempfile); +use Test::More; + +sub fixture { + my ($content) = @_; + my ($fh, $path) = tempfile(); + print {$fh} $content; + close $fh; + return $path; +} + +my $first = fixture("first\n"); +my $second = fixture("second\n"); + +@ARGV = ($first, $second); +my $global_text = ''; +while (<>) { + $global_text .= $_; +} +is $global_text, "first\nsecond\n", + 'diamond reads files assigned to global @ARGV'; + +{ + local @ARGV = ($first, $second); + my ($text, @lines); + while (<>) { + $text .= "$ARGV:$_"; + push @lines, $.; + } + is $text, "$first:first\n$second:second\n", + 'diamond publishes the current filename through $ARGV'; + is_deeply \@lines, [1, 2], 'diamond preserves $. across @ARGV files'; +} + +{ + local @ARGV = ($second); + is scalar(<>), "second\n", 'diamond restarts after a previous @ARGV traversal'; +} + +@ARGV = ($first); +local $/ = undef; +scalar <>; +open STDIN, '<', '/dev/null' or die "cannot reopen STDIN: $!"; +@ARGV = (); +ok eof(), 'argumentless eof uses reopened STDIN after an active diamond reader'; + +done_testing; From e2c13487ee80872fc78bf4ee972d0c80d7fda828 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Thu, 10 Sep 2026 14:11:38 +0200 Subject: [PATCH 7/8] fix(io): preserve files after aborted in-place edits Treat a lone '*' in-place extension as extensionless editing and restore the temporary source when an unhandled error aborts the implicit edit loop. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex --- docs/about/changelog.md | 3 ++ .../java/org/perlonjava/app/cli/Main.java | 2 + .../runtime/runtimetypes/DiamondIO.java | 40 ++++++++++++++++++- src/test/resources/unit/inplace_edit_abort.t | 35 ++++++++++++++++ 4 files changed, 78 insertions(+), 2 deletions(-) create mode 100644 src/test/resources/unit/inplace_edit_abort.t diff --git a/docs/about/changelog.md b/docs/about/changelog.md index 2cf1e35cc..dac681a78 100644 --- a/docs/about/changelog.md +++ b/docs/about/changelog.md @@ -8,6 +8,9 @@ priorities and future plans. - Identify PerlOnJava, its copyright, and its dual-license terms in `jperl -v` output while retaining the standard Perl text. + +- Preserve source files when extensionless in-place editing aborts, and treat + a lone `'*'` in-place extension like Perl's extensionless form. - Prevent eval-created named subs from treating lexical variables as same-named constant calls, restoring `Types::Numbers` loading through `Data::Float`. diff --git a/src/main/java/org/perlonjava/app/cli/Main.java b/src/main/java/org/perlonjava/app/cli/Main.java index 40aa64eba..abfe8b192 100644 --- a/src/main/java/org/perlonjava/app/cli/Main.java +++ b/src/main/java/org/perlonjava/app/cli/Main.java @@ -3,6 +3,7 @@ import org.perlonjava.app.scriptengine.PerlLanguageProvider; import org.perlonjava.runtime.operators.WarnDie; import org.perlonjava.runtime.runtimetypes.ErrorMessageUtil; +import org.perlonjava.runtime.runtimetypes.DiamondIO; import org.perlonjava.runtime.runtimetypes.GlobalVariable; import org.perlonjava.runtime.runtimetypes.PerlExitException; import org.perlonjava.runtime.runtimetypes.PerlRuntime; @@ -155,6 +156,7 @@ private static void run(String[] args) { System.exit(PerlRuntime.current().threadRegistry() .requestedProcessExitOr(e.getExitCode())); } catch (Throwable t) { + DiamondIO.abortInPlaceEditing(); if (parsedArgs.debugEnabled) { // Print full JVM stack t.printStackTrace(); diff --git a/src/main/java/org/perlonjava/runtime/runtimetypes/DiamondIO.java b/src/main/java/org/perlonjava/runtime/runtimetypes/DiamondIO.java index b1febadef..4b0705548 100644 --- a/src/main/java/org/perlonjava/runtime/runtimetypes/DiamondIO.java +++ b/src/main/java/org/perlonjava/runtime/runtimetypes/DiamondIO.java @@ -33,6 +33,7 @@ public static final class State { String inPlaceExtension; boolean inPlaceEdit; Path tempFilePath; + Path inPlaceOriginalPath; RuntimeIO selectedHandleBeforeInPlace; void clear() { @@ -48,6 +49,7 @@ void clear() { inPlaceExtension = null; inPlaceEdit = false; tempFilePath = null; + inPlaceOriginalPath = null; selectedHandleBeforeInPlace = null; } } @@ -244,8 +246,10 @@ private static boolean openNextFile() { // Use RuntimeIO's existing path resolution methods for consistency Path originalPath = RuntimeIO.resolvePath(originalFileName); - if (extension == null || extension.isEmpty()) { - // Create a temporary file for the original file + if (extension == null || extension.isEmpty() || "*".equals(extension)) { + // A lone '*' is Perl's extensionless form. It must use a + // temporary backup rather than substitute the source name + // into the backup path and move a file onto itself. try { state.tempFilePath = Files.createTempFile("temp_", null); backupFileName = state.tempFilePath.toString(); @@ -294,6 +298,8 @@ private static boolean openNextFile() { } } + state.inPlaceOriginalPath = originalPath; + // Open the original file for writing (this is the ARGVOUT equivalent) // Use the resolved path to ensure we write to the correct location state.currentWriter = RuntimeIO.open(originalPath.toString(), ">"); @@ -353,6 +359,35 @@ private static void finishInPlaceEditing() { } } + /** + * Restore the current extensionless in-place source after an unhandled + * exception. Perl's -i uses a temporary backup in this form and leaves + * the source intact when the implicit loop aborts before completion. + */ + public static void abortInPlaceEditing() { + State state = state(); + if (state.currentReader != null) { + state.currentReader.close(); + state.currentReader = null; + } + if (state.currentWriter != null) { + state.currentWriter.close(); + state.currentWriter = null; + } + if (state.tempFilePath != null && state.inPlaceOriginalPath != null) { + try { + Files.move(state.tempFilePath, state.inPlaceOriginalPath, + StandardCopyOption.REPLACE_EXISTING); + } catch (IOException ignored) { + // Preserve the original Perl exception; failed restoration is + // reported by the already-active in-place edit diagnostic. + } + } + state.tempFilePath = null; + state.inPlaceOriginalPath = null; + finishInPlaceEditing(); + } + /** Reset only per-traversal state while retaining command-line -i settings. */ private static void resetTraversalState(State state) { if (state.currentReader != null) { @@ -370,6 +405,7 @@ private static void resetTraversalState(State state) { state.lastDiamondReader = null; state.accumulatedLineNumber = 0; state.tempFilePath = null; + state.inPlaceOriginalPath = null; finishInPlaceEditing(); } diff --git a/src/test/resources/unit/inplace_edit_abort.t b/src/test/resources/unit/inplace_edit_abort.t new file mode 100644 index 000000000..2b3d74a3b --- /dev/null +++ b/src/test/resources/unit/inplace_edit_abort.t @@ -0,0 +1,35 @@ +use strict; +use warnings; +use File::Temp qw(tempfile); +use Test::More tests => 3; + +my ($handle, $path) = tempfile(); +print {$handle} "bar\n"; +close $handle; + +{ + local $^I = '*'; + local @ARGV = ($path); + while (<>) { + print "foo$_"; + } +} + +open $handle, '<', $path or die "Cannot read $path: $!"; +is(do { local $/; <$handle> }, "foobar\n", + q{$^I = '*' performs in-place editing without replacing the input by an empty backup}); +close $handle; + +open $handle, '>', $path or die "Cannot rewrite $path: $!"; +print {$handle} "bar\n"; +close $handle; + +my $runner = $^X eq 'jperl' ? './jperl' : $^X; +my $status = system($runner, '-i', '-n', '-e', 'die', $path); +ok($status != 0, 'an in-place program that dies exits unsuccessfully'); + +open $handle, '<', $path or die "Cannot read aborted edit $path: $!"; +is(do { local $/; <$handle> }, "bar\n", + 'an aborted extensionless in-place edit preserves the original file'); +close $handle; +unlink $path; From 9c7955157cdd25460c6085b99b675ff9ae1ae151 Mon Sep 17 00:00:00 2001 From: "Flavio S. Glock" Date: Thu, 10 Sep 2026 15:04:54 +0200 Subject: [PATCH 8/8] test(io): make diamond lifecycle regression portable Use File::Spec's platform null device so the ARGV lifecycle regression runs on Windows. Generated with [Codex](https://openai.com/codex/) Co-Authored-By: Codex --- src/test/resources/unit/diamond_argv_lifecycle.t | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/test/resources/unit/diamond_argv_lifecycle.t b/src/test/resources/unit/diamond_argv_lifecycle.t index e3bfe675e..f854588fa 100644 --- a/src/test/resources/unit/diamond_argv_lifecycle.t +++ b/src/test/resources/unit/diamond_argv_lifecycle.t @@ -1,5 +1,6 @@ use strict; use warnings; +use File::Spec; use File::Temp qw(tempfile); use Test::More; @@ -42,7 +43,7 @@ is $global_text, "first\nsecond\n", @ARGV = ($first); local $/ = undef; scalar <>; -open STDIN, '<', '/dev/null' or die "cannot reopen STDIN: $!"; +open STDIN, '<', File::Spec->devnull or die "cannot reopen STDIN: $!"; @ARGV = (); ok eof(), 'argumentless eof uses reopened STDIN after an active diamond reader';