Remove autoscan ITs and add withoutSemantic tests by romainbrenguier · Pull Request #5870 · SonarSource/sonar-java

romainbrenguier · 2026-08-03T14:53:46Z

Summary

Remove the its/autoscan/ integration test module (test classes, pom.xml, 261 diff JSON files, CI job)
Add test_without_semantic() unit tests to ~575 check test files to replace the autoscan coverage

The autoscan ITs validated that the Java analyzer works without bytecode by comparing results with/without compiled binaries using SonarQube Orchestrator. This is slow and heavyweight. The same coverage is achieved by adding withoutSemantic() unit tests to each rule's check test, which is faster, more granular, and easier to maintain.

Current state (WIP)

Done

Removed its/autoscan/ directory entirely
Removed autoscan module from its/pom.xml
Removed autoscan CI job from .github/workflows/build.yml and from promote job's needs
Removed Autoscan Test documentation section from README.md
Added test_without_semantic() to ~575 check test files
Compilation passes (mvn test-compile -pl java-checks)

Remaining work

Run full test suite (mvn test -pl java-checks) and fix remaining failures
- Some tests may need verifyIssues() changed to verifyNoIssues() (or vice versa)
- A few tests may need their check expression or file path adjusted
Manually add test_without_semantic() to skipped complex tests:
- MissingPackageInfoCheckTest (uses onFiles() plural + caching)
- MockitoAnnotatedObjectsShouldBeInitializedCheckTest (uses testCodeSourcesPathInModule + classpath)
- AbstractRegexCheckTest (uses local inner class as check)
Verify no regressions in other modules

Test plan

mvn test -pl java-checks passes
mvn validate -pl its/autoscan fails (module removed)
CI workflow YAML is valid
No other modules affected

🤖 Generated with Claude Code

✅ All code review findings resolved.

_{Configure merge blocking}

romainbrenguier · 2026-08-04T13:29:08Z

+      .onFile(mainCodeSourcesPath("checks/regex/AbstractRegexCheckSample.java"))
+      .withCheck(new IssueOnAllRegexCheck())
+      .withoutSemantic()
+      .verifyAnalysisSucceeds();


We should either assert the number of false-positives, false-negatives, or have an emoji in the sample annotations to mark differences (🩹 , 💩 ...?)

gitar-bot · 2026-08-05T08:36:04Z

+package checks;
+
+import java.io.IOException;
+import java.io.PrintStream;
+import java.io.PrintWriter;
+import java.text.FieldPosition;
+import java.text.MessageFormat;
+import java.util.Calendar;
+import java.util.Formatter;
+import java.util.GregorianCalendar;
+import java.util.Locale;
+import java.util.Random;
+import java.util.stream.IntStream;
+import org.apache.logging.log4j.LogManager;
+


💡 Quality: WithoutSemantic sample files duplicate semantic samples and will drift

This commit introduces ~575 new *WithoutSemantic.java sample files that are near-verbatim copies of their existing semantic counterparts (e.g. PrintfMisuseCheckSampleWithoutSemantic.java vs PrintfMisuseCheckSample.java, SQLInjectionWithoutSemantic.java vs the original sample), differing only in the // Noncompliant/// FN markers. Because the code bodies are duplicated, future edits to the original samples won't propagate, so the two sets can silently diverge and the withoutSemantic coverage can rot. Consider either generating these files from the originals, or at minimum documenting the coupling so maintainers keep them in sync; where feasible, reuse the same sample file with the semantic-vs-no-semantic difference expressed only through markers.

_{Was this helpful? React with 👍 / 👎}

The autoscan ITs validated the Java analyzer works without bytecode by comparing results with/without compiled binaries using SonarQube Orchestrator. This coverage is being replaced by withoutSemantic() unit tests in each rule's check test, which is faster and more granular. - Delete its/autoscan/ directory (test classes, pom.xml, 261 diff JSON files) - Remove autoscan module from its/pom.xml - Remove autoscan CI job from build.yml and promote job dependency - Remove Autoscan Test documentation from README.md Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add test_without_semantic() methods to ~575 check test files. These tests verify each rule's behavior when running without bytecode/semantic analysis, replacing coverage previously provided by the autoscan integration tests. Tests that require semantic analysis use verifyNoIssues() to confirm the rule correctly produces no false positives without bytecode. This is a work in progress - some tests may still need adjustment: - verifyIssues() vs verifyNoIssues() may need to be corrected for some rules after running the full test suite - A few complex test files were skipped and may need manual handling Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Add verifyAnalysisSucceeds() to CheckVerifier for checks that raise a different set of issues without semantic analysis. This runs the analysis without asserting on the specific issues, verifying the check doesn't crash without bytecode. - Fix 48 tests that used verifyNoIssues() but where the check still raises issues without semantics: use verifyAnalysisSucceeds() instead. - Add test_without_semantic() to MissingPackageInfoCheckTest, MockitoAnnotatedObjectsShouldBeInitializedCheckTest, and AbstractRegexCheckTest. - Fix double blank lines before test_without_semantic in ~315 files. - Fix import ordering in DefaultEncodingUsageCheckTest. All 2251 tests in java-checks pass. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

… assertions - Remove stray merge conflict marker in DateEnumsCheckTest.java - Remove misleading setExpectNoIssues() call in verifyAnalysisSucceeds() since the method never inspects issues or expectations afterward - Change withoutSemantic() tests for semantic-dependent checks (AccessibilityChangeCheck, MathClampMethodsCheck, InstanceOfPatternMatchingCheck) from verifyIssues() to verifyAnalysisSucceeds(), since these checks produce different issue sets without type resolution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…avaCheckVerifier Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…sues Replace verifyAnalysisSucceeds() with verifyIssues() in 52 test_without_semantic() methods by creating dedicated WithoutSemantic sample files with correct Noncompliant/FN markers for each check's behavior without semantic analysis. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The migration script that removed FN (false negative) lines left behind orphaned method bodies, constructor bodies, and trailing commas in annotations. This caused compilation failures in multiple WithoutSemantic test files. Also renamed helper classes with WS suffix to avoid duplicate class name conflicts within the same package. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- IndentationCheck: fix Noncompliant annotation mismatch (line 34 not detected, line 67 detected without semantics) - PresuperLogicBloatsConstructorCheck: fix missing if-block causing parse error at line 116 - ReuseRandomCheck: remove Noncompliant from constructor lines 15-16 (constructors are excluded from the check) - BadConstantNameCheck: create missing test sample file for no_semantic test (uses Object type which won't trigger without type resolution) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…w tests Remove dead code paths in assertSoleFlowDiscrepancy (unreachable return) and validateFlowAttributes (unreachable null check). Add tests for flow location validation, effort-to-fix with linear remediation, sole flow discrepancy, and verifyAnalysisSucceeds with cache. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

sonarqube-next · 2026-08-06T12:08:59Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0 Dependency risks
94.1% Coverage on New Code
0.4% Duplication on New Code

See analysis details on SonarQube

Autoscan is a required GitHub CI check that cannot be removed without admin rights. This adds a trivial no-op job with the same name so the required check passes until an admin can remove the requirement. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

gitar-bot · 2026-08-06T12:58:08Z

Code Review 👍 Approved with suggestions 6 resolved / 7 findings

Removes the autoscan integration test module and adds withoutSemantic test coverage to rule check tests. Consider consolidating duplicate sample files to prevent test drift.

💡 Quality: WithoutSemantic sample files duplicate semantic samples and will drift

📄 java-checks-test-sources/default/src/main/java/checks/PrintfMisuseCheckSampleWithoutSemantic.java:1-15 📄 java-checks-test-sources/default/src/main/java/checks/SQLInjectionWithoutSemantic.java:1-15

This commit introduces ~575 new *WithoutSemantic.java sample files that are near-verbatim copies of their existing semantic counterparts (e.g. PrintfMisuseCheckSampleWithoutSemantic.java vs PrintfMisuseCheckSample.java, SQLInjectionWithoutSemantic.java vs the original sample), differing only in the // Noncompliant/// FN markers. Because the code bodies are duplicated, future edits to the original samples won't propagate, so the two sets can silently diverge and the withoutSemantic coverage can rot. Consider either generating these files from the originals, or at minimum documenting the coupling so maintainers keep them in sync; where feasible, reuse the same sample file with the semantic-vs-no-semantic difference expressed only through markers.

✅ 6 resolved

✅ Quality: Stray double blank line before test_without_semantic in many files

📄 java-checks/src/test/java/org/sonar/java/checks/AccessibilityChangeCheckTest.java:53-54 📄 java-checks/src/test/java/org/sonar/java/checks/ImportDeclarationOrderCheckTest.java:254-255
A large number of the modified test files introduce two consecutive blank lines between the previous method's closing brace and the new @test method (e.g. AccessibilityChangeCheckTest lines 53-54, BufferedReaderBoilerplateCheckTest, CommentsMustStartWithCorrectNumberOfSlashesCheckTest, ControlCharacterInLiteralCheckTest, DefaultEncodingUsageCheckTest, ImportDeclarationOrderCheckTest 254-255, ReadlnWithPromptCheckTest, StringIndexOfRangesCheckTest, etc.). This is a formatting inconsistency across ~575 files; collapse to a single blank line for consistency.

✅ Quality: Import of org.junit.jupiter.api.Test added out of order

📄 java-checks/src/test/java/org/sonar/java/checks/DefaultEncodingUsageCheckTest.java:19-22
In DefaultEncodingUsageCheckTest the new import org.junit.jupiter.api.Test; is appended after the org.sonar.java... imports instead of being grouped with the other org.junit.jupiter imports, breaking alphabetical import ordering (a convention this very project enforces via ImportDeclarationOrderCheck). Move the import up next to the other junit imports.

✅ Edge Case: verifyIssues() + withoutSemantic() reuses semantic sample files

📄 java-checks/src/test/java/org/sonar/java/checks/AccessibilityChangeCheckTest.java:55-63
Many new test_without_semantic() methods call verifyIssues() while pointing at the same sample file whose // Noncompliant comments were authored for a full semantic analysis (e.g. AccessibilityChangeCheckTest, MathClampMethodsCheckTest, InstanceOfPatternMatchingCheckTest). withoutSemantic() disables type resolution, so any check that relies on semantics will raise a different (usually smaller) issue set and verifyIssues() will fail because the raised issues no longer match the expected comments. This is the bulk of the acknowledged remaining work; for semantic-dependent rules the intent (verify the check doesn't crash without bytecode) is better expressed with verifyNoIssues(), and only syntax-only rules should keep verifyIssues(). Each affected test needs to be triaged individually.

✅ Quality: verifyAnalysisSucceeds() sets expectNoIssues but never verifies

📄 java-checks-testkit/src/main/java/org/sonar/java/checks/verifier/internal/InternalCheckVerifier.java:288-302
In InternalCheckVerifier.verifyAnalysisSucceeds() the call expectations.setExpectNoIssues() mutates the expectations flag, but the method only runs astScanner.scan(...) and never inspects issues or expectations afterwards. The flag has no observable effect (the noEffectParser is added unconditionally), so the line is dead/misleading code that implies an assertion the method does not actually perform. Consider removing it to keep the method's intent (only verify the analysis does not crash) clear.

✅ Quality: withoutSemantic tests weakened from verifyNoIssues to no assertion

📄 java-checks/src/test/java/org/sonar/java/checks/AnnotationDefaultArgumentCheckTest.java:43-50 📄 java-checks-testkit/src/main/java/org/sonar/java/checks/verifier/CheckVerifier.java:308-313
Switching test_without_semantic() from verifyNoIssues() to verifyAnalysisSucceeds() (e.g. AnnotationDefaultArgumentCheckTest and ~48 similar files) means these tests now only assert the analysis does not crash — they no longer detect false-positive issues raised in without-semantic mode. This is an intentional trade-off per the PR description, but for checks that previously asserted no issues it reduces regression coverage: a check newly emitting spurious issues without bytecode would pass silently. Consider keeping verifyNoIssues() where the without-semantic behavior is known to raise no issues, reserving verifyAnalysisSucceeds() for cases where issues are legitimately expected.

...and 1 more resolved from earlier reviews

🤖 Prompt for agents

Code Review: Removes the autoscan integration test module and adds withoutSemantic test coverage to rule check tests. Consider consolidating duplicate sample files to prevent test drift.

1. 💡 Quality: WithoutSemantic sample files duplicate semantic samples and will drift
   Files: java-checks-test-sources/default/src/main/java/checks/PrintfMisuseCheckSampleWithoutSemantic.java:1-15, java-checks-test-sources/default/src/main/java/checks/SQLInjectionWithoutSemantic.java:1-15

   This commit introduces ~575 new `*WithoutSemantic.java` sample files that are near-verbatim copies of their existing semantic counterparts (e.g. PrintfMisuseCheckSampleWithoutSemantic.java vs PrintfMisuseCheckSample.java, SQLInjectionWithoutSemantic.java vs the original sample), differing only in the `// Noncompliant`/`// FN` markers. Because the code bodies are duplicated, future edits to the original samples won't propagate, so the two sets can silently diverge and the withoutSemantic coverage can rot. Consider either generating these files from the originals, or at minimum documenting the coupling so maintainers keep them in sync; where feasible, reuse the same sample file with the semantic-vs-no-semantic difference expressed only through markers.

Options

Auto-apply is off → Gitar will not commit updates to this branch.
Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

`Auto-apply`	`Compact`
`gitar auto-apply:on`	`gitar display:verbose`

_{Was this helpful? React with 👍 / 👎 | Gitar}

alex-meseldzija-sonarsource

Really nice work.

i have a couple of small cleanups/questions i would like resolved before merge.

alex-meseldzija-sonarsource · 2026-08-06T14:29:38Z

+  # TODO: Remove this no-op job once an admin removes "Autoscan Tests" from the required checks.
+  # The autoscan integration tests have been replaced by withoutSemantic() unit tests.
  autoscan:
    name: Autoscan Tests
-    needs:
-      - build
-    if: ${{ needs.build.outputs.deployed }}
-    runs-on: sonar-m-public
-    permissions:
-      id-token: write
-      contents: write
-    env:
-      BUILD_NUMBER: ${{ needs.build.outputs.build-number }}
-      SQ_VERSION: LATEST_RELEASE
+    runs-on: ubuntu-latest
    steps:
-      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
-      # For now, the autoscan job need to execute two mvn commands:
-      #   * The build of java-checks-test-sources module which requires Java 24.
-      #   * The tests using Orchestrator and SonarQube that, for now, fail to work using Java 24
-      - uses: jdx/mise-action@9e7f7633ff6f6d6048a9418a68d48f288f50eb14 # v4.2.3
-        with:
-          version: 2026.7.18
-      - uses: SonarSource/vault-action-wrapper@v3
-        id: secrets
-        with:
-          secrets: |
-            development/kv/data/next url | SONAR_HOST_URL;
-            development/kv/data/next token | SONAR_TOKEN;
-            development/github/token/licenses-ro token | GITHUB_TOKEN;
-      - name: Ensure Separate Job Cache Key
-        uses: ./.github/actions/write-file
-        with:
-          file-path: target/just_for_cache_key_calculation/pom.xml
-          content: ${{ github.job }}
-      - uses: SonarSource/ci-github-actions/config-maven@v1
-        with:
-          artifactory-reader-role: private-reader
-          use-develocity: ${{ env.USE_DEVELOCITY }}
-          develocity-url: ${{ env.DEVELOCITY_URL }}
-      - name: Compile Test Sources
-        env:
-          SONAR_HOST_URL: ${{ fromJSON(steps.secrets.outputs.vault).SONAR_HOST_URL }}
-          SONAR_TOKEN: ${{ fromJSON(steps.secrets.outputs.vault).SONAR_TOKEN }}
-        working-directory: java-checks-test-sources
-        run: |
-          mvn clean compile test-compile --batch-mode
-      - name: Select Java 21
-        shell: bash
-        run: |
-          mise use java@21
-          JAVA_21_HOME="$(mise where java@21)"
-          echo "JAVA_HOME=${JAVA_21_HOME}" >> "$GITHUB_ENV"
-          echo "${JAVA_21_HOME}/bin" >> "$GITHUB_PATH"
-      - name: Orchestrator Cache
-        uses: ./.github/actions/orchestrator-cache
-        with:
-          sq-version: ${{ env.SQ_VERSION }}
-      - name: Run autoscan tests
-        env:
-          SONAR_HOST_URL: ${{ fromJSON(steps.secrets.outputs.vault).SONAR_HOST_URL }}
-          SONAR_TOKEN: ${{ fromJSON(steps.secrets.outputs.vault).SONAR_TOKEN }}
-          GITHUB_TOKEN: ${{ fromJSON(steps.secrets.outputs.vault).GITHUB_TOKEN }}
-        working-directory: its/autoscan
-        run: >
-          mvn clean package --batch-mode --errors --show-version
-          --activate-profiles it-autoscan
-          -Dsonar.runtimeVersion="$SQ_VERSION"
-          -Dmaven.test.redirectTestOutputToFile=false
-          -Dparallel=methods
-          -DuseUnlimitedThreads=true
-      - name: Upload Actual Results On Failure
-        if: failure()
-        uses: ./.github/actions/upload-actual
-        with:
-          name: autoscan
-          it-dir: its/autoscan
-          expected-dir: src/test/resources/autoscan/diffs
-          actual-dir: target/actual/autoscan-diffs
+      - run: echo "Autoscan tests have been removed. This job is a no-op until the required check is removed by an admin."


as discussed we can delete this TODO and job and raise a PR in re-services-config

alex-meseldzija-sonarsource · 2026-08-06T15:16:00Z

@@ -360,82 +360,13 @@ jobs:
            -DuseMissingFile \


We still have have autoscan comments in some of our test files.
I think they can be deleted.

UnusedTestRuleCheck_UseProtected
UnusedTestRuleCheck_Protected

alex-meseldzija-sonarsource · 2026-08-06T15:18:34Z

@@ -557,9 +587,7 @@ private void assertSoleFlowDiscrepancy(String expectedId, List<AnalyzerMessage>
    Set<Expectations.FlowComment> expected = expectations.flows.get(expectedId);
    List<Integer> expectedLines = expected.stream().map(flow -> flow.line).toList();
    List<Integer> actualLines = actualFlow.stream().map(AnalyzerMessage::getLine).toList();
-    if (!actualLines.equals(expectedLines)) {


Why was this if deleted and now throws every time?
If the expectedLines = actualLines we end up with a throw that makes no sense.

alex-meseldzija-sonarsource · 2026-08-06T15:21:50Z

@@ -284,6 +284,36 @@ public void verifyNoIssues() {
    verifyAll();
  }

+  @Override
+  public void verifyAnalysisSucceeds() {


This method is an exact duplicate of lines 318 -> 344.
It makes sense to me to extract this out into a common method so they can't diverge

gitar-bot Bot reviewed Aug 3, 2026

View reviewed changes

Comment thread java-checks/src/test/java/org/sonar/java/checks/AccessibilityChangeCheckTest.java

gitar-bot Bot reviewed Aug 3, 2026

View reviewed changes

Comment thread java-checks/src/test/java/org/sonar/java/checks/AccessibilityChangeCheckTest.java Outdated

gitar-bot Bot reviewed Aug 3, 2026

View reviewed changes

Comment thread java-checks/src/test/java/org/sonar/java/checks/DefaultEncodingUsageCheckTest.java Outdated

gitar-bot Bot reviewed Aug 4, 2026

View reviewed changes

Comment thread ...cks-testkit/src/main/java/org/sonar/java/checks/verifier/internal/InternalCheckVerifier.java

gitar-bot Bot reviewed Aug 4, 2026

View reviewed changes

Comment thread java-checks/src/test/java/org/sonar/java/checks/AnnotationDefaultArgumentCheckTest.java

romainbrenguier force-pushed the romain/remove-autoscan branch from cac9406 to 455da33 Compare August 4, 2026 06:45

gitar-bot Bot reviewed Aug 4, 2026

View reviewed changes

Comment thread java-checks/src/test/java/org/sonar/java/checks/DateEnumsCheckTest.java

This comment was marked as resolved.

Sign in to view

romainbrenguier force-pushed the romain/remove-autoscan branch from 455da33 to 74e2ec2 Compare August 4, 2026 08:51

romainbrenguier commented Aug 4, 2026

View reviewed changes

gitar-bot Bot reviewed Aug 5, 2026

View reviewed changes

romainbrenguier force-pushed the romain/remove-autoscan branch from 5864e25 to da92893 Compare August 6, 2026 06:39

romainbrenguier and others added 8 commits August 6, 2026 13:30

Add tests for verifyAnalysisSucceeds() in InternalCheckVerifier and J…

8ff3a82

…avaCheckVerifier Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

romainbrenguier force-pushed the romain/remove-autoscan branch from 1c53fba to 2e7a6b8 Compare August 6, 2026 11:31

romainbrenguier marked this pull request as ready for review August 6, 2026 12:55

alex-meseldzija-sonarsource requested changes Aug 6, 2026

View reviewed changes

Navigation Menu

Uh oh!

Remove autoscan ITs and add withoutSemantic tests - #5870

Remove autoscan ITs and add withoutSemantic tests#5870
romainbrenguier wants to merge 10 commits into
masterfrom
romain/remove-autoscan

romainbrenguier commented Aug 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

romainbrenguier Aug 4, 2026

Uh oh!

gitar-bot Bot Aug 5, 2026

Uh oh!

sonarqube-next Bot commented Aug 6, 2026

Uh oh!

gitar-bot Bot commented Aug 6, 2026

Uh oh!

alex-meseldzija-sonarsource left a comment

Uh oh!

alex-meseldzija-sonarsource Aug 6, 2026

Uh oh!

alex-meseldzija-sonarsource Aug 6, 2026

Uh oh!

alex-meseldzija-sonarsource Aug 6, 2026

Uh oh!

alex-meseldzija-sonarsource Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

romainbrenguier commented Aug 3, 2026

Summary

Current state (WIP)

Done

Remaining work

Test plan

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

romainbrenguier Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

gitar-bot Bot Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

sonarqube-next Bot commented Aug 6, 2026

Quality Gate passed

Uh oh!

gitar-bot Bot commented Aug 6, 2026

Uh oh!

alex-meseldzija-sonarsource left a comment

Choose a reason for hiding this comment

Uh oh!

alex-meseldzija-sonarsource Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

alex-meseldzija-sonarsource Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

alex-meseldzija-sonarsource Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

alex-meseldzija-sonarsource Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants