Skip to content

Keep discarded findings out of final review comments #5

Open

Description

@christso

Problem

A final review comment can repeat a discarded concern even after the model reads the core skill and reviews reference.

Public case C4 in PR #2 asks for the settled assessment: approval with one schema-dialect follow-up. The candidate starts:

Approved. The freeze deliberately applies only to external-client contracts; standalone contracts remain editable.

It then explains the active schema concern correctly. The opening still carries the discarded freeze concern into a comment for an author who has not seen the drafting discussion.

The existing guidance already says to use the latest assessment and include only history the recipient needs. This is a follow-up reliability investigation; a skill change has not yet been shown to be necessary.

Proposed work

  • Calibrate the case's distinction between useful recipient context and a discarded finding.
  • Repeat the fictional case with and without the skill.
  • Investigate a small change only if the failure persists and points to a gap or conflict in the guidance. Review-specific detail belongs in the reviews reference.

Acceptance

  • The final comment includes the active concern, caller example, current review status and next action.
  • It omits the discarded concern and unnecessary explanations of its resolution.
  • Related cases retain history when the recipient actually needs it to act.
  • Any change has evidence from repeated cases and does not add a case-specific rule to the general skill.

Writer: gpt-6.1-sol; grader: gpt-5.6-luna; reasoning: medium. This example is fictional and contains no private review history.

Additional coverage lives in the private eval repository.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions