Repository navigation
[bot] Merge master/b765e4b5 into rel/dev - #1922
Merged
Merged
Conversation
The conversation evaluator scored only what a turn finally produced, so
it could not tell an agent that asked the clarifying question the
request needs from one that guessed, nor one that asked about something
the request already settled. gen-ai now asks closed choices as a
clarifyingQuestions part, and none of that was visible to a fixture.
A turn gains four opt-in keys, read from its first complete reply
before any simulated-user answer:
- expects_clarification: true means the reply must ask, false means it
must not. A confirmation is not a question, but a question followed
by a confirmation phrase ("A or B? Please confirm.") is.
- expected_clarification_form: questionnaire (a clarifyingQuestions
part) or text (a question in prose only).
- max_clarifying_questions: the questionnaire asks at most this many.
- clarifying_options_grounded: every {type/id} option names an object
of the workspace. The workspace's metrics, attributes, labels, facts,
datasets, parameters, dashboards and visualizations are listed on
each turn that checks, so an object an earlier turn created counts.
An attribute and its label accept each other, {date/...} names a
date dataset and {date_attribute/...} an attribute. gen-ai types no
listing holds (computed_attribute, filter_context, document, report)
stay unchecked.
A mismatch fails context_success and is listed in failure_reasons, and
the turn reports turn_<id>_clarification_correct to Langfuse. The last
three keys need expects_clarification: true and are rejected otherwise,
since they would pass without a question; the last two also fail a
question asked in prose. Legacy mode never lets these checks fail an
item, so a fixture with expects_clarification raises there. Without the
keys nothing changes: no check, no score, the same turn detail.
The entity API links a next page from every page, an empty one too, so
the listing stops on a short page instead. A kind the server lacks
(404) is empty. A listing that fails leaves the turn ungraded rather
than passed or the run ended: no clarification_correct score, and the
reason as clarification_ungraded in the turn detail. A mismatch the
listing is not needed for still fails the turn. Neither the listing nor a
reply the server cut short counts toward what the turn is graded on or
its run latency.
Checked live on an ecommerce_demo workspace with gpt-5.6-luna: three
QA-30039 items (visualization, alert, anomaly detection) with all four
keys set pass, and the workspace listing returns.
jira: QA-30039
risk: low
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The conversation evaluator answered every clarifying question with text, so it never sent what the chat panel sends: a clarifyingAnswers message with the picked positions and the texts shown, where a question left out is skipped. Skipping, and what the agent does after a skip, could not be evaluated. ChatResult keeps the itemId of the assistant item holding a clarifyingQuestions part, and each such part keeps its item's id, since gen-ai numbers questions within the item an answer names. ChatClient.send_clarifying_answers sends the message with it as questionsItemId. send_message and the new method share one request path, so options and user context apply to both. A turn gains opt-in keys, all needing expects_clarification: true: - clarifying_answers: picks for the questionnaire of the turn's first reply, sent even when the judge flags that question as lost context, since the fixture expects it. A pick names an option by a type/id reference or its whole text, or lists both spellings, since the agent writes options either way; it may name the question's position. attribute/x and label/x match each other. A question no pick matches is skipped, and an empty list skips them all. Later questions, a follow-up confirmation included, still get set_answers text. A pick no questionnaire offered fails the turn, and so do answers never sent, with the reason: the first reply asked no questionnaire, delivered the output with it, had no itemId, or the clarification budget ended the turn. - states_skipped_assumption: a reply after the answers names an option of each skipped question, by a whole reference or its text as whole words; the reply asking for confirmation usually states it. A mention stands in for the stated assumption. - skipped_questions_not_reasked: no later questionnaire asks a skipped question again, judged by its text or its option set. Yes/No style options fit any question, a confirmation too, so they count only with the same text. - expects_confirmation_after_answers: the reply to the answers is not the output. A skip is not an approval, so the agent stops first. The transcript records the answers as the text gen-ai builds for the model from them. Without the keys nothing changes. Checked live on an ecommerce_demo workspace with gpt-5.6-luna: a picked sales measure is sent and charted, skipping every question leads to a stated assumption with no re-ask, and a dashboard build stops for confirmation after a skip. jira: QA-30039 risk: low Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
clarifying_options_grounded reads only {type/id} options, but gen-ai
also writes options as plain names. Asked to describe a chart whose
title two charts share, it offers names that point at no chart or at
the wrong one: the second chart's description, which is a third
chart's title, or a paraphrase like "Net Sales version". A user
picking such an option gets another chart described.
The workspace listing now keeps each object's title and each chart's
type, and a turn gains two opt-in keys, both needing
expects_clarification: true:
- clarifying_option_titles: an object kind (visualization, metric,
dashboard, attribute, fact, dataset). Every plain-text option must be
the title of such an object, alone or followed by context in brackets
or after a dash, comma or colon, as two charts with one title need.
- clarifying_option_objects: the type/id objects the question is about.
Every option must name one of them, by reference, by its title with
that context, or by its chart type ("Donut chart") when no other
candidate has that type and the option does not start with another
object's title. A reference or title of any other object fails.
Both read only the questions they are about: a question with at least
one option the check accepts. A questionnaire that also asks for a
channel or a breakdown no longer fails on those options. When no
question is about the check, every option it reads fails. Grounding
still reads every question. Like grounding, both leave the turn
ungraded when the workspace cannot be read.
The listing now goes through gooddata_sdk's EntitiesApi with
load_all_entities, retrying a transient failure as a chat message is.
It is read only when the first reply is a questionnaire with options,
and kept for the run until the run creates or updates a metric or an
alert.
Checked live on an ecommerce_demo workspace with gpt-5.6-luna, asking
to describe the Net Sales Breakdown chart, which a donut and a
waterfall chart are titled: options naming invented versions fail,
options naming the two charts by reference or by chart type pass.
jira: QA-30039
risk: low
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(gooddata-eval): grade clarifying questions in conversations
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## rel/dev #1922 +/- ##
===========================================
+ Coverage 85.83% 86.03% +0.20%
===========================================
Files 343 344 +1
Lines 25322 25698 +376
===========================================
+ Hits 21735 22110 +375
- Misses 3587 3588 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 Automated PR to perform merge from master into rel/dev with changes up to b765e4b (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/41954578155).