Feature/agent test set p1

[yuyixg]

Agent regression test harness
Adds a per-agent regression test harness as a new plugin, plus the one extension point in BotSharp.Core it needs. Scripted multi-turn cases, mocked tools, deterministic assertions, recording cases from real conversations, comparison runs across several models, and AI-assisted case extraction.

The problem it solves: there is currently no way to tell whether editing an agent's instruction, swapping its model, or changing a tool broke something, other than talking to it by hand. That is not repeatable, and for agents with side-effecting tools it is not safe either.

Two separable parts

1. An extension point in the core (6 existing files, +93/-11).

FunctionExecutorFactory was internal static, so nothing outside BotSharp.Core could take part in deciding who executes a function name. It is now a service behind a new IFunctionExecutorFactory, and asks any registered IFunctionExecutorProvider first (ascending Order, so resolution never depends on DI registration order).

BotSharp.Core.Rules' ToolCallAction used to look up IFunctionCallback itself and invoke it directly, bypassing the factory entirely. It now resolves through the factory, which closes the only path by which a function could execute without a provider getting a say. Its original case-insensitive name matching is preserved: registered callbacks are still used to resolve the canonical name, and only the execution moves to the factory.

Compatibility. With no IFunctionExecutorProvider registered, resolution is byte-for-byte what it was: the same three-stage chain in the same order with the same semantics. That is asserted directly by FunctionExecutorFactoryTests, which exists specifically to pin it, since this is the one way this change could introduce a silent regression on the core path.

Two guards came out of testing rather than review: a null/blank function name used to be harmless (the old IsEqualTo lookup simply matched nothing) but would now reach IFunctionExecutorProvider.TryResolve, whose contract declares a non-null string. A provider doing a dictionary lookup by name would throw ArgumentNullException instead of failing gracefully, so both FunctionExecutorFactory.Create and ToolCallAction reject it up front.

2. BotSharp.Plugin.AgentTesting (new, ~7k lines including tests).

Self-contained: depends on BotSharp.Abstraction only, keeps its own four Mongo collections behind AgentTestMongoDbContext, and mirrors BotSharp.Plugin.MongoStorage's conventions — same Database:BotSharpMongoDb setting, same rule for deriving the database name, same TablePrefix convention — so a host that already has Mongo storage configured needs no additional configuration.

Deliberately not added to IBotSharpRepository: four new collections there would mean roughly twenty new members that FileRepository and BotSharpDbContext would each have to implement, for data no other feature reads.

How a test run avoids touching the real world
TestMockExecutorProvider takes over every function inside a conversation under test, so an unmocked tool is blocked rather than executed. Five explicit control-flow functions are allowed through -- notably not by a util- prefix, because util-email-handle_email_sender, util-twilio-outbound_phone_call, util-twilio-text_message, util-http-handle_http_request and util-db-sql_select all start with it and all have real side effects. Allowing by prefix would mean one test run really sends the emails and really places the calls; a test asserts that the allow list is exactly those five names, and fails under a prefix rule.

Interception is keyed by conversationId in a registry, not by AsyncLocal. AsyncLocal depends on ExecutionContext flowing and is silently lost across a background-queue or SideCar boundary, and losing it does not produce a failing test — it produces unmocked tools executing for real.

Before any user message is sent, each case calls a canary function and verifies the content that comes back. If the seam is not live (a build resolving an older BotSharp.Core without provider support, say), mocking fails silently, and the canary turns that into an explicit Error on the case instead of a green run against real tools.

A blocked tool fails the case. Blocking is the seam working correctly, but it also stops that turn, so everything the agent would have done next never happened and every later assertion is evaluated against a conversation that ended early. Reporting Passed there is the same "executed nothing, reports green" defect the no-turns guard and the canary exist to prevent.

Results are attributable
Failed and Error are kept strictly apart: Failed means it ran and an assertion did not hold, Error means it never got that far (timeout, dead canary, a case with no turns, a case filter that matched nothing, a host restart mid-run). Collapsing them would make "the harness broke" read as "the agent regressed", and every run-level Error carries a reason worded for whoever has to act on it, because such a run can produce zero case results and then that field is the only place the reason exists.

Multi-model runs sweep one suite across several models in a single run (cases x models), each result tagged with the model that produced it, so pass rates and durations compare side by side. The override rides on the existing IAgentHook, rewriting agent.LlmConfig as the agent loads. That timing is forced rather than chosen: RoutingService.InvokeAgent passes agent.LlmConfig's provider/model to CompletionProvider explicitly, and GetProviderAndModel only consults the conversation-state override when what it was passed is empty — so seeding provider/model into conversation state has no effect on the main path.

Recording and AI extraction
Recording converts a real conversation into an editable draft case: real function returns become mocks, real state deltas become state writes and initial states, and the two stable assertion kinds (toolCalled, stateEquals) are generated. Output-text assertions are deliberately never generated — using the model's exact wording as a baseline is brittle enough that any rephrasing goes red.

AI extraction splits one conversation into a case per scenario it covers. The model decides only where to cut and what to name each case; mock return values, assertions and state still come verbatim from the conversation. Letting a model author mocks would stop a case being a replay of a call that really happened, and letting it author assertions would reintroduce exactly the brittle output-text assertions above. The segmenter rejects any segmentation it cannot fully verify -- gaps, overlaps, out-of-range or uncovered turns -- because a half-correct one looks entirely normal in the UI and only misbehaves when someone runs it. Only user messages and tool names are sent to the model; tool arguments and results, where the densest PII sits, are withheld.

Every recorded draft lands disabled and has to be reviewed and enabled by hand before it joins a run.

Testing
194 unit tests in tests/BotSharp.Core.UnitTests, of which 153 cover this feature and the rest are the pre-existing suite. The two seam test classes are the ones worth reviewing: they pin no-provider compatibility on the core path and the case-insensitive name matching on the rule path.

Verified end to end against a live host: multi-model comparison across two models produced per-model attributed results with differing durations; an intentionally invalid model name is rejected at trigger time rather than failing every case with an opaque error; AI extraction split a three-topic conversation into three correctly named cases with earlier-turn state carried into each.

Notes for review
The plugin exposes a REST controller. Every endpoint requires authentication, and the two cost/PII-sensitive ones (recording, triggering a run) additionally require admin/root: recording copies raw conversation content into the test store with no ownership check on the conversation id, and triggering really spends token quota with no throttling anywhere.
Runs execute serially. That is a safety choice, not a performance oversight -- cases share external dependencies and would pollute each other's state concurrently.
The run queue is in-process, so a restart abandons runs in flight. They are swept to Error on the next start rather than left claiming to be Running.
llmJudge is accepted as an assertion type but always fails, with a message saying so, rather than passing silently and showing a case that verified nothing as green.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →