Find the sequence that breaks your business rules. Keep the evidence.
中文说明 · Integration guide · Architecture · Validation & limitations
FlowTrial explores sequences of HTTP actions against a disposable test instance. You define the actors, actions and business invariants. Its planner proposes sequences; an independent rule engine decides whether they violate those invariants.
deliver_payment → deliver_payment credited twice
cancel_order → deliver_payment cancelled order becomes paid
revoke_member → read_document removed member still receives 200
Every confirmed finding includes a fresh-state replay, a reduced action sequence, request/response and state evidence, and an executable pytest regression.
Requires Python 3.11+ and uv.
git clone https://github.com/shyangar/flowtrial.git
cd flowtrial
uv sync --locked --all-extras
uv run flowtrial demo --output reports/demoOpen reports/demo/index.html. The command starts real local HTTP servers with
SQLite databases, explores the deliberately vulnerable build and the fixed build,
then replays the saved failures against the fix. It stops both servers when finished.
Expected result: 3 confirmed rule violations → 0 on the fixed build → 3 passing replays. These are deliberately introduced reference defects, not newly discovered third-party vulnerabilities. The demo uses deterministic local search, not an LLM.
No uv? python -m pip install ".[dev]", then python -m flowtrial demo.
The repository lockfile is used by the uv path and CI. PyPI publication is not assumed.
| Capability | Behavior |
|---|---|
| Multi-actor execution | Separate HTTP clients and cookie jars for each actor |
| Stateful rules | Compare before/after state, response, initial state and extracted variables |
| Local planner | Seed mutation, duplicates, order swaps, pairwise coverage and seeded exploration |
| Optional LLM planner | OpenAI-compatible Chat Completions, execution feedback, bounded action selection |
| Confirmation | At least two fresh-state confirmations; the final reduced sequence is reconfirmed |
| Reduction | Single-deletion minimization preserving the same violated rule |
| Evidence | Versioned JSON findings, self-contained HTML report, configuration fingerprint |
| Regression export | Generated pytest cases with a separate test root and explicit target configuration |
| Failure semantics | Infrastructure failures, invalid oracles and untested rules are inconclusive |
| Budgets | Limits on sequences, steps, HTTP calls, model calls, response size and request timeout |
Start with uv run flowtrial init my-app.flowtrial.json, then adapt the specification.
Your isolated test instance needs:
- A reset endpoint that restores a known fixture, including relevant external-service fakes.
- Named actions and actors using your actual HTTP routes.
- Business rules you have reviewed. An optional read-only observer exposes relevant state.
FlowTrial does not infer correct business policy from implementation code. It does not automatically instrument arbitrary repositories. See the complete adapter contract.
uv run flowtrial validate my-app.flowtrial.json
uv run flowtrial scan my-app.flowtrial.json --budget 80 --output reports/my-runRemote test instances require --allow-remote. Reset and mutation requests are real;
use a dedicated instance you control. Do not run independent scans against the same resettable instance.
Exit codes: 0 no violations within the explored scope; 1 confirmed violations; 2 incomplete/invalid execution; 130 interrupted. Incomplete verification takes precedence over code 1, while confirmed findings remain in the report.
Model access is optional. Select the model explicitly and configure
FLOWTRIAL_API_KEY in your local environment or secret manager.
Do not put credentials in committed specifications. Loopback model servers can run without a key.
uv run flowtrial scan my-app.flowtrial.json --planner llm --model YOUR_MODEL_ID --model-base-url https://api.openai.com/v1 --max-model-calls 5 --budget 24 --output reports/agent-runFor a local compatible server, point --model-base-url at its /v1 URL.
The server must support Chat Completions with JSON-object output and max_completion_tokens.
Provider compatibility is not universal.
The model receives the action catalog, rules and recent execution outcomes. It cannot add endpoints, credentials, rules, Python code or shell commands. The complete response batch is schema-validated before any proposed action executes. Provider failures do not silently fall back to local search. Privacy and adapter details.
uv run flowtrial replay reports/my-run/findings/RULE_ID.json --spec my-app.flowtrial.json --base-url http://127.0.0.1:8765To run the exported pytest suite, set FLOWTRIAL_SPEC to your spec file and
FLOWTRIAL_BASE_URL to the fixed test instance, then:
uv run pytest reports/my-run/test_regressions.pyEach test resets the target. Run serially. Missing configuration, an unreachable target or a rule that was never evaluated makes the regression fail, not skip.
uv sync --locked --all-extras
uv run ruff check .
uv run ruff format --check .
uv run mypy
uv run pytest --cov=flowtrial --cov-report=term-missing
uv run python -m buildThe CI matrix covers Linux, Windows and macOS, including Python 3.11 and 3.13. See CONTRIBUTING.md for adding planners, rules and application adapters.
Version 0.1 targets serial state transitions, repeated events, reordered events and role changes. It does not control a thread scheduler, prove absence of bugs, discover unknown business rules, or replace load testing. A reduced sequence is one-deletion-minimal when reduction completes, not necessarily globally shortest. Reports disclose incomplete reduction.
Online model integration has contract tests; the reproducible reference experiment needs no paid model calls. Model-dependent discovery quality requires separate evaluation.
MIT licensed. No telemetry, hosted service or account is required for local use.