You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Scoped audit of the observability topic, run by the nightly rotation. No pull request has been merged into the platform's main branch since #2733, so this run audited the topic whose last audit was the oldest (2026-08-10).
Page audited: observability.mdx, the only page this topic owns.
Platform: mnfst/manifest at main, commit 6b045749b.
Docs: mnfst/docs at 83776bd.
Window: the whole observability surface as it behaves today, plus the merged PRs that built it (#2644, #2648, #2653, #2660, #2662, #2663, #2666, #2675, #2676, #2678, #2683, #2685, #2689, #2691, #2697, #2712, #2717, #2727, #2729, #2733).
8 findings: 4 wrong claims, 4 missing cases. No image finding this run: two image requests already stand on this page and both wait on a capture (issue #54 finding 4, issue #70 finding 5).
Nine out-of-scope observations, seven of them platform-side, were written to references/backlog.md before this issue was filed. One of them matters here: providers/custom-providers.mdx carries the same wrong claim as finding 3.
Findings 1, 2 and 3 are the ones worth reading first. Finding 8 consumes a point parked on 2026-08-17 for this topic's own audit.
Findings
1 · [WRONG] The Spend alerts section calls the limit period rolling. The period is a calendar window that restarts at midnight, on Monday, or on the 1st.
You create a limit rule on a harness: metric cost, threshold $10, period Day.
Manifest adds up that harness's spend from midnight to now, on the clock of the server running Manifest.
The rule acts once that running total crosses $10.
Nothing frees up as the day goes on. The total restarts only at the next midnight.
So a harness blocked at 09:00 stays blocked for fifteen hours. A burst at 23:30 and another at 00:30 can also trip the same rule twice, one hour apart.
Options:
Name the anchor and the restart: a day starts at midnight, a week on Monday, a month on the 1st, an hour at the top of the previous clock hour, and the count restarts at the next boundary. Recommended (it also makes this page agree with llm-gateway.mdx, which already describes the same mechanic from the blocking side).
Delete the word "rolling" and say nothing about the window. Removes the falsehood, leaves "when does my daily count restart" unanswered on the page that owns rule creation.
Keep the opening sentence generic and put the window in the Period row of the field table, where a reader is actually choosing a period.
Suggested wording (option 1), replaces line 66:
A limit rule watches one metric for one harness over the current period, and acts when the threshold is crossed. The period is a calendar window on the clock of the server running Manifest. A day starts at midnight, a week starts on Monday, and a month starts on the 1st. An hour period starts at the top of the previous clock hour. The count restarts at the next boundary.
Evidence (for the agent)
observability.mdx:66 — "A limit rule watches one metric for one harness over a rolling period, and acts when the threshold is crossed."
common/utils/period.util.ts:16-41computePeriodBoundaries: day = new Date(y, m, d) (local midnight today), week = this Monday 00:00, month = the 1st at 00:00, hour = now.getHours() - 1 (top of the previous clock hour), periodEnd = now.
common/utils/period.util.ts:9-15 states the boundaries are computed in the Node process's LOCAL timezone deliberately, because they are compared against local-time agent_messages.timestamp values. Any wording must not say UTC.
Used by both evaluators: notifications/services/limit-check.service.ts:73 (blocking), notifications/services/notification-cron.service.ts:104 (alerting). Consumption window notifications/services/notification-rules.service.ts:144-145.
Restart instant: period.util.ts:47-72computePeriodResetDate. The in-app banner already says it: Limits.tsx:213-215 "until the usage resets in the next period".
Corpus check: llm-gateway.mdx:148 says "The block clears on its own at the start of the next period". The two pages contradict each other today. rolling appears nowhere else in the corpus (the only other hit is "hand-rolling" in errors/M300.mdx:34).
A user who reads "rolling" budgets for a window that slides. It does not.
2 · [WRONG] The page says a failed attempt can still burn tokens. Manifest records every failed attempt with zero tokens and no cost.
Your request reaches a provider and the call fails.
Manifest writes an attempt for that failure and stores the error on it.
The attempt keeps zero input tokens, zero output tokens, and no cost at all.
The request's Cost adds up its attempts, so the failed one contributes nothing.
The spend chart, the model table, and a cost limit rule's consumption all read that same zero.
Options:
Say what the record holds: cost and tokens sit on the attempt, only a successful attempt carries them, and a failed attempt keeps its error with zero usage. Recommended (it keeps the paragraph's teaching purpose, and every word is checkable in the product).
Cut the justification clause and stop at "cost is attributed to attempts". Cheapest, teaches less.
Flip the warning: a provider can still charge for a call that failed, and Manifest records nothing for it, so the dashboard total can read lower than the provider bill. More useful to a cost-conscious reader, but the first half is a claim about provider billing that neither repo can support. It needs your confirmation, not a code citation.
Suggested wording (option 1), replaces line 27:
This is why the request count and the provider call count don't match. Manifest records cost and tokens on the attempt, not on the request. Only a successful attempt carries them. A failed attempt keeps its error and reads zero tokens with no cost.
Evidence (for the agent)
observability.mdx:27 — "This is why the request count and the provider call count don't match, and why cost is attributed to attempts: a failed attempt can still burn tokens."
routing/proxy/proxy-message-recorder.ts: cost_usd is written at exactly two sites, both on the success path, :1105 (inside recordFallbackSuccess, function at :1029) and :1203 (inside recordSuccessMessage, function at :1123). Non-zero token values only at :1101-1102 and :1199-1200.
The four failure builders supply none of the four columns: recordProviderError (:638), recordManifestBlockedRequest (:737), recordFailedFallbacks (:807), recordPrimaryFailure (:934). Rows therefore keep buildMessageRow's defaults input_tokens: 0, output_tokens: 0 (:301-302) and a NULL cost.
Request-level rollups sum those attempt rows: analytics/services/messages-query.service.ts:459,467-470. The spend chart (analytics/services/timeseries-queries.service.ts:89), the per-model table (:242,258) and a cost rule's consumption (notifications/services/notification-rules.service.ts:119-122) sum the same column.
The sentence is offered as the REASON the numbers are shaped this way, so a reader reconciling the dashboard against a provider invoice starts from a wrong premise.
3 · [WRONG] The cost Tip tells custom-provider users Manifest cannot price their models. The connect form asks for those prices, and Manifest uses them.
You connect a custom provider and list its models.
Each model row carries two price fields, Input / 1M tokens and Output / 1M tokens.
You fill in either one and save.
Manifest prices every request to that model from those figures.
The cost shows on the request, in per-model spend, and in a cost limit rule's consumption.
Leave both fields empty and Manifest has nothing to compute from, which is the only case the current Tip describes.
Options:
Split the Tip: keep the local-model sentence as it is, and replace the custom-provider clause with the conditional truth. Recommended (it corrects the claim where the reader meets it).
Drop the custom-provider clause here and let the providers page own custom-provider pricing. That page repeats the same wrong claim today, so this is a two-page fix, not a smaller one.
Keep a two-case sentence here and hand the detail to a link.
<Tip>
Local models record `cost = 0` with real token counts and latency, since
nothing was billed. You price a custom provider yourself: its connect form
takes **Input / 1M tokens** and **Output / 1M tokens** for each model, and
Manifest computes cost from those figures like any other provider. Leave both
prices empty and Manifest has nothing to compute from, so the model records
`cost = 0` too.
</Tip>
Evidence (for the agent)
observability.mdx:59-61 — "Local models record cost = 0 with real token counts and latency, since nothing was billed. Custom providers do the same, because Manifest can't infer pricing for a model it doesn't know."
The visible column headers on the connect form are Model name, Input / 1M tokens, Output / 1M tokens (CustomProviderForm.tsx:410-420, under the Models label at :404). Inputs submitted at :142-147.
model-prices/model-pricing-cache.service.ts:293-313 stores an entry keyed custom:<uuid>/<model_name> whenever at least one price is filled. Only entries with BOTH prices empty are skipped (:303).
That key is the exact string the caller sends, and the recorder looks pricing up on that raw string BEFORE canonicalisation (proxy-message-recorder.ts:1154-1164 vs :1168), so the lookup hits and computeTokenCost returns a real number. The service states the contract in its own comment (model-pricing-cache.service.ts:268-272). The stored agent_messages.model is the canonicalised form (custom-provider.service.ts:183-186), which is why the lookup must run first.
A priced custom model then shows its cost everywhere the dashboard sums that column: the spend chart (analytics/services/timeseries-queries.service.ts:89), the per-model table (:242,258) and a cost rule's consumption (notifications/services/notification-rules.service.ts:119-122).
Scope note: this does NOT re-open the settled NULL-versus-literal-zero DISPLAY question (decisions.md, 2026-08-10 and 2026-08-11). Those clearances reasoned about how an UNPRICED model renders. This finding is about a model the user HAS priced, which no prior clearance's citations reach. The suggested wording deliberately keeps cost = 0 for the unpriced case so the settled point is untouched.
Same wrong claim on providers/custom-providers.mdx:65, out of scope this run, parked in references/backlog.md for the providers topic. Under the never-halfway invariant the two pages should be fixed together.
A user running an in-house endpoint reads that cost tracking is impossible for them and stops looking, after the product asked them for the prices.
4 · [MISSING-CASE] The page promises one alert email per period. A failed send costs you that period's email, and the Limits page still counts the rule as triggered.
Manifest writes the crossing to that rule's history, and then tries to send the email.
The send fails, for example because the email provider is down or its key was revoked.
Manifest does not try again inside that period, because the history row it already wrote is what stops a second email.
The Triggered count on the Limits page still goes up, because it counts crossings and not deliveries.
You see no email, and a rule that reports it fired.
Options:
Say a crossing is recorded once per period whether or not the email leaves, and that Triggered counts crossings rather than deliveries. Point self-hosted readers at Send test email as the way to prove the sender still works. Recommended (one sentence, and it repairs the conclusion the dashboard pushes a user toward).
Say only that a failed send is not retried inside the period, and leave the Triggered count unexplained.
Leave the page alone and fix the platform: write the history row after a successful send and keep a separate crossing counter. A real improvement, but it leaves every install running today undocumented.
Suggested wording (option 1). Append to line 92, after "…won't flood your inbox.":
Manifest records the crossing before it sends the email, and it does not retry inside the period. A send that fails leaves that period without an email. The Triggered count on the Limits page still goes up, because it counts threshold crossings and not deliveries.
And append to the self-hosted paragraph at line 104, after "…where you can edit or remove it.":
Edit it and click Send test email to check the sender still works.
Evidence (for the agent)
observability.mdx:92 — "Each rule emails once per period, so a rule that stays over its threshold won't flood your inbox." Read with :110 — "Without an email provider from either path, alerts are skipped." The page covers the no-sender case and never the failed-send case.
Source order in notifications/services/notification-cron.service.ts: hasAlreadySent guard at :106, insertLog at :115-124, sendThresholdAlert at :126-140. The log row is written BEFORE the send is attempted. Same order in notifications/services/limit-check.service.ts:118, :126-135.
Dedup query notifications/services/notification-log.service.ts:14-20, one row per (rule_id, period_start).
Consequence: the cron's own warning "Failed to send alert for rule …, will retry next cron run" (notification-cron.service.ts:141-143) describes a retry that cannot happen.
The column header is Triggered (LimitRuleTable.tsx:80) and it renders a count of exactly those log rows (notification-rules.service.ts:44-55, cell at LimitRuleTable.tsx:128). The same rows fill the page's History panel (LimitHistoryTable.tsx:28).
Button label confirmed: Send test email (EmailProviderModal.tsx:416). The primary button beside it is Test & Connect / Test & Save (:236), which the page already documents, so the wording points at the ghost button that tests without saving.
Alerting and blocking share the dedup table, so a hard limit that already emailed suppresses the soft alert for the same rule and period.
A user reads silence as "no threshold crossing", and checking the dashboard confirms the wrong conclusion.
Platform-side fix also parked in references/backlog.md.
5 · [MISSING-CASE] The page never mentions pending and cancelled requests. The Requests log shows both, and the Overview's failed count leaves them out.
Manifest records your request as Pending the moment it arrives, before it picks a provider. It has no attempt yet.
A request whose caller disconnects is recorded as Cancelled, keeping the attempt that had already started if there was one.
The Requests log lists both, each with its own status pill.
The log's Failed filter returns everything that did not succeed, so Pending and Cancelled rows come back as failures.
The Overview counts finished requests only. Clicking its Failed requests card opens the log on the Failed filter, and that list is longer than the number on the card.
Opening a Pending request shows the message "Manifest rejected this request before contacting a provider", which reads like a hard limit and is not one.
Options:
Add the two states to the request model and one sentence on what Failed returns versus what the Overview counts. Recommended (the two numbers are a mismatch a user cannot reconcile alone).
Add the two states only, and leave the two numbers unexplained. Fixes the drawer confusion, not the arithmetic.
Leave the page and change the filter platform-side. That is a behaviour change on a filter other screens deep-link into, and the docs should describe today's product either way.
Suggested wording (option 1). Append to line 25, after "…and carries the error that stopped it.":
A request that is still running has no attempt yet and shows as Pending. A request whose caller disconnected shows as Cancelled, and it keeps the attempt that had already started, if there was one.
And a new paragraph between line 44 (</CardGroup>) and line 46, outside both deploy blocks because it holds in either mode:
The Overview counts finished requests only. The Requests log's Failed filter returns everything that did not succeed, including pending and cancelled requests. So the Overview's Failed requests card links to a list longer than the number on the card.
Evidence (for the agent)
observability.mdx:20-27 gives a Request exactly three shapes (succeeded, retried, Manifest-blocked). Read with :36 "The full log, newest first. Filter it, then open any row…". No in-flight state and no cancelled state anywhere on the page.
Pending: a Request is written with zero attempts at ingress, before routing (routing/proxy/proxy.controller.ts:256-268).
Cancelled: a caller disconnect writes a terminal cancelled Request and persists an attempt only if one already existed (proxy-message-recorder.ts:590-608).
Both render their own pills: Success / Pending / Cancelled / Failed (message-table-cells.tsx:402-419). Filter option label Failed at MessageLog.tsx:600-604.
The Failed filter is r.status NOT IN (SUCCESS_STATUS_SQL_LIST) i.e. not ok and not success (analytics/services/messages-query.service.ts:268-272), so pending and cancelled rows come back.
The Overview's predicate excludes them: sqlIsCompletedStatus (analytics/services/query-helpers.ts:73-75, the sibling sqlIsFailedStatus names the same exclusion at :67), used at analytics/services/request-volume.service.ts:136.
The Overview card label is Failed requests and it deep-links into that filter (AutofixKpiCards.tsx:168,170).
The zero-attempt drawer body reads "Manifest rejected this request before contacting a provider" (RequestDrawer.tsx:413-439). A Manifest-blocked request always writes an attempt row (proxy-message-recorder.ts:803), so that sentence is only ever shown to a pending or cancelled request. The platform-side half is parked in references/backlog.md.
Corpus check: no page documents these two request states today. request-logs.mdx:71's "canceled" is about subscriptions, not requests.
6 · [MISSING-CASE] The usage API section promises a budget check the page gives no way to build. It documents no parameter and no response field, and the two endpoints default to different windows.
You call the usage endpoint the way the page shows it, with no parameters, and get the last 24 hours.
You call the costs endpoint the same way and get the last 7 days.
Nothing on the page says the two windows differ, so a dashboard built from both compares 24 hours of tokens against 7 days of spend.
You guess a window value like 1d to fix it. Manifest answers 400 instead of falling back to a default.
Both windows count back from now, while a limit rule's period starts at a calendar boundary (finding 1).
So the "distance to my limit" the page suggests you build is wrong by construction.
Options:
Add the range parameter with its accepted values and the two different defaults, one sample response per endpoint, and one line saying the API window counts back from now while a limit period starts at a calendar boundary. Recommended (it makes the promised use case buildable).
Add only the defaults and the range parameter here, and put the response shapes in the API reference. That page does not carry them today, so the never-halfway invariant makes this a two-page fix for no reader benefit.
Drop the "stopping work before a limit blocks you" half of the promise and keep the section as a pointer to a dashboard-equivalent number.
Suggested wording (option 1) is long. It replaces lines 116-126 with the same intro, a curl carrying ?range=24h, a third column on the endpoint table giving the defaults 24h and 7d, a short paragraph on the accepted values and the 400, a paragraph on rolling-versus-calendar linking to Spend alerts, one JSON sample per endpoint, and one line saying message_count counts successful requests only. Full draft in the details block.
Evidence (for the agent)
observability.mdx:116 — "useful for showing a budget in your own UI or stopping work before a limit blocks you", then a parameter-less curl at :118-121 and a two-row table at :123-126 whose Returns column says only "Token usage for the calling harness" / "Cost data for the calling harness".
Accepted values: @IsIn(RANGE_VALUES) (common/dto/range-query.dto.ts:4-7), RANGE_VALUES = 1h|6h|24h|7d|30d|90d|365d (common/utils/range.util.ts:5). The global pipe runs whitelist + forbidNonWhitelisted (main.ts:113-121), so an unaccepted value is a 400, not a fallback. The DTO also whitelists agent_name (range-query.dto.ts:9-11), which these two endpoints ignore, so that one name does not 400.
API windows roll back from now: analytics/services/agent-analytics.service.ts:41-44. Limit periods are calendar windows: common/utils/period.util.ts:16-41 (finding 1).
Response fields, copied verbatim from AgentUsageResult and AgentCostsResult at analytics/services/agent-analytics.service.ts:14-32. agentName is appended by the controller on /usage only (agent-analytics.controller.ts:25-33); /costs does not carry it.
message_count uses sqlCountMessages (analytics/services/query-helpers.ts:90-96): distinct requests, excluding pending, cancelled and every error status. The token sums in the same response include every attempt row, failed ones included.
Corpus check: agent/usage and agent/costs appear only on observability.mdx, so no other page defines these shapes and this fix creates no halfway state.
Caveat on the draft: the numbers inside the two JSON samples are illustrative, not captured from a live response. They are internally consistent (total_tokens = input + output, the cache rates match the token counts, by_model sorted by cost descending per orderBy('cost_usd','DESC')) and the model IDs echo ones the corpus already uses (openai/gpt-4o at llm-gateway.mdx:82, claude-sonnet-4-6 at reference/headers.mdx:28, shown here with its provider prefix). Capture real values before merging if you want them exact.
Full draft for option 1:
A harness can query its own numbers with its `mnfst_` key, which is useful for showing a budget in your own UI or stopping work before a limit blocks you.
```bash
curl "https://app.manifest.build/api/v1/agent/usage?range=24h" \
-H "Authorization: Bearer mnfst_YOUR_KEY"```| Endpoint | Returns | Default `range`||---|---|---||`GET /api/v1/agent/usage`| Token usage for the calling harness |`24h`||`GET /api/v1/agent/costs`| Cost data for the calling harness |`7d`|
Both endpoints take a `range` parameter. It accepts `1h`, `6h`, `24h`, `7d`, `30d`, `90d` or `365d`, and any other value gets a `400`. The two defaults differ, so send `range` yourself whenever you compare the two responses.
A range counts back from now, while a limit rule's period starts at a calendar boundary ([Spend alerts](#spend-alerts)). No range lines up with a rule's window, so read these numbers as how close you are, not as the figure the rule reads.
`GET /api/v1/agent/usage` returns:
```json
{
"range": "24h",
"total_tokens": 128400,
"input_tokens": 96200,
"output_tokens": 32200,
"cache_read_tokens": 41000,
"cache_creation_tokens": 5200,
"cache_read_rate": 0.43,
"cache_write_rate": 0.05,
"message_count": 312,
"trend_pct": 12.4,
"agentName": "billing-bot"
}
````GET /api/v1/agent/costs` returns:
```json
{
"range": "7d",
"total_cost_usd": 18.42,
"trend_pct": -6.1,
"by_model": [
{ "model": "openai/gpt-4o", "cost_usd": 11.9, "input_tokens": 820000, "output_tokens": 140000 },
{ "model": "anthropic/claude-sonnet-4-6", "cost_usd": 6.52, "input_tokens": 210000, "output_tokens": 48000 }
]
}
````message_count` counts successful requests only. A failed or in-flight request is not in it.
7 · [MISSING-CASE] The Free-plan note covers the Overview only. The same lock sits on the Requests log, where All time stays free.
You open the Overview and find the 30-, 90- and 365-day ranges badged PRO and unselectable.
You open the Requests log and find the same three ranges badged and unselectable there too.
The Requests log also offers All time, which is not locked.
All time lists every request you ever sent, on the Free plan.
Options:
Extend the sentence to both screens and name the escape hatch: the long ranges need Pro on the Overview and on the Requests log, and the log's All time view is not restricted. Recommended (the All time clause is the one fact the badge on screen cannot give the reader).
Generalise without the escape hatch: the long ranges need Pro wherever you pick a range.
Leave it. The badge on the locked option already tells the user in place.
Suggested wording (option 1), replaces line 48 inside the cloud block:
On the Free plan, the 30-, 90- and 365-day ranges need Pro. The lock applies on the Overview and on the Requests log. The log's All time option is not locked, so you can still list every request you ever sent.
Evidence (for the agent)
observability.mdx:48 — "On the Free plan, the Overview covers up to 7 days of history; the 30-, 90-, and 365-day ranges need Pro."
The Overview's own badge and lock: GlobalOverview.tsx:141PRO_DASHBOARD_RANGES, badged and disabled at :219-226, selection refused at :185.
Same lock on the Requests log: MessageLog.tsx:86-87PRO_RANGES = {30d, 90d, 365d} with the comment "Same Pro gating as the Overview range selector: long windows are paid", built at :266, disabled and badged at :584-598, selection refused at :328.
All time is the first option, value '', outside the locked set, so it stays available on Free (MessageLog.tsx:587-598). The Overview has no All-time option at all (GlobalOverview.tsx:134-140).
The lock exists only when billing is enabled, so self-hosted installs have every range (billing/plan.service.ts:172-175, plan-store.ts:41-45). The sentence is correctly inside the cloud block and stays there.
The draft drops "covers up to 7 days of history" because it reads as a retention limit. Checked that All time is safe to promise: request-recording-retention.service.ts deletes stored bodies and nulls recording_key only, so the request rows themselves survive.
Corpus check: "all time" and "all-time" have zero hits in the corpus today, so this is the first mention and nothing elsewhere contradicts it.
Label note: the control carries label="Period filter" but Select.tsx:157 uses that string as an aria-label only, never painted on screen. The wording names the screen, not the control, and bolds only All time, which is real visible option text.
8 · [WRONG] The Limits card tells every reader the page carries email provider setup. A cloud user opens that page and finds a read-only panel.
You read the dashboard map in the Cloud view, which is this page's default view.
The Limits card says the page carries threshold rules per harness plus email provider setup.
You open a harness's Limits page on Manifest Cloud.
You find a read-only panel headed Email alerts that names your account email.
There is nothing to pick and nothing to save.
The page tells you the real story much further down, in the Self-hosted view.
Options:
Trim the card to "Threshold rules per harness." Recommended (three words, and the self-hosted block already names the exact screen).
Self-scope the clause in place: "Threshold rules per harness. Self-hosted installs also configure the email sender here."
Duplicate the card grid inside each deploy block so each mode gets its own four cards. Doubles a 14-line block to fix one clause.
Suggested wording (option 1), replaces line 42:
Threshold rules per harness.
Evidence (for the agent)
observability.mdx:41-43, inside the <CardGroup> at :31-44, which sits outside every data-deploy wrapper — "Limits … Threshold rules per harness, plus email provider setup."
pages/Limits.tsx:246-261 renders the provider picker only when isSelfHosted() is true. Cloud and unknown fall back to CloudEmailInfo, a read-only card reading "Email alerts / Alerts will be sent to <account email>" (CloudEmailInfo.tsx:15-17).
The "Configure email provider" heading and the three service tiles exist only on the self-hosted branch (EmailProviderSetup.tsx:19, mounted via EmailProviderSection.tsx:66-75).
AUDIENCE basis: the Cloud / Self-hosted rule makes a mode-dependent instruction sitting outside the toggle a WRONG finding in its own right.
Backlog re-decision: this point was parked on 2026-08-17 during the full audit of PR #2727 and explicitly deferred to this topic's own audit, which is this run. Re-decided on today's evidence, it becomes a finding. The mitigating fact the clearance relied on, that :98 tells cloud users where alerts go, survives as the reason severity is low and option 1 is cheap.
Cloud is the page's default view, so a cloud reader can conclude alerts are broken until they configure something they will never find.
Reply with the finding number and the chosen option
(e.g. "3 → option 2", or "3: other idea, let's discuss").
PRs are then made manually with the agent, never by the harness.
Rejected findings: reply "N → reject" and they will never be re-raised.
Scoped audit of the observability topic, run by the nightly rotation. No pull request has been merged into the platform's
mainbranch since #2733, so this run audited the topic whose last audit was the oldest (2026-08-10).observability.mdx, the only page this topic owns.mnfst/manifestatmain, commit6b045749b.mnfst/docsat83776bd.8 findings: 4 wrong claims, 4 missing cases. No image finding this run: two image requests already stand on this page and both wait on a capture (issue #54 finding 4, issue #70 finding 5).
Nine out-of-scope observations, seven of them platform-side, were written to
references/backlog.mdbefore this issue was filed. One of them matters here:providers/custom-providers.mdxcarries the same wrong claim as finding 3.Findings 1, 2 and 3 are the ones worth reading first. Finding 8 consumes a point parked on 2026-08-17 for this topic's own audit.
Findings
1 · [WRONG] The Spend alerts section calls the limit period rolling. The period is a calendar window that restarts at midnight, on Monday, or on the 1st.
Page: https://manifest.build/docs/observability (Spend alerts, the opening sentence)
What actually happens:
Options:
llm-gateway.mdx, which already describes the same mechanic from the blocking side).Suggested wording (option 1), replaces line 66:
Evidence (for the agent)
observability.mdx:66— "A limit rule watches one metric for one harness over a rolling period, and acts when the threshold is crossed."common/utils/period.util.ts:16-41computePeriodBoundaries:day=new Date(y, m, d)(local midnight today),week= this Monday 00:00,month= the 1st at 00:00,hour=now.getHours() - 1(top of the previous clock hour),periodEnd= now.common/utils/period.util.ts:9-15states the boundaries are computed in the Node process's LOCAL timezone deliberately, because they are compared against local-timeagent_messages.timestampvalues. Any wording must not say UTC.notifications/services/limit-check.service.ts:73(blocking),notifications/services/notification-cron.service.ts:104(alerting). Consumption windownotifications/services/notification-rules.service.ts:144-145.period.util.ts:47-72computePeriodResetDate. The in-app banner already says it:Limits.tsx:213-215"until the usage resets in the next period".llm-gateway.mdx:148says "The block clears on its own at the start of the next period". The two pages contradict each other today.rollingappears nowhere else in the corpus (the only other hit is "hand-rolling" inerrors/M300.mdx:34).2 · [WRONG] The page says a failed attempt can still burn tokens. Manifest records every failed attempt with zero tokens and no cost.
Page: https://manifest.build/docs/observability (Requests and provider attempts, the last sentence of the section)
What actually happens:
Options:
Suggested wording (option 1), replaces line 27:
Evidence (for the agent)
observability.mdx:27— "This is why the request count and the provider call count don't match, and why cost is attributed to attempts: a failed attempt can still burn tokens."routing/proxy/proxy-message-recorder.ts:cost_usdis written at exactly two sites, both on the success path,:1105(insiderecordFallbackSuccess, function at:1029) and:1203(insiderecordSuccessMessage, function at:1123). Non-zero token values only at:1101-1102and:1199-1200.recordProviderError(:638),recordManifestBlockedRequest(:737),recordFailedFallbacks(:807),recordPrimaryFailure(:934). Rows therefore keepbuildMessageRow's defaultsinput_tokens: 0, output_tokens: 0(:301-302) and a NULL cost.analytics/services/messages-query.service.ts:459,467-470. The spend chart (analytics/services/timeseries-queries.service.ts:89), the per-model table (:242,258) and a cost rule's consumption (notifications/services/notification-rules.service.ts:119-122) sum the same column.3 · [WRONG] The cost Tip tells custom-provider users Manifest cannot price their models. The connect form asks for those prices, and Manifest uses them.
Page: https://manifest.build/docs/observability (What's on each request, the Tip under the field list)
What actually happens:
Options:
Suggested wording (option 1), replaces lines 58-62:
Evidence (for the agent)
observability.mdx:59-61— "Local models recordcost = 0with real token counts and latency, since nothing was billed. Custom providers do the same, because Manifest can't infer pricing for a model it doesn't know."Model name,Input / 1M tokens,Output / 1M tokens(CustomProviderForm.tsx:410-420, under theModelslabel at:404). Inputs submitted at:142-147.model-prices/model-pricing-cache.service.ts:293-313stores an entry keyedcustom:<uuid>/<model_name>whenever at least one price is filled. Only entries with BOTH prices empty are skipped (:303).proxy-message-recorder.ts:1154-1164vs:1168), so the lookup hits andcomputeTokenCostreturns a real number. The service states the contract in its own comment (model-pricing-cache.service.ts:268-272). The storedagent_messages.modelis the canonicalised form (custom-provider.service.ts:183-186), which is why the lookup must run first.analytics/services/timeseries-queries.service.ts:89), the per-model table (:242,258) and a cost rule's consumption (notifications/services/notification-rules.service.ts:119-122).decisions.md, 2026-08-10 and 2026-08-11). Those clearances reasoned about how an UNPRICED model renders. This finding is about a model the user HAS priced, which no prior clearance's citations reach. The suggested wording deliberately keepscost = 0for the unpriced case so the settled point is untouched.providers/custom-providers.mdx:65, out of scope this run, parked inreferences/backlog.mdfor the providers topic. Under the never-halfway invariant the two pages should be fixed together.4 · [MISSING-CASE] The page promises one alert email per period. A failed send costs you that period's email, and the Limits page still counts the rule as triggered.
Page: https://manifest.build/docs/observability (Spend alerts, the paragraph on how often rules are evaluated)
What actually happens:
Options:
Suggested wording (option 1). Append to line 92, after "…won't flood your inbox.":
And append to the self-hosted paragraph at line 104, after "…where you can edit or remove it.":
Evidence (for the agent)
observability.mdx:92— "Each rule emails once per period, so a rule that stays over its threshold won't flood your inbox." Read with:110— "Without an email provider from either path, alerts are skipped." The page covers the no-sender case and never the failed-send case.notifications/services/notification-cron.service.ts:hasAlreadySentguard at:106,insertLogat:115-124,sendThresholdAlertat:126-140. The log row is written BEFORE the send is attempted. Same order innotifications/services/limit-check.service.ts:118,:126-135.notifications/services/notification-log.service.ts:14-20, one row per(rule_id, period_start).notification-cron.service.ts:141-143) describes a retry that cannot happen.Triggered(LimitRuleTable.tsx:80) and it renders a count of exactly those log rows (notification-rules.service.ts:44-55, cell atLimitRuleTable.tsx:128). The same rows fill the page's History panel (LimitHistoryTable.tsx:28).Send test email(EmailProviderModal.tsx:416). The primary button beside it isTest & Connect/Test & Save(:236), which the page already documents, so the wording points at the ghost button that tests without saving.references/backlog.md.5 · [MISSING-CASE] The page never mentions pending and cancelled requests. The Requests log shows both, and the Overview's failed count leaves them out.
Page: https://manifest.build/docs/observability (Requests and provider attempts, and the Requests card in The dashboard)
What actually happens:
Options:
Suggested wording (option 1). Append to line 25, after "…and carries the error that stopped it.":
And a new paragraph between line 44 (
</CardGroup>) and line 46, outside both deploy blocks because it holds in either mode:Evidence (for the agent)
observability.mdx:20-27gives a Request exactly three shapes (succeeded, retried, Manifest-blocked). Read with:36"The full log, newest first. Filter it, then open any row…". No in-flight state and no cancelled state anywhere on the page.routing/proxy/proxy.controller.ts:256-268).cancelledRequest and persists an attempt only if one already existed (proxy-message-recorder.ts:590-608).Success/Pending/Cancelled/Failed(message-table-cells.tsx:402-419). Filter option labelFailedatMessageLog.tsx:600-604.r.status NOT IN (SUCCESS_STATUS_SQL_LIST)i.e. notokand notsuccess(analytics/services/messages-query.service.ts:268-272), so pending and cancelled rows come back.sqlIsCompletedStatus(analytics/services/query-helpers.ts:73-75, the siblingsqlIsFailedStatusnames the same exclusion at:67), used atanalytics/services/request-volume.service.ts:136.Failed requestsand it deep-links into that filter (AutofixKpiCards.tsx:168,170).RequestDrawer.tsx:413-439). A Manifest-blocked request always writes an attempt row (proxy-message-recorder.ts:803), so that sentence is only ever shown to a pending or cancelled request. The platform-side half is parked inreferences/backlog.md.request-logs.mdx:71's "canceled" is about subscriptions, not requests.6 · [MISSING-CASE] The usage API section promises a budget check the page gives no way to build. It documents no parameter and no response field, and the two endpoints default to different windows.
Page: https://manifest.build/docs/observability (Reading usage from your own code)
What actually happens:
1dto fix it. Manifest answers 400 instead of falling back to a default.Options:
rangeparameter with its accepted values and the two different defaults, one sample response per endpoint, and one line saying the API window counts back from now while a limit period starts at a calendar boundary. Recommended (it makes the promised use case buildable).rangeparameter here, and put the response shapes in the API reference. That page does not carry them today, so the never-halfway invariant makes this a two-page fix for no reader benefit.Suggested wording (option 1) is long. It replaces lines 116-126 with the same intro, a
curlcarrying?range=24h, a third column on the endpoint table giving the defaults24hand7d, a short paragraph on the accepted values and the 400, a paragraph on rolling-versus-calendar linking to Spend alerts, one JSON sample per endpoint, and one line sayingmessage_countcounts successful requests only. Full draft in the details block.Evidence (for the agent)
observability.mdx:116— "useful for showing a budget in your own UI or stopping work before a limit blocks you", then a parameter-lesscurlat:118-121and a two-row table at:123-126whose Returns column says only "Token usage for the calling harness" / "Cost data for the calling harness".analytics/controllers/agent-analytics.controller.ts:26query.range ?? '24h';:37query.range ?? '7d'.@IsIn(RANGE_VALUES)(common/dto/range-query.dto.ts:4-7),RANGE_VALUES=1h|6h|24h|7d|30d|90d|365d(common/utils/range.util.ts:5). The global pipe runswhitelist+forbidNonWhitelisted(main.ts:113-121), so an unaccepted value is a 400, not a fallback. The DTO also whitelistsagent_name(range-query.dto.ts:9-11), which these two endpoints ignore, so that one name does not 400.analytics/services/agent-analytics.service.ts:41-44. Limit periods are calendar windows:common/utils/period.util.ts:16-41(finding 1).AgentUsageResultandAgentCostsResultatanalytics/services/agent-analytics.service.ts:14-32.agentNameis appended by the controller on/usageonly (agent-analytics.controller.ts:25-33);/costsdoes not carry it.message_countusessqlCountMessages(analytics/services/query-helpers.ts:90-96): distinct requests, excluding pending, cancelled and every error status. The token sums in the same response include every attempt row, failed ones included.agent/usageandagent/costsappear only onobservability.mdx, so no other page defines these shapes and this fix creates no halfway state.total_tokens= input + output, the cache rates match the token counts,by_modelsorted by cost descending perorderBy('cost_usd','DESC')) and the model IDs echo ones the corpus already uses (openai/gpt-4oatllm-gateway.mdx:82,claude-sonnet-4-6atreference/headers.mdx:28, shown here with its provider prefix). Capture real values before merging if you want them exact.Full draft for option 1:
7 · [MISSING-CASE] The Free-plan note covers the Overview only. The same lock sits on the Requests log, where All time stays free.
Page: https://manifest.build/docs/observability (The dashboard, the Free-plan note under the card grid, Cloud view)
What actually happens:
Options:
Suggested wording (option 1), replaces line 48 inside the cloud block:
Evidence (for the agent)
observability.mdx:48— "On the Free plan, the Overview covers up to 7 days of history; the 30-, 90-, and 365-day ranges need Pro."GlobalOverview.tsx:141PRO_DASHBOARD_RANGES, badged and disabled at:219-226, selection refused at:185.MessageLog.tsx:86-87PRO_RANGES = {30d, 90d, 365d}with the comment "Same Pro gating as the Overview range selector: long windows are paid", built at:266, disabled and badged at:584-598, selection refused at:328.All timeis the first option, value'', outside the locked set, so it stays available on Free (MessageLog.tsx:587-598). The Overview has no All-time option at all (GlobalOverview.tsx:134-140).billing/plan.service.ts:172-175,plan-store.ts:41-45). The sentence is correctly inside the cloud block and stays there.request-recording-retention.service.tsdeletes stored bodies and nullsrecording_keyonly, so the request rows themselves survive.label="Period filter"butSelect.tsx:157uses that string as anaria-labelonly, never painted on screen. The wording names the screen, not the control, and bolds onlyAll time, which is real visible option text.8 · [WRONG] The Limits card tells every reader the page carries email provider setup. A cloud user opens that page and finds a read-only panel.
Page: https://manifest.build/docs/observability (The dashboard, the Limits card in the four-card grid)
What actually happens:
Options:
Suggested wording (option 1), replaces line 42:
Evidence (for the agent)
observability.mdx:41-43, inside the<CardGroup>at:31-44, which sits outside everydata-deploywrapper — "Limits … Threshold rules per harness, plus email provider setup."pages/Limits.tsx:246-261renders the provider picker only whenisSelfHosted()is true. Cloud and unknown fall back toCloudEmailInfo, a read-only card reading "Email alerts / Alerts will be sent to<account email>" (CloudEmailInfo.tsx:15-17).EmailProviderSetup.tsx:19, mounted viaEmailProviderSection.tsx:66-75).:98tells cloud users where alerts go, survives as the reason severity is low and option 1 is cheap.Reply with the finding number and the chosen option
(e.g. "3 → option 2", or "3: other idea, let's discuss").
PRs are then made manually with the agent, never by the harness.
Rejected findings: reply "N → reject" and they will never be re-raised.