Blog: Kubernetes observability in 2026 with OpenObserve 1.0 as the backend - #268
Open
saiyam1814 wants to merge 25 commits into
Open
saiyam1814 wants to merge 25 commits into
saiyam1814 wants to merge 25 commits into
Conversation
Deploying website with
|
| Latest commit: |
3f34357
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://d9f307e7.website-dab.pages.dev |
| Branch Preview URL: | https://blog-kubernetes-observabilit.website-dab.pages.dev |
…shot captions, rc2 note
…, real log line, versions
…AQ, entity schema - Lead the TL;DR and meta description with the 31 ms search and 43m CPU numbers - Align title and seoTitle on "with OpenObserve 1.0 as the backend" - Link OpenObserve pages on first relevant mention instead of only in the footer - Name Loki, Elasticsearch and the LGTM stack next to the comparison table - Open each "How it's built" subsection with one self-contained sentence - Six-question FAQ in frontmatter (renders the FAQ section and FAQPage JSON-LD) - Template: `about` and `mentions` frontmatter entities in the BlogPosting JSON-LD, plus `dateModified` support with a visible Updated date Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Adds the OpenObserve vs Prometheus and Mimir post, which runs both file formats on the same engine, next to the log-records comparison. The section fragment Jacob sent does not resolve (headings on that page have no ids), so the link points at the post. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
OpenObserve's CEO flagged that showing a DuckDB query against a Vortex data file reads as a recommendation to query OpenObserve with DuckDB, which is not the intention. Saying the files are open is fine; showing the demo is what misleads. - Step 5 loses the brew install, the read_vortex query and the result table. Bar item 3 now rests on the magic bytes already printed, stated as the strong hint it is rather than a proof, and the step says plainly that this is an archive property and not a way to query the cluster. - Parquet and Vortex no longer get one portability claim between them. Spark and pandas read Parquet; Vortex's reader list is still short. - Step 6 leads with registering the MCP server in Claude Code, copied from the setup page in the UI, with the raw JSON-RPC kept as the path an agent takes. No agent session is claimed, since none was captured. - duckdb out of the prerequisites, Claude Code in as optional. DuckDB survives only in the lakehouse sentence, which is about Iceberg and the warehouses rather than about OpenObserve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rebuilt the demo cluster on 15 September to capture what the CEO asked for: the MCP server registered in Claude Code, a question asked in plain English, and the answer that came back. The agent listed the streams, read the traces schema and wrote its own SQL, so the post now shows that query instead of asserting anything about it. It also caught two things unprompted: that 154 error spans are 77 failed requests rather than 154 incidents, and that "the last hour" was really thirteen minutes on a cluster that young. The raw JSON-RPC output below it was recaptured on the same cluster so both halves of the step come from one run. datePublished moves to 15 September. PR stays a draft; this only takes effect on merge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Tested against the running instance: "Basic <token>" as a bare --header value returns 401, "Authorization: Basic <token>" returns 200. The UI source confirms the Claude Code template omits the header name while the Cursor, VS Code and Codex templates wrap it in Authorization. The post told readers to copy the command as it stands, which would not have worked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
saiyam1814
changed the title
saiyam1814
marked this pull request as ready for review
September 15, 2026 05:38
Two small changes, neither touching the captured output. The lead-in above the agent's answer now says the failures are the demo app's own 2 percent rate and that the cluster was minutes old. The answer's closing caveat about the traces stream holding only thirteen minutes is accurate and stays verbatim, but lifted out of context it reads like an OpenObserve retention problem rather than a young cluster, and the explanation sat a paragraph too late to prevent that. The 401 note now links openobserve/openobserve#14489, reported upstream. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
I tested the header against my own rc1 instance and read the bundle it
was serving, then wrote it up as a current bug and filed it upstream.
Checking the template at each tag shows rc1 emits `--header "${auth}"`
and rc2 onward emit `--header "Authorization: ${auth}"`, so it was fixed
on 3 September and only ever affected rc1.
The note is now scoped to rc1 as a pinning wrinkle, and the link to the
upstream issue is gone, since I closed it as already fixed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1.0.0 shipped on 11 September, after this post's runs. Upgraded the cluster to GA and re-ran the two steps whose claims depend on the build. The SLO bug survived GA. The release notes say the storage layer split into separate ORM read/write clients and retired the sqlite write lock, so I expected this gone, but the SLO pass still writes through the read-only client. On GA it backfills to full coverage, computes an SLI of 98.009 against a 99 target, then freezes: computed_at did not move for twenty minutes with the load generator failing 60 percent of payments, and the burn-rate alert never fired. Reported as openobserve#14490. That finding now also appears under sharp edges, where a single-node reader will actually see it, and the closing ask is sharpened to say the SQLite path is still stuck on 1.0.0. Intro, versions line, values pin and release link all move to GA, while saying plainly that the walkthrough itself ran on rc1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 6's agent exchange and raw JSON-RPC output were both captured on rc1. Re-ran them against a clean 1.0.0 install so every number in the step comes from the shipped release. The GA answer is a better demonstration than the rc1 one: the agent reads the schema, writes five queries of its own, separates 104 error spans from 52 failed requests, notes that inventory.reserve is clean and so the fault sits at the payment gateway boundary, and recovers the demo app's configured 2 percent failure rate as 2.15 percent measured. It still volunteers that "the last hour" was really eight minutes on a cluster that young. Getting there needed a reinstall: deleting the traces stream to clear the chaos-rate data left it wedged in "being deleted" for half an hour, discarding new spans and surviving a pod restart. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drop the stale rc1 header caveat from step 6: the values file pins 1.0.0, so nobody following the post can hit an rc1-only bug. The bigger problem, which a consistency sweep caught on four independent reads: the intro claimed every number in steps 6 and 7 came from the shipped release. That is true of step 6, which was re-run in full, but false of step 7, which still carries the rc1 SLO page reading of 97.907 percent and an alert payload stamped 2 September. Two SLI figures then sat a few lines apart with nothing saying they were different builds. Step 7 now labels the rc1 run and the GA re-check separately, the intro says exactly what was re-run, and the version list no longer contradicts the values file. Also: `idx_scan_size` reports 0 on the full-text query that does the most pruning and 135 on the two that scan everything. Reproduced on 1.0.0, so the post now tells readers to trust the scan_records drop instead. Plus dangling pronouns left by the DuckDB removal and a repeated sentence in step 6. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Chasing the last inconsistencies turned up the real story. The fix for this was already written: #14192 moved the SLO status write onto the read-write client and merged to main on 9 September. v1.0.0 was tagged on the 11th without it. At the tag commit_status still takes the read-only handle; on main it does not. So the post now says that, which is both more precise and fairer: they solved it, the 1.0.0 line was cut from a branch that never got the backport. Also from the consistency sweep: the compression FAQ said "container logs" for a figure covering all four log streams, and that answer is emitted as FAQPage JSON-LD, so the wrong word travels. Two storage snapshots taken hours apart were easy to mistake for each other. The index-per-data-file annotation claimed an exactness the file counts do not support. The count query is the slowest of the three, not the fastest. And one clause opened two paragraphs in the same step. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Checked the refs rather than assuming. #14192 merged to main on 9 September and #14453 backported it to branch-v1.0.0 on the 12th, one day after v1.0.0 was tagged, and no patch release has shipped. So the branch did get it and the published image did not, which is a release-timing miss rather than a branch that was never updated. Adds the workaround while that patch is pending: point the metadata store at PostgreSQL instead of SQLite, since its read-only client falls back to the read-write connection and the failing write succeeds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ready for review. Do not merge until we decide to publish —
datePublishedis 2026-09-15, so the post goes live with that date whenever this merges.Post: Kubernetes observability in 2026 with OpenObserve 1.0 as the backend
Problem (why the backend is where observability cost lives), landscape, how OpenObserve 1.0 is built (write path, index file, query funnel, Vortex, MCP), and a hands-on run on a kiac cluster with a companion repo: https://github.com/saiyam1814/openobserve-k8s-demo
Preview: https://blog-kubernetes-observabilit.website-dab.pages.dev/blog/kubernetes-observability-in-2026-with-openobserve
Assets: cover, 5 excalidraw-style diagrams, 2 GIF animations, 8 screenshots under
public/img/blog/kubernetes-observability-in-2026-with-openobserve/(2.6 MB total).Review rounds with the OpenObserve team, all applied
Version provenance
1.0.0 went GA on 11 September, after the original runs. The walkthrough and the step 8 measurements are from rc1 on 2 and 4 September. Step 6 was re-run in full on 15 September against GA. In step 7 only the SLO bug was re-checked against GA, and the post labels the rc1 run and the GA re-check separately.
The SLO finding is the post's sharpest. SLO alerts never fire on a single-node install, because the status row write goes through the read-only client and SQLite refuses it. That was reported during the rc1 run (openobserve#14189) and fixed by openobserve#14192, which merged to
mainon 9 September. v1.0.0 was tagged on the 11th without it, so the shipped release is still affected. Tracked upstream in openobserve#14490.Template changes in this PR
lib/blog.jsandapp/blog/[slug]/page.jsgain optionalabout:,mentions:anddateModified:frontmatter, all backward compatible with the other posts.🤖 Generated with Claude Code