[FEATURE] Add SQL, Lucene and Query DSL query languages to the OpenSearch plugin by Oliver-ke · Pull Request #840 · perses/plugins

Oliver-ke · 2026-09-29T08:58:32Z

Description

The OpenSearch log query could only be written in PPL. This PR adds a per-query Query language selector with four options:

Language	Endpoint	Index	Notes
PPL (default)	`POST /_plugins/_ppl`	Optional (usually in `source=`)	Unchanged
SQL	`POST /_plugins/_sql`	From the `FROM` clause	Needs the SQL/PPL plugin, like PPL
Lucene	`POST /{index}/_search`	Required	`query_string` query. An empty query returns everything in the time range
Query DSL	`POST /{index}/_search`	Required	Raw `_search` body. The time range is added to it

The dashboard time range is added automatically for every language, unless Disable automatic time filtering is ticked. Lucene and Query DSL also get an optional Limit field, which sets size (default 500). All four languages return the same LogData shape, so logs panels treat them the same.

Changes

Spec: new optional fields queryLanguage (ppl | sql | lucene | dsl, default ppl) and limit. index is now required for lucene and dsl.
CUE schema: validates the rules above. query may be empty only for lucene. There are new valid and invalid schema test files for each language.
Go SDK: new QueryLanguage, Index and Limit options, with validation that matches the schema.
Client: new sql() and search() calls next to ppl(). The index in the _search URL is checked against an allowlist of characters, so it can't be used for path traversal.
Datasource editor: new proxied datasources allow /_plugins/_sql and /.*/_search as well as /_plugins/_ppl.
Query editor:
- a language selector
- help text for the index field that changes with the language
- an "index required" error for Lucene and Query DSL
- the Limit field
- examples and docs links for each language
Results: _search hits are converted to log entries. Nested _source fields are flattened to dot-separated labels, and _id and _index are added as labels. hasMore is set when OpenSearch stopped counting or the result was cut off by size. Numeric epoch timestamps in seconds, milliseconds, microseconds or nanoseconds are recognised, including fractional ones.
Errors: when the Perses proxy blocks an endpoint with a 403, the error says which endpoints to add to allowedEndpoints

Screenshots

Checklist

Pull request has a descriptive title and context useful to a reviewer.
Pull request title follows the [<catalog_entry>] <commit message> naming convention using one of the
following catalog_entry values: FEATURE, ENHANCEMENT, BUGFIX, BREAKINGCHANGE, DOC,IGNORE.
All commits have DCO signoffs.

UI Changes

Changes that impact the UI include screenshots and/or screencasts of the relevant changes.
Code follows the UI guidelines.

Signed-off-by: Kelechi Oliver Azorji <kelechioliver96@gmail.com>

…ane-dsl-opensearch-plugin Signed-off-by: Kelechi Oliver Azorji <kelechioliver96@gmail.com>

Signed-off-by: Kelechi Oliver Azorji <kelechioliver96@gmail.com>

Copilot

Copilot review overview

🟡 Changes recommended

Public API regressions and limit, validation, and search-construction defects remain unresolved.

Review effort: Balanced
Findings: 7 · 1

Open (8)

Reject explicit zero limits during Go decoding · New
Preserve deprecated OpenSearchPPL type aliases · New
Preserve deprecated OpenSearchPPLError compatibility export · New
Reject decimal limits instead of truncating with parseInt · New
Ensure configured limit overrides raw DSL size · New
Use date for timestamp unmapped_type · New
Preserve deprecated convertPPLToLogs export · New
Document dashboard query language configuration · New

What changed in this PR

Adds SQL, Lucene, and Query DSL support to OpenSearch log queries across runtime, UI, schema, Go SDK, documentation, and tests.

Changes:

Adds per-query language selection, limits, index validation, and time filtering.
Adds SQL and _search clients plus response conversion.
Extends CUE/Go contracts and test coverage.

File	Description
`opensearch/src/queries/opensearch-log-query/OpenSearchLogQueryEditor.tsx`	Adds language and limit controls.
`opensearch/src/queries/opensearch-log-query/OpenSearchLogQueryEditor.test.tsx`	Tests editor behavior.
`opensearch/src/queries/opensearch-log-query/OpenSearchLogQuery.tsx`	Defaults new queries to PPL.
`opensearch/src/queries/opensearch-log-query/opensearch-log-query-types.ts`	Defines languages and fields.
`opensearch/src/queries/opensearch-log-query/opensearch-log-query-types.test.ts`	Tests language resolution.
`opensearch/src/queries/opensearch-log-query/index.ts`	Exports builders and converters.
`opensearch/src/queries/opensearch-log-query/get-opensearch-log-data.ts`	Dispatches requests by language.
`opensearch/src/queries/opensearch-log-query/get-opensearch-log-data.test.ts`	Tests dispatch and conversion.
`opensearch/src/queries/opensearch-log-query/convert/parse-timestamp.ts`	Normalizes timestamp formats.
`opensearch/src/queries/opensearch-log-query/convert/index.ts`	Exports conversion utilities.
`opensearch/src/queries/opensearch-log-query/convert/hits-to-logs.ts`	Converts search hits to logs.
`opensearch/src/queries/opensearch-log-query/convert/hits-to-logs.test.ts`	Tests hit conversion.
`opensearch/src/queries/opensearch-log-query/convert/datarows-to-logs.ts`	Converts PPL/SQL rows.
`opensearch/src/queries/opensearch-log-query/build/sql.ts`	Builds SQL requests.
`opensearch/src/queries/opensearch-log-query/build/sql.test.ts`	Tests SQL requests.
`opensearch/src/queries/opensearch-log-query/build/search.ts`	Provides shared search helpers.
`opensearch/src/queries/opensearch-log-query/build/ppl.ts`	Extracts PPL request construction.
`opensearch/src/queries/opensearch-log-query/build/lucene.ts`	Builds Lucene search bodies.
`opensearch/src/queries/opensearch-log-query/build/lucene.test.ts`	Tests Lucene bodies.
`opensearch/src/queries/opensearch-log-query/build/index.ts`	Exports request builders.
`opensearch/src/queries/opensearch-log-query/build/dsl.ts`	Builds Query DSL bodies.
`opensearch/src/queries/opensearch-log-query/build/dsl.test.ts`	Tests Query DSL handling.
`opensearch/src/queries/constants.ts`	Adds language metadata and defaults.
`opensearch/src/model/opensearch-client.ts`	Adds SQL/search transports and errors.
`opensearch/src/model/opensearch-client.test.ts`	Tests transports and path safety.
`opensearch/src/model/opensearch-client-types.ts`	Adds shared response types.
`opensearch/src/datasources/opensearch-datasource/OpenSearchDatasourceEditor.tsx`	Allows new proxy endpoints.
`opensearch/src/datasources/opensearch-datasource/OpenSearchDatasource.tsx`	Exposes new client methods.
`opensearch/sdk/go/query/log/options.go`	Adds language and limit options.
`opensearch/sdk/go/query/log/log.go`	Extends and validates Go specs.
`opensearch/sdk/go/query/log/log_test.go`	Tests Go SDK additions.
`opensearch/schemas/queries/opensearch-log-query/tests/valid/sql.json`	Covers valid SQL.
`opensearch/schemas/queries/opensearch-log-query/tests/valid/ppl.json`	Covers explicit PPL.
`opensearch/schemas/queries/opensearch-log-query/tests/valid/no-language.json`	Covers legacy defaulting.
`opensearch/schemas/queries/opensearch-log-query/tests/valid/lucene.json`	Covers valid Lucene.
`opensearch/schemas/queries/opensearch-log-query/tests/valid/lucene-empty-query.json`	Covers blank Lucene queries.
`opensearch/schemas/queries/opensearch-log-query/tests/valid/dsl.json`	Covers valid Query DSL.
`opensearch/schemas/queries/opensearch-log-query/tests/invalid/zero-limit.json`	Rejects zero limits.
`opensearch/schemas/queries/opensearch-log-query/tests/invalid/unknown-language.json`	Rejects unknown languages.
`opensearch/schemas/queries/opensearch-log-query/tests/invalid/sql-empty-query.json`	Rejects empty SQL.
`opensearch/schemas/queries/opensearch-log-query/tests/invalid/negative-limit.json`	Rejects negative limits.
`opensearch/schemas/queries/opensearch-log-query/tests/invalid/lucene-without-index.json`	Requires Lucene index.
`opensearch/schemas/queries/opensearch-log-query/tests/invalid/dsl-without-index.json`	Requires Query DSL index.
`opensearch/schemas/queries/opensearch-log-query/query.cue`	Defines the expanded schema.
`opensearch/README.md`	Advertises supported languages.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

ibakshay · 2026-10-03T15:25:55Z

Hi @Oliver-ke,
Thank you very much for the PR! Could you please address all the review comments from the copilot and fix merge conflicts? I will then do a manual review :)

…ane-dsl-opensearch-plugin # Conflicts: # opensearch/src/datasources/opensearch-datasource/OpenSearchDatasource.tsx # opensearch/src/model/opensearch-client.ts

Signed-off-by: Kelechi Oliver Azorji <kelechioliver96@gmail.com>

…ane-dsl-opensearch-plugin

Signed-off-by: Kelechi Oliver Azorji <kelechioliver96@gmail.com>

Oliver-ke · 2026-10-12T13:58:14Z

Hi @Oliver-ke, Thank you very much for the PR! Could you please address all the review comments from the copilot and fix merge conflicts? I will then do a manual review :)

@ibakshay fixes applied, kindly review

ibakshay · 2026-10-14T09:03:27Z

Hi @Oliver-ke, Thank you very much for the PR! Could you please address all the review comments from the copilot and fix merge conflicts? I will then do a manual review :)

@ibakshay fixes applied, kindly review

Added to my TODO list. Looking forward :)

Copilot

🟡 Changes recommended

Query DSL handling can broaden malformed queries and alter scoring semantics, while error and prerequisite documentation also need correction.

4 open findings

8 resolved since last review

🧠 Review effort: Balanced

+  { timestampField = DEFAULT_TIMESTAMP_FIELD, disableTimeFilter = false, limit }: SearchBodyOptions = {},
+): OpenSearchSearchRequestBody {
+  const parsed = parseBody(userJson);
+  const userQuery = parsed.query as Record<string, unknown> | undefined;


+  const filter: Array<Record<string, unknown>> = [buildTimeRangeFilter(timestampField, start, end)];
+  if (userQuery) {
+    filter.push(userQuery);
+  }
+  body.query = { bool: { filter } };


+- A **current Perses release** — the OpenSearch plugin ships bundled, so there's
+  nothing to install (you'll confirm this in [Step 2](#step-2--run-perses)).
+- An **OpenSearch** cluster with the SQL/PPL plugin — bundled by default.
+


+    const rootCause = parsed?.error?.root_cause?.[0]?.reason;
+    const suffix = reason && details && reason !== details ? `${reason} — ${details}` : reason || details || rootCause;


ibakshay · 2026-10-17T19:45:48Z

A few UX suggestions from trying the query editor locally. None are blockers, but they seem cheap to do in this PR:

Move the query box up. It's currently the last field, after datasource, index, timestamp field, message field, the time-filter checkbox and Limit. Since it's the field people use most, I'd put it right after Query language / Datasource / Index. The optional fields (timestamp field, message field, time filtering, Limit) could then go in a collapsible "Options" section below it.
Use a positive label for the time-filter checkbox. "Disable automatic time filtering" is a negative you have to think about for a second. Something like "Use dashboard time range", checked by default, says the same thing more directly. The spec field can stay as it is; this would only change the UI label.
Datasource label is doubled. The field has a "Datasource" heading and an "OpenSearch Datasource" floating label right under it, which looks cramped next to the other fields. One label is enough.
Switching language keeps the old query. If I switch a PPL query to Lucene, the PPL text stays in the box. The query then returns nothing or fails with an unrelated error. It'd be nicer to keep one query per language and restore it when switching back. Or, more simply, clear the box so the language's placeholder or examples show.

ibakshay · 2026-10-17T19:54:21Z

I also found one bug when I was testing locallly.

So, When automatic time filtering is on (the default), the SQL query sends the time range in the _sql request's filter field. OpenSearch only supports filter in its legacy SQL engine, so every SQL query with the time filter enabled runs on the legacy engine. OpenSearch's new SQL engine only accepts query, fetch_size, parameters and cursor

I tested this on OpenSearch 2.19 and saw three problems:

Wrong timestamps. The legacy engine reads microseconds as milliseconds. A document at 16:35:02.798478Z shows in the panel as 16:48:20.478Z, about 13 minutes off. Ticking "Disable automatic time filtering" fixes it. Microsecond and nanosecond timestamps are common, e.g. OTel logs.
Valid queries fail. SELECT * FROM app-logs WHERE match(message, 'timeout') LIMIT 20 returns Invalid SQL query: syntax error, expect AGAINST, actual LIMIT. It works with the time filter disabled.
Different results. DATE_FORMAT([@timestamp](https://github.com/timestamp)`, '%Y')returns%2026instead of2026`.

To reproduce directly against OpenSearch:

# new engine: correct
curl -XPOST localhost:9200/_plugins/_sql -H 'Content-Type: application/json' \
  -d '{"query":"SELECT `@timestamp` FROM app-logs ORDER BY `@timestamp` DESC LIMIT 1"}'
# => "2026-10-17 16:39:55.798478"

# same query with filter: legacy engine, shifted
curl -XPOST localhost:9200/_plugins/_sql -H 'Content-Type: application/json' \
  -d '{"query":"SELECT `@timestamp` FROM app-logs ORDER BY `@timestamp` DESC LIMIT 1","filter":{"range":{"@timestamp":{"gte":0,"format":"epoch_millis"}}}}'
# => "2026-10-17 16:53:13.478"

Oliver-ke added 2 commits September 27, 2026 10:33

feature: Add more query type support to Opensearch plugin

7d368fd

Signed-off-by: Kelechi Oliver Azorji <kelechioliver96@gmail.com>

chore: query improvement fixes

2868846

Signed-off-by: Kelechi Oliver Azorji <kelechioliver96@gmail.com>

Oliver-ke requested review from a team, AntoineThebaud and Nexucis as code owners September 29, 2026 08:58

Oliver-ke requested review from jgbernalp and removed request for a team September 29, 2026 08:58

Oliver-ke mentioned this pull request Sep 29, 2026

OpenSearch: Support SQL, Lucene, and Query DSL perses/perses#4378

Open

11 tasks

Oliver-ke added 2 commits September 29, 2026 11:00

Merge remote-tracking branch 'upstream/main' into feature/add-sql-luc…

0539c1c

…ane-dsl-opensearch-plugin Signed-off-by: Kelechi Oliver Azorji <kelechioliver96@gmail.com>

fix: remove embedded field from selector in opensearch SDK

674ab14

Signed-off-by: Kelechi Oliver Azorji <kelechioliver96@gmail.com>

jgbernalp requested a balanced review from Copilot October 1, 2026 09:57

Copilot started reviewing on behalf of jgbernalp October 1, 2026 09:57

View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

Oliver-ke added 4 commits October 12, 2026 02:08

Merge remote-tracking branch 'upstream/main' into feature/add-sql-luc…

73cc631

…ane-dsl-opensearch-plugin # Conflicts: # opensearch/src/datasources/opensearch-datasource/OpenSearchDatasource.tsx # opensearch/src/model/opensearch-client.ts

fix: address Copilot review feedback

8ea14ef

Signed-off-by: Kelechi Oliver Azorji <kelechioliver96@gmail.com>

Merge remote-tracking branch 'upstream/main' into feature/add-sql-luc…

6d03035

…ane-dsl-opensearch-plugin

chore: fix document formatting

adbe50d

Signed-off-by: Kelechi Oliver Azorji <kelechioliver96@gmail.com>

ibakshay requested a balanced review from Copilot October 14, 2026 09:04

Copilot started reviewing on behalf of ibakshay October 14, 2026 09:05

View session

Copilot AI reviewed Oct 14, 2026

View reviewed changes

Navigation Menu

Repository navigation

[FEATURE] Add SQL, Lucene and Query DSL query languages to the OpenSearch plugin - #840

[FEATURE] Add SQL, Lucene and Query DSL query languages to the OpenSearch plugin#840
Oliver-ke wants to merge 8 commits into
perses:mainfrom
Oliver-ke:feature/add-sql-lucane-dsl-opensearch-plugin

Oliver-ke commented Sep 29, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ibakshay commented Oct 3, 2026 •

edited

Loading

Uh oh!

Oliver-ke commented Oct 12, 2026

Uh oh!

ibakshay commented Oct 14, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

ibakshay commented Oct 17, 2026

Uh oh!

ibakshay commented Oct 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		const rootCause = parsed?.error?.root_cause?.[0]?.reason;
		const suffix = reason && details && reason !== details ? `${reason} — ${details}` : reason \|\| details \|\| rootCause;

Conversation

Oliver-ke commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Screenshots

Checklist

UI Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ibakshay commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Oliver-ke commented Oct 12, 2026

Uh oh!

ibakshay commented Oct 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

ibakshay commented Oct 17, 2026

Uh oh!

ibakshay commented Oct 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Oliver-ke commented Sep 29, 2026 •

edited

Loading

ibakshay commented Oct 3, 2026 •

edited

Loading