diff --git a/docs/opensearch/README.md b/docs/opensearch/README.md index 898286dda..6a2c5c431 100644 --- a/docs/opensearch/README.md +++ b/docs/opensearch/README.md @@ -4,13 +4,21 @@ tags: --- # OpenSearch plugins -The OpenSearch plugin package provides support for [OpenSearch](https://opensearch.org/) in Perses dashboards. Logs are queried with [PPL (Piped Processing Language)](https://opensearch.org/docs/latest/search-plugins/sql/ppl/index/), which requires the SQL/PPL plugin on your cluster — it ships with OpenSearch by default. +The OpenSearch plugin package provides support for [OpenSearch](https://opensearch.org/) in Perses dashboards. Logs can be queried with [PPL (Piped Processing Language)](https://docs.opensearch.org/latest/sql-and-ppl/ppl/index/), [SQL](https://docs.opensearch.org/latest/sql-and-ppl/sql/index/), [Lucene / query string](https://docs.opensearch.org/latest/query-dsl/full-text/query-string/) or raw [Query DSL](https://docs.opensearch.org/latest/query-dsl/), selected per query. PPL and SQL require the SQL/PPL plugin on your cluster — it ships with OpenSearch by default. Lucene and Query DSL use the core `_search` API. ## Datasource (`OpenSearchDatasource`) The OpenSearch datasource enables connection between Perses and your OpenSearch cluster for log aggregation and querying. It works with log-related panels and queries. -It supports the [proxy](https://perses.dev/perses/docs/concepts/proxy/) feature of Perses that allows to restrict the access to your data source. All queries are sent to a single endpoint, `POST /_plugins/_ppl`, so that is the only endpoint you need to allow. +It supports the [proxy](https://perses.dev/perses/docs/concepts/proxy/) feature of Perses that allows to restrict the access to your data source. Each query language uses its own endpoint, so allow the ones you need: + +| Query language | Endpoint | `endpointPattern` | +|-------------------|-------------------------|-------------------| +| PPL | `POST /_plugins/_ppl` | `/_plugins/_ppl` | +| SQL | `POST /_plugins/_sql` | `/_plugins/_sql` | +| Lucene, Query DSL | `POST /{index}/_search` | `/.*/_search` | + +A datasource created in the Perses UI allows all three by default. A datasource created before SQL, Lucene and Query DSL support only allows `/_plugins/_ppl`; queries in the other languages are then rejected by the proxy with a 403 until you add their endpoints. See also technical docs related to this plugin: @@ -19,7 +27,16 @@ See also technical docs related to this plugin: ## Log Query (`OpenSearchLogQuery`) -The OpenSearch log query plugin enables querying log data from your OpenSearch cluster using PPL syntax. It supports filtering, projection and aggregation through PPL pipes. +The OpenSearch log query plugin enables querying log data from your OpenSearch cluster. The `queryLanguage` field selects how `query` is interpreted: + +| `queryLanguage` | `query` is | `index` | +|-----------------|----------------------------------------------|-----------------------------------------------| +| `ppl` (default) | a PPL expression | Optional, see [Index pattern](#index-pattern) | +| `sql` | a SQL statement | Ignored — the index comes from `FROM` | +| `lucene` | a query string; blank matches every document | Required | +| `dsl` | a JSON `_search` request body | Required | + +A query without `queryLanguage` is a PPL query, so dashboards saved before the other languages existed keep working. See also technical docs related to this plugin: @@ -28,11 +45,21 @@ See also technical docs related to this plugin: ### Index pattern -The `index` field of the query is prepended to your expression as a `source=` clause. It is ignored when your expression already starts with `source=` (or `search source=`), so you can either set the index once on the query or write the source clause yourself. +For Lucene and Query DSL, `index` is required: it is the index or index pattern in the `POST /{index}/_search` request path. For SQL it is ignored. For PPL, the `index` field of the query is prepended to your expression as a `source=` clause. It is ignored when your expression already starts with `source=` (or `search source=`), so you can either set the index once on the query or write the source clause yourself. + +List several indices with commas (`logs-*,app-*`; spaces after the commas are ignored), and use `cluster:index` for cross-cluster search. A variable in `index` that has several values selected is joined with commas: `$index` with `logs-a` and `logs-b` becomes `logs-a,logs-b`. + +### Result limit + +`limit` sets how many log documents a Lucene or Query DSL query fetches, as the `_search` request's `size`. It defaults to 500 and must be an integer from 1 to 10,000, because `_search` rejects a larger `size` under OpenSearch's default `index.max_result_window`. For Query DSL, a `size` in your request body is kept, but capped at `limit` when both are set, and at 10,000 when it is set alone. `limit` does not apply to PPL or SQL — use `head` or `LIMIT` in the query instead. ### Automatic time filtering -The panel time range is injected into the query as a `where` clause on the timestamp field, placed immediately after the source clause so it applies before any pipe that drops the timestamp column, such as `stats` or `fields`. A query written as: +The panel time range is added to every query as a filter on the timestamp field: + +- **SQL**: added to the outer query's `WHERE` clause as a condition on the timestamp field, ANDed in front of your own condition (which is wrapped in parentheses), or inserted as a new `WHERE` before `GROUP BY`, `HAVING`, `ORDER BY` or `LIMIT`. Only a single `SELECT … FROM` query can be filtered this way: a statement with `UNION` or `JOIN`, or one that isn't a `SELECT`, is rejected with an error. Set `disableTimeFilter` and filter on the timestamp yourself in that case. The `_sql` API's `filter` field is not used because OpenSearch only supports it on its legacy SQL engine. +- **Lucene and Query DSL**: added as a `range` clause in `bool.filter`. Lucene queries sit in the same `bool.filter`; a Query DSL `query` goes under `bool.must`, so `_score` sorting, `track_scores` and `min_score` still work. A Query DSL body without a `query` key is filtered on the time range alone. +- **PPL**: injected into the query as a `where` clause on the timestamp field, placed immediately after the source clause so it applies before any pipe that drops the timestamp column, such as `stats` or `fields`. A query written as: ```ppl source=otel-logs-demo | where severityText='ERROR' @@ -44,16 +71,47 @@ is sent to OpenSearch as: source=otel-logs-demo | where `@timestamp` >= '2026-07-14T03:53:33.000Z' and `@timestamp` <= '2026-07-14T04:53:33.000Z' | where severityText='ERROR' ``` -Set `disableTimeFilter` to `true` to turn this off and manage the time bounds yourself. +Set `disableTimeFilter` to `true` to turn this off, in every language, and manage the time bounds yourself. + +Lucene and Query DSL results are sorted newest first on the timestamp field. A Query DSL body that sets its own `sort` keeps it. + +### Time range variables + +`$__from` and `$__to` are replaced with the start and end of the panel time range, in epoch milliseconds, in every query language and in `index`. Use them to bound a query yourself when `disableTimeFilter` is set, for example a SQL statement that automatic time filtering rejects: + +| Language | Example | +| --- | --- | +| SQL | ``WHERE `@timestamp` >= FROM_UNIXTIME($__from / 1000.0) AND `@timestamp` <= FROM_UNIXTIME($__to / 1000.0)`` | +| Lucene | `@timestamp:[$__from TO $__to]` | +| Query DSL | `{"range": {"@timestamp": {"gte": $__from, "lte": $__to, "format": "epoch_millis"}}}` | + +In SQL, divide by `1000.0` rather than `1000`: integer division drops the milliseconds. A dashboard variable with the same name takes precedence. + +### Multi-value variables + +A variable with several values selected, referenced without a format, is written in the syntax of the query language: + +| Language | `$level` with `error` and `warn` | Use it as | +| --- | --- | --- | +| SQL | `'error','warn'` | `WHERE level IN ($level)` | +| Lucene | `("error" OR "warn")` | `level:$level` | +| PPL | `(error\|warn)` | Perses' default; give a format such as `${level:singlequote}` | +| `index` | `logs-a,logs-b` | OpenSearch's multi-index syntax | + +An explicit format, such as `${level:csv}`, always takes precedence. + +### Variables in Query DSL + +In a Query DSL body, a variable's value is formatted and then escaped as JSON string content, so a value containing `"` or `\`, or the quotes a format such as `lucene` adds, stays inside the string it is placed in and can't change the structure of the request. Put a variable inside quotes, for example `{"term": {"traceId": "$traceId"}}`. To insert a multi-value variable as a JSON array, use the `json` format, which is not escaped: `{"terms": {"level": ${level:json}}}`. ### Field mapping -OpenSearch does not enforce a schema, and field names vary by exporter. Each row of the PPL response is mapped to a log entry by looking up a timestamp column and a message column, trying these names in order: +OpenSearch does not enforce a schema, and field names vary by exporter. Each PPL or SQL result row, and each Lucene or Query DSL hit's `_source` (flattened to dot-separated keys), is mapped to a log entry by looking up a timestamp column and a message column, trying these names in order (object columns in PPL and SQL results, such as `service`, are flattened to `service.name`-style labels too): - **timestamp**: `@timestamp`, `timestamp`, `time` - **message**: `message`, `log`, `body` -Set `timestampField` or `messageField` on the query to prefer a different name; the defaults above are still used as fallbacks. Every remaining column of the response becomes a label on the log entry. If no message column matches, the whole row is serialized to JSON as the log line so no data is lost. +Set `timestampField` or `messageField` on the query to prefer a different name; the defaults above are still used as fallbacks. Every remaining field becomes a label on the log entry. If no message column matches, the whole row is serialized to JSON as the log line so no data is lost. !!! note - The `timestampField` override also determines the column used by the automatic time filter, which defaults to `@timestamp` alone. If your index stores its timestamp as `time` or `timestamp`, set `timestampField` explicitly, otherwise the injected `where` clause references a column that does not exist and OpenSearch rejects the query. + The `timestampField` override also determines the field used by the automatic time filter and, for Lucene and Query DSL, the sort, which default to `@timestamp` alone. If your index stores its timestamp as `time` or `timestamp`, set `timestampField` explicitly, otherwise the time filter references a field that does not exist and the query fails or returns no logs. diff --git a/docs/opensearch/go-sdk/log-query.md b/docs/opensearch/go-sdk/log-query.md index 2bbdecd2c..737d15d3d 100644 --- a/docs/opensearch/go-sdk/log-query.md +++ b/docs/opensearch/go-sdk/log-query.md @@ -9,7 +9,7 @@ var options []log.Option log.OpenSearchLogQuery(`source=otel-logs-* | where severityText='ERROR'`, options...) ``` -Need to provide the PPL expression and a list of options. The expression cannot be empty. +Need to provide the query expression and a list of options. The expression is PPL unless [QueryLanguage](#querylanguage) says otherwise. It cannot be empty, except for a Lucene query, where a blank expression matches every document. ## Default options @@ -25,7 +25,7 @@ import "github.com/perses/plugins/opensearch/sdk/go/query/log" log.Query(`source=otel-logs-* | where severityText='ERROR' | head 20`) ``` -Define the PPL query expression for log data. +Define the query expression for log data, in the language set by [QueryLanguage](#querylanguage). #### Datasource @@ -45,7 +45,27 @@ import "github.com/perses/plugins/opensearch/sdk/go/query/log" log.Index("otel-logs-*") ``` -Define the index or index pattern to read from. It is prepended to the query as a `source=` clause, and ignored when the query already starts with `source=`. +Define the index or index pattern to read from. Required for the `lucene` and `dsl` languages, where it is the index in `POST /{index}/_search`. For PPL it is prepended to the query as a `source=` clause, and ignored when the query already starts with `source=`. Ignored for SQL, which reads its index from the `FROM` clause. + +#### QueryLanguage + +```golang +import "github.com/perses/plugins/opensearch/sdk/go/query/log" + +log.QueryLanguage("lucene") +``` + +Select how the query expression is interpreted: `ppl`, `sql`, `lucene` or `dsl`. When not set, the query is PPL and `queryLanguage` is omitted from the spec. + +#### Limit + +```golang +import "github.com/perses/plugins/opensearch/sdk/go/query/log" + +log.Limit(200) +``` + +Cap the number of log documents fetched by a `lucene` or `dsl` query (the `_search` request's `size`). Must be between 1 and `log.MaxLimit` (10,000); defaults to 500 when not set. A `size` in a `dsl` body is capped at this value. Ignored for PPL and SQL. #### TimestampField @@ -75,7 +95,7 @@ import "github.com/perses/plugins/opensearch/sdk/go/query/log" log.DisableTimeFilter(true) ``` -Stop the panel time range from being injected as a `where` clause on the timestamp field, so the query manages its own time bounds. +Stop the panel time range from being added as a filter on the timestamp field, so the query manages its own time bounds. ## Example diff --git a/docs/opensearch/model.md b/docs/opensearch/model.md index bdcdb2552..a1199529a 100644 --- a/docs/opensearch/model.md +++ b/docs/opensearch/model.md @@ -25,7 +25,7 @@ spec: See [common plugin definitions](https://perses.dev/perses/docs/plugins/common/#http-proxy-specification). -The plugin sends every query to `POST /_plugins/_ppl`, so that single endpoint is all you need to allow. +Allow the endpoints for the query languages you use: `POST /_plugins/_ppl` for PPL, `POST /_plugins/_sql` for SQL, and `POST /{index}/_search` (pattern `/.*/_search`) for Lucene and Query DSL. ### Example @@ -63,6 +63,10 @@ spec: allowedEndpoints: - endpointPattern: "/_plugins/_ppl" method: "POST" + - endpointPattern: "/_plugins/_sql" + method: "POST" + - endpointPattern: "/.*/_search" + method: "POST" secret: "opensearch_secret_config" ``` @@ -73,18 +77,29 @@ Perses supports log queries for OpenSearch: `OpenSearchLogQuery`. ```yaml kind: "OpenSearchLogQuery" spec: - # `query` is the PPL expression for log data. It cannot be empty. + # `query` is the log query, written in `queryLanguage`. + # It cannot be empty, except for `lucene`, where a blank query matches every document. + # For `dsl` it is a JSON `_search` request body. query: + # `queryLanguage` is one of `ppl`, `sql`, `lucene` or `dsl`. Defaults to `ppl` when omitted. + queryLanguage: # Optional + # `datasource` is a datasource selector. If not provided, the default OpenSearchDatasource is used. # See the documentation about the datasources to understand how it is selected. datasource: # Optional # `index` is the index or index pattern to read from, e.g. `logs-*`. - # It is prepended to the query as a `source=` clause, and ignored when the - # query already starts with `source=`. + # Required for `lucene` and `dsl`, where it is the index in `POST /{index}/_search`. + # For `ppl` it is prepended to the query as a `source=` clause, and ignored when + # the query already starts with `source=`. Ignored for `sql`, which uses its FROM clause. index: # Optional + # `limit` is the maximum number of log documents fetched by a `lucene` or `dsl` query. + # It must be between 1 and 10000 and defaults to 500. A `size` in a `dsl` body is capped at `limit`. + # Ignored for `ppl` and `sql`. + limit: # Optional + # `timestampField` overrides which column carries the log timestamp. # It is also the column used by the automatic time filter. # Defaults to `@timestamp`, then `timestamp`, then `time`. @@ -94,8 +109,8 @@ spec: # Defaults to `message`, then `log`, then `body`. messageField: # Optional - # When `disableTimeFilter` is true, the panel time range is NOT injected as a - # `where` clause on the timestamp field. + # When `disableTimeFilter` is true, the panel time range is NOT added as a filter on + # the timestamp field. disableTimeFilter: # Optional ``` @@ -132,6 +147,33 @@ spec: disableTimeFilter: true ``` +A Lucene query, fetching at most 200 documents: + +```yaml +kind: "LogQuery" +spec: + plugin: + kind: "OpenSearchLogQuery" + spec: + queryLanguage: "lucene" + index: "otel-logs-*" + query: "severityText:ERROR AND service.name:payment-svc" + limit: 200 +``` + +A Query DSL query: + +```yaml +kind: "LogQuery" +spec: + plugin: + kind: "OpenSearchLogQuery" + spec: + queryLanguage: "dsl" + index: "otel-logs-*" + query: '{"query": {"term": {"traceId": "$traceId"}}}' +``` + ## Shared definitions ### OpenSearch Datasource selector diff --git a/opensearch/README.md b/opensearch/README.md index a1c10be30..ebe7db606 100644 --- a/opensearch/README.md +++ b/opensearch/README.md @@ -1,7 +1,20 @@ # Plugin Module: opensearch -Datasource plugin for [OpenSearch](https://opensearch.org/) in Perses. Supports log queries using -[PPL (Piped Processing Language)](https://opensearch.org/docs/latest/search-plugins/sql/ppl/index/). +Datasource plugin for [OpenSearch](https://opensearch.org/) in Perses. Log queries can +be written in [PPL](https://docs.opensearch.org/latest/sql-and-ppl/ppl/index/), +[SQL](https://docs.opensearch.org/latest/sql-and-ppl/sql/index/), +[Lucene / query string](https://docs.opensearch.org/latest/query-dsl/full-text/query-string/), +or raw [Query DSL](https://docs.opensearch.org/latest/query-dsl/), selected per query. +PPL and SQL need the SQL/PPL plugin enabled on the cluster; Lucene and Query DSL use +the core `_search` API. + +Set `queryLanguage` on the query to `ppl` (the default), `sql`, `lucene` or `dsl`. +Lucene and Query DSL queries also need an `index` and accept an optional `limit` +(500 by default). The dashboard time range is added to every query automatically +unless `disableTimeFilter` is set. If you use the Perses proxy, allow the endpoint for +each language: `/_plugins/_ppl`, `/_plugins/_sql` and `/.*/_search`. See the +[plugin documentation](../docs/opensearch/README.md) and the +[data model](../docs/opensearch/model.md) for details. This plugin is intended to display logs alongside the traces surfaced by the Tempo plugin, so a user can pivot from a trace to the related logs in OpenSearch. diff --git a/opensearch/docs/architecture.md b/opensearch/docs/architecture.md new file mode 100644 index 000000000..01f592696 --- /dev/null +++ b/opensearch/docs/architecture.md @@ -0,0 +1,357 @@ +# How the OpenSearch plugin works + +This document explains the internals of the OpenSearch plugin: what it is, how it +plugs into Perses on one side and OpenSearch on the other, and what happens — +step by step — when a log panel asks for data. + +If the [user guide](./using-opensearch-in-perses.md) is "how do I use it", this is +"how does it actually work". + +## The one-sentence model + +> **A Perses plugin is a set of TypeScript contracts on the Perses side and one of +> three HTTP calls on the OpenSearch side. This plugin's whole job is the +> translation in between: turn a query — written in PPL, SQL, Lucene, or Query +> DSL — typed into a panel into log lines fetched from a cluster, and turn +> whichever of the cluster's two answer shapes that language returns back into +> something Perses can render.** + +Keep that sentence in mind — every part below is a detail of it. + +## What the plugin actually is + +The package `@perses-dev/opensearch-plugin` publishes **one plugin module** named +`OpenSearch`. That module registers **two plugins** that Perses loads and uses +independently: + +| Plugin | Kind | Responsibility | +| --- | --- | --- | +| `OpenSearchDatasource` | `Datasource` | Knows *how to reach* a cluster. Produces a client. | +| `OpenSearchLogQuery` | `LogQuery` | Knows *how to ask* for logs with PPL, SQL, Lucene, or Query DSL and normalize the answer. | + +They are declared in the `perses` block of `package.json`, and everything else in +the repo exists to support these two. + +## The big picture + +Here is the whole system in one view. The browser runs the Perses UI; the plugin +code runs *inside* that browser; and it reaches OpenSearch either directly or +through the Perses backend proxy. + +```mermaid +flowchart LR + subgraph Browser["🌐 Browser (Perses UI)"] + panel["Log panel"] + subgraph plugin["OpenSearch plugin"] + query["OpenSearchLogQuery
(builds & normalizes)"] + client["OpenSearchClient
(HTTP)"] + end + end + + subgraph Backend["Perses backend"] + proxy["HTTP proxy"] + end + + os["🔎 OpenSearch
_plugins/_ppl, _plugins/_sql,
or {index}/_search"] + + panel -->|"getLogData(spec, context)"| query + query -->|"query + language"| client + client -->|"proxy mode"| proxy --> os + client -.->|"direct mode"| os + os -->|"datarows or hits"| client + client --> query + query -->|"LogData (entries)"| panel +``` + +The request goes **down** the left edge (panel → query → client → OpenSearch) and +the data comes **back up** (OpenSearch → client → query → panel). The two arrows +into OpenSearch are the two connection modes — more on that below. + +## How it plugs into Perses + +Perses does not know OpenSearch exists. It only knows two interfaces from +`@perses-dev/plugin-system`. The plugin implements them, and Perses calls the +implementations at the right moments. + +```mermaid +flowchart TB + subgraph perses["Perses expects"] + dsp["DatasourcePlugin<Spec, Client>
• createClient(spec, options)
• OptionsEditorComponent
• createInitialOptions()"] + lqp["LogQueryPlugin<Spec>
• getLogData(spec, context, signal)
• OptionsEditorComponent
• dependsOn(spec)
• createInitialOptions()"] + end + + subgraph impl["This plugin provides"] + ds["OpenSearchDatasource
datasources/opensearch-datasource/"] + lq["OpenSearchLogQuery
queries/opensearch-log-query/"] + end + + dsp -.implemented by.-> ds + lqp -.implemented by.-> lq + ds -->|"createClient returns"| clientObj["OpenSearchClient
{ options, ppl(), sql(), search() }"] + lq -->|"getLogData returns"| logdata["LogData
{ entries, totalCount }"] +``` + +The two return types are the contract in practice: + +- **`OpenSearchDatasource.createClient`** returns an `OpenSearchClient` — an object + with three methods, `ppl()`, `sql()`, and `search()`, one per OpenSearch + transport. The query plugin calls whichever one matches the query's resolved + language; that is the only thing it ever needs from the datasource. +- **`OpenSearchLogQuery.getLogData`** returns `LogData` — the normalized + `{ entries, totalCount }` shape from `@perses-dev/spec` that Perses log panels + know how to render. + +The same query spec is also mirrored as a **CUE schema** (for server-side +validation) and a **Go SDK struct** (for defining dashboards as code), so the UI, +the backend, and dashboards-as-code all agree on the shape. + +## How it reaches OpenSearch + +The plugin talks to one of **three endpoints**, chosen by the query's language, +and gets back one of **two response shapes**: + +| Language(s) | Endpoint | Response shape | +| --- | --- | --- | +| PPL | `POST /_plugins/_ppl` | `{ schema, datarows }` — a table | +| SQL | `POST /_plugins/_sql` | `{ schema, datarows }` — a table | +| Lucene, Query DSL | `POST /{index}/_search` | `{ hits: { hits: […] } }` — documents | + +For example, a PPL request/response pair: + +``` +Request: POST /_plugins/_ppl { "query": "source=logs-* | head 20" } +Response: { "schema": [ { "name", "type" }, … ], + "datarows": [ [ v, v, v ], … ] } +``` + +PPL and SQL share the tabular `datarows` shape because both go through the +SQL/PPL plugin; Lucene and Query DSL bypass it and hit the core `_search` API +directly, coming back as hits instead of rows. That split is why there are two +converters rather than one (see below). + +`createClient` decides *where* those POSTs go, based on the datasource spec — +the query language only picks *which* endpoint, not the connection mode: + +```mermaid +flowchart TB + spec{"datasource spec"} + spec -->|"has directUrl"| direct["Direct mode
Browser → OpenSearch
needs CORS on the cluster"] + spec -->|"has proxy"| proxied["Proxy mode (recommended)
Browser → Perses → OpenSearch
no CORS, credentials stay server-side"] + direct --> endpoint["client.ppl() / .sql() / .search()
→ POST /_plugins/_ppl, /_plugins/_sql,
or /{index}/_search"] + proxied --> endpoint +``` + +Either way the query plugin calls whichever client method matches the resolved +language — only the resolved base URL differs, not the transport logic. If a +request fails, the client reads the error body and throws an +`OpenSearchApiError`, which pulls the actionable text out of OpenSearch's +`reason` / `details` fields (or, for `_search` failures, `root_cause[0].reason`) +so you get a diagnosable message instead of a bare status code. + +## What happens when a panel loads — the full lifecycle + +This ties both ends together. When a log panel needs data, Perses calls +`getLogData(spec, context, abortSignal)`. The shape of the call is the same for +every language — resolve variables, get a client, build a request, call +OpenSearch, convert the response — only the build/convert/client step in the +middle changes. Log queries receive only user-defined variables, so `getLogData` +adds the builtin `__from` / `__to` (the time range in epoch milliseconds) before +resolving them. Follow the default (PPL) path end to end: + +```mermaid +sequenceDiagram + participant P as Perses runtime + participant Q as OpenSearchLogQuery
(get-opensearch-log-data) + participant C as OpenSearchClient + participant OS as OpenSearch + + P->>Q: getLogData(spec, context, signal) + Note over Q: context = timeRange,
variableState, datasourceStore + + alt query is blank + Q-->>P: empty LogData (no request) + else has a query + Q->>Q: replaceVariables() — resolve $vars and
$__from / $__to in query and index + Q->>P: datasourceStore.getDatasourceClient() + P-->>Q: OpenSearchClient + Q->>Q: buildBoundedPPL() — inject source=
and the panel time window + Q->>C: client.ppl({ query }) + C->>OS: POST /_plugins/_ppl + OS-->>C: { schema, datarows } + C-->>Q: OpenSearchDatarowsResponse + Q->>Q: convertDatarowsToLogs() — rows → LogEntry[] + Q-->>P: LogData + metadata.executedQueryString + end +``` + +Two of those steps do the real translating — a **build** step on the way out and +a **convert** step on the way back; here, `buildBoundedPPL` and +`convertDatarowsToLogs`. The other three languages take the same shape but swap +in their own builder and, for Lucene/Query DSL, a different converter. They are +worth understanding on their own. + +## The outbound transform: `build/` + +Every query language has its own builder in `build/` that turns the query spec +into whatever request shape its endpoint expects: + +- **`ppl.ts` → `buildBoundedPPL`** — splices a `source=` clause and a + time-bounded `where` clause into the PPL string. +- **`sql.ts` → `buildSqlRequest`** — adds a time-bounded condition to the outer + query's `WHERE` clause (or inserts a `WHERE` before `GROUP BY` / `HAVING` / + `ORDER BY` / `LIMIT`). It does not use the `_sql` API's `filter` field, which + forces OpenSearch onto its legacy SQL engine. +- **`lucene.ts` → `buildLuceneSearchBody`** and **`dsl.ts` → `buildDslSearchBody`** + — wrap the query string (Lucene) or parsed JSON body (Query DSL) in a + `_search` request, ANDing in a `bool.filter` time-range clause. A Query DSL + `query` is nested under `bool.must` so it keeps its scoring. +- **`search.ts`** — shared helpers used by the two `_search` builders above: + `buildTimeRangeFilter`, `buildDefaultSort`, and `resolveSize` (the `limit` → + `size` default, 500). + +`buildBoundedPPL` is the most involved of the five, because the time bound has +to be spliced into a query string rather than added as a separate request +field. It is worth walking through in detail. + +You almost never want *all* logs — you want the ones in the dashboard's time +range. Rather than make you write that clause every time, the plugin rewrites your +query before sending it. Two things get injected: + +1. **A `source=` clause** from the *Index pattern* field — but only if you did not + already write one yourself. +2. **A time bound** — a `where` clause on the timestamp field, bounded to the + panel's current range, inserted **right after the source clause** so it runs + before any pipe (like `stats`) that would drop the timestamp column. + +```mermaid +flowchart TB + start["Your query + index + time range"] + q1{"Query already
starts with source= ?"} + q1 -->|no| addsrc["Prepend source=<index>"] + q1 -->|yes| skipsrc["Leave source as-is"] + addsrc --> q2 + skipsrc --> q2 + q2{"disableTimeFilter ?"} + q2 -->|yes| passthrough["Send query unchanged"] + q2 -->|no| bound["Insert time-range where clause
after the source clause"] + bound --> send["Send to OpenSearch"] + passthrough --> send + start --> q1 +``` + +So a query you write as: + +```ppl +source=otel-logs-demo | where severityText='ERROR' +``` + +is sent (for a "last 60 minutes" range) as: + +```ppl +source=otel-logs-demo + | where `@timestamp` >= '2026-07-14T03:53:33.000Z' and `@timestamp` <= '2026-07-14T04:53:33.000Z' + | where severityText='ERROR' +``` + +Because the index and timestamp field are user-editable, they are treated as +untrusted before being spliced into the string: `escapeIdentifier` doubles any +backtick in the timestamp field, `sanitizeIndex` strips characters that could open +a new PPL stage (`` ` ``, `|`, quotes, whitespace) from the index, and +`joinPPLStages` drops empty segments so there is never a dangling `… |`. + +## The inbound transform: `convert/` + +There are two converters, one per response shape: + +- **`datarows-to-logs.ts` → `convertDatarowsToLogs`** — for PPL and SQL's + tabular `{ schema, datarows }` response. +- **`hits-to-logs.ts` → `convertHitsToLogs`** — for Lucene and Query DSL's + `_search` `{ hits }` response. It flattens each hit's `_source` document to + dot-separated label keys (`flattenSource`), since a `LogEntry`'s `labels` is a + flat map and `_source` is not. + +Both share `parse-timestamp.ts`'s `parseTimestamp` to normalize whatever +timestamp value they pick to seconds. + +OpenSearch answers with columns and positional rows (or, for `_search`, JSON +documents). Perses renders `LogEntry` objects, each with a `timestamp`, a `line`, +and a bag of `labels`. `convertDatarowsToLogs` bridges the tabular shape by +**picking** one column as the timestamp and one as the message, then sweeping +every other column into labels. + +```mermaid +flowchart LR + subgraph resp["OpenSearch response"] + cols["schema: @timestamp, body,
service.name, severityText, traceId
datarows: [ … ]"] + end + subgraph entry["LogEntry (per row)"] + ts["timestamp ← @timestamp
(parsed to seconds)"] + line["line ← body"] + labels["labels ← every other column
{ service.name, severityText, traceId }"] + end + cols --> ts + cols --> line + cols --> labels +``` + +Which column is which is chosen by name, with fallbacks. Your query-level +`timestampField` / `messageField` overrides are tried first, then these defaults: + +- **timestamp:** `@timestamp` → `timestamp` → `time` +- **message:** `message` → `log` → `body` + +If no message column matches, the whole row is JSON-stringified into `line` so +nothing is lost. Timestamps are normalized to seconds by detecting the unit from +the value's magnitude (seconds / milliseconds / microseconds / nanoseconds), and +OpenSearch's zoneless `"2020-08-26 13:49:00"` datetimes are forced to explicit UTC +so the browser's local offset does not shift them. + +## Where each concern lives + +```mermaid +flowchart TB + pkg["package.json (perses block)
getPluginModule.ts
— declares the module"] + subgraph src["src/"] + dsdir["datasources/opensearch-datasource/
OpenSearchDatasource.tsx (+ editor)"] + model["model/opensearch-client.ts
— ppl(), sql(), search(),
buildUrl(), OpenSearchApiError"] + subgraph qdir["queries/opensearch-log-query/"] + qroot["OpenSearchLogQuery.tsx (+ editor)
get-opensearch-log-data.ts"] + builddir["build/
ppl.ts, sql.ts, search.ts,
lucene.ts, dsl.ts"] + convertdir["convert/
datarows-to-logs.ts, hits-to-logs.ts,
parse-timestamp.ts"] + qroot --> builddir + qroot --> convertdir + end + end + subgraph schemas["schemas/ (CUE)"] + dscue["datasources/opensearch.cue"] + qcue["queries/opensearch-log-query/query.cue"] + end + subgraph sdk["sdk/go/ (dashboards-as-code)"] + gods["datasource/"] + golog["query/log/log.go"] + end + + pkg --> dsdir + pkg --> qdir + dsdir --> model + qdir --> model + dsdir -. validated by .-> dscue + qdir -. validated by .-> qcue + qdir -. mirrored by .-> golog + dsdir -. mirrored by .-> gods +``` + +## Recap + +- The plugin is **one module, two plugins**: a datasource (connection) and a log + query (PPL, SQL, Lucene, or Query DSL + normalization). +- Perses talks to it through **two interfaces**; OpenSearch talks to it through + **three endpoints** (`POST /_plugins/_ppl`, `POST /_plugins/_sql`, and + `POST /{index}/_search`), which return one of **two response shapes**. +- At runtime, `getLogData` **resolves variables → gets a client → builds a + language-specific request → calls the matching OpenSearch endpoint → converts + the response to log entries**. +- The real logic lives in two directories: `build/` (five builders — one per + language, plus `search.ts`'s shared `_search` helpers) and `convert/` (two + converters — `convertDatarowsToLogs` for PPL/SQL, `convertHitsToLogs` for + Lucene/Query DSL). diff --git a/opensearch/docs/using-opensearch-in-perses.md b/opensearch/docs/using-opensearch-in-perses.md new file mode 100644 index 000000000..14444238a --- /dev/null +++ b/opensearch/docs/using-opensearch-in-perses.md @@ -0,0 +1,383 @@ +# Get insights from your OpenSearch data in Perses + +[Perses](https://perses.dev/) is an open, GitOps-friendly dashboarding and +observability tool. Historically it has been strongest for metrics and traces — +but logs are usually where an investigation actually ends. The **OpenSearch +datasource plugin** now **ships bundled with Perses**, so out of the box you can +query your OpenSearch logs directly from a dashboard using +[PPL (Piped Processing Language)](https://docs.opensearch.org/latest/sql-and-ppl/ppl/index/) +and put them next to the metrics and traces you already have. + +This guide walks you through it end to end: stand up OpenSearch, run Perses (the +plugin is already built in — nothing to install), wire up a datasource, and run +your first PPL queries. At the end there's a bonus section on correlating a trace +with its logs. + +Every command and every query output below was run against OpenSearch 2.19 and a +current Perses image before it was written down. + +## What you'll build + +A Perses datasource pointing at OpenSearch, and log panels driven by PPL — for +example, "show me the ERROR logs from `payment-svc` in the current time range," +or "show me every log line that belongs to this one trace." + +## Prerequisites + +To *use* the plugin you only need: + +- A **current Perses release** — the OpenSearch plugin ships bundled, so there's + nothing to install (you'll confirm this in [Step 2](#step-2--run-perses)). +- An **OpenSearch** cluster. PPL and SQL queries also need the SQL/PPL plugin — + bundled by default. Lucene and Query DSL use the core `_search` API. + +**If you already have Perses and OpenSearch running,** jump straight to +[Step 3](#step-3--add-the-opensearch-datasource) and add the datasource — you can +ignore everything before it. + +The rest of the tutorial stands up OpenSearch and Perses locally, which also needs: + +- **Docker** (with Compose). + +> **Running an older Perses that predates the plugin?** See the note at the end of +> [Step 2](#step-2--run-perses) for loading it as a released archive — though the +> simplest path is to upgrade Perses. + +## Step 1 — Get OpenSearch running (with some data) + +The plugin ships a ready-to-use Compose file at +[`docs/examples/docker-compose.yml`](./examples/docker-compose.yml). It starts a +single-node OpenSearch (security disabled, CORS enabled for local use) plus +OpenSearch Dashboards: + +```bash +cd opensearch/docs/examples +docker compose up -d opensearch +``` + +Wait for the cluster to go green: + +```bash +curl -s http://localhost:9200/_cluster/health | grep -o '"status":"[a-z]*"' +# "status":"green" +``` + +> **Already have OpenSearch?** Point the datasource at your own cluster in Step 3 +> and skip the seeding below. PPL and SQL queries need the **SQL/PPL plugin** +> enabled, which ships with OpenSearch by default; Lucene and Query DSL don't. + +For the tutorial we'll seed a handful of OTel-style log documents so there's +something to query. The docs carry a `@timestamp`, a `body` (the log message), a +`traceId`, and a `service.name`: + +```bash +curl -s -X PUT http://localhost:9200/otel-logs-demo -H 'Content-Type: application/json' -d '{ + "mappings": { "properties": { + "@timestamp": { "type": "date" }, + "body": { "type": "text" }, + "traceId": { "type": "keyword" }, + "severityText": { "type": "keyword" }, + "service": { "properties": { "name": { "type": "keyword" } } } + }} +}' + +# a couple of documents sharing one traceId, plus some standalone lines +curl -s -X POST http://localhost:9200/_bulk -H 'Content-Type: application/x-ndjson' --data-binary ' +{"index":{"_index":"otel-logs-demo"}} +{"@timestamp":"'"$(date -u -d '3 minutes ago' +%FT%T.000Z)"'","severityText":"INFO","traceId":"4bf92f3577b34da6a3ce929d0e0e4736","service":{"name":"checkout-api"},"body":"GET /cart/items 200 in 34ms"} +{"index":{"_index":"otel-logs-demo"}} +{"@timestamp":"'"$(date -u -d '2 minutes ago' +%FT%T.000Z)"'","severityText":"ERROR","traceId":"4bf92f3577b34da6a3ce929d0e0e4736","service":{"name":"payment-svc"},"body":"charge failed: gateway timeout after 3 retries"} +{"index":{"_index":"otel-logs-demo"}} +{"@timestamp":"'"$(date -u -d '1 minutes ago' +%FT%T.000Z)"'","severityText":"WARN","service":{"name":"search-svc"},"body":"cache miss for key products:featured"} +' +curl -s -X POST http://localhost:9200/otel-logs-demo/_refresh +``` + +## Step 2 — Run Perses + +Current Perses images ship the OpenSearch plugin, so there's nothing to build, +download, or configure — just start Perses: + +```yaml +# perses-compose.yml +services: + perses: + image: persesdev/perses:latest + ports: + - "8080:8080" +``` + +```bash +docker compose -f perses-compose.yml up -d +``` + +Confirm the plugin is there. It registers both a datasource and a query plugin: + +```bash +curl -s http://localhost:8080/api/v1/plugins | \ + python3 -c "import sys,json; [print(p['kind'],'->',p['spec']['name']) for m in json.load(sys.stdin) if m['metadata']['name']=='OpenSearch' for p in m['spec']['plugins']]" +# Datasource -> OpenSearchDatasource +# LogQuery -> OpenSearchLogQuery +``` + +That's the whole "installation" — the plugin is part of the Perses distribution. + +> **Older Perses without the plugin bundled?** Add it from a released +> `OpenSearch-.tar.gz` ([perses/plugins releases](https://github.com/perses/plugins/releases)) +> via Perses' plugin archive path (`PERSES_PLUGIN_ARCHIVE_PATHS`). Note that setting +> is a **list** — include the image's built-in archive path as well, or you'll drop +> the other bundled plugins. Simpler, though, is to upgrade Perses. If you're +> *developing* the plugin from source, build it with `percli plugin build` and use +> the live dev loop (`percli plugin start`); see the repository +> [README](../../README.md#development). + +## Step 3 — Add the OpenSearch datasource + +A datasource tells Perses how to reach your cluster. The recommended setup routes +requests through the **Perses backend proxy** — the browser talks to Perses, +Perses talks to OpenSearch — so you don't need CORS on the cluster and any +credentials stay server-side. + +You can create it in the UI (**Datasources → Add**, pick *OpenSearch Datasource*), +or provision it as YAML. As a `GlobalDatasource`: + +```yaml +kind: GlobalDatasource +metadata: + name: opensearch-demo +spec: + default: true + plugin: + kind: OpenSearchDatasource + spec: + proxy: + kind: HTTPProxy + spec: + url: http://opensearch:9200 # your cluster URL + allowedEndpoints: + - endpointPattern: /_plugins/_ppl + method: POST + - endpointPattern: /_plugins/_sql + method: POST + - endpointPattern: /.*/_search + method: POST +``` + +> **Upgrading an existing datasource?** Datasources created before SQL/Lucene/Query +> DSL support was added allow only `/_plugins/_ppl`, so SQL and `_search` requests +> will be rejected by the Perses proxy with a `403` until you add +> `/_plugins/_sql` and `/.*/_search` to `allowedEndpoints`. + +## Choosing a query language + +Each log query picks its own language with the **Query language** dropdown: + +| Language | Endpoint | Index field | Notes | +| --- | --- | --- | --- | +| PPL (default) | `POST /_plugins/_ppl` | Optional — ignored if your query starts with `source=` | Requires the SQL/PPL plugin on the cluster | +| SQL | `POST /_plugins/_sql` | Ignored — the index is in your `FROM` clause | Requires the SQL/PPL plugin on the cluster | +| Lucene | `POST /{index}/_search` | **Required** | The query string syntax used in Dashboards Discover | +| Query DSL | `POST /{index}/_search` | **Required** | Paste a `_search` request body straight from Dev Tools | + +The dashboard time range is applied automatically in every language — as a `where` +clause for PPL, as a condition added to the outer `WHERE` for SQL, and as a `bool.filter` range +clause for Lucene and Query DSL (a Query DSL `query` goes under `bool.must`, so scoring +still works). Untick **Use dashboard time range** (under **Options**) to manage +the bounds yourself. + +**Ordering differs by language.** Lucene and Query DSL sort newest-first — the +plugin adds an explicit `sort` on the timestamp field. PPL and SQL return results +in cluster order unless you sort in the query yourself (`sort - @timestamp` for +PPL, `ORDER BY` for SQL). Switching a panel's query language can therefore reorder +its rows even though nothing else about the query changed. + +**SQL time filtering rewrites the `WHERE` clause.** The plugin ANDs a condition on +the timestamp field into your outer `WHERE`, or inserts one before `GROUP BY`, +`HAVING`, `ORDER BY` or `LIMIT`. It only does this for a single `SELECT … FROM` +query: a statement with `UNION` or `JOIN`, or one that isn't a `SELECT`, fails with +an error asking you to untick **Use dashboard time range** and filter on the +timestamp yourself. `$__from` and `$__to` hold the panel time range in epoch +milliseconds for that, in every language: + +```sql +SELECT * FROM `logs-*` +WHERE `@timestamp` >= FROM_UNIXTIME($__from / 1000.0) + AND `@timestamp` <= FROM_UNIXTIME($__to / 1000.0) +``` + +> Verified against OpenSearch 2.19.6 (`distribution: opensearch`) using a +> single-node example cluster, including indices with microsecond `date_nanos` +> timestamps. + +For Lucene and Query DSL the **Limit** field sets the `_search` `size`. If you leave it +empty the plugin sends 500, because OpenSearch's own default is only 10 documents — +which would silently truncate a log panel. For PPL and SQL, cap results yourself with +`head` or `LIMIT`. + +**An empty Lucene query still runs.** Every other language short-circuits a blank +query without contacting OpenSearch, but a blank Lucene query legitimately means +"everything in the time range" — so as soon as an index is set, it fetches up to +the Limit. + +Query DSL input is treated as a complete `_search` request body. Your `size` and +`sort` win over the defaults; your `query`, if present, is ANDed with the time range. + +To provision it at startup, drop the file in a folder and add +`PERSES_PROVISIONING_FOLDERS_0=/etc/perses/provisioning` (plus the matching +volume) to the Perses service. + +You can sanity-check the datasource straight through the Perses proxy — this is +the exact path the plugin uses at runtime: + +```bash +curl -s -X POST \ + http://localhost:8080/proxy/globaldatasources/opensearch-demo/_plugins/_ppl \ + -H 'Content-Type: application/json' \ + -d '{"query":"source=otel-logs-demo | fields severityText, `service.name`, body | head 1"}' +``` + +A `200` with a PPL result body means the whole path — browser → Perses proxy → +OpenSearch — is working: + +```json +{ + "schema": [ + { "name": "severityText", "type": "string" }, + { "name": "service.name", "type": "string" }, + { "name": "body", "type": "string" } + ], + "datarows": [ + [ "INFO", "checkout-api", "GET /cart/items 200 in 34ms" ] + ], + "total": 1, + "size": 1 +} +``` + +> **Prefer a direct connection?** Instead of `proxy`, set +> `directUrl: http://localhost:9200`. The browser then calls OpenSearch directly, +> which is why the example Compose file enables CORS. The proxy setup is the +> better default for anything beyond local tinkering. + +## Step 4 — Query your logs with PPL + +This is the core of getting insight out of OpenSearch. Add a panel to a dashboard, +choose the **OpenSearch Log Query** type, and you get an editor with these fields: + +| Field | What it does | +| --- | --- | +| **Query language** | Selects PPL, SQL, Lucene, or Query DSL. It's the first control in the editor, and it changes how the fields below are used and what the query field expects. | +| **Datasource** | Which OpenSearch datasource to query. | +| **Index pattern** | e.g. `logs-*`. For PPL, prepended as `source=` — *ignored if your PPL already starts with `source=`.* For SQL it's ignored (the index comes from your `FROM` clause). Required for Lucene and Query DSL, which send it as `POST /{index}/_search`. | +| **Query** | The query itself, in whichever language you picked above — the field's label changes to match (`PPL Query`, `SQL Query`, `Lucene Query`, or `Query DSL Query`). Each language keeps its own query while you edit, so switching language and back restores what you typed; a language you haven't used yet starts empty with its own placeholder and examples. | +| **Options** → **Timestamp field** *(optional)* | Overrides the default timestamp field. | +| **Options** → **Message field** *(optional)* | Overrides which field becomes the log line. | +| **Options** → **Use dashboard time range** | On by default. Untick it to turn off the automatic time bound (see below). | +| **Options** → **Limit** *(optional — Lucene and Query DSL only)* | Caps the number of documents fetched (the `_search` request's `size`). Leave it empty and the plugin sends 500; the maximum is 10,000, which is all `_search` accepts by default. For PPL and SQL, cap results yourself with `head` or `LIMIT` instead. | + +**Options** is collapsed until you open it, and starts open when the query already +changes one of its settings. + +A first query — the most recent lines, newest first: + +```ppl +source=otel-logs-demo | fields `@timestamp`, severityText, `service.name`, body | sort - `@timestamp` | head 5 +``` + +which returns rows like: + +``` +2026-07-14 04:51:11 WARN payment-svc retry budget low for downstream billing +2026-07-14 04:47:11 INFO search-svc query executed hits=44 took=9ms +2026-07-14 04:41:11 INFO checkout-api GET /cart/items 200 in 29ms +2026-07-14 04:35:11 ERROR search-svc OpenSearch shard unavailable, retrying +2026-07-14 04:28:11 WARN payment-svc connection pool at 85% capacity +``` + +PPL reads left to right: a `source=` selects the index, and each `|` pipes rows +into the next command — `where` to filter, `fields` to project, `stats` to +aggregate, `sort`/`head` to order and limit. A few useful shapes: + +```ppl +source=logs-* | where level="error" | head 20 +source=logs-* | fields @timestamp, service, message | head 50 +source=logs-* | stats count() by service +``` + +### Automatic time filtering + +You almost never want *all* logs — you want the ones in the dashboard's time +range. The plugin does this for you: it injects a `where` clause on the timestamp +field bounded to the panel's current range, right after the `source=` clause +(so it applies before any pipe that drops the timestamp column, like `stats`). +The query you write: + +```ppl +source=otel-logs-demo | where severityText='ERROR' +``` + +is sent to OpenSearch as (for a "last 60 minutes" range): + +```ppl +source=otel-logs-demo + | where `@timestamp` >= '2026-07-14T03:53:33.000Z' and `@timestamp` <= '2026-07-14T04:53:33.000Z' + | where severityText='ERROR' +``` + +Change the dashboard time picker and the logs follow. If you manage time bounds +yourself, untick **Use dashboard time range** under **Options**. + +### Field mapping for schema-flexible indices + +OpenSearch doesn't enforce a schema, and field names vary by exporter. The plugin +maps each result row to a log entry by looking for a timestamp field and a message +field, trying these names in order: + +- **timestamp:** `@timestamp`, `timestamp`, `time` +- **message:** `message`, `log`, `body` + +If your index uses different names — say the log line lives in `msg` — set the +**Message field** (or **Timestamp field**) override on the query, and the plugin +will prefer it. Every other column in the result comes back as a label on the log +entry, so you keep `service.name`, `severityText`, `traceId`, and the rest for +filtering and display. + +## Bonus — Correlate a trace with its logs + +Because a `traceId` is just another field, you can pivot from a trace straight to +the log lines it produced. Give the dashboard a `traceId` variable and filter on +it: + +```ppl +source=otel-logs-demo | where traceId='$traceId' +``` + +With `$traceId` set to a real trace, that returns the story of a single request: + +``` +2026-07-14 04:11:10 INFO checkout-api GET /cart/items 200 in 34ms +2026-07-14 04:12:10 INFO checkout-api validating cart for user 8841 +2026-07-14 04:13:10 WARN payment-svc payment gateway latency 812ms exceeds soft limit +2026-07-14 04:14:10 ERROR payment-svc charge failed: gateway timeout after 3 retries +``` + +Put a trace panel (Tempo or Jaeger) and an OpenSearch logs panel on the same +dashboard, both reading `$traceId`, and selecting a trace updates the logs to +match. A complete example dashboard lives in +[`docs/examples/trace-to-logs.json`](./examples/trace-to-logs.json); see +[`docs/examples/README.md`](./examples/README.md) for field-name conventions +across exporters and how to swap Tempo for Jaeger. + +## Wrap-up + +You now have OpenSearch logs available inside Perses: a datasource wired through +the Perses proxy, PPL queries that stay bounded to the dashboard's time range, and +a path from a trace to its logs. From here, add the logs to the dashboards where +you already watch metrics and traces — so the next time something breaks, the +evidence is one panel away. + +### Reference + +- Plugin source & schema: [`opensearch/`](../) +- OpenSearch PPL reference: +- Perses documentation: diff --git a/opensearch/schemas/queries/opensearch-log-query/query.cue b/opensearch/schemas/queries/opensearch-log-query/query.cue index 8dcd6005e..714294006 100644 --- a/opensearch/schemas/queries/opensearch-log-query/query.cue +++ b/opensearch/schemas/queries/opensearch-log-query/query.cue @@ -19,11 +19,52 @@ import ( ) kind: "OpenSearchLogQuery" + +// `index` is required only for the languages that put it in the request URL +// (`lucene`, `dsl`, which hit `_search`); it's optional for `ppl`/`sql`, where +// the index lives in the query text itself. `query` may be empty only for +// `lucene`, where a blank query string means "everything in the time range". +// +// A disjunction of two closed structs (one branch requiring `index` for +// lucene/dsl, one leaving it optional for ppl/sql) does NOT work: a missing +// *required* field in CUE produces an *incomplete* value, not a conflicting +// one, so the lucene/dsl branch is never eliminated just because +// `queryLanguage` is absent from the input. That leaves the whole disjunction +// unresolved, and every spec that omits `queryLanguage` (which must be legal — +// see tests/valid/no-language.json) fails as "incomplete value" instead of +// validating against the ppl/sql branch. Do not retry this approach. +// +// The `if` comprehension below works instead, but only because of two things +// together: `queryLanguage` is given a default (`*"ppl" | ...`) so it is +// always concrete, even when the input omits it entirely; and every field +// referenced by the comprehension is declared directly in this one struct +// literal. A CUE comprehension condition can only see fields declared +// lexically in the same struct literal — it cannot see fields that arrive via +// unification (`someBase & {...}`) or embedding a separately-declared struct +// (`{someBase, if ...}`). A prior version of this schema factored the shared +// fields out into a `_base` field and unified/embedded it here; that broke +// the comprehension, because `queryLanguage` (declared in `_base`) wasn't +// visible to the `if` conditions declared alongside the embedding. So all +// fields are inlined directly into this single `spec: close({...})` struct +// literal instead of being factored out — do not re-extract a shared base +// without re-testing the comprehension against it. spec: close({ ds.#selector - query: strings.MinRunes(1) - index?: strings.MinRunes(1) timestampField?: strings.MinRunes(1) messageField?: strings.MinRunes(1) disableTimeFilter?: bool + limit?: int & >0 & <=10000 + queryLanguage: *"ppl" | "sql" | "lucene" | "dsl" + if queryLanguage == "lucene" { + query: string + } + if queryLanguage != "lucene" { + query: strings.MinRunes(1) + } + if queryLanguage == "lucene" || queryLanguage == "dsl" { + index: strings.MinRunes(1) + } + if queryLanguage != "lucene" && queryLanguage != "dsl" { + index?: strings.MinRunes(1) + } }) diff --git a/opensearch/schemas/queries/opensearch-log-query/tests/invalid/dsl-without-index.json b/opensearch/schemas/queries/opensearch-log-query/tests/invalid/dsl-without-index.json new file mode 100644 index 000000000..23e0184dc --- /dev/null +++ b/opensearch/schemas/queries/opensearch-log-query/tests/invalid/dsl-without-index.json @@ -0,0 +1,4 @@ +{ + "kind": "OpenSearchLogQuery", + "spec": { "queryLanguage": "dsl", "query": "{\"query\":{\"match\":{\"level\":\"error\"}}}" } +} diff --git a/opensearch/schemas/queries/opensearch-log-query/tests/invalid/limit-above-max.json b/opensearch/schemas/queries/opensearch-log-query/tests/invalid/limit-above-max.json new file mode 100644 index 000000000..b3b6ab882 --- /dev/null +++ b/opensearch/schemas/queries/opensearch-log-query/tests/invalid/limit-above-max.json @@ -0,0 +1,4 @@ +{ + "kind": "OpenSearchLogQuery", + "spec": { "queryLanguage": "lucene", "query": "level:error", "index": "logs-*", "limit": 10001 } +} diff --git a/opensearch/schemas/queries/opensearch-log-query/tests/invalid/lucene-without-index.json b/opensearch/schemas/queries/opensearch-log-query/tests/invalid/lucene-without-index.json new file mode 100644 index 000000000..ec7bf41c1 --- /dev/null +++ b/opensearch/schemas/queries/opensearch-log-query/tests/invalid/lucene-without-index.json @@ -0,0 +1,4 @@ +{ + "kind": "OpenSearchLogQuery", + "spec": { "queryLanguage": "lucene", "query": "level:error" } +} diff --git a/opensearch/schemas/queries/opensearch-log-query/tests/invalid/negative-limit.json b/opensearch/schemas/queries/opensearch-log-query/tests/invalid/negative-limit.json new file mode 100644 index 000000000..b83bdfb89 --- /dev/null +++ b/opensearch/schemas/queries/opensearch-log-query/tests/invalid/negative-limit.json @@ -0,0 +1,4 @@ +{ + "kind": "OpenSearchLogQuery", + "spec": { "queryLanguage": "lucene", "query": "level:error", "index": "logs-*", "limit": -3 } +} diff --git a/opensearch/schemas/queries/opensearch-log-query/tests/invalid/sql-empty-query.json b/opensearch/schemas/queries/opensearch-log-query/tests/invalid/sql-empty-query.json new file mode 100644 index 000000000..cc6708835 --- /dev/null +++ b/opensearch/schemas/queries/opensearch-log-query/tests/invalid/sql-empty-query.json @@ -0,0 +1,4 @@ +{ + "kind": "OpenSearchLogQuery", + "spec": { "queryLanguage": "sql", "query": "" } +} diff --git a/opensearch/schemas/queries/opensearch-log-query/tests/invalid/unknown-language.json b/opensearch/schemas/queries/opensearch-log-query/tests/invalid/unknown-language.json new file mode 100644 index 000000000..172a82651 --- /dev/null +++ b/opensearch/schemas/queries/opensearch-log-query/tests/invalid/unknown-language.json @@ -0,0 +1,4 @@ +{ + "kind": "OpenSearchLogQuery", + "spec": { "queryLanguage": "kql", "query": "level:error", "index": "logs-*" } +} diff --git a/opensearch/schemas/queries/opensearch-log-query/tests/invalid/zero-limit.json b/opensearch/schemas/queries/opensearch-log-query/tests/invalid/zero-limit.json new file mode 100644 index 000000000..1db868137 --- /dev/null +++ b/opensearch/schemas/queries/opensearch-log-query/tests/invalid/zero-limit.json @@ -0,0 +1,4 @@ +{ + "kind": "OpenSearchLogQuery", + "spec": { "queryLanguage": "lucene", "query": "level:error", "index": "logs-*", "limit": 0 } +} diff --git a/opensearch/schemas/queries/opensearch-log-query/tests/valid/dsl.json b/opensearch/schemas/queries/opensearch-log-query/tests/valid/dsl.json new file mode 100644 index 000000000..70c3b3096 --- /dev/null +++ b/opensearch/schemas/queries/opensearch-log-query/tests/valid/dsl.json @@ -0,0 +1,4 @@ +{ + "kind": "OpenSearchLogQuery", + "spec": { "queryLanguage": "dsl", "query": "{\"query\":{\"match\":{\"level\":\"error\"}}}", "index": "logs-*" } +} diff --git a/opensearch/schemas/queries/opensearch-log-query/tests/valid/lucene-empty-query.json b/opensearch/schemas/queries/opensearch-log-query/tests/valid/lucene-empty-query.json new file mode 100644 index 000000000..34047889c --- /dev/null +++ b/opensearch/schemas/queries/opensearch-log-query/tests/valid/lucene-empty-query.json @@ -0,0 +1,4 @@ +{ + "kind": "OpenSearchLogQuery", + "spec": { "queryLanguage": "lucene", "query": "", "index": "logs-*" } +} diff --git a/opensearch/schemas/queries/opensearch-log-query/tests/valid/lucene.json b/opensearch/schemas/queries/opensearch-log-query/tests/valid/lucene.json new file mode 100644 index 000000000..df3a8a334 --- /dev/null +++ b/opensearch/schemas/queries/opensearch-log-query/tests/valid/lucene.json @@ -0,0 +1,4 @@ +{ + "kind": "OpenSearchLogQuery", + "spec": { "queryLanguage": "lucene", "query": "level:error AND service:api", "index": "logs-*", "limit": 250 } +} diff --git a/opensearch/schemas/queries/opensearch-log-query/tests/valid/max-limit.json b/opensearch/schemas/queries/opensearch-log-query/tests/valid/max-limit.json new file mode 100644 index 000000000..bcf939626 --- /dev/null +++ b/opensearch/schemas/queries/opensearch-log-query/tests/valid/max-limit.json @@ -0,0 +1,4 @@ +{ + "kind": "OpenSearchLogQuery", + "spec": { "queryLanguage": "lucene", "query": "level:error AND service:api", "index": "logs-*", "limit": 10000 } +} diff --git a/opensearch/schemas/queries/opensearch-log-query/tests/valid/no-language.json b/opensearch/schemas/queries/opensearch-log-query/tests/valid/no-language.json new file mode 100644 index 000000000..deff77e09 --- /dev/null +++ b/opensearch/schemas/queries/opensearch-log-query/tests/valid/no-language.json @@ -0,0 +1,4 @@ +{ + "kind": "OpenSearchLogQuery", + "spec": { "query": "source=logs-* | head 10" } +} diff --git a/opensearch/schemas/queries/opensearch-log-query/tests/valid/ppl.json b/opensearch/schemas/queries/opensearch-log-query/tests/valid/ppl.json new file mode 100644 index 000000000..927e76f89 --- /dev/null +++ b/opensearch/schemas/queries/opensearch-log-query/tests/valid/ppl.json @@ -0,0 +1,12 @@ +{ + "kind": "OpenSearchLogQuery", + "spec": { + "datasource": { "kind": "OpenSearchDatasource", "name": "opensearch-demo" }, + "queryLanguage": "ppl", + "query": "source=logs-* | where level='error'", + "index": "logs-*", + "timestampField": "@timestamp", + "messageField": "message", + "disableTimeFilter": false + } +} diff --git a/opensearch/schemas/queries/opensearch-log-query/tests/valid/sql.json b/opensearch/schemas/queries/opensearch-log-query/tests/valid/sql.json new file mode 100644 index 000000000..4cae7fea2 --- /dev/null +++ b/opensearch/schemas/queries/opensearch-log-query/tests/valid/sql.json @@ -0,0 +1,4 @@ +{ + "kind": "OpenSearchLogQuery", + "spec": { "queryLanguage": "sql", "query": "SELECT @timestamp, message FROM `logs-2025`" } +} diff --git a/opensearch/sdk/go/query/log/log.go b/opensearch/sdk/go/query/log/log.go index 20b653675..111a54426 100644 --- a/opensearch/sdk/go/query/log/log.go +++ b/opensearch/sdk/go/query/log/log.go @@ -27,18 +27,44 @@ const PluginKind = "OpenSearchLogQuery" type PluginSpec struct { Datasource *datasource.Selector `json:"datasource,omitempty" yaml:"datasource,omitempty"` Query string `json:"query" yaml:"query"` + QueryLanguage string `json:"queryLanguage,omitempty" yaml:"queryLanguage,omitempty"` + Limit int `json:"limit,omitempty" yaml:"limit,omitempty"` Index string `json:"index,omitempty" yaml:"index,omitempty"` TimestampField string `json:"timestampField,omitempty" yaml:"timestampField,omitempty"` MessageField string `json:"messageField,omitempty" yaml:"messageField,omitempty"` DisableTimeFilter bool `json:"disableTimeFilter,omitempty" yaml:"disableTimeFilter,omitempty"` } +// limitPresence is decoded alongside PluginSpec to tell an explicit `limit: 0` apart from an +// omitted limit, which both leave the bare-int PluginSpec.Limit at 0. +type limitPresence struct { + Limit *int `json:"limit" yaml:"limit"` +} + +// MaxLimit is the largest `limit` accepted: `_search` rejects a `size` above the index's +// max_result_window, which defaults to 10,000. +const MaxLimit = 10000 + +func (l limitPresence) validate() error { + if l.Limit != nil && *l.Limit <= 0 { + return fmt.Errorf("limit must be greater than 0, got %d", *l.Limit) + } + return nil +} + func (s *PluginSpec) UnmarshalJSON(data []byte) error { type plain PluginSpec var tmp PluginSpec if err := json.Unmarshal(data, (*plain)(&tmp)); err != nil { return err } + var limit limitPresence + if err := json.Unmarshal(data, &limit); err != nil { + return err + } + if err := limit.validate(); err != nil { + return err + } if err := (&tmp).validate(); err != nil { return err } @@ -52,6 +78,13 @@ func (s *PluginSpec) UnmarshalYAML(unmarshal func(interface{}) error) error { if err := unmarshal((*plain)(&tmp)); err != nil { return err } + var limit limitPresence + if err := unmarshal(&limit); err != nil { + return err + } + if err := limit.validate(); err != nil { + return err + } if err := (&tmp).validate(); err != nil { return err } @@ -59,10 +92,29 @@ func (s *PluginSpec) UnmarshalYAML(unmarshal func(interface{}) error) error { return nil } +// validLanguages mirrors the CUE schema. An empty value means PPL. +var validLanguages = map[string]bool{"": true, "ppl": true, "sql": true, "lucene": true, "dsl": true} + func (s *PluginSpec) validate() error { - if len(s.Query) == 0 { + if !validLanguages[s.QueryLanguage] { + return fmt.Errorf("unknown query language %q: want one of ppl, sql, lucene, dsl", s.QueryLanguage) + } + // A blank Lucene query string means "everything in the time range". + if len(s.Query) == 0 && s.QueryLanguage != "lucene" { return fmt.Errorf("query cannot be empty") } + if (s.QueryLanguage == "lucene" || s.QueryLanguage == "dsl") && len(s.Index) == 0 { + return fmt.Errorf("index is required when queryLanguage is %q", s.QueryLanguage) + } + // Only negative values are rejected here: with a bare int + omitempty, an omitted + // `limit` is also 0, so rejecting 0 would reject every spec that simply omits the + // field. An explicit zero is caught while unmarshalling, via limitPresence. + if s.Limit < 0 { + return fmt.Errorf("limit cannot be negative") + } + if s.Limit > MaxLimit { + return fmt.Errorf("limit must be at most %d, got %d", MaxLimit, s.Limit) + } return nil } @@ -83,6 +135,10 @@ func create(query string, options ...Option) (Builder, error) { } } + if err := builder.validate(); err != nil { + return *builder, err + } + return *builder, nil } diff --git a/opensearch/sdk/go/query/log/log_test.go b/opensearch/sdk/go/query/log/log_test.go index ef761d76e..088fe408a 100644 --- a/opensearch/sdk/go/query/log/log_test.go +++ b/opensearch/sdk/go/query/log/log_test.go @@ -15,6 +15,7 @@ package log import ( "encoding/json" + "strings" "testing" ) @@ -78,6 +79,22 @@ func TestPluginSpecRejectsEmptyQueryOnUnmarshal(t *testing.T) { } } +func TestPluginSpecAcceptsEmptyLuceneQueryOnUnmarshal(t *testing.T) { + raw := []byte(`{"query":"","queryLanguage":"lucene","index":"logs-*"}`) + var spec PluginSpec + if err := json.Unmarshal(raw, &spec); err != nil { + t.Fatalf("unexpected error: %v", err) + } +} + +func TestPluginSpecRejectsEmptySQLQueryOnUnmarshal(t *testing.T) { + raw := []byte(`{"query":"","queryLanguage":"sql"}`) + var spec PluginSpec + if err := json.Unmarshal(raw, &spec); err == nil { + t.Fatalf("expected error unmarshalling sql spec with empty query, got nil") + } +} + func TestPluginSpecAcceptsNonEmptyQueryOnUnmarshal(t *testing.T) { raw := []byte(`{"query":"source=logs-*"}`) var spec PluginSpec @@ -103,3 +120,152 @@ func TestOpenSearchLogQueryDisableTimeFilter(t *testing.T) { t.Errorf("disableTimeFilter mismatch: %v", out["disableTimeFilter"]) } } + +func TestQueryLanguageAndLimit(t *testing.T) { + q := OpenSearchLogQuery("level:error", + QueryLanguage("lucene"), + Index("logs-*"), + Limit(250), + ) + if q.Error != nil { + t.Fatalf("unexpected error: %v", q.Error) + } + + raw, err := json.Marshal(q.Plugin.Spec) + if err != nil { + t.Fatalf("marshal: %v", err) + } + var out map[string]any + if err := json.Unmarshal(raw, &out); err != nil { + t.Fatalf("unmarshal: %v", err) + } + + if out["queryLanguage"] != "lucene" { + t.Errorf("queryLanguage mismatch: %v", out["queryLanguage"]) + } + // JSON numbers decode as float64 into map[string]any. + if out["limit"] != float64(250) { + t.Errorf("limit mismatch: %v", out["limit"]) + } +} + +func TestDefaultLanguageIsOmitted(t *testing.T) { + q := OpenSearchLogQuery("source=logs-*") + if q.Error != nil { + t.Fatalf("unexpected error: %v", q.Error) + } + + raw, err := json.Marshal(q.Plugin.Spec) + if err != nil { + t.Fatalf("marshal: %v", err) + } + var out map[string]any + if err := json.Unmarshal(raw, &out); err != nil { + t.Fatalf("unmarshal: %v", err) + } + + // No queryLanguage or limit key at all: the plugin treats absence as PPL. + if _, ok := out["queryLanguage"]; ok { + t.Errorf("queryLanguage should be omitted, got %v", out["queryLanguage"]) + } + if _, ok := out["limit"]; ok { + t.Errorf("limit should be omitted, got %v", out["limit"]) + } +} + +func TestRejectsUnknownLanguage(t *testing.T) { + q := OpenSearchLogQuery("level:error", QueryLanguage("kql"), Index("logs-*")) + if q.Error == nil { + t.Fatal("expected an error for an unknown query language") + } + if !strings.Contains(q.Error.Error(), "unknown query language") { + t.Errorf("unexpected error text: %v", q.Error) + } +} + +func TestRequiresIndexForLucene(t *testing.T) { + var spec PluginSpec + err := json.Unmarshal([]byte(`{"query":"level:error","queryLanguage":"lucene"}`), &spec) + if err == nil { + t.Fatal("expected an error when index is missing for lucene") + } + if !strings.Contains(err.Error(), "index is required") { + t.Errorf("unexpected error text: %v", err) + } +} + +func TestLimitRejectsZero(t *testing.T) { + q := OpenSearchLogQuery("level:error", QueryLanguage("lucene"), Index("logs-*"), Limit(0)) + if q.Error == nil { + t.Fatal("expected an error for Limit(0)") + } + if !strings.Contains(q.Error.Error(), "limit must be greater than 0") { + t.Errorf("unexpected error text: %v", q.Error) + } +} + +func TestLimitRejectsNegative(t *testing.T) { + q := OpenSearchLogQuery("level:error", QueryLanguage("lucene"), Index("logs-*"), Limit(-3)) + if q.Error == nil { + t.Fatal("expected an error for Limit(-3)") + } + if !strings.Contains(q.Error.Error(), "limit must be greater than 0") { + t.Errorf("unexpected error text: %v", q.Error) + } +} + +func TestLimitRejectsAboveMax(t *testing.T) { + q := OpenSearchLogQuery("level:error", QueryLanguage("lucene"), Index("logs-*"), Limit(MaxLimit+1)) + if q.Error == nil { + t.Fatal("expected an error for a limit above MaxLimit") + } + if !strings.Contains(q.Error.Error(), "limit must be at most 10000") { + t.Errorf("unexpected error text: %v", q.Error) + } +} + +func TestPluginSpecLimitOnUnmarshal(t *testing.T) { + tests := []struct { + name string + raw string + wantErr bool + want int + }{ + {name: "explicit zero", raw: `{"query":"q","limit":0}`, wantErr: true}, + {name: "negative", raw: `{"query":"q","limit":-1}`, wantErr: true}, + {name: "omitted", raw: `{"query":"q"}`, want: 0}, + {name: "positive", raw: `{"query":"q","limit":25}`, want: 25}, + {name: "maximum", raw: `{"query":"q","limit":10000}`, want: 10000}, + {name: "above maximum", raw: `{"query":"q","limit":10001}`, wantErr: true}, + } + for _, tt := range tests { + t.Run("json/"+tt.name, func(t *testing.T) { + var spec PluginSpec + err := json.Unmarshal([]byte(tt.raw), &spec) + checkLimitResult(t, spec, err, tt.wantErr, tt.want) + }) + // UnmarshalYAML receives a decoder callback; drive it with a JSON-backed one so the + // YAML code path is exercised without adding a YAML library dependency. + t.Run("yaml/"+tt.name, func(t *testing.T) { + var spec PluginSpec + err := spec.UnmarshalYAML(func(v interface{}) error { return json.Unmarshal([]byte(tt.raw), v) }) + checkLimitResult(t, spec, err, tt.wantErr, tt.want) + }) + } +} + +func checkLimitResult(t *testing.T, spec PluginSpec, err error, wantErr bool, want int) { + t.Helper() + if wantErr { + if err == nil { + t.Fatal("expected an error, got nil") + } + return + } + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if spec.Limit != want { + t.Errorf("limit = %d, want %d", spec.Limit, want) + } +} diff --git a/opensearch/sdk/go/query/log/options.go b/opensearch/sdk/go/query/log/options.go index 520deeead..689975ae7 100644 --- a/opensearch/sdk/go/query/log/options.go +++ b/opensearch/sdk/go/query/log/options.go @@ -14,6 +14,8 @@ package log import ( + "fmt" + opensearchDatasource "github.com/perses/plugins/opensearch/sdk/go/datasource" ) @@ -62,3 +64,29 @@ func DisableTimeFilter(disable bool) Option { return nil } } + +// QueryLanguage sets the language the query is written in: "ppl" (the default when +// unset), "sql", "lucene", or "dsl". "lucene" and "dsl" also require Index. +func QueryLanguage(lang string) Option { + return func(builder *Builder) error { + builder.QueryLanguage = lang + return nil + } +} + +// Limit caps the number of log documents fetched. Applies to the "lucene" and "dsl" +// languages, which map it to the _search request's `size`. Must be between 1 and MaxLimit: +// the CUE schema constrains `limit` to `int & >0 & <=10000`, and because the field is a bare +// int with omitempty, a zero would be serialized as an omitted value. +func Limit(n int) Option { + return func(builder *Builder) error { + if n <= 0 { + return fmt.Errorf("limit must be greater than 0, got %d", n) + } + if n > MaxLimit { + return fmt.Errorf("limit must be at most %d, got %d", MaxLimit, n) + } + builder.Limit = n + return nil + } +} diff --git a/opensearch/src/datasources/opensearch-datasource/OpenSearchDatasource.tsx b/opensearch/src/datasources/opensearch-datasource/OpenSearchDatasource.tsx index 4943a8f15..b94f5ce13 100644 --- a/opensearch/src/datasources/opensearch-datasource/OpenSearchDatasource.tsx +++ b/opensearch/src/datasources/opensearch-datasource/OpenSearchDatasource.tsx @@ -14,7 +14,7 @@ import type { DatasourcePlugin } from '@perses-dev/plugin-system'; import type { OpenSearchClient } from '../../model/opensearch-client'; -import { ppl } from '../../model/opensearch-client'; +import { ppl, search, sql } from '../../model/opensearch-client'; import type { OpenSearchDatasourceSpec } from './opensearch-datasource-types'; import { OpenSearchDatasourceEditor } from './OpenSearchDatasourceEditor'; @@ -35,6 +35,10 @@ const createClient: DatasourcePlugin }, ppl: (params, headers, signal) => ppl(params, { datasourceUrl, headers: headers ?? specHeaders, fetch: customFetch }, signal), + sql: (params, headers, signal) => + sql(params, { datasourceUrl, headers: headers ?? specHeaders, fetch: customFetch }, signal), + search: (params, headers, signal) => + search(params, { datasourceUrl, headers: headers ?? specHeaders, fetch: customFetch }, signal), }; }; diff --git a/opensearch/src/datasources/opensearch-datasource/OpenSearchDatasourceEditor.tsx b/opensearch/src/datasources/opensearch-datasource/OpenSearchDatasourceEditor.tsx index 0f59b1326..91450c597 100644 --- a/opensearch/src/datasources/opensearch-datasource/OpenSearchDatasourceEditor.tsx +++ b/opensearch/src/datasources/opensearch-datasource/OpenSearchDatasourceEditor.tsx @@ -35,6 +35,16 @@ export function OpenSearchDatasourceEditor(props: OpenSearchDatasourceEditorProp endpointPattern: '/_plugins/_ppl', method: 'POST', }, + { + endpointPattern: '/_plugins/_sql', + method: 'POST', + }, + { + // Lucene and Query DSL queries POST to /{index}/_search, so the index + // segment has to be matched as a pattern. + endpointPattern: '/.*/_search', + method: 'POST', + }, ], url: '', }, diff --git a/opensearch/src/deprecated-exports.test.ts b/opensearch/src/deprecated-exports.test.ts new file mode 100644 index 000000000..66715cbed --- /dev/null +++ b/opensearch/src/deprecated-exports.test.ts @@ -0,0 +1,34 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import type { OpenSearchDatarowsResponse, OpenSearchPPLColumn, OpenSearchPPLResponse } from './index'; +import { convertDatarowsToLogs, convertPPLToLogs, OpenSearchApiError, OpenSearchPPLError } from './index'; + +// Names renamed when SQL/Lucene/DSL support generalised the PPL-only API. They are part of the +// published package surface, so they stay exported from the package root as deprecated aliases. +describe('deprecated package-root exports', () => { + it('keeps convertPPLToLogs as an alias of convertDatarowsToLogs', () => { + expect(convertPPLToLogs).toBe(convertDatarowsToLogs); + }); + + it('keeps OpenSearchPPLError as a subclass of OpenSearchApiError', () => { + expect(new OpenSearchPPLError(500, '')).toBeInstanceOf(OpenSearchApiError); + }); + + it('keeps OpenSearchPPLResponse and OpenSearchPPLColumn assignable to the new types', () => { + const column: OpenSearchPPLColumn = { name: 'message', type: 'string' }; + const response: OpenSearchPPLResponse = { schema: [column], datarows: [['hello']] }; + const generalised: OpenSearchDatarowsResponse = response; + expect(generalised.schema[0]?.name).toBe('message'); + }); +}); diff --git a/opensearch/src/model/opensearch-client-types.ts b/opensearch/src/model/opensearch-client-types.ts index 81fd2de25..e0c58a26f 100644 --- a/opensearch/src/model/opensearch-client-types.ts +++ b/opensearch/src/model/opensearch-client-types.ts @@ -13,18 +13,26 @@ export type OpenSearchRequestHeaders = Record; -export interface OpenSearchPPLColumn { +export interface OpenSearchDatarowsColumn { name: string; type: string; } -export interface OpenSearchPPLResponse { - schema: OpenSearchPPLColumn[]; - datarows: Array>; +/** The schema + datarows shape returned by _ppl and _sql. OpenSearch calls this its `jdbc` response format (?format=jdbc), which is the default. */ +export interface OpenSearchDatarowsResponse { + schema: OpenSearchDatarowsColumn[]; + // Object/struct and array columns (e.g. OTel's `service`) come back as nested JSON values. + datarows: Array | unknown[]>>; total?: number; size?: number; } +/** @deprecated Use `OpenSearchDatarowsColumn`. */ +export type OpenSearchPPLColumn = OpenSearchDatarowsColumn; + +/** @deprecated Use `OpenSearchDatarowsResponse`. */ +export type OpenSearchPPLResponse = OpenSearchDatarowsResponse; + export interface OpenSearchErrorResponse { error?: { type?: string; @@ -33,3 +41,20 @@ export interface OpenSearchErrorResponse { }; status?: number; } + +export interface OpenSearchSearchHit { + _index?: string; + _id?: string; + _source?: Record; +} + +export interface OpenSearchSearchResponse { + hits: { + /** Absent when the request sets `track_total_hits: false`. */ + total?: { value: number; relation: 'eq' | 'gte' }; + hits: OpenSearchSearchHit[]; + }; +} + +/** A `_search` request body. Kept loose so a user-supplied Query DSL body passes through unchanged. */ +export type OpenSearchSearchRequestBody = Record; diff --git a/opensearch/src/model/opensearch-client.test.ts b/opensearch/src/model/opensearch-client.test.ts index 9d60af651..f7064a982 100644 --- a/opensearch/src/model/opensearch-client.test.ts +++ b/opensearch/src/model/opensearch-client.test.ts @@ -13,7 +13,7 @@ import type { Mock } from 'vitest'; -import { ppl, OpenSearchPPLError } from './opensearch-client'; +import { ppl, sql, search, OpenSearchApiError, OpenSearchPPLError } from './opensearch-client'; describe('opensearch-client', () => { const originalFetch = global.fetch; @@ -62,15 +62,28 @@ describe('opensearch-client', () => { expect((init.headers as Record).Authorization).toBe('Basic xyz'); }); - it('throws OpenSearchPPLError carrying status and body on non-200', async () => { + it('throws the deprecated OpenSearchPPLError, also an OpenSearchApiError, carrying status and body on non-200', async () => { mockFetch({ ok: false, status: 400, body: '{"error":{"reason":"bad PPL"}}' }); - await expect(ppl({ query: 'broken' }, { datasourceUrl: 'http://localhost:9200' })).rejects.toMatchObject({ + const err = await ppl({ query: 'broken' }, { datasourceUrl: 'http://localhost:9200' }).catch((e: unknown) => e); + expect(err).toMatchObject({ name: 'OpenSearchPPLError', status: 400, body: '{"error":{"reason":"bad PPL"}}', }); - await expect(ppl({ query: 'broken' }, { datasourceUrl: 'http://localhost:9200' })).rejects.toBeInstanceOf( - OpenSearchPPLError, + expect(err).toBeInstanceOf(OpenSearchPPLError); + expect(err).toBeInstanceOf(OpenSearchApiError); + }); + + it('keeps the deprecated OpenSearchPPLError (status, body) constructor working', () => { + const err = new OpenSearchPPLError(400, '{"error":{"reason":"bad PPL"}}'); + expect(err.name).toBe('OpenSearchPPLError'); + expect(err.message).toBe('OpenSearch PPL request failed (400): bad PPL'); + }); + + it('labels the error message PPL', async () => { + mockFetch({ ok: false, status: 400, body: '{"error":{"reason":"bad PPL"}}' }); + await expect(ppl({ query: 'broken' }, { datasourceUrl: 'http://localhost:9200' })).rejects.toThrow( + 'OpenSearch PPL request failed (400): bad PPL', ); }); @@ -81,55 +94,181 @@ describe('opensearch-client', () => { expect(url).toContain('/api/datasources/proxy/1/_plugins/_ppl'); }); - describe('OpenSearchPPLError.message', () => { + describe('OpenSearchApiError.message', () => { it('uses error.reason from a JSON body when present', () => { - const err = new OpenSearchPPLError(400, '{"error":{"reason":"bad PPL","type":"SyntaxCheckException"}}'); - expect(err.message).toBe('OpenSearch PPL request failed (400): bad PPL'); + const err = new OpenSearchApiError(400, '{"error":{"reason":"bad PPL","type":"SyntaxCheckException"}}'); + expect(err.message).toBe('OpenSearch API request failed (400): bad PPL'); expect(err.body).toBe('{"error":{"reason":"bad PPL","type":"SyntaxCheckException"}}'); }); it('falls back to error.details when reason is absent', () => { - const err = new OpenSearchPPLError(400, '{"error":{"details":"timestamp:2026-... in unsupported format"}}'); - expect(err.message).toBe('OpenSearch PPL request failed (400): timestamp:2026-... in unsupported format'); + const err = new OpenSearchApiError(400, '{"error":{"details":"timestamp:2026-... in unsupported format"}}'); + expect(err.message).toBe('OpenSearch API request failed (400): timestamp:2026-... in unsupported format'); }); it('omits the body entirely when the body is not JSON', () => { - const err = new OpenSearchPPLError(502, 'Bad Gateway'); - expect(err.message).toBe('OpenSearch PPL request failed (502)'); + const err = new OpenSearchApiError(502, 'Bad Gateway'); + expect(err.message).toBe('OpenSearch API request failed (502)'); expect(err.body).toBe('Bad Gateway'); }); it('omits the body when JSON has no error field', () => { - const err = new OpenSearchPPLError(500, '{"status":"weird"}'); - expect(err.message).toBe('OpenSearch PPL request failed (500)'); + const err = new OpenSearchApiError(500, '{"status":"weird"}'); + expect(err.message).toBe('OpenSearch API request failed (500)'); }); it('combines reason and details when both are present and differ', () => { - const err = new OpenSearchPPLError( + const err = new OpenSearchApiError( 400, '{"error":{"reason":"Invalid Query","details":"can\'t resolve Symbol(name=@timestamp)"}}', ); expect(err.message).toBe( - "OpenSearch PPL request failed (400): Invalid Query — can't resolve Symbol(name=@timestamp)", + "OpenSearch API request failed (400): Invalid Query — can't resolve Symbol(name=@timestamp)", ); }); it('does not duplicate text when reason equals details', () => { - const err = new OpenSearchPPLError(400, '{"error":{"reason":"same text","details":"same text"}}'); - expect(err.message).toBe('OpenSearch PPL request failed (400): same text'); + const err = new OpenSearchApiError(400, '{"error":{"reason":"same text","details":"same text"}}'); + expect(err.message).toBe('OpenSearch API request failed (400): same text'); }); it('uses reason alone when details is an empty string', () => { - const err = new OpenSearchPPLError(400, '{"error":{"reason":"Invalid Query","details":""}}'); - expect(err.message).toBe('OpenSearch PPL request failed (400): Invalid Query'); + const err = new OpenSearchApiError(400, '{"error":{"reason":"Invalid Query","details":""}}'); + expect(err.message).toBe('OpenSearch API request failed (400): Invalid Query'); + }); + + it('explains a Perses proxy 403 for an endpoint missing from allowedEndpoints', () => { + const err = new OpenSearchApiError( + 403, + '{"message":"you are not allowed to use this endpoint \\"/_plugins/_sql\\" with the HTTP method POST"}', + ); + expect(err.message).toBe( + 'OpenSearch API request was blocked by the Perses datasource proxy (403): ' + + 'you are not allowed to use this endpoint "/_plugins/_sql" with the HTTP method POST. ' + + "Add the endpoint to the datasource's proxy allowedEndpoints (POST /_plugins/_ppl, " + + '/_plugins/_sql and /.*/_search).', + ); + }); + + it('keeps the OpenSearch reason for a 403 that did not come from the proxy', () => { + const err = new OpenSearchApiError(403, '{"error":{"reason":"no permissions for [indices:data/read/search]"}}'); + expect(err.message).toBe('OpenSearch API request failed (403): no permissions for [indices:data/read/search]'); }); it('falls through to details when reason is an empty string', () => { - const err = new OpenSearchPPLError( + const err = new OpenSearchApiError( 400, '{"error":{"reason":"","details":"can\'t resolve Symbol(name=@timestamp)"}}', ); - expect(err.message).toBe("OpenSearch PPL request failed (400): can't resolve Symbol(name=@timestamp)"); + expect(err.message).toBe("OpenSearch API request failed (400): can't resolve Symbol(name=@timestamp)"); + }); + }); + + describe('sql', () => { + it('POSTs only the query to /_plugins/_sql', async () => { + const mock = mockFetch({ ok: true, status: 200, body: { schema: [], datarows: [] } }); + await sql({ query: 'SELECT * FROM logs' }, { datasourceUrl: 'http://localhost:9200' }); + + const [url, init] = mock.mock.calls[0] as [string, RequestInit]; + expect(url).toBe('http://localhost:9200/_plugins/_sql'); + expect(init.method).toBe('POST'); + expect(JSON.parse(init.body as string)).toEqual({ query: 'SELECT * FROM logs' }); + }); + + it('throws a plain OpenSearchApiError, not the PPL subclass', async () => { + mockFetch({ ok: false, status: 400, body: '{"error":{"reason":"bad SQL"}}' }); + const err = await sql({ query: 'x' }, { datasourceUrl: 'http://localhost:9200' }).catch((e: unknown) => e); + expect(err).toBeInstanceOf(OpenSearchApiError); + expect(err).not.toBeInstanceOf(OpenSearchPPLError); + expect(err).toMatchObject({ name: 'OpenSearchApiError' }); + }); + + it('labels the error message SQL', async () => { + mockFetch({ ok: false, status: 400, body: '{"error":{"reason":"bad SQL"}}' }); + await expect(sql({ query: 'x' }, { datasourceUrl: 'http://localhost:9200' })).rejects.toThrow( + 'OpenSearch SQL request failed (400): bad SQL', + ); + }); + }); + + describe('search', () => { + it('POSTs the body to /{index}/_search', async () => { + const mock = mockFetch({ ok: true, status: 200, body: { hits: { hits: [] } } }); + const body = { size: 500, query: { match_all: {} } }; + await search({ index: 'logs-*', body }, { datasourceUrl: 'http://localhost:9200' }); + + const [url, init] = mock.mock.calls[0] as [string, RequestInit]; + expect(url).toBe('http://localhost:9200/logs-*/_search'); + expect(JSON.parse(init.body as string)).toEqual(body); + }); + + it('preserves commas for multi-index patterns', async () => { + const mock = mockFetch({ ok: true, status: 200, body: { hits: { hits: [] } } }); + await search({ index: 'a-*,b-*', body: {} }, { datasourceUrl: 'http://localhost:9200' }); + const [url] = mock.mock.calls[0] as [string]; + expect(url).toBe('http://localhost:9200/a-*,b-*/_search'); + }); + + it.each([ + ['a cross-cluster index', 'remote:logs-*', 'remote:logs-*'], + ['spaces after the commas of a multi-index list', 'logs-*, app-*', 'logs-*,app-*'], + ])('accepts %s', async (_, index, expected) => { + const mock = mockFetch({ ok: true, status: 200, body: { hits: { hits: [] } } }); + await search({ index, body: {} }, { datasourceUrl: 'http://localhost:9200' }); + const [url] = mock.mock.calls[0] as [string]; + expect(url).toBe(`http://localhost:9200/${expected}/_search`); + }); + + it('allows dots within an index name (date-suffixed indices)', async () => { + const mock = mockFetch({ ok: true, status: 200, body: { hits: { hits: [] } } }); + await search({ index: 'logs-2025.01.01', body: {} }, { datasourceUrl: 'http://localhost:9200' }); + const [url] = mock.mock.calls[0] as [string]; + expect(url).toBe('http://localhost:9200/logs-2025.01.01/_search'); + }); + + it.each([ + '../_cluster/settings', + 'logs/../..', + 'a?b', + 'a#b', + 'logs *', + '', + '%2e%2e', + '%2E%2E', + '%2e', + '.', + '..', + '\\', + 'logs-*,.', + 'a%20b', + ])('rejects unsafe index %p without issuing a request', async (index) => { + const mock = mockFetch({ ok: true, status: 200, body: { hits: { hits: [] } } }); + await expect(search({ index, body: {} }, { datasourceUrl: 'http://localhost:9200' })).rejects.toThrow( + /invalid OpenSearch index/i, + ); + expect(mock).not.toHaveBeenCalled(); + }); + + it('surfaces root_cause reason from a _search error body', async () => { + mockFetch({ + ok: false, + status: 400, + body: '{"error":{"type":"search_phase_execution_exception","root_cause":[{"reason":"failed to parse date field"}]},"status":400}', + }); + await expect(search({ index: 'logs-*', body: {} }, { datasourceUrl: 'http://localhost:9200' })).rejects.toThrow( + 'OpenSearch Search request failed (400): failed to parse date field', + ); + }); + + it('surfaces root_cause reason alongside a generic top-level reason', async () => { + mockFetch({ + ok: false, + status: 400, + body: '{"error":{"root_cause":[{"type":"query_shard_exception","reason":"failed to create query: For input string: \\"abc\\""}],"type":"search_phase_execution_exception","reason":"all shards failed"},"status":400}', + }); + await expect(search({ index: 'logs-*', body: {} }, { datasourceUrl: 'http://localhost:9200' })).rejects.toThrow( + 'OpenSearch Search request failed (400): all shards failed — failed to create query: For input string: "abc"', + ); }); }); }); diff --git a/opensearch/src/model/opensearch-client.ts b/opensearch/src/model/opensearch-client.ts index 17d60c8e1..8767a1a0a 100644 --- a/opensearch/src/model/opensearch-client.ts +++ b/opensearch/src/model/opensearch-client.ts @@ -13,7 +13,12 @@ import type { FetchFn } from '@perses-dev/client'; -import type { OpenSearchPPLResponse, OpenSearchRequestHeaders } from './opensearch-client-types'; +import type { + OpenSearchDatarowsResponse, + OpenSearchRequestHeaders, + OpenSearchSearchRequestBody, + OpenSearchSearchResponse, +} from './opensearch-client-types'; export interface OpenSearchPPLParams { query: string; @@ -25,6 +30,15 @@ export interface OpenSearchApiOptions { fetch?: FetchFn; } +export interface OpenSearchSQLParams { + query: string; +} + +export interface OpenSearchSearchParams { + index: string; + body: OpenSearchSearchRequestBody; +} + export interface OpenSearchClient { options: { datasourceUrl: string; @@ -33,36 +47,82 @@ export interface OpenSearchClient { params: OpenSearchPPLParams, headers?: OpenSearchRequestHeaders, signal?: AbortSignal, - ) => Promise; + ) => Promise; + sql: ( + params: OpenSearchSQLParams, + headers?: OpenSearchRequestHeaders, + signal?: AbortSignal, + ) => Promise; + search: ( + params: OpenSearchSearchParams, + headers?: OpenSearchRequestHeaders, + signal?: AbortSignal, + ) => Promise; } -export class OpenSearchPPLError extends Error { +export class OpenSearchApiError extends Error { constructor( public readonly status: number, public readonly body: string, message?: string, ) { - super(message ?? buildShortMessage(status, body)); + super(message ?? buildShortMessage('API', status, body)); + this.name = 'OpenSearchApiError'; + } +} + +/** + * @deprecated Use `OpenSearchApiError`. Kept so existing `instanceof OpenSearchPPLError` + * checks and `err.name === 'OpenSearchPPLError'` comparisons keep working: `ppl` still + * throws this subclass, so a PPL failure is an instance of both classes. + */ +export class OpenSearchPPLError extends OpenSearchApiError { + constructor(status: number, body: string, message?: string) { + super(status, body, message ?? buildShortMessage('PPL', status, body)); this.name = 'OpenSearchPPLError'; } } -function buildShortMessage(status: number, body: string): string { +/** + * Label used in error messages so the user knows which API rejected the request. + * `'API'` is a neutral fallback for a caller that constructs `OpenSearchApiError` + * without an explicit message — unreachable today since all three transports pass + * one, but a future caller doing so should get a generic label rather than a + * misleading `'PPL'`. + */ +type ApiLabel = 'PPL' | 'SQL' | 'Search' | 'API'; + +function buildShortMessage(label: ApiLabel, status: number, body: string): string { try { - const parsed = JSON.parse(body) as { error?: { reason?: string; details?: string } }; - const reason = parsed?.error?.reason; - const details = parsed?.error?.details; - // OpenSearch often puts a generic phrase in `reason` ("Invalid Query") and the - // actionable text in `details` ("can't resolve Symbol(name=@timestamp)"). Surface - // both so the user can diagnose the failure. - const suffix = reason && details && reason !== details ? `${reason} — ${details}` : reason || details; + const parsed = JSON.parse(body) as { + error?: { reason?: string; details?: string; root_cause?: Array<{ reason?: string }> }; + message?: string; + }; + // The Perses datasource proxy rejects endpoints missing from `allowedEndpoints` with a + // 403 and a top-level `message`. Datasources created before SQL/Lucene/DSL support only + // allow `/_plugins/_ppl`, so say how to fix it rather than surfacing a bare 403. + if (status === 403 && parsed?.message?.includes('not allowed to use this endpoint')) { + return ( + `OpenSearch ${label} request was blocked by the Perses datasource proxy (403): ${parsed.message}. ` + + "Add the endpoint to the datasource's proxy allowedEndpoints (POST /_plugins/_ppl, " + + '/_plugins/_sql and /.*/_search).' + ); + } + // OpenSearch often puts a generic phrase in `reason` ("Invalid Query", "all shards + // failed") and the actionable text in `details` or root_cause[0].reason ("can't resolve + // Symbol(name=@timestamp)", "failed to parse date field"). Surface every distinct one + // so the user can diagnose the failure. + const messages = [parsed?.error?.reason, parsed?.error?.details, parsed?.error?.root_cause?.[0]?.reason].filter( + (m): m is string => typeof m === 'string' && m !== '', + ); + const suffix = [...new Set(messages)].join(' — '); if (suffix) { - return `OpenSearch PPL request failed (${status}): ${suffix}`; + return `OpenSearch ${label} request failed (${status}): ${suffix}`; } } catch { // body wasn't JSON — fall through to the bare-status form } - return `OpenSearch PPL request failed (${status})`; + return `OpenSearch ${label} request failed (${status})`; } function buildUrl(path: string, datasourceUrl: string): URL { @@ -82,28 +142,102 @@ function buildUrl(path: string, datasourceUrl: string): URL { return new URL(fullPath, window.location.origin); } -export async function ppl( - params: OpenSearchPPLParams, +async function postJson( + label: ApiLabel, + path: string, + body: unknown, options: OpenSearchApiOptions, signal?: AbortSignal, -): Promise { - const url = buildUrl('/_plugins/_ppl', options.datasourceUrl); +): Promise { + const url = buildUrl(path, options.datasourceUrl); const doFetch = options.fetch ?? fetch; const response = await doFetch(url.toString(), { method: 'POST', - headers: { - 'Content-Type': 'application/json', - ...options.headers, - }, - body: JSON.stringify({ query: params.query }), + headers: { 'Content-Type': 'application/json', ...options.headers }, + body: JSON.stringify(body), signal, }); if (!response.ok) { - const body = await response.text(); - throw new OpenSearchPPLError(response.status, body); + const text = await response.text(); + const message = buildShortMessage(label, response.status, text); + throw label === 'PPL' + ? new OpenSearchPPLError(response.status, text, message) + : new OpenSearchApiError(response.status, text, message); } return response.json(); } + +/** + * `_search` puts the index in the URL *path*, so an index like `../_cluster/settings` + * would reach a different endpoint entirely. `sanitizeIndex` in build/ppl.ts is the + * wrong filter here (it targets PPL metacharacters), so guard the path separately. + * + * This MUST be an allowlist, not a denylist of specific characters: the value is + * interpolated into a URL path and then re-parsed by `new URL` (via `buildUrl`), and + * that WHATWG parser normalises the path *after* this guard would run — it collapses + * percent-encoded and bare dot segments (`%2e%2e`, `.`, `..`) and treats a backslash + * as a path separator, so a denylist that only checks for literal `/`, `?`, `#`, `..` + * misses `%2e%2e` (decodes to `..` and climbs out of the index segment to the + * cluster-wide root) and `\` (reparsed as an authority separator, escaping the host + * entirely). Only an allowlist of characters that OpenSearch itself permits in an + * index name closes this off in one rule, because it rejects `%` and `\` outright + * regardless of what they'd decode or reparse to. `:` is allowed for cross-cluster + * search (`cluster:logs-*`); behind the leading `/` it stays inside the path segment. + */ +export function assertSafeIndexPath(index: string): void { + const invalid = (): never => { + throw new Error( + `invalid OpenSearch index for a _search request: ${JSON.stringify(index)}. ` + + 'An index (or comma-separated list of indices) must be non-empty and may only contain ' + + 'letters, digits, "_", ".", "+", "-", "*" and ":" (as in a cross-cluster "cluster:index"), ' + + 'with no empty or dots-only segment.', + ); + }; + + if (!index || /[^A-Za-z0-9_.+*,:-]/.test(index)) { + invalid(); + } + + // Reject a comma-separated segment that is empty or consists only of dots — this + // catches the bare `.` / `..` traversal forms (and a dot segment hidden inside a + // multi-index list like `logs-*,.`) that the character allowlist above lets through, + // since `.` is itself a legitimate character in an index name like `logs-2025.01.01`. + for (const segment of index.split(',')) { + if (segment === '' || /^\.+$/.test(segment)) { + invalid(); + } + } +} + +export async function ppl( + params: OpenSearchPPLParams, + options: OpenSearchApiOptions, + signal?: AbortSignal, +): Promise { + return postJson('PPL', '/_plugins/_ppl', { query: params.query }, options, signal); +} + +export async function sql( + params: OpenSearchSQLParams, + options: OpenSearchApiOptions, + signal?: AbortSignal, +): Promise { + return postJson('SQL', '/_plugins/_sql', { query: params.query }, options, signal); +} + +export async function search( + params: OpenSearchSearchParams, + options: OpenSearchApiOptions, + signal?: AbortSignal, +): Promise { + // A multi-index list is often written with spaces after the commas (`logs-*, app-*`). + const index = params.index + .split(',') + .map((segment) => segment.trim()) + .join(','); + assertSafeIndexPath(index); + return postJson('Search', `/${index}/_search`, params.body, options, signal); +} diff --git a/opensearch/src/queries/constants.ts b/opensearch/src/queries/constants.ts index be59e5dbe..199ad69ee 100644 --- a/opensearch/src/queries/constants.ts +++ b/opensearch/src/queries/constants.ts @@ -11,6 +11,8 @@ // See the License for the specific language governing permissions and // limitations under the License. +import type { OpenSearchQueryLanguage } from './opensearch-log-query/opensearch-log-query-types'; + export const DATASOURCE_KIND = 'OpenSearchDatasource'; export const DEFAULT_DATASOURCE = { kind: DATASOURCE_KIND }; @@ -19,11 +21,74 @@ export const DEFAULT_DATASOURCE = { kind: DATASOURCE_KIND }; export const DEFAULT_TIMESTAMP_FIELDS = ['@timestamp', 'timestamp', 'time']; export const DEFAULT_MESSAGE_FIELDS = ['message', 'log', 'body']; -// OpenSearch PPL reference documentation. -export const PPL_DOCS_URL = 'https://opensearch.org/docs/latest/search-plugins/sql/ppl/index/'; +export interface LanguageMeta { + label: string; + docsUrl: string; + docsLabel: string; + placeholder: string; + examples: string; + indexHelperText: string; + usageNote: string; +} -// A handful of copy-pasteable PPL examples shown in the editor's "Query Examples" disclosure. -export const PPL_QUERY_EXAMPLES = `source=logs-* | where level="error" | head 20 +/** + * Single source of truth for the editor's language dropdown and every piece of + * language-specific copy. Adding a language is one entry here plus a `case` in the + * dispatcher. + */ +export const QUERY_LANGUAGES: Record = { + ppl: { + label: 'PPL', + docsUrl: 'https://docs.opensearch.org/latest/sql-and-ppl/ppl/index/', + docsLabel: 'PPL', + placeholder: 'e.g. source=logs-* | where level="error"', + examples: `source=logs-* | where level="error" | head 20 source=logs-* | fields @timestamp, service, message | head 50 source=logs-* | stats count() by service -source=logs-* | where traceId='$traceId'`; +source=logs-* | where traceId='$traceId'`, + indexHelperText: 'Ignored when the PPL query starts with source=.', + usageNote: 'Requires the SQL/PPL plugin enabled on the OpenSearch cluster.', + }, + sql: { + label: 'SQL', + docsUrl: 'https://docs.opensearch.org/latest/sql-and-ppl/sql/index/', + docsLabel: 'SQL', + placeholder: "e.g. SELECT @timestamp, message FROM `logs-*` WHERE level = 'error'", + examples: `SELECT @timestamp, message, level FROM \`logs-*\` WHERE level = 'error' LIMIT 20 +SELECT service, COUNT(*) FROM \`logs-*\` GROUP BY service +SELECT * FROM \`otel-logs-*\` WHERE traceId = '$traceId' LIMIT 50`, + indexHelperText: 'Ignored — the index comes from the FROM clause of your SQL.', + usageNote: 'Requires the SQL/PPL plugin enabled on the OpenSearch cluster.', + }, + lucene: { + label: 'Lucene', + docsUrl: 'https://docs.opensearch.org/latest/query-dsl/full-text/query-string/', + docsLabel: 'query string', + placeholder: 'e.g. level:error AND service:api', + examples: `level:error AND service:api +message:"connection refused" +status:[500 TO 599] +traceId:$traceId +NOT level:debug`, + indexHelperText: 'Required. Sent as the index in POST /{index}/_search.', + usageNote: 'The dashboard time range is added to your query automatically.', + }, + dsl: { + label: 'Query DSL', + docsUrl: 'https://docs.opensearch.org/latest/query-dsl/', + docsLabel: 'Query DSL', + placeholder: 'e.g. {"query": {"match": {"level": "error"}}}', + examples: `{"query": {"match": {"level": "error"}}} +{"query": {"bool": {"must_not": [{"term": {"level": "debug"}}]}}} +{"size": 100, "query": {"term": {"traceId": "$traceId"}}}`, + indexHelperText: 'Required. Sent as the index in POST /{index}/_search.', + usageNote: 'The dashboard time range is added to your query automatically.', + }, +}; + +// `_search` returns only 10 hits by default, which would silently truncate a log panel +// to ten lines. 500 matches the log default used by Grafana's OpenSearch datasource. +export const DEFAULT_SEARCH_SIZE = 500; + +// `_search` rejects `from + size` above the index's `max_result_window`, 10,000 by default. +export const MAX_SEARCH_SIZE = 10000; diff --git a/opensearch/src/queries/opensearch-log-query/OpenSearchLogQuery.tsx b/opensearch/src/queries/opensearch-log-query/OpenSearchLogQuery.tsx index 94963ab9a..f54bd4681 100644 --- a/opensearch/src/queries/opensearch-log-query/OpenSearchLogQuery.tsx +++ b/opensearch/src/queries/opensearch-log-query/OpenSearchLogQuery.tsx @@ -21,7 +21,7 @@ import { OpenSearchLogQueryEditor } from './OpenSearchLogQueryEditor'; export const OpenSearchLogQuery: LogQueryPlugin = { getLogData: getOpenSearchLogData, OptionsEditorComponent: OpenSearchLogQueryEditor, - createInitialOptions: () => ({ query: '' }), + createInitialOptions: () => ({ query: '', queryLanguage: 'ppl' }), dependsOn: (spec) => { const queryVariables = parseVariables(spec.query); const indexVariables = spec.index ? parseVariables(spec.index) : []; diff --git a/opensearch/src/queries/opensearch-log-query/OpenSearchLogQueryEditor.test.tsx b/opensearch/src/queries/opensearch-log-query/OpenSearchLogQueryEditor.test.tsx index 4b924feb3..55c4c86a7 100644 --- a/opensearch/src/queries/opensearch-log-query/OpenSearchLogQueryEditor.test.tsx +++ b/opensearch/src/queries/opensearch-log-query/OpenSearchLogQueryEditor.test.tsx @@ -12,16 +12,19 @@ // limitations under the License. import type * as PluginSystemModule from '@perses-dev/plugin-system'; -import { render, screen, fireEvent } from '@testing-library/react'; +import { render, screen, fireEvent, waitFor } from '@testing-library/react'; import type { Mock } from 'vitest'; -import { PPL_DOCS_URL } from '../constants'; +import { QUERY_LANGUAGES } from '../constants'; import type { OpenSearchLogQuerySpec } from './opensearch-log-query-types'; -import { OpenSearchLogQueryEditor } from './OpenSearchLogQueryEditor'; +import { OpenSearchLogQueryEditor, parseLimitInput } from './OpenSearchLogQueryEditor'; vi.mock('@perses-dev/plugin-system', async (importOriginal) => ({ ...(await importOriginal()), - DatasourceSelect: (): JSX.Element =>
, + // Like the real component, the mock names its input only through `label`. + DatasourceSelect: ({ label }: { label?: string }): JSX.Element => ( + sanitizes an invalid string ("abc", " 5" — leading whitespace + // is not a valid floating-point-number character) to "" before React ever sees a change event, + // so those two inputs cannot be produced by firing a DOM change on the rendered field itself. + // We cover them with a direct unit test of parseLimitInput below instead, and cover the + // DOM-reachable values (empty, zero, negative, decimal) end-to-end through the rendered input. + it.each([ + ['', undefined], + ['0', undefined], + ['-5', undefined], + ['3.7', undefined], + ['20000', 10000], + ])('sets limit to %p -> %p via the rendered field', (input, expected) => { + // Start from a non-empty value so every target value below is a genuine DOM change — + // starting from "" would make the "" case a same-value no-op that never fires onChange. + const { onChange } = setup({ query: 'level:error', queryLanguage: 'lucene', index: 'logs-*', limit: 999 }); + + fireEvent.change(screen.getByLabelText(/Limit/), { target: { value: input } }); + + expect(onChange).toHaveBeenCalledWith(expect.objectContaining({ limit: expected })); + }); + + it.each([ + ['', undefined], + ['0', undefined], + ['-5', undefined], + ['abc', undefined], + ['1.0', 1], + ['3.7', undefined], + [' 5', 5], + ['10000', 10000], + ['10001', 10000], + ])('parseLimitInput(%p) -> %p', (input, expected) => { + expect(parseLimitInput(input)).toBe(expected); + }); +}); diff --git a/opensearch/src/queries/opensearch-log-query/OpenSearchLogQueryEditor.tsx b/opensearch/src/queries/opensearch-log-query/OpenSearchLogQueryEditor.tsx index b048bcc43..c6229f5ea 100644 --- a/opensearch/src/queries/opensearch-log-query/OpenSearchLogQueryEditor.tsx +++ b/opensearch/src/queries/opensearch-log-query/OpenSearchLogQueryEditor.tsx @@ -11,21 +11,62 @@ // See the License for the specific language governing permissions and // limitations under the License. -import { Box, Checkbox, FormControlLabel, InputLabel, Link, Stack, TextField, Typography } from '@mui/material'; +import { + Box, + Checkbox, + FormControlLabel, + InputLabel, + Link, + MenuItem, + Select, + Stack, + TextField, + Typography, +} from '@mui/material'; import { createModEnterHandler } from '@perses-dev/dashboards'; import type { DatasourceSelectProps, OptionsEditorProps } from '@perses-dev/plugin-system'; import { DatasourceSelect, isVariableDatasource, useDatasourceSelectValueToSelector } from '@perses-dev/plugin-system'; import { produce } from 'immer'; import type { ReactElement } from 'react'; +import { useState } from 'react'; import type { OpenSearchDatasourceSelector } from '../../model'; import { isDefaultOpenSearchSelector, OPENSEARCH_DATASOURCE_KIND } from '../../model'; -import { DATASOURCE_KIND, DEFAULT_DATASOURCE, PPL_DOCS_URL, PPL_QUERY_EXAMPLES } from '../constants'; +import { DATASOURCE_KIND, DEFAULT_DATASOURCE, MAX_SEARCH_SIZE, QUERY_LANGUAGES } from '../constants'; import { useQueryState } from '../query-editor-model'; -import type { OpenSearchLogQuerySpec } from './opensearch-log-query-types'; +import type { OpenSearchLogQuerySpec, OpenSearchQueryLanguage } from './opensearch-log-query-types'; +import { OPENSEARCH_QUERY_LANGUAGES, requiresIndex, resolveQueryLanguage } from './opensearch-log-query-types'; type OpenSearchQueryEditorProps = OptionsEditorProps; +// Constant across renders — derived once from the language registry so a fifth +// `_search`-based language only needs a `QUERY_LANGUAGES` entry, not an editor edit. +const REQUIRED_INDEX_MESSAGE = `Required for ${OPENSEARCH_QUERY_LANGUAGES.filter(requiresIndex) + .map((l) => QUERY_LANGUAGES[l].label) + .join(' and ')}`; + +/** + * Parses the Limit field's raw text into a spec-safe value: `undefined` for anything empty, + * non-numeric, or not a positive integer, since the CUE schema constrains `limit` to `int & >0`. + * Values above MAX_SEARCH_SIZE are clamped to it, because `_search` rejects a larger `size`. + * Exported (rather than inlined in the change handler) so it can be unit tested directly — + * a native `` sanitizes invalid strings like `"abc"` or `" 5"` to `""` + * before a change event ever reaches React, so those inputs can't be exercised through the + * rendered field itself. + */ +export function parseLimitInput(raw: string): number | undefined { + // `Number` rather than `parseInt`: parseInt would silently truncate "3.7" to 3, whereas + // a decimal should be rejected like any other non-integer. `Number("")` is 0, so empty is rejected too. + const parsed = Number(raw); + return Number.isInteger(parsed) && parsed > 0 ? Math.min(parsed, MAX_SEARCH_SIZE) : undefined; +} + +const labelSx = { display: 'block', marginBottom: '4px', fontWeight: 500 } as const; +const languageInputProps = { 'aria-label': 'Query language' } as const; +const limitInputProps = { min: 1, max: MAX_SEARCH_SIZE, 'aria-label': 'Limit (optional)' } as const; + +const summarySx = { cursor: 'pointer', fontSize: '12px', color: 'text.secondary', marginBottom: '8px' } as const; + const examplesSx = { fontSize: '11px', color: 'text.secondary', @@ -48,6 +89,45 @@ export function OpenSearchLogQueryEditor(props: OpenSearchQueryEditorProps): Rea const { query, handleQueryChange, handleQueryBlur } = useQueryState(props); + const language = resolveQueryLanguage(value); + const meta = QUERY_LANGUAGES[language]; + const indexMissing = requiresIndex(language) && !value.index; + + // Queries typed in other languages during this editing session, so switching away and back + // restores them. Kept in editor state only: the spec holds a single query. + const [queriesByLanguage, setQueriesByLanguage] = useState>>({}); + + // Captured once on mount so the Options section starts open when it holds a non-default + // setting, and is never collapsed under the user while they edit it. + const [optionsInitiallyOpen] = useState( + () => + value.timestampField !== undefined || + value.messageField !== undefined || + value.disableTimeFilter === true || + value.limit !== undefined, + ); + + const handleLanguageChange = (next: OpenSearchQueryLanguage): void => { + if (next === language) return; + // Save the live local text (useQueryState commits on blur), so an uncommitted edit survives + // a round trip. A language not used yet starts empty, so its placeholder and examples show. + setQueriesByLanguage((saved) => ({ ...saved, [language]: query })); + onChange( + produce(value, (draft) => { + draft.query = queriesByLanguage[next] ?? ''; + draft.queryLanguage = next; + }), + ); + }; + + const handleLimitChange = (e: React.ChangeEvent): void => { + onChange( + produce(value, (draft) => { + draft.limit = parseLimitInput(e.target.value); + }), + ); + }; + const handleDatasourceChange: DatasourceSelectProps['onChange'] = (newDatasourceSelection) => { if (!isVariableDatasource(newDatasourceSelection) && newDatasourceSelection.kind === DATASOURCE_KIND) { onChange( @@ -80,11 +160,11 @@ export function OpenSearchLogQueryEditor(props: OpenSearchQueryEditorProps): Rea ); }; - const handleDisableTimeFilterChange = (e: React.ChangeEvent): void => { + const handleUseTimeRangeChange = (e: React.ChangeEvent): void => { const checked = e.target.checked; onChange( produce(value, (draft) => { - draft.disableTimeFilter = checked ? true : undefined; + draft.disableTimeFilter = checked ? undefined : true; }), ); }; @@ -101,63 +181,51 @@ export function OpenSearchLogQueryEditor(props: OpenSearchQueryEditorProps): Rea return (
- Datasource + + Query language + + +
+ +
+ {/* DatasourceSelect names its input only through its own `label`, so that is the one + label here: a separate heading above it would leave the input unnamed. */}
- Index pattern + Index pattern -
- -
- - Timestamp field (optional) - - -
- -
- - Message field (optional) - -
- - } - label="Disable automatic time filtering" - /> -
- PPL Query + {`${meta.label} Query`} handleQueryChange(e.target.value)} onBlur={handleQueryBlur} onKeyDown={createModEnterHandler(handleQueryExecute)} - placeholder='e.g. source=logs-* | where level="error"' + placeholder={meta.placeholder} inputProps={{ style: { fontFamily: 'monospace' } }} /> Uses OpenSearch{' '} - - PPL + + {meta.docsLabel} - . Requires the PPL plugin enabled on the OpenSearch cluster. + . {meta.usageNote}
- + Query Examples - {PPL_QUERY_EXAMPLES} + {meta.examples} +
+ +
+ + Options + + +
+ Timestamp field (optional) + +
+ +
+ Message field (optional) + +
+ + } + label="Use dashboard time range" + /> + + {/* `requiresIndex` is true for exactly the `_search`-based languages (lucene/dsl), which + is also where a result-count limit applies, so it doubles as the gate for this field. */} + {requiresIndex(language) && ( +
+ Limit (optional) + +
+ )} +
); diff --git a/opensearch/src/queries/opensearch-log-query/build/dsl.test.ts b/opensearch/src/queries/opensearch-log-query/build/dsl.test.ts new file mode 100644 index 000000000..9523c9b33 --- /dev/null +++ b/opensearch/src/queries/opensearch-log-query/build/dsl.test.ts @@ -0,0 +1,115 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import { buildDslSearchBody } from './dsl'; + +const start = new Date('2025-01-01T00:00:00.000Z'); // 1735689600000 +const end = new Date('2025-01-01T01:00:00.000Z'); // 1735693200000 + +const timeFilter = { + range: { '@timestamp': { gte: 1735689600000, lte: 1735693200000, format: 'epoch_millis' } }, +}; + +describe('buildDslSearchBody', () => { + it("ANDs the user's query under bool.must with the time range in bool.filter", () => { + const body = buildDslSearchBody('{"query":{"match":{"level":"error"}}}', start, end); + expect(body).toEqual({ + size: 500, + sort: [{ '@timestamp': { order: 'desc', unmapped_type: 'date' } }], + query: { bool: { must: [{ match: { level: 'error' } }], filter: [timeFilter] } }, + }); + }); + + it('applies only the time filter when the body has no query key', () => { + const body = buildDslSearchBody('{}', start, end); + expect(body.query).toEqual({ bool: { filter: [timeFilter] } }); + }); + + it("lets the user's own sort win", () => { + const body = buildDslSearchBody('{"sort":[{"seq":"asc"}],"query":{"match_all":{}}}', start, end); + expect(body.sort).toEqual([{ seq: 'asc' }]); + }); + + it.each([ + ['a smaller body size under the limit', '{"size":7}', 500, 7], + ['the limit over a larger body size', '{"size":1000}', 10, 10], + ['the body size when no limit is set', '{"size":1000}', undefined, 1000], + ['the limit when the body has no size', '{}', 10, 10], + ['the limit over a non-numeric body size', '{"size":"1000"}', 10, 10], + ['10,000 over a hand-edited limit above it', '{"size":50000}', 20000, 10000], + ['10,000 over a body size above it when no limit is set', '{"size":50000}', undefined, 10000], + ])('caps size: keeps %s', (_, json, limit, expected) => { + expect(buildDslSearchBody(json, start, end, { limit }).size).toBe(expected); + }); + + it('passes unrelated top-level keys through untouched', () => { + const body = buildDslSearchBody( + '{"_source":["message"],"aggs":{"by_level":{"terms":{"field":"level"}}}}', + start, + end, + ); + expect(body['_source']).toEqual(['message']); + expect(body.aggs).toEqual({ by_level: { terms: { field: 'level' } } }); + }); + + it('nests a user bool query inside our must rather than merging it', () => { + const body = buildDslSearchBody('{"query":{"bool":{"must_not":[{"term":{"level":"debug"}}]}}}', start, end); + expect(body.query).toEqual({ + bool: { must: [{ bool: { must_not: [{ term: { level: 'debug' } }] } }], filter: [timeFilter] }, + }); + }); + + it('keeps the query untouched when disableTimeFilter is set', () => { + const body = buildDslSearchBody('{"query":{"match_all":{}}}', start, end, { disableTimeFilter: true }); + expect(body.query).toEqual({ match_all: {} }); + }); + + it('defaults to match_all when there is neither a query nor a time filter', () => { + const body = buildDslSearchBody('{}', start, end, { disableTimeFilter: true }); + expect(body.query).toEqual({ match_all: {} }); + }); + + it('preserves an explicit size of 0 rather than falling back to the default', () => { + expect(buildDslSearchBody('{"size":0,"query":{"match_all":{}}}', start, end).size).toBe(0); + }); + + it('preserves an explicit empty sort array rather than falling back to the default', () => { + expect(buildDslSearchBody('{"sort":[],"query":{"match_all":{}}}', start, end).sort).toEqual([]); + }); + + it.each(['not json', '', ' ', '{"query":}', '{'])('throws a JSON syntax error for %p', (input) => { + expect(() => buildDslSearchBody(input, start, end)).toThrow(/Query DSL is not valid JSON/i); + }); + + it.each(['[1,2]', '"a string"', 'null', '42', 'true'])( + 'throws a shape error for valid JSON that is not an object: %p', + (input) => { + expect(() => buildDslSearchBody(input, start, end)).toThrow(/Query DSL must be a JSON object/i); + }, + ); + + it.each([ + ['false', false], + ['null', false], + ['""', false], + ['0', false], + ['[]', false], + ['false', true], + ['null', true], + ['[]', true], + ])('rejects a present but non-object query %s (disableTimeFilter=%s)', (query, disableTimeFilter) => { + expect(() => buildDslSearchBody(`{"query":${query}}`, start, end, { disableTimeFilter })).toThrow( + /Query DSL "query" must be a JSON object/i, + ); + }); +}); diff --git a/opensearch/src/queries/opensearch-log-query/build/dsl.ts b/opensearch/src/queries/opensearch-log-query/build/dsl.ts new file mode 100644 index 000000000..841f427fa --- /dev/null +++ b/opensearch/src/queries/opensearch-log-query/build/dsl.ts @@ -0,0 +1,86 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import type { OpenSearchSearchRequestBody } from '../../../model/opensearch-client-types'; +import { MAX_SEARCH_SIZE } from '../../constants'; +import type { SearchBodyOptions } from './search'; +import { buildDefaultSort, buildTimeRangeFilter, DEFAULT_TIMESTAMP_FIELD, resolveSize } from './search'; + +function parseBody(userJson: string): Record { + let parsed: unknown; + try { + parsed = JSON.parse(userJson); + } catch (e) { + const detail = e instanceof Error ? e.message : String(e); + throw new Error(`Query DSL is not valid JSON: ${detail}`, { cause: e }); + } + + if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed)) { + throw new Error('Query DSL must be a JSON object shaped like a _search request body, e.g. {"query": {...}}.'); + } + + return parsed as Record; +} + +function resolveDslSize(userSize: unknown, limit?: number): unknown { + if (userSize === undefined) return resolveSize(limit); + // Without a limit, a numeric body size is still kept within what `_search` accepts. + if (limit === undefined) return typeof userSize === 'number' ? Math.min(userSize, MAX_SEARCH_SIZE) : userSize; + // A non-numeric size (e.g. a string) can't be compared, so the limit applies to keep the cap. + const cap = resolveSize(limit); + return typeof userSize === 'number' ? Math.min(userSize, cap) : cap; +} + +/** + * Build a `_search` body from a raw Query DSL document. The user's JSON is treated as a + * complete `_search` request body — the thing you would paste out of Dev Tools — so any + * key we do not manage (`_source`, `aggs`, `highlight`, …) passes straight through. + * + * `limit` caps `size`: with both set the smaller wins, so the body can ask for fewer documents + * but never more than the configured limit. With only one set, that one applies. Either way a + * numeric size never exceeds MAX_SEARCH_SIZE, the most `_search` accepts by default. The user wins + * on `sort`; we only supply a default. Their `query`, if present, is nested under our + * `bool.must` while the time range goes in `bool.filter`. That still means exactly "AND", + * but keeps the user's query in scoring context so `_score` sorts, `track_scores` and + * `min_score` behave as in the raw request. We deliberately do not try to merge into their + * bool clause, because rewriting a user's query semantics is far worse than one extra + * level of nesting. + */ +export function buildDslSearchBody( + userJson: string, + start: Date, + end: Date, + { timestampField = DEFAULT_TIMESTAMP_FIELD, disableTimeFilter = false, limit }: SearchBodyOptions = {}, +): OpenSearchSearchRequestBody { + const parsed = parseBody(userJson); + const userQuery = parsed.query; + // A present but falsey or non-object `query` (false, null, "", 0, []) must not be read as + // "no query" — that would silently widen the request to every document in the range. + if ('query' in parsed && (userQuery === null || typeof userQuery !== 'object' || Array.isArray(userQuery))) { + throw new Error('Query DSL "query" must be a JSON object, e.g. {"query": {"match": {...}}}.'); + } + + const body: OpenSearchSearchRequestBody = { ...parsed }; + body.size = resolveDslSize(parsed.size, limit); + body.sort = parsed.sort ?? buildDefaultSort(timestampField); + + if (disableTimeFilter) { + body.query = userQuery ?? { match_all: {} }; + return body; + } + + const timeFilter = buildTimeRangeFilter(timestampField, start, end); + body.query = userQuery ? { bool: { must: [userQuery], filter: [timeFilter] } } : { bool: { filter: [timeFilter] } }; + + return body; +} diff --git a/opensearch/src/queries/opensearch-log-query/build/identifier.ts b/opensearch/src/queries/opensearch-log-query/build/identifier.ts new file mode 100644 index 000000000..492f66f8d --- /dev/null +++ b/opensearch/src/queries/opensearch-log-query/build/identifier.ts @@ -0,0 +1,23 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +/** + * Escape a user-supplied identifier (such as the timestamp field name) for safe + * interpolation inside a backtick-quoted PPL or SQL identifier. A backtick in the value + * would otherwise close the quoting early and let the rest of the string be parsed as + * query text, producing an invalid — or potentially injected — query. Both languages + * escape a literal backtick by doubling it. + */ +export function escapeIdentifier(name: string): string { + return name.replace(/`/g, '``'); +} diff --git a/opensearch/src/queries/opensearch-log-query/build/index.ts b/opensearch/src/queries/opensearch-log-query/build/index.ts new file mode 100644 index 000000000..4d8502ae1 --- /dev/null +++ b/opensearch/src/queries/opensearch-log-query/build/index.ts @@ -0,0 +1,18 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +export * from './ppl'; +export * from './sql'; +export * from './search'; +export * from './lucene'; +export * from './dsl'; diff --git a/opensearch/src/queries/opensearch-log-query/build/lucene.test.ts b/opensearch/src/queries/opensearch-log-query/build/lucene.test.ts new file mode 100644 index 000000000..44e56a266 --- /dev/null +++ b/opensearch/src/queries/opensearch-log-query/build/lucene.test.ts @@ -0,0 +1,75 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import { buildLuceneSearchBody } from './lucene'; + +const start = new Date('2025-01-01T00:00:00.000Z'); // 1735689600000 +const end = new Date('2025-01-01T01:00:00.000Z'); // 1735693200000 + +const timeFilter = { + range: { '@timestamp': { gte: 1735689600000, lte: 1735693200000, format: 'epoch_millis' } }, +}; + +describe('buildLuceneSearchBody', () => { + it('wraps the user query in a query_string clause beside the time range', () => { + expect(buildLuceneSearchBody('level:error AND service:api', start, end)).toEqual({ + size: 500, + sort: [{ '@timestamp': { order: 'desc', unmapped_type: 'date' } }], + query: { + bool: { + filter: [timeFilter, { query_string: { query: 'level:error AND service:api', analyze_wildcard: true } }], + }, + }, + }); + }); + + it('drops the query_string clause for a blank query, leaving the time window', () => { + const body = buildLuceneSearchBody(' ', start, end); + expect(body.query).toEqual({ bool: { filter: [timeFilter] } }); + }); + + it('honours limit as the search size', () => { + expect(buildLuceneSearchBody('*', start, end, { limit: 25 }).size).toBe(25); + }); + + it('caps a limit above what _search accepts at 10,000', () => { + expect(buildLuceneSearchBody('*', start, end, { limit: 20000 }).size).toBe(10000); + }); + + // The CUE schema constrains `limit` to `int & >0`, so `limit: 0` cannot reach here from a + // validated dashboard. This test pins the `??` operator choice in resolveSize, not `size: 0` + // as a supported use case — do not "fix" it by switching resolveSize to `||`. + it('preserves an explicit limit of 0 rather than falling back to the default size', () => { + expect(buildLuceneSearchBody('*', start, end, { limit: 0 }).size).toBe(0); + }); + + it('sorts and filters on the configured timestamp field', () => { + const body = buildLuceneSearchBody('*', start, end, { timestampField: 'event.time' }); + expect(body.sort).toEqual([{ 'event.time': { order: 'desc', unmapped_type: 'date' } }]); + expect(body.query).toEqual({ + bool: { + filter: [ + { range: { 'event.time': { gte: 1735689600000, lte: 1735693200000, format: 'epoch_millis' } } }, + { query_string: { query: '*', analyze_wildcard: true } }, + ], + }, + }); + }); + + it('omits the range clause when disableTimeFilter is set', () => { + const body = buildLuceneSearchBody('level:error', start, end, { disableTimeFilter: true }); + expect(body.query).toEqual({ + bool: { filter: [{ query_string: { query: 'level:error', analyze_wildcard: true } }] }, + }); + }); +}); diff --git a/opensearch/src/queries/opensearch-log-query/build/lucene.ts b/opensearch/src/queries/opensearch-log-query/build/lucene.ts new file mode 100644 index 000000000..2e9911a1d --- /dev/null +++ b/opensearch/src/queries/opensearch-log-query/build/lucene.ts @@ -0,0 +1,44 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import type { OpenSearchSearchRequestBody } from '../../../model/opensearch-client-types'; +import type { SearchBodyOptions } from './search'; +import { buildDefaultSort, buildTimeRangeFilter, DEFAULT_TIMESTAMP_FIELD, resolveSize } from './search'; + +/** + * Build a `_search` body from a Lucene (query string) query: the user's string becomes a + * `query_string` clause ANDed with the panel time range inside `bool.filter`. A blank + * query is legitimate — it means "everything in this time window". + */ +export function buildLuceneSearchBody( + userQuery: string, + start: Date, + end: Date, + { timestampField = DEFAULT_TIMESTAMP_FIELD, disableTimeFilter = false, limit }: SearchBodyOptions = {}, +): OpenSearchSearchRequestBody { + const trimmed = userQuery.trim(); + + const filter: Array> = []; + if (!disableTimeFilter) { + filter.push(buildTimeRangeFilter(timestampField, start, end)); + } + if (trimmed.length > 0) { + filter.push({ query_string: { query: trimmed, analyze_wildcard: true } }); + } + + return { + size: resolveSize(limit), + sort: buildDefaultSort(timestampField), + query: { bool: { filter } }, + }; +} diff --git a/opensearch/src/queries/opensearch-log-query/build/ppl.ts b/opensearch/src/queries/opensearch-log-query/build/ppl.ts new file mode 100644 index 000000000..76059e1fe --- /dev/null +++ b/opensearch/src/queries/opensearch-log-query/build/ppl.ts @@ -0,0 +1,82 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import { escapeIdentifier } from './identifier'; +import { DEFAULT_TIMESTAMP_FIELD } from './search'; + +/** + * Bound the query to the panel time range using a PPL `where` clause on the + * configured timestamp field (`@timestamp` by default). + * The bound is injected immediately after the source clause so it runs before any + * pipe that drops the timestamp column from the schema (stats, fields, top, etc.). + * If the user already filters on the timestamp field themselves, PPL ANDs the two clauses. + */ +export interface BoundedPPLOptions { + index?: string; + timestampField?: string; + disableTimeFilter?: boolean; +} + +/** + * Sanitize a user-supplied index name/pattern before embedding it in a `source=` + * clause. Unlike the timestamp field, `source=` is not backtick-quoted and must + * keep wildcards (`*`) and multi-index commas intact, so we can't just quote it. + * Instead we strip the characters that could terminate the clause and inject an + * extra PPL stage (backtick, pipe, quotes) or that are simply invalid in an index + * name (whitespace) — e.g. `logs-* | delete` would otherwise smuggle in a pipe. + */ +function sanitizeIndex(index: string): string { + return index.replace(/[`|'"\s]/g, ''); +} + +/** Join PPL stages with ` | `, dropping any empty segments so we never emit a dangling pipe. */ +function joinPPLStages(stages: string[]): string { + return stages.filter((stage) => stage.trim().length > 0).join(' | '); +} + +export function buildBoundedPPL( + userQuery: string, + start: Date, + end: Date, + { index, timestampField = DEFAULT_TIMESTAMP_FIELD, disableTimeFilter = false }: BoundedPPLOptions = {}, +): string { + let trimmed = userQuery.trim(); + + if (index && !/^(?:search\s+)?source\s*=/i.test(trimmed)) { + trimmed = `source=${sanitizeIndex(index)} | ${trimmed}`; + } + + // Skip the auto-injected time-range clause when the caller manages their own time + // bounds (disableTimeFilter). + if (disableTimeFilter) { + return trimmed; + } + + const startIso = start.toISOString(); + const endIso = end.toISOString(); + // timestampField is user-editable, so escape it before embedding it in the + // backtick-quoted identifier (the ISO bounds are machine-generated and safe). + const tsField = escapeIdentifier(timestampField); + const bound = `where \`${tsField}\` >= '${startIso}' and \`${tsField}\` <= '${endIso}'`; + + const firstPipe = trimmed.indexOf('|'); + if (firstPipe === -1) { + return joinPPLStages([trimmed, bound]); + } + + const sourceClause = trimmed.slice(0, firstPipe).trimEnd(); + const rest = trimmed.slice(firstPipe + 1).trimStart(); + // `rest` can be empty (e.g. a trailing pipe or an empty query), so join through a + // filter to avoid producing an invalid `... | ` dangling stage. + return joinPPLStages([sourceClause, bound, rest]); +} diff --git a/opensearch/src/queries/opensearch-log-query/build/search.ts b/opensearch/src/queries/opensearch-log-query/build/search.ts new file mode 100644 index 000000000..e56a94074 --- /dev/null +++ b/opensearch/src/queries/opensearch-log-query/build/search.ts @@ -0,0 +1,53 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import { DEFAULT_SEARCH_SIZE, DEFAULT_TIMESTAMP_FIELDS, MAX_SEARCH_SIZE } from '../../constants'; + +export interface SearchBodyOptions { + timestampField?: string; + disableTimeFilter?: boolean; + /** `_search` size. Defaults to DEFAULT_SEARCH_SIZE. */ + limit?: number; +} + +// Single source of truth for the default timestamp field, shared with build/ppl.ts. +// Non-null assertion: DEFAULT_TIMESTAMP_FIELDS is a non-empty literal array, so index 0 +// always exists — `noUncheckedIndexedAccess` just can't see that from the `string[]` type. +export const DEFAULT_TIMESTAMP_FIELD = DEFAULT_TIMESTAMP_FIELDS[0]!; + +/** A `range` clause bounding the timestamp field to the panel window, in epoch milliseconds. */ +export function buildTimeRangeFilter(timestampField: string, start: Date, end: Date): Record { + return { + range: { + [timestampField]: { + gte: start.getTime(), + lte: end.getTime(), + format: 'epoch_millis', + }, + }, + }; +} + +/** + * Newest-first, which is what a log panel wants — log panels do not sort entries + * themselves. `unmapped_type` keeps a wildcard index pattern working when one of the + * matched indices has no such field, which would otherwise fail the shard. + */ +export function buildDefaultSort(timestampField: string): Array> { + return [{ [timestampField]: { order: 'desc', unmapped_type: 'date' } }]; +} + +/** The configured limit, or the default, never above what `_search` accepts. */ +export function resolveSize(limit?: number): number { + return Math.min(limit ?? DEFAULT_SEARCH_SIZE, MAX_SEARCH_SIZE); +} diff --git a/opensearch/src/queries/opensearch-log-query/build/sql.test.ts b/opensearch/src/queries/opensearch-log-query/build/sql.test.ts new file mode 100644 index 000000000..8039654ed --- /dev/null +++ b/opensearch/src/queries/opensearch-log-query/build/sql.test.ts @@ -0,0 +1,144 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import { buildSqlRequest, UNSUPPORTED_SQL_TIME_FILTER_MESSAGE } from './sql'; + +const start = new Date('2025-01-01T00:00:00.000Z'); +const end = new Date('2025-01-01T01:00:00.000Z'); + +const bound = "(`@timestamp` >= '2025-01-01T00:00:00.000Z' AND `@timestamp` <= '2025-01-01T01:00:00.000Z')"; + +const build = (sql: string): string => buildSqlRequest(sql, start, end).query; + +describe('buildSqlRequest', () => { + it('never sends the legacy-engine filter field', () => { + expect(buildSqlRequest('SELECT * FROM logs', start, end)).toEqual({ query: `SELECT * FROM logs WHERE ${bound}` }); + }); + + it.each([ + ['appends a WHERE at the end', 'SELECT * FROM logs', `SELECT * FROM logs WHERE ${bound}`], + [ + 'inserts a WHERE before LIMIT', + 'SELECT * FROM app-logs LIMIT 20', + `SELECT * FROM app-logs WHERE ${bound} LIMIT 20`, + ], + [ + 'inserts a WHERE before GROUP BY', + 'SELECT level, count(*) FROM logs GROUP BY level ORDER BY level', + `SELECT level, count(*) FROM logs WHERE ${bound} GROUP BY level ORDER BY level`, + ], + [ + 'inserts a WHERE before ORDER BY', + 'select * from logs order by `@timestamp` desc', + `select * from logs WHERE ${bound} order by \`@timestamp\` desc`, + ], + [ + 'ANDs in front of an existing WHERE and parenthesises it', + "SELECT * FROM logs WHERE level = 'ERROR' OR level = 'WARN' LIMIT 5", + `SELECT * FROM logs WHERE ${bound} AND (level = 'ERROR' OR level = 'WARN') LIMIT 5`, + ], + [ + 'keeps new-engine functions in the WHERE intact', + "SELECT * FROM app-logs WHERE match(message, 'timeout') LIMIT 20", + `SELECT * FROM app-logs WHERE ${bound} AND (match(message, 'timeout')) LIMIT 20`, + ], + [ + 'ignores keywords inside string literals', + "SELECT * FROM logs WHERE message = 'limit where order by' LIMIT 3", + `SELECT * FROM logs WHERE ${bound} AND (message = 'limit where order by') LIMIT 3`, + ], + [ + 'ignores keywords inside escaped quotes', + "SELECT * FROM logs WHERE a = 'it''s limit' AND b = 'x\\' limit'", + `SELECT * FROM logs WHERE ${bound} AND (a = 'it''s limit' AND b = 'x\\' limit')`, + ], + [ + 'ignores keywords inside backtick identifiers', + 'SELECT `limit` FROM logs WHERE `order` > 1', + `SELECT \`limit\` FROM logs WHERE ${bound} AND (\`order\` > 1)`, + ], + [ + 'ignores keywords inside parentheses', + 'SELECT EXTRACT(YEAR FROM ts) FROM logs WHERE x IN (SELECT y FROM other LIMIT 1)', + `SELECT EXTRACT(YEAR FROM ts) FROM logs WHERE ${bound} AND (x IN (SELECT y FROM other LIMIT 1))`, + ], + [ + 'places the WHERE before a trailing line comment', + 'SELECT * FROM logs -- recent errors', + `SELECT * FROM logs WHERE ${bound} -- recent errors`, + ], + [ + 'closes the condition before a comment ahead of the next clause', + 'SELECT * FROM logs WHERE a = 1 -- note\nLIMIT 5', + `SELECT * FROM logs WHERE ${bound} AND (a = 1) -- note\nLIMIT 5`, + ], + [ + 'places the WHERE before a trailing # comment', + 'SELECT * FROM logs # recent errors', + `SELECT * FROM logs WHERE ${bound} # recent errors`, + ], + [ + 'ignores keywords inside a # comment', + "SELECT * FROM logs # where x limit 1\nWHERE level = 'ERROR'", + `SELECT * FROM logs # where x limit 1\nWHERE ${bound} AND (level = 'ERROR')`, + ], + [ + 'reads a word after a dot as part of a field name, not a keyword', + "SELECT * FROM logs WHERE user.group = 'admin' AND rate.limit > 1", + `SELECT * FROM logs WHERE ${bound} AND (user.group = 'admin' AND rate.limit > 1)`, + ], + [ + 'accepts a dotted field named like FROM in the select list', + 'SELECT source.from FROM logs', + `SELECT source.from FROM logs WHERE ${bound}`, + ], + [ + 'treats a bare -- as two minus signs, not a comment', + 'SELECT * FROM logs WHERE x = 5--1 LIMIT 3', + `SELECT * FROM logs WHERE ${bound} AND (x = 5--1) LIMIT 3`, + ], + ['treats -- at the end of the query as a comment', 'SELECT * FROM logs --', `SELECT * FROM logs WHERE ${bound} --`], + [ + 'ignores keywords inside block comments', + 'SELECT * /* from limit */ FROM logs', + `SELECT * /* from limit */ FROM logs WHERE ${bound}`, + ], + ['drops a trailing semicolon', 'SELECT * FROM logs LIMIT 5;', `SELECT * FROM logs WHERE ${bound} LIMIT 5`], + ])('%s', (_, input, expected) => { + expect(build(input)).toBe(expected); + }); + + it('uses the configured timestamp field and escapes backticks in it', () => { + expect(buildSqlRequest('SELECT * FROM logs', start, end, { timestampField: 'event.time' }).query).toContain( + "`event.time` >= '2025-01-01T00:00:00.000Z'", + ); + expect(buildSqlRequest('SELECT * FROM logs', start, end, { timestampField: 'a`b' }).query).toContain('`a``b` >='); + }); + + it.each([ + ['a UNION', 'SELECT a FROM x UNION SELECT a FROM y'], + ['a JOIN', 'SELECT * FROM x JOIN y ON x.id = y.id'], + ['a query without FROM', 'SELECT 1'], + ['a non-SELECT statement', 'SHOW TABLES LIKE logs%'], + ['a DESCRIBE statement', 'DESCRIBE TABLES LIKE logs'], + ])('rejects %s rather than guessing where the bound goes', (_, input) => { + expect(() => build(input)).toThrow(UNSUPPORTED_SQL_TIME_FILTER_MESSAGE); + }); + + it('sends the query unchanged apart from trimming when disableTimeFilter is set', () => { + const request = buildSqlRequest(' SELECT a FROM x UNION SELECT a FROM y; ', start, end, { + disableTimeFilter: true, + }); + expect(request).toEqual({ query: 'SELECT a FROM x UNION SELECT a FROM y' }); + }); +}); diff --git a/opensearch/src/queries/opensearch-log-query/build/sql.ts b/opensearch/src/queries/opensearch-log-query/build/sql.ts new file mode 100644 index 000000000..20e9fe7a8 --- /dev/null +++ b/opensearch/src/queries/opensearch-log-query/build/sql.ts @@ -0,0 +1,173 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import type { OpenSearchSQLParams } from '../../../model/opensearch-client'; +import { escapeIdentifier } from './identifier'; +import { DEFAULT_TIMESTAMP_FIELD } from './search'; + +export interface SqlRequestOptions { + timestampField?: string; + disableTimeFilter?: boolean; +} + +export const UNSUPPORTED_SQL_TIME_FILTER_MESSAGE = + 'Automatic time filtering supports a single SELECT … FROM query without UNION or JOIN. ' + + 'Untick "Use dashboard time range" and filter on the timestamp field in your own WHERE clause instead, ' + + 'e.g. WHERE `@timestamp` >= FROM_UNIXTIME($__from / 1000.0) AND `@timestamp` <= FROM_UNIXTIME($__to / 1000.0).'; + +/** Keywords that end a WHERE clause (or mark where one must be inserted) in the outer query. */ +const CLAUSES_AFTER_WHERE = new Set(['GROUP', 'HAVING', 'ORDER', 'LIMIT']); +/** Top-level keywords the splice can't reason about, so the query is rejected instead of guessed at. */ +const UNSUPPORTED_KEYWORDS = new Set(['UNION', 'INTERSECT', 'EXCEPT', 'MINUS', 'JOIN']); + +interface SqlKeyword { + word: string; + start: number; + end: number; + /** End of the last code character before this keyword, so a splice never lands inside a comment. */ + codeEndBefore: number; +} + +interface ScannedSql { + keywords: SqlKeyword[]; + /** End of the last code character (not whitespace or comment) in the whole query. */ + codeEnd: number; +} + +/** + * Collect the words outside parentheses, quotes and comments, which is enough to find the + * outer query's clauses. Quoting follows the OpenSearch SQL lexer: '…', "…" and `…` may + * escape a character with a backslash or by doubling the quote. `#`, `/* *\/`, and `--` followed + * by a space or the end of a line are comments; a bare `--` is two minus signs (`5--1`). A word + * right after a `.` is part of a dotted field name (`user.group`), never a keyword. + */ +function scanTopLevelWords(sql: string): ScannedSql { + const keywords: SqlKeyword[] = []; + let depth = 0; + let codeEnd = 0; + let i = 0; + + while (i < sql.length) { + const ch = sql[i]!; + + const afterDashes = sql[i + 2]; + const isDashComment = + ch === '-' && + sql[i + 1] === '-' && + (afterDashes === undefined || afterDashes === ' ' || afterDashes === '\n' || afterDashes === '\r'); + if (isDashComment || ch === '#') { + const newline = sql.indexOf('\n', i); + i = newline === -1 ? sql.length : newline; + continue; + } + if (ch === '/' && sql[i + 1] === '*') { + const close = sql.indexOf('*/', i + 2); + i = close === -1 ? sql.length : close + 2; + continue; + } + if (/\s/.test(ch)) { + i++; + continue; + } + + if (ch === "'" || ch === '"' || ch === '`') { + i++; + while (i < sql.length) { + if (sql[i] === '\\') { + i += 2; + } else if (sql[i] === ch && sql[i + 1] === ch) { + i += 2; + } else if (sql[i] === ch) { + i++; + break; + } else { + i++; + } + } + codeEnd = i; + continue; + } + + if (/[A-Za-z_]/.test(ch)) { + const start = i; + while (i < sql.length && /\w/.test(sql[i]!)) i++; + if (depth === 0 && sql[start - 1] !== '.') { + keywords.push({ word: sql.slice(start, i).toUpperCase(), start, end: i, codeEndBefore: codeEnd }); + } + codeEnd = i; + continue; + } + + if (ch === '(') depth++; + if (ch === ')') depth = Math.max(0, depth - 1); + i++; + codeEnd = i; + } + + return { keywords, codeEnd }; +} + +/** + * Add `bound` to the outer query's WHERE clause: ANDed in front of an existing condition + * (wrapped in parentheses so its own ORs stay inside), or as a new WHERE placed before + * GROUP BY / HAVING / ORDER BY / LIMIT, or at the end of the query. + */ +function spliceWhere(sql: string, bound: string): string { + const { keywords, codeEnd } = scanTopLevelWords(sql); + + const froms = keywords.filter((k) => k.word === 'FROM'); + if (keywords[0]?.word !== 'SELECT' || froms.length !== 1 || keywords.some((k) => UNSUPPORTED_KEYWORDS.has(k.word))) { + throw new Error(UNSUPPORTED_SQL_TIME_FILTER_MESSAGE); + } + + const afterFrom = keywords.filter((k) => k.start > froms[0]!.start); + const where = afterFrom.find((k) => k.word === 'WHERE'); + const next = afterFrom.find((k) => CLAUSES_AFTER_WHERE.has(k.word) && (!where || k.start > where.start)); + const clauseEnd = next ? next.codeEndBefore : codeEnd; + + if (where) { + const condition = sql.slice(where.end, clauseEnd).trim(); + return `${sql.slice(0, where.start)}WHERE ${bound} AND (${condition})${sql.slice(clauseEnd)}`; + } + return `${sql.slice(0, clauseEnd)} WHERE ${bound}${sql.slice(clauseEnd)}`; +} + +/** + * Bound a SQL query to the panel time range by adding a condition on the timestamp field to + * its WHERE clause, as PPL gets a `where` stage. The `_sql` API's `filter` request field is + * deliberately not used: OpenSearch only supports it on the legacy SQL engine, which then + * misreads sub-millisecond timestamps, rejects `date_nanos` fields and fails on new-engine + * syntax such as `match(field, 'text') LIMIT n`. + */ +export function buildSqlRequest( + userQuery: string, + start: Date, + end: Date, + { timestampField = DEFAULT_TIMESTAMP_FIELD, disableTimeFilter = false }: SqlRequestOptions = {}, +): OpenSearchSQLParams { + // A trailing `;` would end the statement before an appended WHERE. + const query = userQuery.trim().replace(/;+$/, '').trimEnd(); + + if (disableTimeFilter) { + return { query }; + } + + // timestampField is user-editable, so escape it before embedding it in the backtick-quoted + // identifier (the bounds are machine-generated and safe). + const tsField = `\`${escapeIdentifier(timestampField)}\``; + // ISO 8601 bounds: the new SQL engine compares them as timestamps, and when a query falls + // back to the legacy engine they are pushed down to a `range` query, whose default date + // format accepts ISO but not `yyyy-MM-dd HH:mm:ss`. + const bound = `(${tsField} >= '${start.toISOString()}' AND ${tsField} <= '${end.toISOString()}')`; + return { query: spliceWhere(query, bound) }; +} diff --git a/opensearch/src/queries/opensearch-log-query/convert/datarows-to-logs.ts b/opensearch/src/queries/opensearch-log-query/convert/datarows-to-logs.ts new file mode 100644 index 000000000..e3c29bda3 --- /dev/null +++ b/opensearch/src/queries/opensearch-log-query/convert/datarows-to-logs.ts @@ -0,0 +1,78 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import type { LogEntry, LogData } from '@perses-dev/spec'; + +import type { OpenSearchDatarowsResponse } from '../../../model/opensearch-client-types'; +import type { ConvertOptions } from './fields'; +import { fieldCandidates, flattenSource } from './fields'; +import { parseTimestamp } from './parse-timestamp'; + +function pickIndex(cols: Array<{ name: string }>, candidates: string[]): number { + for (const candidate of candidates) { + const idx = cols.findIndex((c) => c.name === candidate); + if (idx !== -1) return idx; + } + return -1; +} + +export function convertDatarowsToLogs(response: OpenSearchDatarowsResponse, options: ConvertOptions = {}): LogData { + const { schema = [], datarows = [] } = response; + + const { timestamp: tsCandidates, message: msgCandidates } = fieldCandidates(options); + + const tsIdx = pickIndex(schema, tsCandidates); + const msgIdx = pickIndex(schema, msgCandidates); + + const entries: LogEntry[] = datarows.map((row) => { + const rawTs = tsIdx !== -1 ? row[tsIdx] : null; + const rawMsg = msgIdx !== -1 ? row[msgIdx] : null; + + const timestamp = parseTimestamp(rawTs); + let line: string; + if (rawMsg === null || rawMsg === undefined) { + line = JSON.stringify(rowToObject(schema, row)); + } else { + line = typeof rawMsg === 'object' ? JSON.stringify(rawMsg) : String(rawMsg); + } + + // Object and struct columns (e.g. OTel's `service`) come back as nested objects, so flatten + // them to `service.name`-style labels, as the `_search` converter does for `_source`. + const labels: Record = {}; + schema.forEach((col, i) => { + if (i === tsIdx || i === msgIdx) return; + Object.assign(labels, flattenSource({ [col.name]: row[i] })); + }); + + return { timestamp, line, labels }; + }); + + // A PPL response carries no separate "total hits" field, so totalCount reflects + // the number of rows actually returned by the query (bounded by any `head`/limit + // the user added), not a grand total of matching documents. + return { entries, totalCount: entries.length }; +} + +function rowToObject( + schema: OpenSearchDatarowsResponse['schema'], + row: OpenSearchDatarowsResponse['datarows'][number], +): Record { + const out: Record = {}; + schema.forEach((col, i) => { + out[col.name] = row[i]; + }); + return out; +} + +/** @deprecated Use `convertDatarowsToLogs`. */ +export const convertPPLToLogs = convertDatarowsToLogs; diff --git a/opensearch/src/queries/opensearch-log-query/convert/fields.ts b/opensearch/src/queries/opensearch-log-query/convert/fields.ts new file mode 100644 index 000000000..0bc2e79dd --- /dev/null +++ b/opensearch/src/queries/opensearch-log-query/convert/fields.ts @@ -0,0 +1,64 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import { DEFAULT_MESSAGE_FIELDS, DEFAULT_TIMESTAMP_FIELDS } from '../../constants'; + +export interface ConvertOptions { + timestampField?: string; + messageField?: string; +} + +/** + * The field names to try, in order, for the timestamp and the log line: the configured + * override first, then the defaults as fallbacks. Shared by both converters so PPL/SQL and + * Lucene/DSL pick the same fields for the same index. + */ +export function fieldCandidates(options: ConvertOptions): { timestamp: string[]; message: string[] } { + return { + timestamp: options.timestampField + ? [options.timestampField, ...DEFAULT_TIMESTAMP_FIELDS] + : DEFAULT_TIMESTAMP_FIELDS, + message: options.messageField ? [options.messageField, ...DEFAULT_MESSAGE_FIELDS] : DEFAULT_MESSAGE_FIELDS, + }; +} + +/** + * Flatten a document to dot-separated keys, because LogEntry labels are a flat + * `Record`. Arrays are JSON-stringified rather than indexed, since + * `tags.0`/`tags.1` labels are noise; null/undefined values are dropped so they do not + * render as the strings "null"/"undefined". + */ +export function flattenSource(source: Record): Record { + const out: Record = {}; + + const walk = (value: unknown, prefix: string): void => { + if (value === null || value === undefined) return; + if (Array.isArray(value)) { + out[prefix] = JSON.stringify(value); + return; + } + if (typeof value === 'object') { + for (const [key, child] of Object.entries(value as Record)) { + walk(child, prefix ? `${prefix}.${key}` : key); + } + return; + } + out[prefix] = String(value); + }; + + for (const [key, value] of Object.entries(source)) { + walk(value, key); + } + + return out; +} diff --git a/opensearch/src/queries/opensearch-log-query/convert/hits-to-logs.test.ts b/opensearch/src/queries/opensearch-log-query/convert/hits-to-logs.test.ts new file mode 100644 index 000000000..4c73ff2f8 --- /dev/null +++ b/opensearch/src/queries/opensearch-log-query/convert/hits-to-logs.test.ts @@ -0,0 +1,138 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import { flattenSource } from './fields'; +import { convertHitsToLogs } from './hits-to-logs'; + +describe('flattenSource', () => { + it('flattens nested objects to dot paths and stringifies scalars', () => { + expect(flattenSource({ a: { b: 1, c: 'x' }, d: true })).toEqual({ 'a.b': '1', 'a.c': 'x', d: 'true' }); + }); + + it('JSON-stringifies arrays rather than exploding them', () => { + expect(flattenSource({ tags: ['a', 'b'] })).toEqual({ tags: '["a","b"]' }); + }); + + it('drops null and undefined values', () => { + expect(flattenSource({ a: null, b: undefined, c: 0 })).toEqual({ c: '0' }); + }); +}); + +describe('convertHitsToLogs', () => { + const response = { + hits: { + total: { value: 2, relation: 'eq' as const }, + hits: [ + { + _index: 'logs-2025.01.01', + _id: 'abc', + _source: { + '@timestamp': '2025-01-01T00:00:00.000Z', + message: 'Error processing request', + level: 'error', + service: { name: 'api' }, + }, + }, + { + _index: 'logs-2025.01.01', + _id: 'def', + _source: { '@timestamp': '2025-01-01T00:00:01.000Z', message: 'Retrying', level: 'warn' }, + }, + ], + }, + }; + + it('maps _source to timestamp, line, and labels', () => { + const { entries } = convertHitsToLogs(response); + expect(entries).toHaveLength(2); + expect(entries[0]).toEqual({ + timestamp: 1735689600, + line: 'Error processing request', + labels: { level: 'error', 'service.name': 'api', _id: 'abc', _index: 'logs-2025.01.01' }, + }); + }); + + it('excludes the chosen timestamp and message fields from labels', () => { + const { entries } = convertHitsToLogs(response); + expect(entries[0]?.labels['@timestamp']).toBeUndefined(); + expect(entries[0]?.labels.message).toBeUndefined(); + }); + + it('reports totalCount from hits.total and no hasMore when the count is exact', () => { + const result = convertHitsToLogs(response); + expect(result.totalCount).toBe(2); + expect(result.hasMore).toBeUndefined(); + }); + + it('sets hasMore when hits.total.relation is gte', () => { + const result = convertHitsToLogs({ + hits: { total: { value: 10000, relation: 'gte' }, hits: [] }, + }); + expect(result.totalCount).toBe(10000); + expect(result.hasMore).toBe(true); + }); + + it('sets hasMore when an exact total exceeds the hits returned', () => { + const result = convertHitsToLogs({ + hits: { total: { value: 5000, relation: 'eq' }, hits: [{ _source: { message: 'a' } }] }, + }); + expect(result.totalCount).toBe(5000); + expect(result.hasMore).toBe(true); + }); + + it('parses a fractional epoch-seconds timestamp', () => { + const { entries } = convertHitsToLogs({ + hits: { hits: [{ _source: { '@timestamp': 1735689600.5, message: 'a' } }] }, + }); + expect(entries[0]?.timestamp).toBe(1735689600.5); + }); + + it('falls back to the entry count when hits.total is absent', () => { + const result = convertHitsToLogs({ hits: { hits: [{ _source: { message: 'a' } }] } }); + expect(result.totalCount).toBe(1); + }); + + it('honours explicit timestampField and messageField overrides', () => { + const { entries } = convertHitsToLogs( + { hits: { hits: [{ _source: { ts: 1735689600000, body: 'hello', other: 'x' } }] } }, + { timestampField: 'ts', messageField: 'body' }, + ); + expect(entries[0]?.timestamp).toBe(1735689600); + expect(entries[0]?.line).toBe('hello'); + expect(entries[0]?.labels).toEqual({ other: 'x' }); + }); + + it('falls back through the default field names', () => { + const { entries } = convertHitsToLogs({ + hits: { hits: [{ _source: { time: '2025-01-01T00:00:00.000Z', body: 'from body' } }] }, + }); + expect(entries[0]?.timestamp).toBe(1735689600); + expect(entries[0]?.line).toBe('from body'); + }); + + it('stringifies the whole source when no message field matches', () => { + const { entries } = convertHitsToLogs({ hits: { hits: [{ _source: { foo: 'bar' } }] } }); + expect(entries[0]?.line).toBe(JSON.stringify({ foo: 'bar' })); + }); + + it('yields timestamp 0 and an empty-object line for a hit with no _source', () => { + const { entries } = convertHitsToLogs({ hits: { hits: [{ _id: 'x' }] } }); + expect(entries[0]?.timestamp).toBe(0); + expect(entries[0]?.line).toBe('{}'); + expect(entries[0]?.labels).toEqual({ _id: 'x' }); + }); + + it('returns no entries for an empty hits array', () => { + expect(convertHitsToLogs({ hits: { hits: [] } })).toEqual({ entries: [], totalCount: 0 }); + }); +}); diff --git a/opensearch/src/queries/opensearch-log-query/convert/hits-to-logs.ts b/opensearch/src/queries/opensearch-log-query/convert/hits-to-logs.ts new file mode 100644 index 000000000..f9e240143 --- /dev/null +++ b/opensearch/src/queries/opensearch-log-query/convert/hits-to-logs.ts @@ -0,0 +1,72 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import type { LogData, LogEntry } from '@perses-dev/spec'; + +import type { OpenSearchSearchResponse } from '../../../model/opensearch-client-types'; +import type { ConvertOptions } from './fields'; +import { fieldCandidates, flattenSource } from './fields'; +import { parseTimestamp } from './parse-timestamp'; + +/** First candidate present in the flattened document, or undefined if none match. */ +function pickKey(flat: Record, candidates: string[]): string | undefined { + return candidates.find((candidate) => candidate in flat); +} + +/** + * Convert a `_search` response into the same LogData shape the datarows converter produces, so + * Lucene and Query DSL results are indistinguishable from PPL/SQL results downstream. + */ +export function convertHitsToLogs(response: OpenSearchSearchResponse, options: ConvertOptions = {}): LogData { + const hits = response?.hits?.hits ?? []; + + const { timestamp: tsCandidates, message: msgCandidates } = fieldCandidates(options); + + // OpenSearch's metadata fields are underscore-prefixed, so rename them on the way in. + const entries: LogEntry[] = hits.map(({ _id: id, _index: index, _source: rawSource }) => { + const source = rawSource ?? {}; + const flat = flattenSource(source); + + const tsKey = pickKey(flat, tsCandidates); + const msgKey = pickKey(flat, msgCandidates); + + const timestamp = tsKey ? parseTimestamp(flat[tsKey]) : 0; + const line = msgKey ? (flat[msgKey] as string) : JSON.stringify(source); + + const labels: Record = {}; + for (const [key, value] of Object.entries(flat)) { + if (key === tsKey || key === msgKey) continue; + labels[key] = value; + } + // Document identity is genuinely useful: _index disambiguates a wildcard pattern and + // _id lets a user correlate a line back to the source document. + if (id !== undefined) labels['_id'] = id; + if (index !== undefined) labels['_index'] = index; + + return { timestamp, line, labels }; + }); + + const total = response?.hits?.total; + const result: LogData = { + entries, + totalCount: total?.value ?? entries.length, + }; + // `relation: 'gte'` means OpenSearch stopped counting (default cap 10,000), so the true + // total is at least this — surface that rather than paying for track_total_hits. An exact + // total larger than the page means `size` truncated the result. + if (total?.relation === 'gte' || (total?.value ?? 0) > entries.length) { + result.hasMore = true; + } + + return result; +} diff --git a/opensearch/src/queries/opensearch-log-query/convert/index.ts b/opensearch/src/queries/opensearch-log-query/convert/index.ts new file mode 100644 index 000000000..fdc3b3340 --- /dev/null +++ b/opensearch/src/queries/opensearch-log-query/convert/index.ts @@ -0,0 +1,17 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +export * from './datarows-to-logs'; +export * from './fields'; +export * from './parse-timestamp'; +export * from './hits-to-logs'; diff --git a/opensearch/src/queries/opensearch-log-query/convert/parse-timestamp.ts b/opensearch/src/queries/opensearch-log-query/convert/parse-timestamp.ts new file mode 100644 index 000000000..6036e238b --- /dev/null +++ b/opensearch/src/queries/opensearch-log-query/convert/parse-timestamp.ts @@ -0,0 +1,53 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +export function parseTimestamp(v: unknown): number { + if (v === null || v === undefined) return 0; + + // Numeric epochs arrive in seconds, milliseconds, microseconds, or nanoseconds + // (logs frequently carry ns precision). They come as a JS number, or — when the + // value exceeds Number.MAX_SAFE_INTEGER (a ns epoch is ~1e18, well past 2^53) — as + // a numeric string so it survives JSON parsing without silent overflow. `_search` + // hits are flattened to strings, so a fractional epoch (`1700000000.5`) also arrives + // as a numeric string and must not fall through to Date.parse. + // We emit seconds (LogEntry is second-resolution), so we only need the correct + // magnitude, not sub-second precision: a float's rounding at the ns scale is far + // below one second and is discarded anyway, so BigInt isn't required here. + let numeric = NaN; + if (typeof v === 'number') { + numeric = v; + } else if (/^\d+(\.\d+)?$/.test(String(v))) { + numeric = Number(v); + } + if (!Number.isNaN(numeric)) { + // Detect the unit by magnitude — a value past year ~5138 for a given unit must + // really be a finer unit — and normalize to seconds. Without the µs/ns tiers + // those values would parse far in the future. + if (numeric >= 1e17) return numeric / 1e9; // nanoseconds → seconds + if (numeric >= 1e14) return numeric / 1e6; // microseconds → seconds + if (numeric >= 1e11) return numeric / 1e3; // milliseconds → seconds + return numeric; // seconds + } + + // OpenSearch PPL returns timestamp/datetime fields as space-separated UTC strings + // with no timezone designator (e.g. "2020-08-26 13:49:00"). Date.parse treats a + // zoneless date-time as *local* time, which would shift the value by the browser's + // offset, so normalize it to explicit UTC (replace the space with `T`, append `Z`) + // before parsing. Strings that already carry a zone (…Z / ±hh:mm) or are date-only + // are left untouched — the spec parses those as UTC already. + const str = String(v).trim(); + const zonelessDateTime = /^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2}(\.\d+)?$/; + const iso = zonelessDateTime.test(str) ? `${str.replace(' ', 'T')}Z` : str; + const parsed = Date.parse(iso); + return Number.isNaN(parsed) ? 0 : parsed / 1000; +} diff --git a/opensearch/src/queries/opensearch-log-query/get-opensearch-log-data.test.ts b/opensearch/src/queries/opensearch-log-query/get-opensearch-log-data.test.ts index 84e892511..eafde971b 100644 --- a/opensearch/src/queries/opensearch-log-query/get-opensearch-log-data.test.ts +++ b/opensearch/src/queries/opensearch-log-query/get-opensearch-log-data.test.ts @@ -16,8 +16,9 @@ import type { Mock } from 'vitest'; import { OpenSearchDatasource } from '../../datasources/opensearch-datasource'; import type { OpenSearchDatasourceSpec } from '../../datasources/opensearch-datasource/opensearch-datasource-types'; -import type { OpenSearchPPLResponse } from '../../model/opensearch-client-types'; -import { buildBoundedPPL, convertPPLToLogs, parseTimestamp } from './get-opensearch-log-data'; +import type { OpenSearchDatarowsResponse } from '../../model/opensearch-client-types'; +import { buildBoundedPPL } from './build/ppl'; +import { convertDatarowsToLogs, parseTimestamp } from './convert'; import { OpenSearchLogQuery } from './OpenSearchLogQuery'; const datasource: OpenSearchDatasourceSpec = { @@ -27,7 +28,7 @@ const datasource: OpenSearchDatasourceSpec = { const stubClient = OpenSearchDatasource.createClient(datasource, {}); stubClient.ppl = vi.fn(async () => { - const response: OpenSearchPPLResponse = { + const response: OpenSearchDatarowsResponse = { schema: [ { name: '@timestamp', type: 'timestamp' }, { name: 'message', type: 'text' }, @@ -43,6 +44,23 @@ stubClient.ppl = vi.fn(async () => { return response; }); +stubClient.sql = vi.fn(async () => ({ + schema: [ + { name: '@timestamp', type: 'timestamp' }, + { name: 'message', type: 'text' }, + ], + datarows: [['2025-01-01T00:00:00.000Z', 'from sql']], +})); + +stubClient.search = vi.fn(async () => ({ + hits: { + total: { value: 1, relation: 'eq' as const }, + hits: [ + { _index: 'logs-2025', _id: 'h1', _source: { '@timestamp': '2025-01-01T00:00:00.000Z', message: 'from search' } }, + ], + }, +})); + const getDatasourceClient: Mock = vi.fn(() => stubClient); function createStubContext(): LogQueryContext { @@ -68,10 +86,12 @@ function createStubContext(): LogQueryContext { describe('OpenSearchLogQuery', () => { afterEach(() => { (stubClient.ppl as Mock).mockClear(); + (stubClient.sql as Mock).mockClear(); + (stubClient.search as Mock).mockClear(); }); - it('creates initial options with empty query', () => { - expect(OpenSearchLogQuery.createInitialOptions()).toEqual({ query: '' }); + it('creates initial options with an empty PPL query', () => { + expect(OpenSearchLogQuery.createInitialOptions()).toEqual({ query: '', queryLanguage: 'ppl' }); }); it('resolves variable dependencies from query and index', () => { @@ -347,9 +367,30 @@ describe('parseTimestamp', () => { }); }); -describe('convertPPLToLogs', () => { +describe('convertDatarowsToLogs', () => { + it('flattens object columns into dotted labels instead of "[object Object]"', () => { + const logs = convertDatarowsToLogs({ + schema: [ + { name: '@timestamp', type: 'timestamp' }, + { name: 'body', type: 'text' }, + { name: 'service', type: 'object' }, + { name: 'tags', type: 'array' }, + ], + datarows: [['2025-01-01T00:00:00.000Z', 'hello', { name: 'api', ns: { env: 'prod' } }, ['a', 'b']]], + }); + expect(logs.entries[0]?.labels).toEqual({ 'service.name': 'api', 'service.ns.env': 'prod', tags: '["a","b"]' }); + }); + + it('serializes an object message column as JSON', () => { + const logs = convertDatarowsToLogs({ + schema: [{ name: 'message', type: 'object' }], + datarows: [[{ text: 'hi' }]], + }); + expect(logs.entries[0]?.line).toBe('{"text":"hi"}'); + }); + it('maps @timestamp and message fields and collects remaining fields as labels', () => { - const logs = convertPPLToLogs({ + const logs = convertDatarowsToLogs({ schema: [ { name: '@timestamp', type: 'timestamp' }, { name: 'message', type: 'text' }, @@ -363,7 +404,7 @@ describe('convertPPLToLogs', () => { }); it('parses numeric epoch-millis timestamps into seconds', () => { - const logs = convertPPLToLogs({ + const logs = convertDatarowsToLogs({ schema: [ { name: '@timestamp', type: 'long' }, { name: 'message', type: 'text' }, @@ -374,7 +415,7 @@ describe('convertPPLToLogs', () => { }); it('falls back to a JSON dump of the row when no message field is present', () => { - const logs = convertPPLToLogs({ + const logs = convertDatarowsToLogs({ schema: [ { name: '@timestamp', type: 'timestamp' }, { name: 'event', type: 'text' }, @@ -385,12 +426,12 @@ describe('convertPPLToLogs', () => { }); it('handles empty datarows', () => { - const logs = convertPPLToLogs({ schema: [], datarows: [] }); + const logs = convertDatarowsToLogs({ schema: [], datarows: [] }); expect(logs).toEqual({ entries: [], totalCount: 0 }); }); it('honors explicit timestampField and messageField overrides', () => { - const logs = convertPPLToLogs( + const logs = convertDatarowsToLogs( { schema: [ { name: 'time', type: 'timestamp' }, @@ -407,7 +448,7 @@ describe('convertPPLToLogs', () => { }); it('falls back to defaults when the override field is not in the schema', () => { - const logs = convertPPLToLogs( + const logs = convertDatarowsToLogs( { schema: [ { name: '@timestamp', type: 'timestamp' }, @@ -421,7 +462,7 @@ describe('convertPPLToLogs', () => { }); it('puts trace_id columns into labels for trace pivot', () => { - const logs = convertPPLToLogs({ + const logs = convertDatarowsToLogs({ schema: [ { name: '@timestamp', type: 'timestamp' }, { name: 'message', type: 'text' }, @@ -432,3 +473,180 @@ describe('convertPPLToLogs', () => { expect(logs.entries[0]?.labels.traceId).toBe('abc123'); }); }); + +describe('getLogData language dispatch', () => { + const context = createStubContext(); + + afterEach(() => { + (stubClient.ppl as Mock).mockClear(); + (stubClient.sql as Mock).mockClear(); + (stubClient.search as Mock).mockClear(); + }); + + it('defaults to PPL when queryLanguage is absent (backward compatibility)', async () => { + await OpenSearchLogQuery.getLogData({ query: 'source=logs-*' }, context); + expect(stubClient.ppl).toHaveBeenCalledTimes(1); + expect(stubClient.sql).not.toHaveBeenCalled(); + expect(stubClient.search).not.toHaveBeenCalled(); + }); + + it('routes queryLanguage: sql to client.sql with the time range in its WHERE clause', async () => { + const result = await OpenSearchLogQuery.getLogData({ query: 'SELECT * FROM logs', queryLanguage: 'sql' }, context); + expect(stubClient.ppl).not.toHaveBeenCalled(); + const [params] = (stubClient.sql as Mock).mock.calls[0]!; + expect(params).toEqual({ + query: + "SELECT * FROM logs WHERE (`@timestamp` >= '2025-01-01T00:00:00.000Z' AND `@timestamp` <= '2025-01-01T01:00:00.000Z')", + }); + expect(result.logs.entries[0]?.line).toBe('from sql'); + }); + + it('routes queryLanguage: lucene to client.search with the resolved index', async () => { + const result = await OpenSearchLogQuery.getLogData( + { query: 'level:error', queryLanguage: 'lucene', index: 'logs-*' }, + context, + ); + const [params] = (stubClient.search as Mock).mock.calls[0]!; + expect(params.index).toBe('logs-*'); + expect(params.body.query.bool.filter).toHaveLength(2); + expect(result.logs.entries[0]?.line).toBe('from search'); + expect(result.logs.entries[0]?.labels._index).toBe('logs-2025'); + }); + + it('routes queryLanguage: dsl to client.search', async () => { + await OpenSearchLogQuery.getLogData( + { query: '{"query":{"match":{"level":"error"}}}', queryLanguage: 'dsl', index: 'logs-*' }, + context, + ); + const [params] = (stubClient.search as Mock).mock.calls[0]!; + expect(params.body.query.bool.must).toEqual([{ match: { level: 'error' } }]); + }); + + it('passes limit through as the search size', async () => { + await OpenSearchLogQuery.getLogData( + { query: 'level:error', queryLanguage: 'lucene', index: 'logs-*', limit: 42 }, + context, + ); + const [params] = (stubClient.search as Mock).mock.calls[0]!; + expect(params.body.size).toBe(42); + }); + + it('resolves variables in a DSL body', async () => { + await OpenSearchLogQuery.getLogData( + { query: '{"query":{"term":{"traceId":"$traceId"}}}', queryLanguage: 'dsl', index: 'logs-*' }, + { + ...context, + variableState: { + traceId: { value: 'abc123', loading: false }, + } as unknown as LogQueryContext['variableState'], + }, + ); + const [params] = (stubClient.search as Mock).mock.calls[0]!; + expect(params.body.query.bool.must).toEqual([{ term: { traceId: 'abc123' } }]); + }); + + it('escapes variable values so they stay inside a DSL JSON string', async () => { + await OpenSearchLogQuery.getLogData( + { query: '{"query":{"term":{"path":"$path"}}}', queryLanguage: 'dsl', index: 'logs-*' }, + { + ...context, + variableState: { + path: { value: 'C:\\temp\\logs "x"', loading: false }, + } as unknown as LogQueryContext['variableState'], + }, + ); + const [params] = (stubClient.search as Mock).mock.calls[0]!; + expect(params.body.query.bool.must).toEqual([{ term: { path: 'C:\\temp\\logs "x"' } }]); + }); + + it('does not let a DSL variable value add keys to the request body', async () => { + await OpenSearchLogQuery.getLogData( + { query: '{"query":{"term":{"level":"$level"}}}', queryLanguage: 'dsl', index: 'logs-*' }, + { + ...context, + variableState: { + level: { value: 'x"}},"size":10000,"q":{"a":"y', loading: false }, + } as unknown as LogQueryContext['variableState'], + }, + ); + const [params] = (stubClient.search as Mock).mock.calls[0]!; + expect(params.body.size).toBe(500); + expect(params.body.query.bool.must).toEqual([{ term: { level: 'x"}},"size":10000,"q":{"a":"y' } }]); + }); + + it('leaves a DSL variable with the json format as the JSON it emits', async () => { + await OpenSearchLogQuery.getLogData( + { query: '{"query":{"terms":{"level":${levels:json}}}}', queryLanguage: 'dsl', index: 'logs-*' }, + { + ...context, + variableState: { + levels: { value: ['error', 'warn'], loading: false }, + } as unknown as LogQueryContext['variableState'], + }, + ); + const [params] = (stubClient.search as Mock).mock.calls[0]!; + expect(params.body.query.bool.must).toEqual([{ terms: { level: ['error', 'warn'] } }]); + }); + + it('joins a multi-value index variable with commas', async () => { + await OpenSearchLogQuery.getLogData( + { query: 'level:error', queryLanguage: 'lucene', index: '$index' }, + { + ...context, + variableState: { + index: { value: ['logs-a', 'logs-b'], loading: false }, + } as unknown as LogQueryContext['variableState'], + }, + ); + const [params] = (stubClient.search as Mock).mock.calls[0]!; + expect(params.index).toBe('logs-a,logs-b'); + }); + + it('resolves the builtin $__from / $__to to the time range in epoch milliseconds', async () => { + await OpenSearchLogQuery.getLogData( + { + query: + 'SELECT * FROM logs WHERE `@timestamp` >= FROM_UNIXTIME($__from / 1000.0) AND `@timestamp` <= FROM_UNIXTIME(${__to} / 1000.0)', + queryLanguage: 'sql', + disableTimeFilter: true, + }, + context, + ); + const [params] = (stubClient.sql as Mock).mock.calls[0]!; + expect(params.query).toBe( + 'SELECT * FROM logs WHERE `@timestamp` >= FROM_UNIXTIME(1735689600000 / 1000.0) AND `@timestamp` <= FROM_UNIXTIME(1735693200000 / 1000.0)', + ); + }); + + it('lets a user-defined variable named __from win over the builtin', async () => { + await OpenSearchLogQuery.getLogData( + { query: '@timestamp:[$__from TO *]', queryLanguage: 'lucene', index: 'logs-*', disableTimeFilter: true }, + { + ...context, + variableState: { + __from: { value: 'now-1d', loading: false }, + } as unknown as LogQueryContext['variableState'], + }, + ); + const [params] = (stubClient.search as Mock).mock.calls[0]!; + expect(JSON.stringify(params.body.query)).toContain('@timestamp:[now-1d TO *]'); + }); + + it.each(['lucene', 'dsl'] as const)('rejects %s without an index', async (queryLanguage) => { + await expect(OpenSearchLogQuery.getLogData({ query: 'level:error', queryLanguage }, context)).rejects.toThrow( + /index is required/i, + ); + expect(stubClient.search).not.toHaveBeenCalled(); + }); + + it.each([ + { query: 'source=logs-*', queryLanguage: 'ppl' as const }, + { query: 'SELECT * FROM logs', queryLanguage: 'sql' as const }, + { query: 'level:error', queryLanguage: 'lucene' as const, index: 'logs-*' }, + { query: '{"query":{"match_all":{}}}', queryLanguage: 'dsl' as const, index: 'logs-*' }, + ])('records the executed request in metadata for $queryLanguage', async (spec) => { + const result = await OpenSearchLogQuery.getLogData(spec, context); + expect(typeof result.metadata?.executedQueryString).toBe('string'); + expect(result.metadata?.executedQueryString.length).toBeGreaterThan(0); + }); +}); diff --git a/opensearch/src/queries/opensearch-log-query/get-opensearch-log-data.ts b/opensearch/src/queries/opensearch-log-query/get-opensearch-log-data.ts index af085d7a6..1248a370d 100644 --- a/opensearch/src/queries/opensearch-log-query/get-opensearch-log-data.ts +++ b/opensearch/src/queries/opensearch-log-query/get-opensearch-log-data.ts @@ -11,229 +11,86 @@ // See the License for the specific language governing permissions and // limitations under the License. -import type { LogQueryPlugin, LogQueryContext } from '@perses-dev/plugin-system'; -import { replaceVariables } from '@perses-dev/plugin-system'; -import type { LogEntry, LogData } from '@perses-dev/spec'; +import type { LogQueryContext, LogQueryPlugin } from '@perses-dev/plugin-system'; +import type { LogData } from '@perses-dev/spec'; import type { OpenSearchClient } from '../../model/opensearch-client'; -import type { OpenSearchPPLResponse } from '../../model/opensearch-client-types'; -import { DEFAULT_DATASOURCE, DEFAULT_MESSAGE_FIELDS, DEFAULT_TIMESTAMP_FIELDS } from '../constants'; +import { DEFAULT_DATASOURCE } from '../constants'; +import { buildBoundedPPL, buildDslSearchBody, buildLuceneSearchBody, buildSqlRequest } from './build'; +import { convertDatarowsToLogs, convertHitsToLogs } from './convert'; import type { OpenSearchLogQuerySpec } from './opensearch-log-query-types'; - -/** - * Bound the query to the panel time range using a PPL `where` clause on the - * configured timestamp field (`@timestamp` by default). - * The bound is injected immediately after the source clause so it runs before any - * pipe that drops the timestamp column from the schema (stats, fields, top, etc.). - * If the user already filters on the timestamp field themselves, PPL ANDs the two clauses. - */ -interface BoundedPPLOptions { - index?: string; - timestampField?: string; - disableTimeFilter?: boolean; -} - -/** - * Escape a user-supplied identifier (timestamp field name) for safe interpolation - * inside a PPL backtick-quoted identifier. A backtick in the value would otherwise - * close the quoting early and let the rest of the string be parsed as PPL, producing - * an invalid — or potentially injected — query. PPL escapes a literal backtick by - * doubling it. - */ -function escapeIdentifier(name: string): string { - return name.replace(/`/g, '``'); -} - -/** - * Sanitize a user-supplied index name/pattern before embedding it in a `source=` - * clause. Unlike the timestamp field, `source=` is not backtick-quoted and must - * keep wildcards (`*`) and multi-index commas intact, so we can't just quote it. - * Instead we strip the characters that could terminate the clause and inject an - * extra PPL stage (backtick, pipe, quotes) or that are simply invalid in an index - * name (whitespace) — e.g. `logs-* | delete` would otherwise smuggle in a pipe. - */ -function sanitizeIndex(index: string): string { - return index.replace(/[`|'"\s]/g, ''); -} - -/** Join PPL stages with ` | `, dropping any empty segments so we never emit a dangling pipe. */ -function joinPPLStages(stages: string[]): string { - return stages.filter((stage) => stage.trim().length > 0).join(' | '); -} - -export function buildBoundedPPL( - userQuery: string, - start: Date, - end: Date, - { index, timestampField = '@timestamp', disableTimeFilter = false }: BoundedPPLOptions = {}, -): string { - let trimmed = userQuery.trim(); - - if (index && !/^(?:search\s+)?source\s*=/i.test(trimmed)) { - trimmed = `source=${sanitizeIndex(index)} | ${trimmed}`; - } - - // Skip the auto-injected time-range clause when the caller manages their own time - // bounds (disableTimeFilter). - if (disableTimeFilter) { - return trimmed; - } - - const startIso = start.toISOString(); - const endIso = end.toISOString(); - // timestampField is user-editable, so escape it before embedding it in the - // backtick-quoted identifier (the ISO bounds are machine-generated and safe). - const tsField = escapeIdentifier(timestampField); - const bound = `where \`${tsField}\` >= '${startIso}' and \`${tsField}\` <= '${endIso}'`; - - const firstPipe = trimmed.indexOf('|'); - if (firstPipe === -1) { - return joinPPLStages([trimmed, bound]); - } - - const sourceClause = trimmed.slice(0, firstPipe).trimEnd(); - const rest = trimmed.slice(firstPipe + 1).trimStart(); - // `rest` can be empty (e.g. a trailing pipe or an empty query), so join through a - // filter to avoid producing an invalid `... | ` dangling stage. - return joinPPLStages([sourceClause, bound, rest]); -} - -function pickIndex(cols: Array<{ name: string }>, candidates: string[]): number { - for (const candidate of candidates) { - const idx = cols.findIndex((c) => c.name === candidate); - if (idx !== -1) return idx; - } - return -1; -} - -interface ConvertOptions { - timestampField?: string; - messageField?: string; -} - -export function convertPPLToLogs(response: OpenSearchPPLResponse, options: ConvertOptions = {}): LogData { - const { schema = [], datarows = [] } = response; - - const tsCandidates = options.timestampField - ? [options.timestampField, ...DEFAULT_TIMESTAMP_FIELDS] - : DEFAULT_TIMESTAMP_FIELDS; - const msgCandidates = options.messageField - ? [options.messageField, ...DEFAULT_MESSAGE_FIELDS] - : DEFAULT_MESSAGE_FIELDS; - - const tsIdx = pickIndex(schema, tsCandidates); - const msgIdx = pickIndex(schema, msgCandidates); - - const entries: LogEntry[] = datarows.map((row) => { - const rawTs = tsIdx !== -1 ? row[tsIdx] : null; - const rawMsg = msgIdx !== -1 ? row[msgIdx] : null; - - const timestamp = parseTimestamp(rawTs); - const line = rawMsg !== null && rawMsg !== undefined ? String(rawMsg) : JSON.stringify(rowToObject(schema, row)); - - const labels: Record = {}; - schema.forEach((col, i) => { - if (i === tsIdx || i === msgIdx) return; - const v = row[i]; - if (v !== null && v !== undefined) labels[col.name] = String(v); - }); - - return { timestamp, line, labels }; - }); - - // A PPL response carries no separate "total hits" field, so totalCount reflects - // the number of rows actually returned by the query (bounded by any `head`/limit - // the user added), not a grand total of matching documents. - return { entries, totalCount: entries.length }; -} - -export function parseTimestamp(v: unknown): number { - if (v === null || v === undefined) return 0; - - // Numeric epochs arrive in seconds, milliseconds, microseconds, or nanoseconds - // (logs frequently carry ns precision). They come as a JS number, or — when the - // value exceeds Number.MAX_SAFE_INTEGER (a ns epoch is ~1e18, well past 2^53) — as - // a numeric string so it survives JSON parsing without silent overflow. - // We emit seconds (LogEntry is second-resolution), so we only need the correct - // magnitude, not sub-second precision: a float's rounding at the ns scale is far - // below one second and is discarded anyway, so BigInt isn't required here. - let numeric = NaN; - if (typeof v === 'number') { - numeric = v; - } else if (/^\d+$/.test(String(v))) { - numeric = Number(v); - } - if (!Number.isNaN(numeric)) { - // Detect the unit by magnitude — a value past year ~5138 for a given unit must - // really be a finer unit — and normalize to seconds. Without the µs/ns tiers - // those values would parse far in the future. - if (numeric >= 1e17) return numeric / 1e9; // nanoseconds → seconds - if (numeric >= 1e14) return numeric / 1e6; // microseconds → seconds - if (numeric >= 1e11) return numeric / 1e3; // milliseconds → seconds - return numeric; // seconds - } - - // OpenSearch PPL returns timestamp/datetime fields as space-separated UTC strings - // with no timezone designator (e.g. "2020-08-26 13:49:00"). Date.parse treats a - // zoneless date-time as *local* time, which would shift the value by the browser's - // offset, so normalize it to explicit UTC (replace the space with `T`, append `Z`) - // before parsing. Strings that already carry a zone (…Z / ±hh:mm) or are date-only - // are left untouched — the spec parses those as UTC already. - const str = String(v).trim(); - const zonelessDateTime = /^\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2}(\.\d+)?$/; - const iso = zonelessDateTime.test(str) ? `${str.replace(' ', 'T')}Z` : str; - const parsed = Date.parse(iso); - return Number.isNaN(parsed) ? 0 : parsed / 1000; -} - -function rowToObject( - schema: OpenSearchPPLResponse['schema'], - row: Array, -): Record { - const out: Record = {}; - schema.forEach((col, i) => { - out[col.name] = row[i]; - }); - return out; -} +import { requiresIndex, resolveQueryLanguage } from './opensearch-log-query-types'; +import { replaceIndexVariables, replaceQueryVariables, withTimeRangeVariables } from './variables'; export const getOpenSearchLogData: LogQueryPlugin['getLogData'] = async ( spec: OpenSearchLogQuerySpec, context: LogQueryContext, abortSignal?: AbortSignal, ) => { - // Treat a blank or whitespace-only query as "no query": trimming it would leave - // an empty string that produces invalid PPL, so short-circuit before hitting the API. - if (!spec.query?.trim()) { - return { - logs: { entries: [], totalCount: 0 }, - timeRange: { start: context.timeRange.start, end: context.timeRange.end }, - }; + const { start, end } = context.timeRange; + const language = resolveQueryLanguage(spec); + + // Treat a blank or whitespace-only query as "no query": trimming it would leave an empty + // string that produces an invalid request, so short-circuit before hitting the API. + // A blank Lucene query is the one exception — it legitimately means "everything in range". + if (!spec.query?.trim() && language !== 'lucene') { + return { logs: { entries: [], totalCount: 0 }, timeRange: { start, end } }; + } + + const variableState = withTimeRangeVariables(context.variableState, start, end); + const query = replaceQueryVariables(spec.query, variableState, language); + const resolvedIndex = spec.index ? replaceIndexVariables(spec.index, variableState) : undefined; + + if (requiresIndex(language) && !resolvedIndex) { + throw new Error(`An index is required when the query language is ${language}.`); } - const query = replaceVariables(spec.query, context.variableState); - const resolvedIndex = spec.index ? replaceVariables(spec.index, context.variableState) : undefined; const client = (await context.datasourceStore.getDatasourceClient( spec.datasource ?? DEFAULT_DATASOURCE, )) as OpenSearchClient; - const { start, end } = context.timeRange; - const boundedQuery = buildBoundedPPL(query, start, end, { - index: resolvedIndex, + const convertOptions = { timestampField: spec.timestampField, messageField: spec.messageField }; + const searchOptions = { timestampField: spec.timestampField, disableTimeFilter: spec.disableTimeFilter, - }); + limit: spec.limit, + }; - const response = await client.ppl({ query: boundedQuery }, undefined, abortSignal); + let logs: LogData; + let executedQueryString: string; + + switch (language) { + case 'sql': { + const request = buildSqlRequest(query, start, end, { + timestampField: spec.timestampField, + disableTimeFilter: spec.disableTimeFilter, + }); + logs = convertDatarowsToLogs(await client.sql(request, undefined, abortSignal), convertOptions); + executedQueryString = `POST /_plugins/_sql ${JSON.stringify(request)}`; + break; + } + case 'lucene': + case 'dsl': { + const body = + language === 'lucene' + ? buildLuceneSearchBody(query, start, end, searchOptions) + : buildDslSearchBody(query, start, end, searchOptions); + const index = resolvedIndex as string; + logs = convertHitsToLogs(await client.search({ index, body }, undefined, abortSignal), convertOptions); + executedQueryString = `POST /${index}/_search ${JSON.stringify(body)}`; + break; + } + default: { + const boundedQuery = buildBoundedPPL(query, start, end, { + index: resolvedIndex, + timestampField: spec.timestampField, + disableTimeFilter: spec.disableTimeFilter, + }); + logs = convertDatarowsToLogs(await client.ppl({ query: boundedQuery }, undefined, abortSignal), convertOptions); + executedQueryString = boundedQuery; + break; + } + } - return { - logs: convertPPLToLogs(response, { - timestampField: spec.timestampField, - messageField: spec.messageField, - }), - timeRange: { start, end }, - metadata: { - executedQueryString: boundedQuery, - }, - }; + return { logs, timeRange: { start, end }, metadata: { executedQueryString } }; }; diff --git a/opensearch/src/queries/opensearch-log-query/index.ts b/opensearch/src/queries/opensearch-log-query/index.ts index 58706dec6..254a55970 100644 --- a/opensearch/src/queries/opensearch-log-query/index.ts +++ b/opensearch/src/queries/opensearch-log-query/index.ts @@ -15,3 +15,5 @@ export * from './get-opensearch-log-data'; export * from './OpenSearchLogQuery'; export * from './OpenSearchLogQueryEditor'; export * from './opensearch-log-query-types'; +export * from './build'; +export * from './convert'; diff --git a/opensearch/src/queries/opensearch-log-query/opensearch-log-query-types.test.ts b/opensearch/src/queries/opensearch-log-query/opensearch-log-query-types.test.ts new file mode 100644 index 000000000..58191ad9c --- /dev/null +++ b/opensearch/src/queries/opensearch-log-query/opensearch-log-query-types.test.ts @@ -0,0 +1,31 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import type { OpenSearchQueryLanguage } from './opensearch-log-query-types'; +import { resolveQueryLanguage } from './opensearch-log-query-types'; + +describe('resolveQueryLanguage', () => { + it('defaults to ppl when queryLanguage is absent', () => { + expect(resolveQueryLanguage({})).toBe('ppl'); + }); + + it.each(['ppl', 'sql', 'lucene', 'dsl'] as const)('resolves a recognised language %s unchanged', (language) => { + expect(resolveQueryLanguage({ queryLanguage: language })).toBe(language); + }); + + it('falls back to ppl for an out-of-enum queryLanguage', () => { + // Cast through `as` since 'kql' is not a valid OpenSearchQueryLanguage — this simulates a + // hand-edited or hand-crafted spec reaching the resolver at runtime, bypassing the type system. + expect(resolveQueryLanguage({ queryLanguage: 'kql' as OpenSearchQueryLanguage })).toBe('ppl'); + }); +}); diff --git a/opensearch/src/queries/opensearch-log-query/opensearch-log-query-types.ts b/opensearch/src/queries/opensearch-log-query/opensearch-log-query-types.ts index 386360994..054312d36 100644 --- a/opensearch/src/queries/opensearch-log-query/opensearch-log-query-types.ts +++ b/opensearch/src/queries/opensearch-log-query/opensearch-log-query-types.ts @@ -13,16 +13,43 @@ import type { DatasourceSelector } from '@perses-dev/spec'; -import type { OpenSearchPPLResponse } from '../../model/opensearch-client-types'; +import type { OpenSearchDatarowsResponse } from '../../model/opensearch-client-types'; + +/** + * Query language a single log query is written in. + * `ppl` and `sql` go to the SQL-plugin endpoints and return the `datarows` format; + * `lucene` and `dsl` go to `_search` and return hits. + */ +export type OpenSearchQueryLanguage = 'ppl' | 'sql' | 'lucene' | 'dsl'; + +export const OPENSEARCH_QUERY_LANGUAGES: readonly OpenSearchQueryLanguage[] = ['ppl', 'sql', 'lucene', 'dsl']; export interface OpenSearchLogQuerySpec { query: string; + /** Defaults to `ppl` when absent, which keeps pre-existing dashboards working unchanged. */ + queryLanguage?: OpenSearchQueryLanguage; datasource?: DatasourceSelector; + /** Required for `lucene` and `dsl` (it forms the `_search` path). Ignored for `sql`, where the index lives in the FROM clause. */ index?: string; timestampField?: string; messageField?: string; - /** When true, the panel time range is NOT injected as a `where` clause on the timestamp field. */ + /** When true, the panel time range is NOT injected into the request. */ disableTimeFilter?: boolean; + /** `_search` size for `lucene` and `dsl`. Defaults to DEFAULT_SEARCH_SIZE. */ + limit?: number; +} + +export type OpenSearchLogQueryResponse = OpenSearchDatarowsResponse; + +export function resolveQueryLanguage(spec: Pick): OpenSearchQueryLanguage { + const language = spec.queryLanguage; + // Fall back to PPL for an unrecognised value as well as an absent one: the CUE schema rejects + // unknown languages on save, but a hand-edited or hand-crafted spec can still reach the editor, + // and an unknown key would otherwise resolve to `undefined` metadata and throw during render. + return language !== undefined && OPENSEARCH_QUERY_LANGUAGES.includes(language) ? language : 'ppl'; } -export type OpenSearchLogQueryResponse = OpenSearchPPLResponse; +/** Languages whose index goes into the request URL and therefore cannot be omitted. */ +export function requiresIndex(language: OpenSearchQueryLanguage): boolean { + return language === 'lucene' || language === 'dsl'; +} diff --git a/opensearch/src/queries/opensearch-log-query/variables.test.ts b/opensearch/src/queries/opensearch-log-query/variables.test.ts new file mode 100644 index 000000000..e187e8596 --- /dev/null +++ b/opensearch/src/queries/opensearch-log-query/variables.test.ts @@ -0,0 +1,85 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import type { VariableStateMap } from '@perses-dev/plugin-system'; + +import { replaceIndexVariables, replaceQueryVariables } from './variables'; + +const state = (vars: Record): VariableStateMap => + Object.fromEntries(Object.entries(vars).map(([name, value]) => [name, { value, loading: false }])); + +describe('replaceQueryVariables', () => { + const levels = state({ level: ['error', 'warn'] }); + + it.each([ + ['sql', 'SELECT * FROM logs WHERE level IN ($level)', "SELECT * FROM logs WHERE level IN ('error','warn')"], + ['lucene', 'level:$level', 'level:("error" OR "warn")'], + ['ppl', "source=logs | where level='$level'", "source=logs | where level='(error|warn)'"], + ] as const)('formats a multi-value variable for %s', (language, query, expected) => { + expect(replaceQueryVariables(query, levels, language)).toBe(expected); + }); + + it('keeps an explicit format over the language default', () => { + expect(replaceQueryVariables('level:${level:csv}', levels, 'lucene')).toBe('level:error,warn'); + }); + + it('escapes single-value variables in SQL strings the same way as before', () => { + expect(replaceQueryVariables("WHERE traceId = '$traceId'", state({ traceId: 'abc' }), 'sql')).toBe( + "WHERE traceId = 'abc'", + ); + }); + + it.each([ + ['the lucene format', '{"query":{"query_string":{"query":"svc:${svc:lucene}"}}}', { svc: ['api', 'web'] }], + ['the regex format', '{"regexp":{"host":"${host:regex}"}}', { host: 'a.b' }], + ['a value with quotes and backslashes', '{"term":{"path":"$path"}}', { path: 'C:\\logs "x"' }], + ])('keeps a DSL body valid JSON with %s', (_, query, vars) => { + const resolved = replaceQueryVariables(query, state(vars), 'dsl'); + expect(() => JSON.parse(resolved)).not.toThrow(); + }); + + it('puts the formatted value inside the DSL string, quotes included', () => { + const resolved = replaceQueryVariables( + '{"query":{"query_string":{"query":"svc:${svc:lucene}"}}}', + state({ svc: ['api', 'web'] }), + 'dsl', + ); + expect(JSON.parse(resolved).query.query_string.query).toBe('svc:("api" OR "web")'); + }); + + it('leaves a json-formatted DSL reference as the JSON it emits', () => { + expect(replaceQueryVariables('{"terms":{"level":${level:json}}}', levels, 'dsl')).toBe( + '{"terms":{"level":["error","warn"]}}', + ); + }); + + it('replaces a null value with an empty string', () => { + expect(replaceQueryVariables('level:$level', state({ level: null }), 'lucene')).toBe('level:'); + }); + + it('replaces the longest variable name first', () => { + expect(replaceQueryVariables('$__range_ms', state({ __range: '1h', __range_ms: '3600000' }), 'sql')).toBe( + '3600000', + ); + }); +}); + +describe('replaceIndexVariables', () => { + it('joins a multi-value index variable with commas', () => { + expect(replaceIndexVariables('$index', state({ index: ['logs-a', 'logs-b'] }))).toBe('logs-a,logs-b'); + }); + + it('keeps an explicit format', () => { + expect(replaceIndexVariables('${index:pipe}', state({ index: ['logs-a', 'logs-b'] }))).toBe('logs-a|logs-b'); + }); +}); diff --git a/opensearch/src/queries/opensearch-log-query/variables.ts b/opensearch/src/queries/opensearch-log-query/variables.ts new file mode 100644 index 000000000..9f5712222 --- /dev/null +++ b/opensearch/src/queries/opensearch-log-query/variables.ts @@ -0,0 +1,105 @@ +// Copyright The Perses Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +import type { VariableStateMap } from '@perses-dev/plugin-system'; +import { InterpolationFormat, interpolate, parseVariablesAndFormat, replaceVariables } from '@perses-dev/plugin-system'; + +import type { OpenSearchQueryLanguage } from './opensearch-log-query-types'; + +/** + * Add the Perses builtin `__from` / `__to` (epoch milliseconds) to the variable state. Log + * queries receive only user-defined variables, unlike time series queries, so without this + * `$__from` would stay literal text. User-defined variables of the same name win. + */ +export function withTimeRangeVariables(variableState: VariableStateMap, start: Date, end: Date): VariableStateMap { + return { + __from: { value: start.valueOf().toString(), loading: false }, + __to: { value: end.valueOf().toString(), loading: false }, + ...variableState, + }; +} + +/** Escape a string for use inside a JSON string literal, without the surrounding quotes. */ +function escapeJsonString(value: string): string { + return JSON.stringify(value).slice(1, -1); +} + +/** + * Same as Perses' `replaceVariables`, but with a language-specific default format for + * multi-value variables, and a hook that post-processes each formatted value. + */ +function replaceVariablesWith( + text: string, + variableState: VariableStateMap, + defaultListFormat: InterpolationFormat, + finish: (formatted: string, format: InterpolationFormat | undefined) => string = (formatted) => formatted, +): string { + const formats = parseVariablesAndFormat(text); + let result = text; + // Longest name first, so `$__range` can't replace the start of `$__range_ms`. + for (const name of [...formats.keys()].toSorted((a, b) => b.length - a.length)) { + const state = variableState[name]; + if (state === undefined) continue; + const format = formats.get(name); + const { value } = state; + let formatted = ''; + if (Array.isArray(value)) { + formatted = interpolate(value, name, format ?? defaultListFormat); + } else if (typeof value === 'string') { + formatted = interpolate([value], name, format ?? InterpolationFormat.RAW); + } + const replacement = finish(formatted, format); + result = result + .replaceAll(`$${name}`, replacement) + .replaceAll(`\${${name}${format ? `:${format}` : ''}}`, replacement); + } + return result; +} + +/** + * Resolve variables in a query. A multi-value variable referenced without a format gets the + * syntax its language understands, instead of Perses' default `(a|b)`: + * - SQL: `'a','b'` (`sqlstring`), for `IN ($level)`. + * - Lucene: `("a" OR "b")` (`lucene`). + * - Query DSL: values are escaped as JSON string content after formatting, so a value (or the + * quotes a format such as `lucene` adds) can't break the JSON or add keys to the request. + * A `json` reference is left as the JSON it already is. + * - PPL keeps Perses' standard behaviour, as before this plugin supported other languages. + */ +export function replaceQueryVariables( + query: string, + variableState: VariableStateMap, + language: OpenSearchQueryLanguage, +): string { + switch (language) { + case 'sql': + return replaceVariablesWith(query, variableState, InterpolationFormat.SQLSTRING); + case 'lucene': + return replaceVariablesWith(query, variableState, InterpolationFormat.LUCENE); + case 'dsl': + return replaceVariablesWith(query, variableState, InterpolationFormat.PROMETHEUS, (formatted, format) => + format === InterpolationFormat.JSON ? formatted : escapeJsonString(formatted), + ); + case 'ppl': + return replaceVariables(query, variableState); + } +} + +/** + * Resolve variables in the index. A multi-value variable referenced without a format is + * joined with commas, OpenSearch's multi-index syntax, instead of the default `(a|b)` that + * the index check would reject. + */ +export function replaceIndexVariables(index: string, variableState: VariableStateMap): string { + return replaceVariablesWith(index, variableState, InterpolationFormat.CSV); +}