[FEATURE] Add SQL, Lucene and Query DSL query languages to the OpenSearch plugin

docs/opensearch/README.md

@@ -4,13 +4,21 @@ tags:
 ---
 # OpenSearch plugins
 
-The OpenSearch plugin package provides support for [OpenSearch](https://opensearch.org/) in Perses dashboards. Logs are queried with [PPL (Piped Processing Language)](https://opensearch.org/docs/latest/search-plugins/sql/ppl/index/), which requires the SQL/PPL plugin on your cluster — it ships with OpenSearch by default.
+The OpenSearch plugin package provides support for [OpenSearch](https://opensearch.org/) in Perses dashboards. Logs can be queried with [PPL (Piped Processing Language)](https://docs.opensearch.org/latest/sql-and-ppl/ppl/index/), [SQL](https://docs.opensearch.org/latest/sql-and-ppl/sql/index/), [Lucene / query string](https://docs.opensearch.org/latest/query-dsl/full-text/query-string/) or raw [Query DSL](https://docs.opensearch.org/latest/query-dsl/), selected per query. PPL and SQL require the SQL/PPL plugin on your cluster — it ships with OpenSearch by default. Lucene and Query DSL use the core `_search` API.
 
 ## Datasource (`OpenSearchDatasource`)
 
 The OpenSearch datasource enables connection between Perses and your OpenSearch cluster for log aggregation and querying. It works with log-related panels and queries.
 
-It supports the [proxy](https://perses.dev/perses/docs/concepts/proxy/) feature of Perses that allows to restrict the access to your data source. All queries are sent to a single endpoint, `POST /_plugins/_ppl`, so that is the only endpoint you need to allow.
+It supports the [proxy](https://perses.dev/perses/docs/concepts/proxy/) feature of Perses that allows to restrict the access to your data source. Each query language uses its own endpoint, so allow the ones you need:
+
+| Query language    | Endpoint                | `endpointPattern` |
+|-------------------|-------------------------|-------------------|
+| PPL               | `POST /_plugins/_ppl`   | `/_plugins/_ppl`  |
+| SQL               | `POST /_plugins/_sql`   | `/_plugins/_sql`  |
+| Lucene, Query DSL | `POST /{index}/_search` | `/.*/_search`     |
+
+A datasource created in the Perses UI allows all three by default. A datasource created before SQL, Lucene and Query DSL support only allows `/_plugins/_ppl`; queries in the other languages are then rejected by the proxy with a 403 until you add their endpoints.
 
 See also technical docs related to this plugin:
 
@@ -19,7 +27,16 @@ See also technical docs related to this plugin:
 
 ## Log Query (`OpenSearchLogQuery`)
 
-The OpenSearch log query plugin enables querying log data from your OpenSearch cluster using PPL syntax. It supports filtering, projection and aggregation through PPL pipes.
+The OpenSearch log query plugin enables querying log data from your OpenSearch cluster. The `queryLanguage` field selects how `query` is interpreted:
+
+| `queryLanguage` | `query` is                                   | `index`                                       |
+|-----------------|----------------------------------------------|-----------------------------------------------|
+| `ppl` (default) | a PPL expression                             | Optional, see [Index pattern](#index-pattern) |
+| `sql`           | a SQL statement                              | Ignored — the index comes from `FROM`         |
+| `lucene`        | a query string; blank matches every document | Required                                      |
+| `dsl`           | a JSON `_search` request body                | Required                                      |
+
+A query without `queryLanguage` is a PPL query, so dashboards saved before the other languages existed keep working.
 
 See also technical docs related to this plugin:
 
@@ -28,11 +45,19 @@ See also technical docs related to this plugin:
 
 ### Index pattern
 
-The `index` field of the query is prepended to your expression as a `source=<index>` clause. It is ignored when your expression already starts with `source=` (or `search source=`), so you can either set the index once on the query or write the source clause yourself.
+For Lucene and Query DSL, `index` is required: it is the index or index pattern in the `POST /{index}/_search` request path. For SQL it is ignored. For PPL, the `index` field of the query is prepended to your expression as a `source=<index>` clause. It is ignored when your expression already starts with `source=` (or `search source=`), so you can either set the index once on the query or write the source clause yourself.
+
+### Result limit
+
+`limit` sets how many log documents a Lucene or Query DSL query fetches, as the `_search` request's `size`. It defaults to 500 and must be a positive integer. For Query DSL, a `size` in your request body is kept, but capped at `limit` when both are set. `limit` does not apply to PPL or SQL — use `head` or `LIMIT` in the query instead.
 
 ### Automatic time filtering
 
-The panel time range is injected into the query as a `where` clause on the timestamp field, placed immediately after the source clause so it applies before any pipe that drops the timestamp column, such as `stats` or `fields`. A query written as:
+The panel time range is added to every query as a filter on the timestamp field:
+
+- **SQL**: sent in the `_sql` request's `filter` field, so your statement is sent unchanged.
+- **Lucene and Query DSL**: added as a `range` filter next to your query inside a `bool.filter`. A Query DSL body without a `query` key is filtered on the time range alone.
+- **PPL**: injected into the query as a `where` clause on the timestamp field, placed immediately after the source clause so it applies before any pipe that drops the timestamp column, such as `stats` or `fields`. A query written as:
 
 ```ppl
 source=otel-logs-demo | where severityText='ERROR'
@@ -44,16 +69,18 @@ is sent to OpenSearch as:
 source=otel-logs-demo | where `@timestamp` >= '2026-07-14T03:53:33.000Z' and `@timestamp` <= '2026-07-14T04:53:33.000Z' | where severityText='ERROR'
 ```
 
-Set `disableTimeFilter` to `true` to turn this off and manage the time bounds yourself.
+Set `disableTimeFilter` to `true` to turn this off, in every language, and manage the time bounds yourself.
+
+Lucene and Query DSL results are sorted newest first on the timestamp field. A Query DSL body that sets its own `sort` keeps it.
 
 ### Field mapping
 
-OpenSearch does not enforce a schema, and field names vary by exporter. Each row of the PPL response is mapped to a log entry by looking up a timestamp column and a message column, trying these names in order:
+OpenSearch does not enforce a schema, and field names vary by exporter. Each PPL or SQL result row, and each Lucene or Query DSL hit's `_source` (flattened to dot-separated keys), is mapped to a log entry by looking up a timestamp column and a message column, trying these names in order:
 
 - **timestamp**: `@timestamp`, `timestamp`, `time`
 - **message**: `message`, `log`, `body`
 
-Set `timestampField` or `messageField` on the query to prefer a different name; the defaults above are still used as fallbacks. Every remaining column of the response becomes a label on the log entry. If no message column matches, the whole row is serialized to JSON as the log line so no data is lost.
+Set `timestampField` or `messageField` on the query to prefer a different name; the defaults above are still used as fallbacks. Every remaining field becomes a label on the log entry. If no message column matches, the whole row is serialized to JSON as the log line so no data is lost.
 
 !!! note
-    The `timestampField` override also determines the column used by the automatic time filter, which defaults to `@timestamp` alone. If your index stores its timestamp as `time` or `timestamp`, set `timestampField` explicitly, otherwise the injected `where` clause references a column that does not exist and OpenSearch rejects the query.
+    The `timestampField` override also determines the field used by the automatic time filter and, for Lucene and Query DSL, the sort, which default to `@timestamp` alone. If your index stores its timestamp as `time` or `timestamp`, set `timestampField` explicitly, otherwise the time filter references a field that does not exist and the query fails or returns no logs.

docs/opensearch/go-sdk/log-query.md

@@ -9,7 +9,7 @@ var options []log.Option
 log.OpenSearchLogQuery(`source=otel-logs-* | where severityText='ERROR'`, options...)
 ```
 
-Need to provide the PPL expression and a list of options. The expression cannot be empty.
+Need to provide the query expression and a list of options. The expression is PPL unless [QueryLanguage](#querylanguage) says otherwise. It cannot be empty, except for a Lucene query, where a blank expression matches every document.
 
 ## Default options
 
@@ -25,7 +25,7 @@ import "github.com/perses/plugins/opensearch/sdk/go/query/log"
 log.Query(`source=otel-logs-* | where severityText='ERROR' | head 20`)
 ```
 
-Define the PPL query expression for log data.
+Define the query expression for log data, in the language set by [QueryLanguage](#querylanguage).
 
 #### Datasource
 
@@ -45,7 +45,27 @@ import "github.com/perses/plugins/opensearch/sdk/go/query/log"
 log.Index("otel-logs-*")
 ```
 
-Define the index or index pattern to read from. It is prepended to the query as a `source=<index>` clause, and ignored when the query already starts with `source=`.
+Define the index or index pattern to read from. Required for the `lucene` and `dsl` languages, where it is the index in `POST /{index}/_search`. For PPL it is prepended to the query as a `source=<index>` clause, and ignored when the query already starts with `source=`. Ignored for SQL, which reads its index from the `FROM` clause.
+
+#### QueryLanguage
+
+```golang
+import "github.com/perses/plugins/opensearch/sdk/go/query/log"
+
+log.QueryLanguage("lucene")
+```
+
+Select how the query expression is interpreted: `ppl`, `sql`, `lucene` or `dsl`. When not set, the query is PPL and `queryLanguage` is omitted from the spec.
+
+#### Limit
+
+```golang
+import "github.com/perses/plugins/opensearch/sdk/go/query/log"
+
+log.Limit(200)
+```
+
+Cap the number of log documents fetched by a `lucene` or `dsl` query (the `_search` request's `size`). Must be greater than 0; defaults to 500 when not set. A `size` in a `dsl` body is capped at this value. Ignored for PPL and SQL.
 
 #### TimestampField
 
@@ -75,7 +95,7 @@ import "github.com/perses/plugins/opensearch/sdk/go/query/log"
 log.DisableTimeFilter(true)
 ```
 
-Stop the panel time range from being injected as a `where` clause on the timestamp field, so the query manages its own time bounds.
+Stop the panel time range from being added as a filter on the timestamp field, so the query manages its own time bounds.
 
 ## Example
 

docs/opensearch/model.md

@@ -25,7 +25,7 @@ spec:
 
 See [common plugin definitions](https://perses.dev/perses/docs/plugins/common/#http-proxy-specification).
 
-The plugin sends every query to `POST /_plugins/_ppl`, so that single endpoint is all you need to allow.
+Allow the endpoints for the query languages you use: `POST /_plugins/_ppl` for PPL, `POST /_plugins/_sql` for SQL, and `POST /{index}/_search` (pattern `/.*/_search`) for Lucene and Query DSL.
 
 ### Example
 
@@ -63,6 +63,10 @@ spec:
           allowedEndpoints:
             - endpointPattern: "/_plugins/_ppl"
               method: "POST"
+            - endpointPattern: "/_plugins/_sql"
+              method: "POST"
+            - endpointPattern: "/.*/_search"
+              method: "POST"
           secret: "opensearch_secret_config"
 ```
 
@@ -73,18 +77,29 @@ Perses supports log queries for OpenSearch: `OpenSearchLogQuery`.
 ```yaml
 kind: "OpenSearchLogQuery"
 spec:
-  # `query` is the PPL expression for log data. It cannot be empty.
+  # `query` is the log query, written in `queryLanguage`.
+  # It cannot be empty, except for `lucene`, where a blank query matches every document.
+  # For `dsl` it is a JSON `_search` request body.
   query: <string>
 
+  # `queryLanguage` is one of `ppl`, `sql`, `lucene` or `dsl`. Defaults to `ppl` when omitted.
+  queryLanguage: <string> # Optional
+
   # `datasource` is a datasource selector. If not provided, the default OpenSearchDatasource is used.
   # See the documentation about the datasources to understand how it is selected.
   datasource: <OpenSearch Datasource selector> # Optional
 
   # `index` is the index or index pattern to read from, e.g. `logs-*`.
-  # It is prepended to the query as a `source=<index>` clause, and ignored when the
-  # query already starts with `source=`.
+  # Required for `lucene` and `dsl`, where it is the index in `POST /{index}/_search`.
+  # For `ppl` it is prepended to the query as a `source=<index>` clause, and ignored when
+  # the query already starts with `source=`. Ignored for `sql`, which uses its FROM clause.
   index: <string> # Optional
 
+  # `limit` is the maximum number of log documents fetched by a `lucene` or `dsl` query.
+  # It must be greater than 0 and defaults to 500. A `size` in a `dsl` body is capped at `limit`.
+  # Ignored for `ppl` and `sql`.
+  limit: <int> # Optional
+
   # `timestampField` overrides which column carries the log timestamp.
   # It is also the column used by the automatic time filter.
   # Defaults to `@timestamp`, then `timestamp`, then `time`.
@@ -94,8 +109,8 @@ spec:
   # Defaults to `message`, then `log`, then `body`.
   messageField: <string> # Optional
 
-  # When `disableTimeFilter` is true, the panel time range is NOT injected as a
-  # `where` clause on the timestamp field.
+  # When `disableTimeFilter` is true, the panel time range is NOT added as a filter on
+  # the timestamp field.
   disableTimeFilter: <boolean> # Optional
 ```
 
@@ -132,6 +147,33 @@ spec:
       disableTimeFilter: true
 ```
 
+A Lucene query, fetching at most 200 documents:
+
+```yaml
+kind: "LogQuery"
+spec:
+  plugin:
+    kind: "OpenSearchLogQuery"
+    spec:
+      queryLanguage: "lucene"
+      index: "otel-logs-*"
+      query: "severityText:ERROR AND service.name:payment-svc"
+      limit: 200
+```
+
+A Query DSL query:
+
+```yaml
+kind: "LogQuery"
+spec:
+  plugin:
+    kind: "OpenSearchLogQuery"
+    spec:
+      queryLanguage: "dsl"
+      index: "otel-logs-*"
+      query: '{"query": {"term": {"traceId": "$traceId"}}}'
+```
+
 ## Shared definitions
 
 ### OpenSearch Datasource selector

opensearch/README.md

@@ -1,7 +1,20 @@
 # Plugin Module: opensearch
 
-Datasource plugin for [OpenSearch](https://opensearch.org/) in Perses. Supports log queries using
-[PPL (Piped Processing Language)](https://opensearch.org/docs/latest/search-plugins/sql/ppl/index/).
+Datasource plugin for [OpenSearch](https://opensearch.org/) in Perses. Log queries can
+be written in [PPL](https://docs.opensearch.org/latest/sql-and-ppl/ppl/index/),
+[SQL](https://docs.opensearch.org/latest/sql-and-ppl/sql/index/),
+[Lucene / query string](https://docs.opensearch.org/latest/query-dsl/full-text/query-string/),
+or raw [Query DSL](https://docs.opensearch.org/latest/query-dsl/), selected per query.
+PPL and SQL need the SQL/PPL plugin enabled on the cluster; Lucene and Query DSL use
+the core `_search` API.
+
+Set `queryLanguage` on the query to `ppl` (the default), `sql`, `lucene` or `dsl`.
+Lucene and Query DSL queries also need an `index` and accept an optional `limit`
+(500 by default). The dashboard time range is added to every query automatically
+unless `disableTimeFilter` is set. If you use the Perses proxy, allow the endpoint for
+each language: `/_plugins/_ppl`, `/_plugins/_sql` and `/.*/_search`. See the
+[plugin documentation](../docs/opensearch/README.md) and the
+[data model](../docs/opensearch/model.md) for details.
 
 This plugin is intended to display logs alongside the traces surfaced by the Tempo plugin, so a user can
 pivot from a trace to the related logs in OpenSearch.