diff --git a/solutions/observability/apm/opentelemetry/attributes.md b/solutions/observability/apm/opentelemetry/attributes.md index 2002386751..875570adfc 100644 --- a/solutions/observability/apm/opentelemetry/attributes.md +++ b/solutions/observability/apm/opentelemetry/attributes.md @@ -15,6 +15,12 @@ products: In OpenTelemetry, an attribute is a key-value pair. Attributes are similar to [labels](/solutions/observability/apm/metadata.md#apm-data-model-labels) in that they add metadata to transactions, spans, and other entities. +:::{note} +This page describes how attributes are handled when sending OTLP data to {{apm-server-or-mis}} (the `.apm` endpoint). On this path, attributes are mapped to ECS fields, and unmapped attributes are stored under `labels.*`. + +If you use the [{{motlp}}](opentelemetry://reference/motlp.md) or [{{agent}} in OTel mode](elastic-agent://reference/edot-collector/modes.md) as your ingest path, your data is stored in OTel-native format. Attributes are preserved under `resource.attributes.*` and `attributes.*`. For a comparison of both formats and guidance on choosing your ingest path, refer to [How your ingest path determines the data format](opentelemetry://reference/compatibility/data-streams.md#ingest-path-data-format). +::: + Resource attributes are a type of attribute that contains information about the entities that produce telemetry. Resource attributes map to Elastic Common Schema (ECS) fields like `service.*`, `cloud.*`, `process.*`, and so on. These fields describe the service and its environment. For example, Elastic APM maps the OpenTelemetry `deployment.environment` field to the ECS `service.environment` field on ingestion. diff --git a/solutions/observability/apm/opentelemetry/index.md b/solutions/observability/apm/opentelemetry/index.md index a2cefce859..d383444db0 100644 --- a/solutions/observability/apm/opentelemetry/index.md +++ b/solutions/observability/apm/opentelemetry/index.md @@ -28,7 +28,7 @@ In the Kibana Integrations UI, search for `otel` to find and install available i Other Integrations which are beats-based include dashboards based on ECS data and are not compatible with OpenTelemetry semantic conventions. ::: -Elastic offers several [{{edot}}](opentelemetry://reference/index.md) distributions. Each is a customized version of an OpenTelemetry language SDK and the OpenTelemetry Collector, ready to send data to the [Managed OTLP endpoint](opentelemetry://reference/motlp.md), APM Server, or directly to {{es}}. +Elastic offers several [{{edot}}](opentelemetry://reference/index.md) distributions. Each is a customized version of an OpenTelemetry language SDK and the OpenTelemetry Collector, ready to send data to the [{{motlp}}](opentelemetry://reference/motlp.md), {{apm-server-or-mis}}, or directly to {{es}}. **The destination you choose affects the format your data is stored in.** For details, refer to [How your ingest path determines the data format](opentelemetry://reference/compatibility/data-streams.md#ingest-path-data-format). :::{include} /solutions/_snippets/edot-reference-arch.md ::: @@ -45,13 +45,14 @@ There are several ways to send OpenTelemetry data to Elastic. The right choice d ### Collect, process, and export data -* **OTel Collector in {{agent}}** {applies_to}`stack: ga 9.2+`: The OTel Collector runs embedded inside {{agent}}, sharing a single `elastic-agent.yml` configuration file. No separate Collector installation is needed. Refer to [{{agent}} as an OpenTelemetry Collector](/reference/fleet/elastic-agent-as-otel-collector.md) for more information. -* **Standalone {{agent}}**: Run {{agent}} independently as its own process. Refer to [{{agent}}](elastic-agent://reference/edot-collector/index.md) for more information. +* **OTel Collector in {{agent}}** {applies_to}`stack: ga 9.2+`: The OTel Collector runs embedded inside {{agent}}, sharing a single `elastic-agent.yml` configuration file. No separate Collector installation is needed. Data is stored in **OTel-native format** by default. Refer to [{{agent}} as an OpenTelemetry Collector](/reference/fleet/elastic-agent-as-otel-collector.md) for more information. +* **Standalone {{agent}}**: Run {{agent}} independently as its own process. Data is stored in **OTel-native format** by default. Refer to [{{agent}}](elastic-agent://reference/edot-collector/index.md) for more information. * **Upstream `otelcol-contrib` Collector**: Use the community-built Collector to forward data to {{agent}} or directly to {{apm-server-or-mis}} using OTLP. Useful for a vendor-neutral pipeline or fanning out to multiple observability backends, but it's community-supported only. Refer to [Contrib OpenTelemetry Collectors and language SDKs](/solutions/observability/apm/opentelemetry/upstream-opentelemetry-collectors-language-sdks.md) for more information. ### Send data directly -* **Managed OTLP endpoint** ({{serverless-short}} and {{ech}}): Send OpenTelemetry data directly to the [Managed OTLP endpoint](opentelemetry://reference/motlp.md) without managing your own Collector. +* **{{motlp}}** ({{serverless-short}} and {{ech}}): Send OpenTelemetry data directly to the [{{motlp}}](opentelemetry://reference/motlp.md) without managing your own Collector. Data is stored in **OTel-native format**. +* **{{apm-server-or-mis}} OTLP intake** (legacy): Send OTLP data directly to {{apm-server-or-mis}} using its built-in OTLP endpoint. Data is **translated to ECS format** before storage. This path is not recommended for new users, and EDOT SDKs are not supported here. For details on the format difference, refer to [How your ingest path determines the data format](opentelemetry://reference/compatibility/data-streams.md#ingest-path-data-format). ## Why use the Elastic Distributions of OpenTelemetry? diff --git a/solutions/observability/apm/opentelemetry/limitations.md b/solutions/observability/apm/opentelemetry/limitations.md index f7e1cce271..fa14e975c9 100644 --- a/solutions/observability/apm/opentelemetry/limitations.md +++ b/solutions/observability/apm/opentelemetry/limitations.md @@ -14,6 +14,19 @@ products: # OpenTelemetry limitations in Elastic APM [apm-open-telemetry-known-limitations] +## Data format and compatibility with the `.apm` OTLP intake [apm-open-telemetry-data-format] + +When sending OTLP data to {{apm-server-or-mis}} using its built-in OTLP endpoint (the `.apm` path), your data is **translated to ECS format** before storage. This is different from the {{motlp}} and {{agent}} in OTel mode, which store data in OTel-native format. + +Consequences of the ECS translation: + +* **Custom attributes**: Unmapped span and transaction attributes are stored under `labels.*` (strings) or `numeric_labels.*` (numbers), with dots replaced by underscores. For example, `order.id` becomes `labels.order_id`. On OTel-native paths, they are stored under `attributes.order.id`, with dots preserved. +* **Dashboards and content packs**: OpenTelemetry content packs, which provide OTel-compatible dashboards, are not applicable to ECS-translated data. Use prebuilt ECS-based integrations and dashboards, or the classic {{product.apm}} UI. +* **Ingest pipelines**: ECS-translated data passes through {{es}} ingest pipelines in the usual way. OTel-native data bypasses ingest pipelines. +* **EDOT SDK support**: EDOT SDKs are not supported with the {{apm-server-or-mis}} OTLP intake. Use the {{motlp}} or {{agent}} in OTel mode instead. + +For a detailed comparison and guidance on choosing your ingest path, refer to [How your ingest path determines the data format](opentelemetry://reference/compatibility/data-streams.md#ingest-path-data-format). + ## OpenTelemetry traces [apm-open-telemetry-traces-limitations] * Traces of applications using `messaging` semantics might be wrongly displayed as `transactions` in the Applications UI, while they should be considered `spans` (see issue [#7001](https://github.com/elastic/apm-server/issues/7001)). diff --git a/solutions/observability/get-started/opentelemetry/_snippets/motlp-vs-apm-endpoint.md b/solutions/observability/get-started/opentelemetry/_snippets/motlp-vs-apm-endpoint.md new file mode 100644 index 0000000000..482dd26df4 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/_snippets/motlp-vs-apm-endpoint.md @@ -0,0 +1,5 @@ +The {{motlp}} stores OpenTelemetry data **without schema translation**, preserving OpenTelemetry semantic conventions and resource attributes in OTel-native format. Custom attributes are stored under `attributes.*` with dots preserved. It supports ingesting OTLP logs, metrics, and traces. + +The {{apm-server-or-mis}} OTLP intake (the `.apm` endpoint) is a legacy path that **translates OTLP data to ECS format** before storage. Custom attributes land in `labels.*` with dots replaced by underscores. This path is not recommended for new users, and EDOT SDKs are not supported here. + +For a full comparison of ingest paths and their data format implications, refer to [How your ingest path determines the data format](opentelemetry://reference/compatibility/data-streams.md#ingest-path-data-format). diff --git a/solutions/observability/get-started/opentelemetry/quickstart/serverless/index.md b/solutions/observability/get-started/opentelemetry/quickstart/serverless/index.md index b91744c948..4bc2b12c7e 100644 --- a/solutions/observability/get-started/opentelemetry/quickstart/serverless/index.md +++ b/solutions/observability/get-started/opentelemetry/quickstart/serverless/index.md @@ -45,9 +45,10 @@ Select the quickstart guide for your environment from the following list: - [Docker on serverless](docker.md) - [Hosts or VMs on serverless](hosts_vms.md) -## Differences with Elastic APM Endpoint +## Differences from the Elastic APM endpoint -The {{motlp}} ensures that OpenTelemetry data is stored without any schema translation, preserving both OpenTelemetry semantic conventions and resource attributes. It supports ingesting OTLP logs, metrics, and traces in a unified manner, ensuring consistent treatment across all telemetry data. +:::{include} /solutions/observability/get-started/opentelemetry/_snippets/motlp-vs-apm-endpoint.md +::: ## Provide feedback diff --git a/solutions/observability/get-started/quickstart-elastic-cloud-otel-endpoint.md b/solutions/observability/get-started/quickstart-elastic-cloud-otel-endpoint.md index a9f30df704..2f8ad372dd 100644 --- a/solutions/observability/get-started/quickstart-elastic-cloud-otel-endpoint.md +++ b/solutions/observability/get-started/quickstart-elastic-cloud-otel-endpoint.md @@ -242,9 +242,10 @@ When creating a Kubernetes secret, always encode the full string in Base64, incl ::::::: -## Differences from the Elastic APM Endpoint +## Differences from the Elastic APM endpoint -The Elastic Cloud Managed OTLP Endpoint ensures that OpenTelemetry data is stored without any schema translation, preserving both OpenTelemetry semantic conventions and resource attributes. It supports ingesting OTLP logs, metrics, and traces in a unified manner, ensuring consistent treatment across all telemetry data. +:::{include} /solutions/observability/get-started/opentelemetry/_snippets/motlp-vs-apm-endpoint.md +::: ## Troubleshooting