From 31fb1943739e5542292355d63a34f151583dbb82 Mon Sep 17 00:00:00 2001 From: Carson Ip Date: Fri, 17 Apr 2026 12:14:21 +0100 Subject: [PATCH 1/6] docs: add APM HTTP/2 strict client known issue for 8.x Document the HTTP/2 framing/protocol failure symptoms, workaround guidance, and 8.19.15 fix version in APM known issues without 9.x references. --- docs/en/observability/apm/known-issues.asciidoc | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index 21040409b0..510fd008c7 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -21,6 +21,20 @@ _Versions: XX.XX.XX, YY.YY.YY, ZZ.ZZ.ZZ_ // If applicable, link to fix //// +[discrete] +== HTTP/2 connections may fail with strict clients due to framing issues + +_Elastic Stack versions: 8.19.12, 8.19.13, 8.19.14_ + +_Fixed in Elastic Stack version 8.19.15_ + +The issue occurs when using APM Server over HTTP/2 with strict clients. +Affected clients may report errors such as `Error in the HTTP2 framing layer`, +`Failure when receiving data from the peer`, or browser `net::ERR_HTTP2_PROTOCOL_ERROR`. + +As a workaround, use HTTP/1.1 between clients and APM Server, or offload HTTP/2 through a non-strict load balancer or proxy. + +This issue is fixed in 8.19.15 (https://github.com/elastic/apm-server/pull/20908[elastic/apm-server#20908]). + [discrete] == Tail Sampling may not compact / expired TTLs as quickly as desired, causing increased storage usage. From d7c7b9a8f7c1db4cd73154235991aff5b1de2d67 Mon Sep 17 00:00:00 2001 From: Carson Ip Date: Fri, 17 Apr 2026 13:31:29 +0100 Subject: [PATCH 2/6] docs: align HTTP/2 known-issue wording with apm-server Match the source known-issue details and wording for strict HTTP/2 client framing failures, while keeping the entry scoped to 8.x versions. Made-with: Cursor --- .../observability/apm/known-issues.asciidoc | 21 ++++++++++++------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index 510fd008c7..7f2a352a49 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -22,18 +22,23 @@ _Versions: XX.XX.XX, YY.YY.YY, ZZ.ZZ.ZZ_ //// [discrete] -== HTTP/2 connections may fail with strict clients due to framing issues +== HTTP/2 connections can fail with strict clients due to framing errors -_Elastic Stack versions: 8.19.12, 8.19.13, 8.19.14_ + -_Fixed in Elastic Stack version 8.19.15_ +_Elastic Stack versions: >=8.19.12 and <8.19.15_ -The issue occurs when using APM Server over HTTP/2 with strict clients. -Affected clients may report errors such as `Error in the HTTP2 framing layer`, -`Failure when receiving data from the peer`, or browser `net::ERR_HTTP2_PROTOCOL_ERROR`. +APM Server can fail HTTP/2 requests from strict clients (for example, curl/nghttp2) after ALPN negotiates `h2`. +In affected versions, APM Server can send inconsistent SETTINGS values at connection start (an initial empty/default SETTINGS frame followed by a different SETTINGS set), and strict clients treat that sequence as an HTTP/2 protocol error and close the connection. +When this occurs, clients can report framing/connection errors such as `Error in the HTTP2 framing layer` or `Failure when receiving data from the peer`. +Browser-based HTTP requests from the RUM agent can also fail with `net::ERR_HTTP2_PROTOCOL_ERROR`. +When clients close the HTTP/2 connection due to this issue, APM Server logs an error: `http2: received GOAWAY [FrameHeader GOAWAY len=66], starting graceful shutdown`. -As a workaround, use HTTP/1.1 between clients and APM Server, or offload HTTP/2 through a non-strict load balancer or proxy. +For more information, check https://github.com/elastic/apm-server/issues/20887[issue #20887]. -This issue is fixed in 8.19.15 (https://github.com/elastic/apm-server/pull/20908[elastic/apm-server#20908]). +Use HTTP/1.1 to communicate with APM Server on affected versions. + +If clients require HTTP/2, place a non-strict load balancer in front of APM Server and stop HTTP/2 at the load balancer (for example, on some load balancers this can be done by switching from TCP mode to HTTP mode). + +This bug will be fixed in 8.19.15. [discrete] == Tail Sampling may not compact / expired TTLs as quickly as desired, causing increased storage usage. From e13e68d4f0869fcf6c03c5f98a7e4a247706fb69 Mon Sep 17 00:00:00 2001 From: Carson Ip Date: Fri, 17 Apr 2026 13:31:42 +0100 Subject: [PATCH 3/6] docs: use source wording for HTTP/2 workaround Made-with: Cursor --- docs/en/observability/apm/known-issues.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index 7f2a352a49..b832924fa4 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -36,7 +36,7 @@ For more information, check https://github.com/elastic/apm-server/issues/20887[i Use HTTP/1.1 to communicate with APM Server on affected versions. -If clients require HTTP/2, place a non-strict load balancer in front of APM Server and stop HTTP/2 at the load balancer (for example, on some load balancers this can be done by switching from TCP mode to HTTP mode). +If clients require HTTP/2, place a non-strict load balancer in front of APM Server and terminate HTTP/2 at the load balancer (for example, on some load balancers this can be done by switching from TCP mode to HTTP mode). This bug will be fixed in 8.19.15. From 4d28043a6a1390f6b2f6adaf0301d2633299311f Mon Sep 17 00:00:00 2001 From: Carson Ip Date: Fri, 17 Apr 2026 13:32:48 +0100 Subject: [PATCH 4/6] docs: explicitly label HTTP/2 guidance as workaround Made-with: Cursor --- docs/en/observability/apm/known-issues.asciidoc | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index b832924fa4..ad99ff4e6f 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -34,6 +34,8 @@ When clients close the HTTP/2 connection due to this issue, APM Server logs an e For more information, check https://github.com/elastic/apm-server/issues/20887[issue #20887]. +*Workaround* + Use HTTP/1.1 to communicate with APM Server on affected versions. If clients require HTTP/2, place a non-strict load balancer in front of APM Server and terminate HTTP/2 at the load balancer (for example, on some load balancers this can be done by switching from TCP mode to HTTP mode). From f54faa100e38d81bafa507e07096e2da0c9bb9d4 Mon Sep 17 00:00:00 2001 From: Carson Ip Date: Fri, 17 Apr 2026 13:33:21 +0100 Subject: [PATCH 5/6] docs: align workaround section with known-issues prose style Made-with: Cursor --- docs/en/observability/apm/known-issues.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index ad99ff4e6f..d7148e677e 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -34,7 +34,7 @@ When clients close the HTTP/2 connection due to this issue, APM Server logs an e For more information, check https://github.com/elastic/apm-server/issues/20887[issue #20887]. -*Workaround* +There are two workarounds: Use HTTP/1.1 to communicate with APM Server on affected versions. From 7e9f133b6cc6e4b324662e1b8d12dac6138af899 Mon Sep 17 00:00:00 2001 From: Carson Ip Date: Tue, 21 Apr 2026 13:27:37 +0100 Subject: [PATCH 6/6] Apply suggestions from code review Co-authored-by: Visha Angelova <91186315+vishaangelova@users.noreply.github.com> --- docs/en/observability/apm/known-issues.asciidoc | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/en/observability/apm/known-issues.asciidoc b/docs/en/observability/apm/known-issues.asciidoc index d7148e677e..7174df0dbb 100644 --- a/docs/en/observability/apm/known-issues.asciidoc +++ b/docs/en/observability/apm/known-issues.asciidoc @@ -27,18 +27,19 @@ _Versions: XX.XX.XX, YY.YY.YY, ZZ.ZZ.ZZ_ _Elastic Stack versions: >=8.19.12 and <8.19.15_ APM Server can fail HTTP/2 requests from strict clients (for example, curl/nghttp2) after ALPN negotiates `h2`. + In affected versions, APM Server can send inconsistent SETTINGS values at connection start (an initial empty/default SETTINGS frame followed by a different SETTINGS set), and strict clients treat that sequence as an HTTP/2 protocol error and close the connection. -When this occurs, clients can report framing/connection errors such as `Error in the HTTP2 framing layer` or `Failure when receiving data from the peer`. -Browser-based HTTP requests from the RUM agent can also fail with `net::ERR_HTTP2_PROTOCOL_ERROR`. + +When this occurs, clients can report framing/connection errors such as `Error in the HTTP2 framing layer` or `Failure when receiving data from the peer`. Browser-based HTTP requests from the RUM agent can also fail with `net::ERR_HTTP2_PROTOCOL_ERROR`. + When clients close the HTTP/2 connection due to this issue, APM Server logs an error: `http2: received GOAWAY [FrameHeader GOAWAY len=66], starting graceful shutdown`. For more information, check https://github.com/elastic/apm-server/issues/20887[issue #20887]. There are two workarounds: -Use HTTP/1.1 to communicate with APM Server on affected versions. - -If clients require HTTP/2, place a non-strict load balancer in front of APM Server and terminate HTTP/2 at the load balancer (for example, on some load balancers this can be done by switching from TCP mode to HTTP mode). +* Use HTTP/1.1 to communicate with APM Server on affected versions. +* If clients require HTTP/2, place a non-strict load balancer in front of APM Server and terminate HTTP/2 at the load balancer (for example, on some load balancers this can be done by switching from TCP mode to HTTP mode). This bug will be fixed in 8.19.15.