Skip to content

fix(nodes): route load balancer traffic according to externalTrafficPolicy - #31

Open
sircthulhu wants to merge 10 commits into
masterfrom
fix/external-traffic-policy
Open

sircthulhu wants to merge 10 commits into
masterfrom
fix/external-traffic-policy

Conversation

@sircthulhu

@sircthulhu sircthulhu commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Problem

RobotLB always picks target nodes from where the service's pods run, which is the
behaviour of externalTrafficPolicy: Local. Services using the default Cluster
policy therefore lose every node that kube-proxy could have forwarded from.

This breaks a service whose only pod runs on a node where its own nodePort is not
reachable. A KubeVirt VM behind a LoadBalancer service hits exactly this: the VM's
nodePort answers on every node except the one hosting the VM, and since that node was
the load balancer's only target, the target stayed unhealthy and every connection was
accepted and immediately closed. HTTP services on the same cluster kept working only
because their deployments have replicas on several nodes.

Changes

Node selection

  • Target nodes follow the service's externalTrafficPolicy. Cluster, the Kubernetes
    default, uses every node of the cluster; Local keeps the previous pod-based
    discovery. ROBOTLB_DYNAMIC_NODE_SELECTOR=false still selects nodes by the
    robotlb/node-selector annotation.
  • Nodes labelled node.kubernetes.io/exclude-from-external-load-balancers, which
    kubeadm puts on control-plane nodes, stay out of every balancer under either policy.
    Cordoned and not-ready nodes are dropped under Cluster, where they would only take
    up target slots.

Targets

  • Targets are planned before they are applied: deduplicated, sorted, and trimmed to the
    number the balancer type holds, so a cluster larger than the limit keeps a stable set
    of targets instead of retrying calls the API will reject.
  • A target Hetzner rejects, such as an IP outside the vSwitch subnet of the attached
    network, no longer aborts the reconciliation and no longer fails every later run.
    Reconciliation fails only when the balancer is left without a single live target, and
    the error carries the reason the API reported.

Services without a usable port

  • A Hetzner load balancer forwards traffic to a node IP, so a port is reachable only
    through its nodePort. A service whose ports all lack one no longer gets a balancer
    created for it — it used to get an empty one, billed by the hour — and a service that
    already advertises an external IP loses it, so nothing keeps sending traffic to a
    balancer that no longer forwards.

Tests

  • cargo test covers target planning, node selection, node eligibility and port
    mapping. The repository had no test module before, and cargo test is now part of the
    pre-commit hooks the CI workflow runs.

Upgrade impact

Cluster is the Kubernetes default, so services that never set externalTrafficPolicy
get the full node list on upgrade, which rewrites the targets of their existing
balancers.

Target nodes were always restricted to the nodes running the service's
pods, which is the behaviour of externalTrafficPolicy: Local. Services
using the default Cluster policy lost every node that kube-proxy could
have forwarded from, so a service whose only pod sits on a node where
its nodePort is unreachable ends up with no healthy target at all.

Follow the policy declared by the service instead, and cover the port
mapping and the policy check with unit tests.

Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Hetzner refuses IP targets outside the vSwitch subnet of the attached
network. Such a rejection aborted the whole reconciliation, so a single
node the API declines kept every other node out of the load balancer.
Log the rejected target and carry on with the rest.

Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Following externalTrafficPolicy made every node a target under the
Cluster policy, including nodes a cluster marks as unfit for external
load balancers: kubeadm labels control-plane nodes with
node.kubernetes.io/exclude-from-external-load-balancers, and a cordoned
node is draining. Filter both out.

A service left without a single exposable port no longer gets an
external IP patched onto its status, so a load balancer that forwards
nothing is not reported as ready. Port mapping now carries the existing
LBService type instead of a bare pair of integers.

Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Tolerating a rejected target also hid the case where every target is
rejected: reconciliation reported success and the service was patched
with an external IP for a load balancer that had nowhere to forward to.
Fail instead when no target could be added at all, so the controller
requeues and the service stays unready.

Warn as well when more nodes are selected than the balancer type holds,
using the limit the API reports for that type.

Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Counting the attempts of a single run treated a permanently rejected
node as a total failure: once the acceptable targets were in place,
every following run retried only the rejected ones and reported failure
forever. Count the targets that are actually live instead, and carry the
reason of the last rejection into the error.

Targets are now planned up front: deduplicated, sorted, and trimmed to
what the balancer type holds, so a cluster larger than the limit stops
retrying doomed calls and keeps a stable set of targets. The limit is
not reported while a type change is still in flight, when the balancer
answers with the type it is leaving.

Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
A service without a single exposable port still got a Hetzner load
balancer created for it, charged by the hour, with no service behind it
and nothing but a log line to say so. Skip the balancer entirely in that
case, and drop the external IP the service advertises, so that clients
and ExternalDNS stop pointing at a balancer that no longer forwards.

Node selection moves behind a NodeSource decision that can be tested on
its own, not-ready nodes no longer take up target slots, and the
exclusion label now holds under every traffic policy, the way upstream
cloud providers treat it.

Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
…lancer

Assisted-By: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
@sircthulhu
sircthulhu force-pushed the fix/external-traffic-policy branch from ec039e0 to c37b2b6 Compare September 15, 2026 07:43
@sircthulhu
sircthulhu marked this pull request as ready for review September 15, 2026 07:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant