fix(nodes): route load balancer traffic according to externalTrafficPolicy - #31
Open
sircthulhu wants to merge 10 commits into
Open
sircthulhu wants to merge 10 commits into
sircthulhu wants to merge 10 commits into
Conversation
Target nodes were always restricted to the nodes running the service's pods, which is the behaviour of externalTrafficPolicy: Local. Services using the default Cluster policy lost every node that kube-proxy could have forwarded from, so a service whose only pod sits on a node where its nodePort is unreachable ends up with no healthy target at all. Follow the policy declared by the service instead, and cover the port mapping and the policy check with unit tests. Assisted-By: LLM Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Hetzner refuses IP targets outside the vSwitch subnet of the attached network. Such a rejection aborted the whole reconciliation, so a single node the API declines kept every other node out of the load balancer. Log the rejected target and carry on with the rest. Assisted-By: LLM Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Assisted-By: LLM Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Following externalTrafficPolicy made every node a target under the Cluster policy, including nodes a cluster marks as unfit for external load balancers: kubeadm labels control-plane nodes with node.kubernetes.io/exclude-from-external-load-balancers, and a cordoned node is draining. Filter both out. A service left without a single exposable port no longer gets an external IP patched onto its status, so a load balancer that forwards nothing is not reported as ready. Port mapping now carries the existing LBService type instead of a bare pair of integers. Assisted-By: LLM Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Tolerating a rejected target also hid the case where every target is rejected: reconciliation reported success and the service was patched with an external IP for a load balancer that had nowhere to forward to. Fail instead when no target could be added at all, so the controller requeues and the service stays unready. Warn as well when more nodes are selected than the balancer type holds, using the limit the API reports for that type. Assisted-By: LLM Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Assisted-By: LLM Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Assisted-By: LLM Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
Counting the attempts of a single run treated a permanently rejected node as a total failure: once the acceptable targets were in place, every following run retried only the rejected ones and reported failure forever. Count the targets that are actually live instead, and carry the reason of the last rejection into the error. Targets are now planned up front: deduplicated, sorted, and trimmed to what the balancer type holds, so a cluster larger than the limit stops retrying doomed calls and keeps a stable set of targets. The limit is not reported while a type change is still in flight, when the balancer answers with the type it is leaving. Assisted-By: LLM Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
A service without a single exposable port still got a Hetzner load balancer created for it, charged by the hour, with no service behind it and nothing but a log line to say so. Skip the balancer entirely in that case, and drop the external IP the service advertises, so that clients and ExternalDNS stop pointing at a balancer that no longer forwards. Node selection moves behind a NodeSource decision that can be tested on its own, not-ready nodes no longer take up target slots, and the exclusion label now holds under every traffic policy, the way upstream cloud providers treat it. Assisted-By: LLM Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
…lancer Assisted-By: LLM Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
sircthulhu
force-pushed
the
fix/external-traffic-policy
branch
from
September 15, 2026 07:43
ec039e0 to
c37b2b6
Compare
sircthulhu
marked this pull request as ready for review
September 15, 2026 07:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
RobotLB always picks target nodes from where the service's pods run, which is the
behaviour of
externalTrafficPolicy: Local. Services using the defaultClusterpolicy therefore lose every node that kube-proxy could have forwarded from.
This breaks a service whose only pod runs on a node where its own nodePort is not
reachable. A KubeVirt VM behind a
LoadBalancerservice hits exactly this: the VM'snodePort answers on every node except the one hosting the VM, and since that node was
the load balancer's only target, the target stayed unhealthy and every connection was
accepted and immediately closed. HTTP services on the same cluster kept working only
because their deployments have replicas on several nodes.
Changes
Node selection
externalTrafficPolicy.Cluster, the Kubernetesdefault, uses every node of the cluster;
Localkeeps the previous pod-baseddiscovery.
ROBOTLB_DYNAMIC_NODE_SELECTOR=falsestill selects nodes by therobotlb/node-selectorannotation.node.kubernetes.io/exclude-from-external-load-balancers, whichkubeadm puts on control-plane nodes, stay out of every balancer under either policy.
Cordoned and not-ready nodes are dropped under
Cluster, where they would only takeup target slots.
Targets
number the balancer type holds, so a cluster larger than the limit keeps a stable set
of targets instead of retrying calls the API will reject.
network, no longer aborts the reconciliation and no longer fails every later run.
Reconciliation fails only when the balancer is left without a single live target, and
the error carries the reason the API reported.
Services without a usable port
through its
nodePort. A service whose ports all lack one no longer gets a balancercreated for it — it used to get an empty one, billed by the hour — and a service that
already advertises an external IP loses it, so nothing keeps sending traffic to a
balancer that no longer forwards.
Tests
cargo testcovers target planning, node selection, node eligibility and portmapping. The repository had no test module before, and
cargo testis now part of thepre-commit hooks the CI workflow runs.
Upgrade impact
Clusteris the Kubernetes default, so services that never setexternalTrafficPolicyget the full node list on upgrade, which rewrites the targets of their existing
balancers.