Skip to content

agent docs: instrument workloads on GitOps-managed clusters #51

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
amitava82 merged 1 commit into from
Sep 9, 2026
Merged

agent docs: instrument workloads on GitOps-managed clusters #51

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
203 changes: 203 additions & 0 deletions src/content/docs/docs/kloudmate-agent/auto-instrumentation/gitops-clusters.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,203 @@
---
title: "GitOps-managed clusters"
description: "Trace Kubernetes workloads that Argo CD or Flux reconciles, with eBPF, a namespace annotation, a committed annotation, or your own policy engine."
sidebar:
label: "GitOps clusters"
order: 11
---

import { LinkCard, CardGrid } from '@astrojs/starlight/components';

Argo CD and Flux reconcile a cluster back to what's in git. The KloudMate agent detects the workloads they own and doesn't write to them, so you can't turn on **SDK** tracing for them from KloudMate. You can still trace them. Start with eBPF mode and work down the list.

| Approach | Pods restart | Repo change | Argo CD or Flux change |
|---|---|---|---|
| [eBPF mode](#turn-on-ebpf-mode) | No | None | None |
| [Namespace annotation](#annotate-the-namespace) | Once | Usually none | None |
| [Committed annotation](#commit-the-annotation) | Once | One line per workload | None |
| [Agent patching](#allow-the-agent-to-patch-these-workloads) | Once | None | Two settings |
| [Policy engine](#use-your-own-policy-engine) | Once | A policy manifest | None |

## Why the agent doesn't patch these workloads

To turn on SDK tracing, the agent adds an OpenTelemetry Operator annotation to the workload's pod template. Your controller reverts anything that didn't come from git, and the agent would re-add it on the next check-in. Every revert and re-patch changes the pod template, so your pods would roll once a minute, and nothing would show up as an error.

The agent reads ownership from the markers each controller adds to the workload. Argo CD's tracking annotation, an Argo-prefixed instance label, and Flux's kustomize-controller and helm-controller labels all name their controller outright, so they count on any cluster.

Argo CD installs that use label-based resource tracking are the exception. They mark a workload with `app.kubernetes.io/instance` and nothing else, and plain Helm charts set that same label, so the agent counts it only on a cluster where Argo CD itself is running. That cuts both ways: on such a cluster, a workload you installed with Helm and never put under Argo CD carries the label too, and shows as Argo-managed. Any of the options below still instruments it.

:::note[When Argo CD runs outside the cluster]
A central Argo CD that manages this cluster from elsewhere installs nothing in it, so the agent can't tell that Argo CD is involved. With annotation tracking, detection still works, because the tracking annotation travels with each resource. With label tracking there's nothing for the agent to go on, so it keeps patching those workloads and rolling their pods.

Annotation tracking is the default from Argo CD 3.0. On an older instance, switch `application.resourceTrackingMethod` to `annotation`, migrating through `annotation+label`. Or use any of the options below, none of which depend on detection.
:::

## Check whether a workload is externally managed

Open the cluster's agent settings in KloudMate and find **Application instrumentation**. Workloads that Argo CD or Flux owns show as managed by that controller. The rest of the list behaves normally, so a cluster running Argo CD can still have workloads the agent instruments for you.

## Turn on eBPF mode

Set the workload to **eBPF** in **Application instrumentation** and apply. The change takes effect on the next agent check-in. The agent writes nothing to your cluster objects, your manifests don't change, and no pods restart, because eBPF attaches to processes that are already running.

You get request rate, errors, duration, and trace spans. What you don't get is the in-process detail an SDK adds, such as spans for your database queries, framework internals, and outbound HTTP calls. For Go, Ruby, and PHP, eBPF is the only option on Kubernetes, whatever manages the workload, because the OpenTelemetry Operator has no injector for those runtimes. Those workloads aren't traced until you select eBPF for each one.

## Annotate the namespace

The OpenTelemetry Operator reads the inject annotation from the Namespace object as well as from a pod template, so one annotation covers every pod in the namespace:

```bash
kubectl annotate namespace payments \
instrumentation.opentelemetry.io/inject-java=km-agent/km-agent-instrumentation-crd
```

Swap `inject-java` for the runtime you need: `inject-nodejs`, `inject-python`, or `inject-dotnet`. In a manifest, the same annotation looks like this:

```yaml
apiVersion: v1
kind: Namespace
metadata:
name: payments
annotations:
instrumentation.opentelemetry.io/inject-java: km-agent/km-agent-instrumentation-crd
```

Under Argo CD, the namespace is often created by the `CreateNamespace=true` sync option rather than declared in the app repo. If yours is created that way, the Namespace object isn't part of any Application's desired state, so annotating it doesn't count as drift and nothing reverts it.

The annotation doesn't change running pods. New pods get the injection, so delete the running ones or wait for the next deploy. Deleting pods doesn't create drift, because your controller reconciles the Deployment, not the pods it creates.

Know the limits before you use this:

- **Every pod in the namespace gets injected**, including short-lived jobs.
- **One runtime per namespace.** The annotation names a single language. If a namespace holds a Java service and a Python service, this covers only one of them.
- **.NET runtime options come from the pod only.** If a .NET workload needs one, put it on that workload's pod template.

## Commit the annotation

Add the annotation to the workload's pod template in your repo, where it goes through review like any other change:

```yaml
spec:
template:
metadata:
annotations:
instrumentation.opentelemetry.io/inject-java: km-agent/km-agent-instrumentation-crd
```

Your controller applies it on the next sync, and the pods roll once. Argo CD and Flux need no extra configuration, because the annotation is now part of the desired state rather than a difference from it.

Set that workload to **SDK** in **Application instrumentation** as well. The agent still won't write to it. The selection keeps the workload tracked in KloudMate, and once your controller syncs the annotation the workload reports as **Instrumented**.

## Allow the agent to patch these workloads

If you'd rather keep the per-workload toggle in KloudMate, configure Argo CD to accept the agent's annotation, then turn the agent's patching back on. You need both of the Argo CD settings below.

First, ignore the inject annotations in the `argocd-cm` ConfigMap:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cm
namespace: argocd
data:
resource.customizations.ignoreDifferences.apps_Deployment: |
jsonPointers:
- /spec/template/metadata/annotations/instrumentation.opentelemetry.io~1inject-java
- /spec/template/metadata/annotations/instrumentation.opentelemetry.io~1inject-nodejs
- /spec/template/metadata/annotations/instrumentation.opentelemetry.io~1inject-python
- /spec/template/metadata/annotations/instrumentation.opentelemetry.io~1inject-dotnet
```

Inside a JSON pointer, `~1` stands for the `/` in the annotation key. Keep only the runtimes you instrument.

Second, add the sync option to every Application whose workloads you instrument:

```yaml
spec:
syncPolicy:
syncOptions:
- RespectIgnoreDifferences=true
```

:::caution
`ignoreDifferences` on its own only changes what Argo CD reports as drift. Self-heal still reverts the annotation, so the pods keep rolling while the Argo CD UI shows the Application as synced. `RespectIgnoreDifferences=true` keeps the field out of the sync itself.
:::

With both in place, turn the agent's patching back on with the `patchExternallyManagedWorkloads` Helm value:

```bash
helm upgrade kloudmate-release kloudmate/km-kube-agent \
--namespace km-agent --reuse-values \
--set patchExternallyManagedWorkloads=true
```

In the fleet-manager pod spec, this shows up as `KM_K8S_APM_PATCH_MANAGED=true`. It's a cluster-wide switch, so any Application left without the exception above reverts the annotation again and rolls its pods. Those workloads also stop showing as managed by Argo CD in **Application instrumentation**, and go back to reporting the usual instrumentation status.

These steps are specific to Argo CD. On Flux, exclude the same annotation from drift correction before you set the variable, or Flux reverts the agent's patches again.

## Use your own policy engine

If you already run Kyverno, have it add the annotation to pods at admission:

```yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: kloudmate-instrumentation
spec:
rules:
- name: annotate-java-pods
match:
any:
- resources:
kinds:
- Pod
namespaces:
- payments
mutate:
patchStrategicMerge:
metadata:
annotations:
instrumentation.opentelemetry.io/inject-java: km-agent/km-agent-instrumentation-crd
```

Argo CD and Flux compare Deployments, not the pods those Deployments create, so a policy that annotates pods never shows up as drift. The OpenTelemetry documentation suggests the same pattern, and KloudMate needs no configuration for it. As with the namespace annotation, running pods pick it up when they're recreated.

## Install the agent on a GitOps cluster

`helm install` is the tested path, and it keeps the agent-update button in the KloudMate dashboard working. See the [Kubernetes installation guide](../../installation/kubernetes-agent/).

To manage the agent through Argo CD instead, its Application needs these settings:

- **`ignoreDifferences` on the agent's own ConfigMaps.** The agent writes each collector's live configuration into the `/data/agent-daemonset.yaml` and `/data/agent-deployment.yaml` keys of `km-agent-configmap-daemonset` and `km-agent-configmap-deployment`. Argo CD renders the chart without reading the cluster, so every sync would otherwise push those collectors back to the configuration the chart ships.
- **The destination namespace set to `km-agent`.** The chart hardcodes that namespace on the Instrumentation resource that the inject annotation points at (`km-agent/km-agent-instrumentation-crd`).
- **`SkipDryRunOnMissingResource=true`** in the sync options. On the first sync, Argo CD applies the Instrumentation resource in the same wave that installs the OpenTelemetry Operator's CRDs, and the dry run fails without it.

```yaml
spec:
destination:
namespace: km-agent
syncPolicy:
syncOptions:
- SkipDryRunOnMissingResource=true
ignoreDifferences:
- group: ""
kind: ConfigMap
name: km-agent-configmap-daemonset
jsonPointers:
- /data/agent-daemonset.yaml
- group: ""
kind: ConfigMap
name: km-agent-configmap-deployment
jsonPointers:
- /data/agent-deployment.yaml
```

## Next steps

<CardGrid>
<LinkCard title="Kubernetes setup" href="../kubernetes/" description="The OpenTelemetry Operator, monitored namespaces, and the Off, eBPF, and SDK modes." />
<LinkCard title="Application APM overview" href="../overview/" description="The per-workload opt-in flow and status chips." />
<LinkCard title="Install on Kubernetes" href="../../installation/kubernetes-agent/" description="Helm installation and values." />
</CardGrid>
Loading