Zero-Downtime O11y (eBPF)

Getting observability into a running system usually costs a deployment. You add an SDK, or attach an auto-instrumentation agent, and either way that means a code change and a restart. On the systems where restarts are most expensive, that cost is usually what keeps them uninstrumented.

The eBPF receiver avoids it. eBPF (Extended Berkeley Packet Filter) runs sandboxed programs inside the Linux kernel, so the KloudMate agent observes syscalls and network traffic from below your application rather than from inside it. Nothing links into your process, and nothing restarts.

What you get without touching the application

RED metrics

Request rate, error rate, and duration for every observed service. The agent parses HTTP/HTTP2, gRPC, MySQL, PostgreSQL, Redis, MongoDB, Kafka, and Elasticsearch, so the metrics carry real operational detail: HTTP status codes, gRPC status flags, and database operations, rather than opaque byte counts.

Distributed tracing

Incoming requests are correlated with the outgoing calls they cause, so an HTTP handler and the database query it issues land in the same trace. Spans follow OpenTelemetry semantic conventions, which is what lets the service graph build itself.

Service inventory and metadata

The agent identifies each process's runtime language (telemetry.sdk.language) without modifying the binary, and tags traces with host and process IDs plus cloud provider metadata. Under Kubernetes it adds namespace, pod_name, deployment, and node_name.

Network flows

L3/L4 flow metrics between endpoints: bytes transferred, TCP retransmits, connection state changes, and drops between isolated services. This is the service-to-service communication map, which is usually the first thing you want when an incident crosses a boundary you did not know existed.

eBPF compared with SDK instrumentation

What eBPF cannot do

eBPF sees syscalls and network boundaries, not the inside of your application. Two things follow from that.

Business logic stays out of reach. Attributes like user_id or cart_value live in application memory, never in a syscall, so eBPF has no way to read them.

You get boundaries, not stack traces. A request's entry and exit are visible; the execution time of individual functions within it is not.

How to deploy it

Start with eBPF. One agent per host covers every service on that host, in any language, including the ones nobody has gotten around to instrumenting. That gets you the service graph and baseline RED metrics without a deployment.

Then add SDKs where they earn their keep. For the handful of services that need business logic on the span, a user ID or a transaction state, manual instrumentation is still the only way to get it. KloudMate merges both sources, so an SDK-instrumented service sits in the same traces as its eBPF-observed neighbours.

Metrics reference

Nothing is exported unless the feature that owns it is listed under metrics.features.

Application RED: application

Add application_host to get the same RED metrics aggregated per host rather than per service, which is the view you want when asking whether a node is saturated rather than whether a service is slow.

Span and dependency metrics

The service graph metrics are what the topology view is drawn from. application_span and application_span_sizes still work and emit the older traces_spanmetrics_* names, but both are deprecated; application_span_otel matches the OpenTelemetry Collector's spanmetrics connector, so the same dashboards work against either source.

Runtime metrics: application_runtime

Per-process runtime internals, read without an SDK or an agent inside the process. Off by default.

Network flows

TCP stats

stats turns on all four. Be deliberate about stats_tcp_io: it fires on every tcp_sendmsg and tcp_cleanup_rbuf, where the other three fire only on close, failure, or retransmit. If event volume is a concern, enable the other three and opt into stats_tcp_io separately.

Migrating from an earlier agent

Three metrics changed name to follow the OpenTelemetry semantic conventions.

Configuration and setup

The agent runs as a DaemonSet in a Kubernetes cluster, or as a standalone process on a Linux host.

Prerequisites

• A Linux host with kernel >= 5.8 for full feature support. Some features work on 4.18+.
• Root privileges, or the CAP_SYS_ADMIN and CAP_BPF capabilities for the agent process.
• On Kubernetes, RBAC permissions to view Nodes, Pods, Services, and ReplicaSets, which is what the metadata enrichment reads.

Basic configuration

This enables the common features across a whole host or node, sending data to a local OpenTelemetry Collector or straight to the KloudMate backend. Add it to your receiver definition.

# ============================================================================
# METRICS & OBSERVABILITY FEATURES
# ============================================================================
metrics:
  features:
    - application              # Exports standard RED (Request rate, Error rate, Duration) metrics for HTTP, gRPC, DB, and Messaging requests at the service level.
    - application_host         # Exports RED metrics aggregated at the host/node level, providing a node-centric view of application performance.
    - network                  # Exports granular L3/L4 network flow metrics (TCP/UDP bytes, drops, retransmits) between connection endpoints.
    - network_inter_zone       # Aggregates network flow metrics specifically based on traffic crossing defined geographic or logical zones (CIDRs).
    - application_span_otel    # Exports OpenTelemetry span metrics (traces_span_metrics_*) derived from the generated spans.
    - application_service_graph # Exports specifically formatted dependency metrics (service_graph_request_total, etc.) used to build visual topological service maps.

# ============================================================================
# TARGET DISCOVERY
# ============================================================================
discovery:
  kubernetes:
    enable: true
  services:
    # Monitor everything on the node automatically
    - name: all-services
      namespace: default, kube-system, my-app
      open_ports: '80, 443, 8080, 8443, 5432, 3306, 6379, 9092, 27017'
      # k8s_pod_name: '.*'    # Optional: specifically match all Kubernetes pods

# ============================================================================
# NETWORK OBSERVABILITY
# ============================================================================
network:
  enable: true
  source: tc
  direction: both

# ============================================================================
# KUBERNETES ENRICHMENT
# ============================================================================
attributes:
  kubernetes:
    enable: true

# ============================================================================
# OPENTELEMETRY EXPORT SETTINGS
# ============================================================================

Fine-tuning

Cache sizes matter under load, and attributes.select controls exactly which attributes are exported per signal, which is the lever to pull when cardinality gets expensive.

ebpf:
  # Enable heuristic detection for internal DB protocols
  heuristic_sql_detect: true
  # Configure maximum DB operation caches (important for high load)
  mysql_prepared_statements_cache_size: 1024
  postgres_prepared_statements_cache_size: 1024

attributes:
  # Explicit filtering of the attributes appended to your telemetry.
  # For example, only send specific K8s tags on standard HTTP traffic:
  select:
    http.server.request.duration:
      include:
        - http.request.method
        - http.response.status_code
        - url.path
        - k8s.namespace
        - k8s.pod.name
        - service.name
    db.client.operation.duration:
      include:
        - db.operation.name
        - db.system.name
        - server.address
        - k8s.pod.name
        - service.name

Running the agent on Kubernetes

The official KloudMate Helm chart (see the installation guide) configures eBPF support with the volume mounts and permissions it needs, so there are no DaemonSets or mounts to write by hand. Set kubernetes.enable: true in the discovery block of your receivers::ebpfreceiver section and list the namespaces you want monitored.

Trace context propagation

Enable application_span under metrics.features, and set network.source to socket_filter.