How to monitor Go applications in Kubernetes
Monitoring Go applications in Kubernetes enables you to see how your services are performing without modifying your code. With the Site24x7 APM Go agent, you can track Go routines, traces and database queries. This document guides you through deploying the agent and starting monitoring with minimal overhead.
Prerequisites
Before deploying the monitoring solution, ensure the following prerequisites are met:
- Kubernetes: Version 1.19 or later
- Linux kernel: Version 5.8 or later, required for eBPF CO-RE support
- kubectl: Configured to access your Kubernetes cluster
- Site24x7 account: Active account with a valid APM license key
- Go application: Go 1.18 or higher is recommended
- Cluster permissions: Ability to create DaemonSets, ClusterRoles, and privileged pods
Application pod
Your application pod must meet several specific requirements to ensure proper operation. First, a hostPath volume must be created on the host node with a unique path. Next, an initContainer should copy the application binary from the container image into this hostPath. The copied binary must have executable permissions.
Finally, the primary container should run the application binary directly from the hostPath volume rather than from within the container image itself. This setup ensures that the application is accessible and executable on the host as required.
APM agent DaemonSet
The Site24x7 APM Agent DaemonSet should meet the following requirements:
- Privileged access or full capability set
- runAsUser: 0 to execute as root
- hostPID: true for access to node-level process information
- All mandatory volume mounts are present
- DaemonSet is deployed on every node
Application deployment configuration
Step 1: Configure hostPath volume
A hostPath volume is mandatory to ensure stable file metadata (device and inode numbers) required for eBPF uprobes.
volumes:
- name: app-files
hostPath:
path: /var/lib/apm-binaries/my-app
type: DirectoryOrCreate
Step 2: Add initContainer to copy-binary
This step is mandatory. The initContainer copies the application binary to the hostPath before the main container starts. This ensures the binary is accessible outside container overlay layers, allowing eBPF to instrument the process reliably.
initContainers:
- name: copy-binary
image: your-app-image:latest
command: ["/bin/sh", "-c"]
args:
- |
mkdir -p /host/bin
cp /app/bin/my-app /host/bin/my-app
chmod +x /host/bin/my-app
volumeMounts:
- name: app-files
mountPath: /host
APM agent DaemonSet configuration
The APM Agent requires elevated privileges and specific volume mounts for eBPF instrumentation.
Step 1: Privileged access
This step is mandatory. This allows the agent to discover all processes and attach eBPF uprobes to them.
securityContext:
runAsUser: 0
privileged: true
Step 2: Host PID namespace
This step is required to discover all processes running on the node and to enable PID-based uprobe attachment.
hostPID: true
Step 3: Mandatory volume mounts
The DaemonSet must mount the following directories from the host:
| Mount path | Purpose |
| /host/proc | Provides access to process information and metadata |
| /sys/kernel/debug | Required for accessing uprobe interfaces and kernel debugging |
| /sys/fs/bpf | Used to store eBPF maps and programs |
|
/sys |
Enables detection of cgroups and container boundaries |
| /var/lib/apm-binaries | Provides access to shared application binaries |
volumeMounts:
- name: host-proc # Process visibility
mountPath: /host/proc
- name: sys-kernel-debug # Uprobe interface
mountPath: /sys/kernel/debug
- name: sys-fs-bpf # eBPF map/program pinning
mountPath: /sys/fs/bpf
- name: sys # Cgroup & namespace discovery
mountPath: /sys
- name: shared-app-binaries # Shared hostPath binaries
mountPath: /var/lib/apm-binaries
Example for deploying and monitoring a Go application in Kubernetes
This example demonstrates how to update your existing Go application deployment to work with Site24x7 APM Insight using eBPF instrumentation.
Step 1: Update your existing application deployment
To enable Go application monitoring, start by updating your existing application deployment. Use an initContainer to copy the Go binary to a hostPath volume on the node before the main container starts. This hostPath volume acts as a shared directory on the node, such as /var/lib/apm-binaries.
Finally, configure the main container to run the Go binary directly from this hostPath location, ensuring the application executes the binary from the host.
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-go-app
namespace: production
spec:
replicas: 3
selector:
matchLabels:
app: my-go-app
template:
metadata:
labels:
app: my-go-app
spec:
# InitContainer: Copy binary to hostPath
initContainers:
- name: copy-binary-to-host
image: your-registry/my-go-app:latest
imagePullPolicy: Always
command: ["/bin/sh", "-c"]
args:
- |
echo "Copying binary to hostPath for eBPF instrumentation..."
# Create directory structure on host
mkdir -p /host-binaries/bin
mkdir -p /host-binaries/conf
mkdir -p /host-binaries/logs
# Copy binary with execute permissions
cp /app/bin/my-go-app /host-binaries/bin/my-go-app
chmod +x /host-binaries/bin/my-go-app
# Copy config files if they exist
if [ -d "/app/conf" ]; then
cp -r /app/conf/* /host-binaries/conf/ 2>/dev/null || true
fi
echo "Binary copy complete"
ls -lh /host-binaries/bin/my-go-app
volumeMounts:
- name: app-files
mountPath: /host-binaries
containers:
- name: my-go-app
image: your-registry/my-go-app:latest
imagePullPolicy: Always
# IMPORTANT: Override command to run from hostPath
command: ["/app/bin/my-go-app"]
ports:
- containerPort: 8080
name: http
# Mount the same hostPath volume
volumeMounts:
- name: app-files
mountPath: /app
env:
- name: APP_PORT
value: "8080"
# ... other env vars
resources:
requests:
memory: "256Mi"
cpu: "200m"
limits:
memory: "512Mi"
cpu: "500m"
volumes:
# hostPath volume - CRITICAL for eBPF compatibility
- name: app-files
hostPath:
path: /var/lib/apm-binaries # Shared directory on node
type: DirectoryOrCreate
Key configuration points
- Use an initContainer to copy the application binary into a hostPath directory before the main container starts.
- Store the binary in a hostPath volume (for example, /var/lib/apm-binaries) to maintain stable inode and device metadata outside OverlayFS.
- Configure the application container to run the binary directly from the hostPath location using the appropriate command.
- Ensure the initContainer sets executable permissions (chmod +x) on the copied binary so it can run successfully.
Step 2: Create configuration resources
This step includes three sub-tasks to set up the necessary Kubernetes resources for monitoring your Go applications.
2.1 Create a namespace for monitoring
To create a namespace for monitoring, define a YAML file with the apiVersion, kind, and metadata, including the namespace name and relevant labels, as shown below.
apiVersion: v1
kind: Namespace
metadata:
name: monitoring
labels:
name: monitoring
purpose: apm-infrastructure
Apply the namespace by running the following command:
kubectl apply -f monitoring-namespace.yaml
2.2 Create a secret for the license key
To create a secret for your Site24x7 license key, use the following command, replacing your_actual_license_key_here with your actual license key.
# Replace with your actual Site24x7 license key
kubectl create secret generic apm-license-key \
--from-literal=license-key='your_actual_license_key_here' \
-n monitoring
After creating the secret, verify that it has been successfully created by running the following command, which will display the secret in the monitoring namespace:
kubectl get secret apm-license-key -n monitoring
2.3 Create a ConfigMap for application configuration
The GO_APPS environment variable specifies which Go applications the agent should monitor. To set this up, create a ConfigMap that lists your applications along with their exact process names and the ports they listen on using the format and components described below.
Format
AppName=ProcessName:Port1,Port2,Port3;AnotherApp=ProcessName:Port;
Components
The key components for configuring your Go application include:
- AppName: A name that will appear on the Site24x7 dashboard.
- ProcessName: This must match the exact name of your Go binary (for example, my-go-app or server).
- Port(s): The ports that your application listens on, using a comma-separated list if there are multiple ports.
apiVersion: v1
kind: ConfigMap
metadata:
name: apm-apps-config
namespace: monitoring
data:
# Single application with one port
go_apps: "MyGoApp=my-go-app:8080"
# Multiple applications
# go_apps: "WebAPI=webapi:8080,8443;AdminAPI=adminapi:9000;Worker=worker:8081"
# Application with multiple ports
# go_apps: "MyComplexApp=myapp:8080,8443,9090,50051"
Apply the ConfigMap by running the following command:
kubectl apply -f apm-apps-config.yaml
Examples
- For a single application running on a single port:
MyApp=myapp:8080 - For a single application running on multiple ports:
MyApp=myapp:8080,8443,9090 - For multiple applications running on different ports:
API=api:8080; Worker=worker:8081; Admin=admin:9000
Step 3: Deploy the Site24x7 APM agent DaemonSet
3.1 Create DaemonSet
To deploy the Site24x7 APM agent across all nodes in your cluster, create a DaemonSet that ensures the agent runs on every node. The DaemonSet configuration includes necessary privileges, hostPath mounts, and environment variables required for monitoring your Go applications.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: exporter-agent
namespace: monitoring
labels:
app: exporter-agent
component: monitoring
spec:
selector:
matchLabels:
app: exporter-agent
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
template:
metadata:
labels:
app: exporter-agent
component: monitoring
spec:
serviceAccountName: exporter-agent-sa
# CRITICAL: Host namespaces for eBPF monitoring
hostPID: true # REQUIRED: See all processes on node
containers:
- name: exporter-agent
image: site24x7/apminsight-go-agent:latest
imagePullPolicy: IfNotPresent
# Environment variables from Secret and ConfigMap
env:
- name: S247_LICENSE_KEY
valueFrom:
secretKeyRef:
name: apm-license-key
key: license-key
- name: GO_APPS
valueFrom:
configMapKeyRef:
name: apm-apps-config
key: go_apps
# CRITICAL: Security context with elevated privileges
securityContext:
runAsUser: 0 # Must run as root
privileged: true # REQUIRED for eBPF
# Resource limits
resources:
requests:
cpu: 200m
memory: 256Mi
limits:
cpu: 1000m
memory: 1Gi
# CRITICAL: Volume mounts for eBPF access
volumeMounts:
# Host process information
- name: host-proc
mountPath: /host/proc
readOnly: true
# Kernel debug filesystem for eBPF
- name: sys-kernel-debug
mountPath: /sys/kernel/debug
readOnly: false
# BPF filesystem
- name: sys-fs-bpf
mountPath: /sys/fs/bpf
readOnly: false
# System filesystem (for cgroup detection)
- name: sys
mountPath: /sys
readOnly: false
# CRITICAL: Shared binaries from hostPath
- name: shared-app-binaries
mountPath: /var/lib/apm-binaries
readOnly: true
# CRITICAL: Volumes for eBPF and binary access
volumes:
- name: host-proc
hostPath:
path: /proc
type: Directory
- name: sys-kernel-debug
hostPath:
path: /sys/kernel/debug
type: Directory
- name: sys-fs-bpf
hostPath:
path: /sys/fs/bpf
type: DirectoryOrCreate
- name: sys
hostPath:
path: /sys
type: Directory
# CRITICAL: Same hostPath as application pods
- name: shared-app-binaries
hostPath:
path: /var/lib/apm-binaries
type: DirectoryOrCreate
Key DaemonSet configuration
- hostPID: true — Provides visibility into all processes running on the node, enabling accurate pod discovery
- privileged: true — Required because eBPF operations need elevated kernel capabilities
- runAsUser: 0 — Runs the agent as root to perform eBPF-related syscalls successfully
- /proc:/host/proc (hostPath) — Allows process discovery, metadata access, and namespace resolution
- /sys/kernel/debug (hostPath) — Required for eBPF debugging, probe loading, and managing uprobe/kprobe attachments
- /sys/fs/bpf (hostPath) — Provides access to the BPF filesystem where maps and programs are pinned
- /var/lib/apm-binaries (hostPath) — Shared storage for application binaries to ensure stable inode and device metadata across pods
Step 4: Deploy and verify
4.1 Apply all resources
First, create the monitoring namespace by applying the namespace YAML file:
kubectl apply -f monitoring-namespace.yaml
Next, create a secret for your Site24x7 license key and apply the ConfigMap for application configuration:
kubectl create secret generic apm-license-key \
--from-literal=license-key='YOUR_SITE24X7_LICENSE_KEY' \
-n monitoringkubectl apply -f apm-apps-config.yaml
Then, deploy the Site24x7 APM agent DaemonSet:
kubectl apply -f exporter-agent-daemonset.yaml
Finally, deploy your Go application using the updated deployment YAML:
kubectl apply -f my-app-deployment.yaml
4.2 Verify DaemonSet deployment
To check the status of the DaemonSet and ensure it is running on all nodes, run:
kubectl get daemonset -n monitoring
Next, verify that a pod is running on each node:
kubectl get pods -n monitoring -l app=exporter-agent -o wide
You can also view the logs of the DaemonSet pods to check for any errors or issues:
kubectl logs -n monitoring -l app=exporter-agent --tail=100 -f
4.3 Verify eBPF instrumentation
To confirm that eBPF probes are loaded correctly, check the instrumentor logs:
kubectl exec -n monitoring exporter-agent-xxxxx -- \
ls -lt /opt/site24x7/apm-insight-go-agent/instrumentor/logs/
You can view the output of the instrumentor to see which probes have been loaded and whether telemetry is being exported:
kubectl exec -n monitoring exporter-agent-xxxxx -- \
tail -f /opt/site24x7/apm-insight-go-agent/instrumentor/logs/MyGoApp-k8s-*/agent.txt
Expected output:
Probe loaded successfully: net/http/client
Probe loaded successfully: database/sql/client
Exporting 10 spans to the server
This output confirms that your Go application is successfully instrumented and monitored by the Site24x7 APM Go agent.
Related articles