Monitor GPU usage (NVIDIA GPUs) and GPU-related metrics in Grafana

✅ Overview of the Stack

🛠 Step-by-Step Setup

✅ 1. Install NVIDIA DCGM Exporter

The DCGM Exporter (Data Center GPU Manager) exposes GPU metrics in Prometheus format.

kubectl apply -f https://raw.githubusercontent.com/NVIDIA/dcgm-exporter/main/deployments/k8s/dcgm-exporter.yaml

Verify:

kubectl get pods -n gpu-operator
kubectl port-forward svc/dcgm-exporter 9400:9400 -n gpu-operator
curl http://localhost:9400/metrics

You should see Prometheus-format metrics like DCGM_FI_DEV_GPU_UTIL.

✅ 2. Ensure Prometheus is Installed and Scraping

If using Prometheus Operator (from Helm):

Add scrape config to ServiceMonitor

If using kube-prometheus-stack, this is automatic with correct labels.

To scrape manually, add this ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: dcgm-exporter
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: dcgm-exporter
  namespaceSelector:
    matchNames:
      - gpu-operator
  endpoints:
    - port: "metrics"
      interval: 30s

✅ 3. Install Grafana Dashboard

Use NVIDIA's official GPU dashboards:

• Open Grafana
• Go to Dashboards > Import
• Use one of the following dashboard IDs:

You can also customize and clone these dashboards.

✅ 4. Verify GPU Metrics in Prometheus

Run in Prometheus UI or via Grafana Explore:

DCGM_FI_DEV_GPU_UTIL
DCGM_FI_DEV_MEM_COPY_UTIL
DCGM_FI_DEV_FB_USED

These will show % utilization, memory used, etc.

🔒 Optional: Node Exporter GPU Plugin (Advanced)

If you want host-level detail alongside GPU, you can also use node_exporter with custom GPU scripts, but dcgm-exporter is the preferred method for Kubernetes.

✅ Summary

Would you like a Helm chart-based setup for Prometheus + Grafana + DCGM on EKS?