How Can Prometheus Metrics Be Used to Monitor Pod CPU Usage Effectively?
In the dynamic world of containerized applications, understanding and monitoring resource consumption is crucial for maintaining performance and reliability. Among the various metrics that developers and operators track, CPU usage stands out as a vital indicator of a pod’s health and efficiency within a Kubernetes cluster. Leveraging Prometheus metrics for pod CPU usage provides a powerful lens through which teams can gain real-time insights, optimize workloads, and preemptively address potential bottlenecks.
Prometheus, as a leading open-source monitoring and alerting toolkit, offers a rich ecosystem for collecting and querying metrics from Kubernetes environments. When it comes to pods—the smallest deployable units in Kubernetes—tracking CPU usage through Prometheus metrics enables granular visibility into how resources are allocated and consumed. This not only aids in troubleshooting performance issues but also informs scaling decisions and cost management strategies.
Understanding the nuances of Prometheus metrics related to pod CPU usage opens the door to more effective cluster management and application tuning. By exploring how these metrics are gathered, interpreted, and utilized, readers can better harness Prometheus to maintain optimal pod performance and ensure their Kubernetes workloads run smoothly under varying demands.
Common Prometheus Metrics for Monitoring Pod CPU Usage
Prometheus collects a variety of metrics related to CPU usage in Kubernetes pods, primarily through the cAdvisor component integrated into the kubelet. These metrics provide insights into the CPU consumption patterns and resource allocation of pods, containers, and nodes. Understanding these metrics is crucial for effective monitoring and troubleshooting.
Key metrics used for pod CPU usage include:
- container_cpu_usage_seconds_total: This is a cumulative counter metric that tracks the total CPU time consumed by a container, measured in seconds. It increases monotonically and can be used to calculate CPU usage rates over time.
- container_cpu_user_seconds_total: CPU time consumed in user space.
- container_cpu_system_seconds_total: CPU time consumed in kernel space.
- container_cpu_cfs_throttled_seconds_total: Time during which the container’s CPU usage was throttled due to CFS (Completely Fair Scheduler) limits.
- container_cpu_cfs_periods_total and container_cpu_cfs_throttled_periods_total: These counters indicate the number of CFS periods and how many of those periods were throttled, useful for identifying CPU throttling events.
These container-level metrics are often prefixed with labels such as `namespace`, `pod`, `container`, and `instance`, which help in filtering and aggregating data specific to pods or containers.
Calculating CPU Usage Percentage from Prometheus Metrics
Raw CPU time metrics are not directly interpretable as CPU usage percentages. To derive meaningful CPU usage values, it’s necessary to calculate rates over time and normalize by the number of CPU cores available to the pod or node.
The typical approach involves:
- Using the `rate()` or `irate()` function in Prometheus to calculate the per-second increase of the cumulative CPU usage counter.
- Dividing this rate by the number of CPU cores allocated or available to the pod to get a usage ratio.
- Multiplying by 100 to express the ratio as a percentage.
A common query to calculate the CPU usage percentage for a pod might look like this:
“`promql
sum by (pod) (
rate(container_cpu_usage_seconds_total{namespace=”your-namespace”, pod=~”your-pod-regex”}[5m])
)
/ sum by (pod) (kube_pod_container_resource_limits_cpu_cores{namespace=”your-namespace”, pod=~”your-pod-regex”})
- 100
“`
This query sums the CPU usage rate for all containers within a pod, then divides it by the CPU core limits assigned to those containers, yielding the percentage of CPU usage relative to the pod’s limits.
Labels and Their Importance in Prometheus Queries
Labels in Prometheus metrics are essential for filtering and grouping data. When monitoring pod CPU usage, common labels include:
- `namespace`: Indicates the Kubernetes namespace of the pod.
- `pod`: The name of the pod.
- `container`: The name of the container within the pod.
- `instance`: The node or host where the pod is running.
- `cpu`: CPU core identifier (less commonly used at the pod level).
Using these labels appropriately allows you to:
- Isolate metrics for specific pods or namespaces.
- Aggregate CPU usage at container, pod, or node levels.
- Correlate CPU usage with other metrics such as memory, network, or throttling.
Example Table of Prometheus Metrics for Pod CPU Monitoring
| Metric Name | Description | Unit | Typical Labels | Use Case |
|---|---|---|---|---|
| container_cpu_usage_seconds_total | Total CPU time consumed by a container | Seconds (cumulative) | namespace, pod, container | Calculate CPU usage rates over time |
| container_cpu_user_seconds_total | CPU time spent in user space | Seconds (cumulative) | namespace, pod, container | Analyze user-level CPU consumption |
| container_cpu_system_seconds_total | CPU time spent in kernel space | Seconds (cumulative) | namespace, pod, container | Analyze system-level CPU consumption |
| container_cpu_cfs_throttled_seconds_total | Time the CPU was throttled due to CFS limits | Seconds (cumulative) | namespace, pod, container | Identify CPU throttling and resource contention |
| kube_pod_container_resource_limits_cpu_cores | CPU core limits assigned to containers | Cores | namespace, pod, container | Normalize CPU usage against limits |
Understanding Prometheus Metrics for Pod CPU Usage
Prometheus collects time-series data by scraping metrics endpoints exposed by various components in a Kubernetes cluster. When monitoring CPU usage of pods, Prometheus relies primarily on metrics provided by the kubelet and cAdvisor, which are exposed through the Kubernetes metrics API or directly on node exporters.
Key Prometheus Metrics for Pod CPU Usage
The most relevant metrics for pod CPU usage typically include the following:
container_cpu_usage_seconds_total: Cumulative CPU time consumed by a container in seconds.container_spec_cpu_quotaandcontainer_spec_cpu_period: CPU quota and period settings used to calculate CPU limits for containers.container_cpu_user_seconds_totalandcontainer_cpu_system_seconds_total: CPU time consumed in user mode and system mode respectively.container_cpu_cfs_throttled_seconds_total: Total time the container was throttled due to CPU limits.
These metrics are usually labeled with Kubernetes-specific labels such as `pod`, `namespace`, `container`, and `node`, allowing fine-grained filtering and aggregation.
Calculating Instantaneous CPU Usage
Since `container_cpu_usage_seconds_total` is a cumulative counter, deriving the actual CPU usage rate requires calculating the rate of change over time. Prometheus provides the `rate()` and `irate()` functions for this purpose.
Example query to calculate CPU usage (in cores) per pod:
“`promql
sum by (namespace, pod) (
rate(container_cpu_usage_seconds_total{image!=””,container!=”POD”}[5m])
)
“`
**Explanation:**
- The filter `image!=””` excludes infrastructure containers without an image.
- The filter `container!=”POD”` excludes the pause container that Kubernetes uses for pod network namespaces.
- The `rate()` function calculates the per-second average rate of increase over the last 5 minutes.
- Summing by `namespace` and `pod` aggregates CPU usage across all containers in the pod.
CPU Usage Relative to Pod Limits
To measure the CPU usage relative to the pod or container CPU limits, you can combine usage and quota metrics:
| Metric | Description |
|---|---|
| `container_cpu_usage_seconds_total` | CPU time consumed (usage) |
| `container_spec_cpu_quota` | CPU time quota (in microseconds) |
| `container_spec_cpu_period` | Period of CPU quota (in microseconds) |
CPU limit in cores is computed as:
“`
CPU Limit = container_spec_cpu_quota / container_spec_cpu_period
“`
Example query to calculate CPU usage as a percentage of the CPU limit:
“`promql
sum by (namespace, pod) (
rate(container_cpu_usage_seconds_total{image!=””,container!=”POD”}[5m])
)
/
sum by (namespace, pod) (
container_spec_cpu_quota{container!=”POD”} / container_spec_cpu_period{container!=”POD”}
)
“`
This expression provides the fraction of requested CPU used by the pod relative to its configured CPU limit.
Using Metrics from kube_pod_container_resource_limits
In some Kubernetes setups, metrics about resource limits and requests are exposed via the `kube-state-metrics` component, providing richer metadata on pod resource configurations.
Metrics include:
- `kube_pod_container_resource_limits_cpu_cores` – CPU limits in cores.
- `kube_pod_container_resource_requests_cpu_cores` – CPU requests in cores.
Example query comparing usage to requests:
“`promql
sum by (namespace, pod) (
rate(container_cpu_usage_seconds_total{image!=””,container!=”POD”}[5m])
)
/
sum by (namespace, pod) (
kube_pod_container_resource_requests_cpu_cores
)
“`
This helps identify pods exceeding their requested CPU resources or underutilizing them.
Considerations for Accurate CPU Monitoring
– **Container Filters:** Always exclude infrastructure containers (e.g., pause containers) to prevent skewed CPU usage metrics.
– **Scrape Interval:** Ensure Prometheus scrape interval is frequent enough (e.g., 15s or 30s) to capture granular CPU usage fluctuations.
– **Duration Window:** The `rate()` function’s time window (e.g., `[5m]`) should balance between smoothing spikes and reflecting current usage.
– **Throttling Metrics:** Monitor CPU throttling using `container_cpu_cfs_throttled_seconds_total` to detect if pods are being limited by CPU quotas.
– **Node vs Pod Metrics:** Node-level metrics can provide cluster-wide CPU utilization context but are less granular for pod-level troubleshooting.
Sample Dashboard Metrics Breakdown
| Metric Name | Query Example | Usage Description |
|---|---|---|
| Pod CPU Usage (cores) | `sum by(pod) (rate(container_cpu_usage_seconds_total[5m]))` | Instantaneous CPU cores consumed per pod |
| Pod CPU Limit (cores) | `sum by(pod) (container_spec_cpu_quota / container_spec_cpu_period)` | CPU cores allocated per pod |
| CPU Usage % of Limit | Usage / Limit (as above) | Percentage of CPU limit currently used |
| CPU Throttling Duration | `rate(container_cpu_cfs_throttled_seconds_total[5m])` | Duration of CPU throttling per pod |
| CPU Requests (cores) | `sum by(pod) (kube_pod_container_resource_requests_cpu_cores)` | Requested CPU cores per pod |
These metrics form the foundation for effective CPU monitoring and alerting on Kubernetes pods using Prometheus.