Overview
kube-state-metrics (KSM) is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. (See examples in the Metrics section below.) It is not focused on the health of the individual Kubernetes components, but rather on the health of the various objects inside, such as deployments, nodes and pods.
kube-state-metrics is about generating metrics from Kubernetes API objects without modification. This ensures that features provided by kube-state-metrics have the same grade of stability as the Kubernetes API objects themselves. In turn, this means that kube-state-metrics in certain situations may not show the exact same values as kubectl, as kubectl applies certain heuristics to display comprehensible messages. kube-state-metrics exposes raw data unmodified from the Kubernetes API, this way users have all the data they require and perform heuristics as they see fit.
The metrics are exported on the HTTP endpoint /metrics on the listening port
(default 8080). They are served as plaintext. They are designed to be consumed
either by Prometheus itself or by a scraper that is compatible with scraping a
Prometheus client endpoint. You can also open /metrics in a browser to see
the raw metrics. Note that the metrics exposed on the /metrics endpoint
reflect the current state of the Kubernetes cluster. When Kubernetes objects
are deleted they are no longer visible on the /metrics endpoint.
[!NOTE] This README is generated from a template. Please make your changes there and run
make generate-template.
Table of Contents
- Versioning
- Metrics Documentation
- Kube-state-metrics self metrics
- kube-state-metrics vs. metrics-server
- Scaling kube-state-metrics
- Setup
- Usage
Versioning
Kubernetes Version
kube-state-metrics uses client-go to talk with
Kubernetes clusters. The supported Kubernetes cluster version is determined by
client-go.
All additional compatibility is only best effort, or happens to still/already be supported.
Compatibility matrix
At most, 5 kube-state-metrics and 5 kubernetes releases will be recorded below. Generally, it is recommended to use the latest release of kube-state-metrics. If you run a very recent version of Kubernetes, you might want to use an unreleased version to have the full range of supported resources. If you run an older version of Kubernetes, you might need to run an older version in order to have full support for all resources. Be aware, that the maintainers will only support the latest release. Older versions might be supported by interested users of the community.
| kube-state-metrics | Kubernetes client-go Version |
|---|---|
| v2.16.0 | v1.32 |
| v2.17.0 | v1.33 |
| v2.18.0 | v1.34 |
| v2.19.0 | v1.35 |
| v2.20.0 | v1.36 |
| main | v1.36 |
Resource group version compatibility
Resources in Kubernetes can evolve, i.e., the group version for a resource may change from alpha to beta and finally GA in different Kubernetes versions. For now, kube-state-metrics will only use the oldest API available in the latest release.
Container Image
The latest container image can be found at:
registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0(arch:amd64,arm,arm64,ppc64leands390x)- Multi-architecture images
Metrics Documentation
Any resources and metrics based on alpha Kubernetes APIs are excluded from any stability guarantee, which may be changed at any given release.
See the docs directory for more information on the exposed metrics.
Custom Resource State Metrics
[!NOTE]
custom-resource-stateis feature-frozen in favor of resource-state-metrics. Onceresource-state-metricsis stable,custom-resource-statewill be deprecated.
Conflict resolution in label names
The *_labels family of metrics exposes Kubernetes labels as Prometheus labels.
As Kubernetes
is more liberal than
Prometheus
in terms of allowed characters in label names,
we automatically convert unsupported characters to underscores.
For example, app.kubernetes.io/name becomes label_app_kubernetes_io_name.
This conversion can create conflicts when multiple Kubernetes labels like
foo-bar and foo_bar would be converted to the same Prometheus label label_foo_bar.
Kube-state-metrics automatically adds a suffix _conflictN to resolve this conflict,
so it converts the above labels to
label_foo_bar_conflict1 and label_foo_bar_conflict2.
If you'd like to have more control over how this conflict is resolved, you might want to consider addressing this issue on a different level of the stack, e.g. by standardizing Kubernetes labels using an Admission Webhook that ensures that there are no possible conflicts.
ECMAScript regular expression support for allow and deny lists
Starting from #2616, kube-state-metrics supports ECMAScript's regexp for allow and deny lists. This was incorporated as a workaround for the limitations of the regexp package in Go, which does not support lookarounds due to their non-linear time complexity. Please note that while lookarounds are now supported for allow and deny lists, regular expressions' evaluation time is capped at a minute to prevent performance issues.
Kube-state-metrics self metrics
kube-state-metrics exposes its own general process metrics under --telemetry-host and --telemetry-port (default 8081).
kube-state-metrics also exposes list and watch success and error metrics. These can be used to calculate the error rate of list or watch resources. If you encounter those errors in the metrics, it is most likely a configuration or perm