grafana-dashboards-kubernetes

Table of contents

• Description
• Releases
• Features
• Dashboards
• Installation
  • Install manually
  • Install via grafana.com
  • Install with ArgoCD
  • Install with Helm values
  • Install as ConfigMaps
  • Install as ConfigMaps with Terraform
  • Install as GrafanaDashboard with Grafana Operator
• Known issue(s)
  • Broken panels due to a too-high resolution
  • Broken panels on k8s-views-nodes when a node changes its IP address
  • Broken panels on k8s-views-nodes due to the nodename label
  • Windows support
• Contributing

Description

This repository contains a modern set of Grafana dashboards for Kubernetes.
They are inspired by many other dashboards from kubernetes-mixin and grafana.com.

More information about them in my article: A set of modern Grafana dashboards for Kubernetes

You can also download them on Grafana.com.

Releases

This repository follows semantic versioning for releases.
It relies on conventional commits to automate releases using semantic-release.

Features

These dashboards are made and tested for the kube-prometheus-stack chart, but they should work well with others as soon as you have kube-state-metrics and prometheus-node-exporter installed on your Kubernetes cluster.

They are not backward compatible with older Grafana versions because they try to take advantage of Grafana's newest features like:

• gradient mode introduced in Grafana 8.1 (Grafana Blog post)
• time series visualization panel introduced in Grafana 7.4 (Grafana Blog post)
• $__rate_interval variable introduced in Grafana 7.2 (Grafana Blog post)

They also have a Prometheus Datasource variable so they will work on a federated Grafana instance.

As an example, here's how the Kubernetes / Views / Global dashboard looks like:

Dashboards

Installation

In most cases, you will need to clone this repository (or your fork):

git clone https://github.com/dotdc/grafana-dashboards-kubernetes.git
cd grafana-dashboards-kubernetes

If you plan to deploy these dashboards using ArgoCD, ConfigMaps or Terraform, you will also need to enable and configure the dashboards sidecar on the Grafana Helm chart to get the dashboards loaded in your Grafana instance:

# kube-prometheus-stack values
grafana:
  sidecar:
    dashboards:
      enabled: true
      defaultFolderName: "General"
      label: grafana_dashboard
      labelValue: "1"
      folderAnnotation: grafana_folder
      searchNamespace: ALL
      provider:
        foldersFromFilesStructure: true

Install manually

In your Grafana instance, put your mouse over the + sign on the left menu, then click on Import.
Once you are on the Import page, you can upload the JSON files one by one from your local copy using the Upload JSON file button.

Install via grafana.com

In your Grafana instance, put your mouse over the + sign on the left menu, then click on Import.
Once you are on the Import page, you can put the grafana.com dashboard ID (see table below) under Import via grafana.com then click on the Load button. Repeat for each dashboard.

Grafana.com dashboard id list:

Install with ArgoCD

If you are using ArgoCD, this will deploy the dashboards in the default project of ArgoCD:

kubectl apply -f argocd-app.yml

You will also need to enable and configure the Grafana dashboards sidecar as described in Installation.

Install with Helm values

If you use the official Grafana helm chart or kube-prometheus-stack, you can install the dashboards directly using the dashboardProviders & dashboards helm chart values.

Depending on your setup, add or merge the following block example to your helm chart values.
The example is for kube-prometheus-stack, for the official Grafana helm chart, remove the first line (grafana:), and reduce the indentation level of the entire block.

grafana:
  # Provision grafana-dashboards-kubernetes
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
      - name: 'grafana-dashboards-kubernetes'
        orgId: 1
        folder: 'Kubernetes'
        type: file
        disableDeletion: true
        editable: true
        options:
          path: /var/lib/grafana/dashboards/grafana-dashboards-kubernetes
  dashboards:
    grafana-dashboards-kubernetes:
      k8s-system-api-server:
        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-system-api-server.json
        token: ''
      k8s-system-coredns:
        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-system-coredns.json
        token: ''
      k8s-views-global:
        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-global.json
        token: ''
      k8s-views-namespaces:
        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-namespaces.json
        token: ''
      k8s-views-nodes:
        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-nodes.json
        token: ''
      k8s-views-pods:
        url: https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-pods.json
        token: ''

Install as ConfigMaps

Grafana dashboards can be provisioned as Kubernetes ConfigMaps if you configure the dashboard sidecar available on the official Grafana Helm Chart.

To build the ConfigMaps and output them on STDOUT :

kubectl kustomize .

Note: no namespace is set by default, you can change that in the kustomization.yaml file.

To build and deploy them directly on your Kubernetes cluster :

kubectl apply -k . -n monitoring

You will also need to enable and configure the Grafana dashboards sidecar as described in Installation.

Note: you can change the namespace if needed.

Install as ConfigMaps with Terraform

If you use Terraform to provision your Kubernetes resources, you can convert the generated ConfigMaps to Terraform code using tfk8s.

To build and convert ConfigMaps to Terraform code :

kubectl kustomize . | tfk8s

You will also need to enable and configure the Grafana dashboards sidecar as described in Installation.

Note: no namespace is set by default, you can change that in the kustomization.yaml file.

Install as GrafanaDashboard with Grafana Operator

If you use Grafana Operator to provision your Grafana dashboards, you can use the following manifests:

Make sure to use your proper namespace.

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: k8s-system-api-server
  namespace: monitoring
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  url: "https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-system-api-server.json"
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: k8s-system-coredns
  namespace: monitoring
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  url: "https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-system-coredns.json"
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: k8s-views-global
  namespace: monitoring
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  url: "https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-global.json"
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: k8s-views-namespaces
  namespace: monitoring
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  url: "https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-namespaces.json"
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: k8s-views-nodes
  namespace: monitoring
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  url: "https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-nodes.json"
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: k8s-views-pods
  namespace: monitoring
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  url: "https://raw.githubusercontent.com/dotdc/grafana-dashboards-kubernetes/master/dashboards/k8s-views-pods.json"

Known issue(s)

Broken panels due to a too-high resolution

A user reported in #50 that some panels were broken because the default value of the $resolution variable was too low. The root cause hasn't been identified precisely, but he was using Grafana Agent & Grafana Mimir. Changing the $resolution variable to a higher value (a lower resolution) will likely solve the issue. To make the fix permanent, you can configure the Scrape interval in your Grafana Datasource to a working value for your setup.

Broken panels on k8s-views-nodes when a node changes its IP address

To make this dashboard more convenient, there's a small variable hack to display node instead of instance. Because of that, some panels could lack data when a node changes its IP address as reported in #102.

There is currently no known fix for this scenario, though it should be a corner case for most deployments. Contributions with ideas or solutions are welcome.

Broken panels on k8s-views-nodes due to the nodename label

The k8s-views-nodes dashboard will have many broken panels if the node label from kube_node_info doesn't match the nodename label from node_uname_info.

This situation can happen on certain deployments of the node exporter running inside Kubernetes(e.g. via a DaemonSet), where nodename takes a different value than the node name as understood by the Kubernetes API.

Below are some ways to relabel the metric to force the nodename label to the appropriate value, depending on the way the collection agent is deployed:

Directly through the Prometheus configuration file

Assuming the node exporter job is defined through kubernetes_sd_config, you can take advantage of the internal discovery labels and fix this by adding the following relabeling rule to the job:

# File: prometheus.yaml
scrape_configs:
- job_name: node-exporter
  relabel_configs:
  # Add this
  - action: replace
    source_labels: [ __meta_kubernetes_pod_node_name]
    target_label: nodename

Through a ServiceMonitor

If using the Prometheus operator or the Grafana agent in operator mode, the scrape job should instead be configured via a ServiceMonitor that will dynamically edit the Prometheus configuration file. In that case, the relabeling has a slightly different syntax:

# File: service-monitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: node-exporter
spec:
  endpoints:
  - port: http-metrics
    relabelings:
    # Add this
    - action: replace
      sourceLabels: [__meta_kubernetes_node_name]
      targetLabel: nodename

As a convenience, if using the kube-prometheus-stack helm chart, this added rule can be directly specified in your values.yaml:

# File: kube-prometheus-stack-values.yaml
prometheus-node-exporter:
  prometheus:
    monitor:
      relabelings:
      - action: replace
        sourceLabels: [__meta_kubernetes_pod_node_name]
        targetLabel: nodename

With Grafana Alloy

Grafana Alloy can bundle its own node_exporter. In that case, relabeling can be done this way:

prometheus.exporter.unix "node_exporter" {
}

prometheus.scrape "node_exporter" {
  targets    = prometheus.exporter.unix.node_exporter.targets
  forward_to = [prometheus.relabel.node_exporter.receiver]
  job_name   = "node-exporter"
}

prometheus.relabel "node_exporter" {
  forward_to = [prometheus.remote_write.<your_component>.receiver]

  rule {
    action       = "replace"
    replacement  = sys.env("HOSTNAME")
    target_label = "nodename"
  }
}

If you deploy Alloy with the Helm chart, the HOSTNAME environment variable is set to the Kubernetes node name by default. If you use a custom deployment, adjust the value used for nodename to match your node naming scheme.

Windows support

The dashboards currently don't support Windows. We experimented with Windows support in the k8s-views-global dashboard, but it ended up breaking some panels, so it has been removed for now. You can download the last Windows-compatible version of the dashboard or download revision 43 on Grafana.com.

See #79

Contributing

Feel free to contribute to this project:

• Give a GitHub ⭐ if you like it
• Create an Issue to make a feature request, report a bug or share an idea.
• Create a Pull Request if you want to share code or anything useful to this project.