[opensourcedex]

kindling is a free, open source monitoring & observability project written in Go and released under Apache-2.0. It has 1,063 GitHub stars, 166 forks and 27 open issues, and was last pushed 3 months ago. On this registry it ranks #220 of 270 tracked projects in Monitoring & Observability, with 5 head-to-head comparisons available.

What is kindling?

Kindling is an Apache-2.0 licensed, eBPF-based cloud native monitoring tool written in Go that helps Kubernetes operators, SREs and application developers understand application behaviour from the kernel up to the code stack.

What it is

Kindling is a monitoring agent for Kubernetes that uses eBPF to observe the kernel. The agent runs as a DaemonSet on the cluster and collects syscalls together with selected tracepoints, turning that kernel-level data into traces, network metrics and profiling output. Different distributions are handled by different exporters: a Prometheus exporter writes the collected data into Prometheus, where a Grafana plugin renders it.

The concrete problem Kindling addresses is root-cause analysis in production. Traces, metrics and logs on their own often leave an incident unexplained, so Kindling introduces trace profiling, which integrates OnCPU and OffCPU events inside a trace and collects the logs emitted during the trace's execution window. Trace profiling has its own standalone UI rather than living inside the Grafana plugin. The tool replaces the manual, disconnected work of correlating a slow trace with the CPU busy time, the network syscalls, the file syscalls and the per-thread logs — Kindling records how every thread executed so the sequence can be replayed.

Key capabilities

  • eBPF agent deployed as a DaemonSet in Kubernetes, collecting syscalls and additional tracepoints from the node.
  • Trace profiling that integrates OnCPU and OffCPU events into a trace, collecting OnCPU data at thread level rather than process level.
  • OffCPU analysis covering thread waiting states such as lock waits, database queries, remote procedure calls, file reading and file writing.
  • Correlation of a trace's exact executing thread with the flame graph of CPU-busy time, the network time series for network syscalls and the file time series for file syscalls.
  • Per-thread log collection, correlated to the relative thread together with its timestamp.
  • Built-in network monitor dashboards for TCP retransmit, DNS, throughput and TPS.
  • Network exploration of a single RPC call, used in the same way as network exploration in Chrome, to identify which part of a request to investigate.
  • Prometheus exporter that supplies data to Prometheus for display through a Grafana Plugin.

Who uses it and how

  • Kubernetes operators investigating production incidents, where the stated aim is to find the root cause in seconds using trace profiling.
  • Platform teams already standardised on Prometheus and Grafana, who add Kindling as the collection agent and use the existing dashboards.
  • Teams running heterogeneous kernel fleets: eBPF requires a kernel newer than 4.14, trace profiling requires newer than 4.17, and older kernels fall back to a kernel module.
  • Developers inspecting a single RPC call through network explore to decide which part of the request path to dig into.
  • SREs who need an overview of network flows in the cluster before drilling into an individual trace.

Getting started

The documented deployment is the agent running as a DaemonSet inside Kubernetes, with the Prometheus exporter feeding Prometheus and the Grafana plugin for display; installation and configuration details are published on the Kindling website and documentation at http://kindling.harmonycloud.cn/docs/overview-and-concepts/overview/.

How it compares

For older kernels that eBPF cannot support, Kindling uses an enhanced and verified kernel module taken from the Sysdig open-source project, so it acknowledges the limits of its own tracing approach rather than excluding those hosts. It is intended to sit alongside Prometheus and Grafana rather than replace them, using the Prometheus exporter as the data path and Grafana as one of its display surfaces.

When to use it — and when not to

A self-hoster must operate a Kubernetes cluster plus Prometheus for storage and Grafana for dashboards, and accept that trace profiling ships as a separate standalone UI. Hosts on kernels at or below 4.14 can only run the kernel module path, which the project describes as less safe than eBPF in production and which does not cover trace profiling. Anyone without a Kubernetes environment, or anyone unwilling to run privileged node-level instrumentation, should look elsewhere.

project readme (upstream, from github) — read inline

Kindling

License Doc Go Report Card

Visit our Kindling website for more information.

What is Kindling

Kindling is an eBPF-based cloud native monitoring tool, which aims to help users understand the app behavior from kernel to code stack. With trace profiling, we hope the user can understand the application's behavior easily and find the root cause in seconds. Besides trace profiling, Kindling provides an easy way to get an overview of network flows in the Kubernetes environment, and many built-in network monitor dashboards like TCP retransmit, DNS, throughput, and TPS. Not only as a network monitor, but Kindling is also trying to analyze one detail RPC call and get the following information, which can be found in network explore in chrome. And the usage is the same as network exploration in chrome, with which users can tell which part to dig into to find the root cause of the problem in the production environment.

img

What is Kindling Trace-profiling

With traces, metrics, and logs, many issues still can’t be understood easily. Trace-profiling is trying to integrate the OnCPU and OffCPU events within the traces and collect the logs output during the trace execution timeframe.

OnCPU events are just like the flame graph, but the code has been collected at the thread level instead of the process level. And the trace was executed by one thread, so users can understand how the trace was executing on the CPU.

OffCPU events are the opposite of OnCPU events. As for the trace analysis, most traces spend a lifetime waiting for the locks, database query, remote process call, file reading, or file writing. All of these events cause the thread in waiting status, and they are considered as OffCPU events.

So for trace profiling, how all threads were executed is recorded and can be replayed.

  • The exact thread which executed the trace span is highlighted.
  • The logs printed by each thread are collected and correlated to the relative thread with its timestamp.
  • The code execution flame graph is correlated to the time series where the CPU is busy.
  • The network-related metrics are correlated to the time series where the network syscalls are executing.
  • The file-related metrics are correlated to the time series where the file syscalls are executing.

Architecture

From a high-level view, the agent runs as DeamonSet in Kubernetes. It collects all syscalls and some other tracepoints. We use different exporters for different distributions. For example, we build a Prometheus exporter to export the data which can be stored in Prometheus and displayed in Grafana Plugin. But for the trace profiling module, the UI is different, that's a standalone module.

image.png

Linux kernel version support

The Kindling eBPF module depends on the kernel version which is newer than 4.14. But for trace-profiling, the kernel version has to be newer than 4.17, hoping with more work, the trace-profiling can work on older kernel versions. As an eBPF constraint, the eBPF module can't work for older kernel versions. But for the users who want to try the functionality with the old kernel, we use the kernel module from Sysdig open-source project with enhancement and verification. Basically, the idea is to use a kernel module for tracking the kernel tracepoints. Thanks to Sysdig open-source project, it provides a tracepoint instrument framework for older kernel versions.

For now, the kernel module works as expected as the eBPF module during our test except for trace-profiling, but it is recommended to use the eBPF module in the production environment as it is safer than the kernel module. In order to help the older kernel version user experience the eBPF magic, we also support the kernel model. And you are welcome to report issues with the kernel module. For functionality, the kernel module and eBPF module capture the same data and behave exactly the same.

Why do we build Kindling?

When we talk about observability, we already have plenty of tools to use, like Skywalking for tracing, ELK for logging, and Prometheus for metrics. Why do we need to build an eBPF-based monitoring tool?

The majority issue for user adoption of k8s is the complexity. For the applications on Kubernetes, we don't know the network flows between the services until we can instrument the apps. We can't tell which part to blame when there is a product issue arise. Do we configure Kubernetes correctly? Are there any bugs in the virtual network like Calico or Flannel that caused this problem? Does the application code cause this issue?

We are a company based in Hangzhou, China, and used to provide Kubernetes distribution for our customers. Our customers used to have those questions, and we don't have proper solutions to answer those questions.

APM(Application Performance Monitoring) is good for those applications in Java which can be instrumented automatically, while for the Go programs, the code has to be rewritten for instrumenting. And even if we adopt the APM solution, we still can't tell whether an issue is caused by network problems, and many issues can’t be pinpoint the root cause easily.

We found it may be helpful that we triage the issue first by checking the issue from the network view to identify issues roughly like "oh, it's a network problem, the code works fine, and we should dig into the configuration of calico" or "the infrastructure works fine, so the app code should be blamed, let's dig into the logs or the APM dashboard for further information".

After we triage the issue, we need to pinpoint the root cause of the issue. That's why we need the trace-profiling module.

Why eBPF?

The libpcap way of analyzing the flows in the Kubernetes environment is too expensive for the CPU and network. The eBPF way of data capture cost much less than libpcap. eBPF is the most popular technology to track the Linux kernel where the virtual network, built by veth-pair and iptables, works. So eBPF is a proper technique to be used for tracking how the kernel responds to application requests.

Core Features

With the trace-profiling module, we can understand how ElasticSearch works easily. The following image shows how ElasticSearch is executing the bulk insert operation.

trace-profiling

The next image shows a dependency map in Kubernetes.

img

Kindling can be easily integrated with Prometheus, and we uses PromQL to query the data in the frontend, so it should be adopted easily. But due to the cardinality constraint of Prometheus, we group the detailed data into buckets which throw away the detailed information.

Get started

You can deploy Kindling easily, check out the Installation Guide for details.

Documentation

The Kindling documentation is available on our Kindling website

Contributing

Contributions are welcome, you can contribute in many ways: report issues, help us reproduce issues, fix bugs, add features, give us advice on GitHub discussion, and so on. If you are interested in joining us to unveil the eBPF in the Kubernetes area, you can start by reading the Contributing Guide.

Contact

If you have questions or ideas, please feel free to reach out to us in the following ways:

img

License

Kindling is distributed under Apache License, Version2.0.

Frequently asked questions

Is kindling free to use?

kindling is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does kindling do?

eBPF-based Cloud Native Monitoring Tool

What is kindling written in?

kindling is primarily written in Go. Its source is publicly available at https://github.com/harmonycloud/kindling, and it has 1,063 GitHub stars.