whisper: Open Source Monitoring & Observability

whisper is a free, open source monitoring & observability project written in Python and released under Apache-2.0. It has 1,261 GitHub stars, 322 forks and 8 open issues, and was last pushed 10 months ago. On this registry it ranks #206 of 271 tracked projects in Monitoring & Observability, with 5 head-to-head comparisons available.

What is whisper?

Whisper is the file-based time-series database library at the storage layer of Graphite, built for operators and developers who need to record numeric metrics over time as plain files on disk rather than in a separate database server.

What it is

Whisper is one of three components of the Graphite project. The other two are Graphite-Web, a Django-based web application that renders graphs and dashboards, and Carbon, the metric processing daemons. Whisper itself is the time-series database library, written in Python and released under the Apache-2.0 licence. It is a fixed-size database, similar in design and purpose to RRD, the round-robin database, and it provides fast, reliable storage of numeric data over time. Each series lives in its own file with the .wsp extension, and the surrounding Graphite tooling reads and writes those files directly.

The concrete problem it solves is retention and rollup without an external database. Whisper allows higher resolution, measured in seconds per point, for recent data to degrade into lower resolutions for long-term retention of historical data. An operator can therefore keep a few days of fine-grained samples and years of coarse ones for the same metric, without running a query server or a separate storage cluster. For teams already using RRD and hitting its limits inside the Graphite stack, Whisper is the specific thing that replaces the RRD file format in the pipeline, with rrd2whisper.py provided to move existing .rrd data across.

Key capabilities

whisper-create.py builds a new database file from archive definitions such as 60:1440 (60 seconds per datapoint, 1440 datapoints, one day) or 12h:2y (12 hours per datapoint, two years of retention), with --estimate to calculate storage requirements without creating a file.
Multi-resolution retention is expressed at creation time: a file can hold 15m:8, 1h:7d and 12h:2y archives side by side so recent data stays fine-grained and historical data is consolidated.
whisper-fetch.py retrieves metrics over a requested interval using --from and --until, and emits results as JSON with --json or with human-readable timestamps using --pretty, plus --drop=nulls or --drop=zeroes to filter series.
whisper-dump.py writes the entire content of a file to stdout, optionally with --pretty timestamps or --raw output formatted for whisper-update.
rrd2whisper.py converts an RRD file into a Whisper file, carrying over or overriding --xFilesFactor and --aggregationMethod, and accepting --destinationPath for the output location.
Aggregation functions available on a database include average, sum, last, max, min, avg_zero, absmax and absmin.
xFilesFactor controls how partial intervals are handled, setting the proportion of defined points required before a consolidated value is produced.

Who uses it and how

Graphite deployments use Whisper as the storage backend: Carbon daemons write incoming metrics into .wsp files and Graphite-Web reads them back to render graphs and dashboards.
Infrastructure and operations teams store system and application metrics on the same hosts that collect them, since every series is a file rather than a row in a remote database.
Teams migrating off RRD run rrd2whisper.py against existing .rrd files, choosing an aggregation method and destination path, then point their tooling at the resulting .wsp files.
Automation and provisioning workflows call whisper-create.py to lay down databases ahead of collection, and whisper-fetch.py --json to export or check stored values from scripts.
Operators investigating retention or data quality use whisper-dump.py and whisper-fetch.py directly against a file to inspect what was actually recorded.

Getting started

The README defers installation, configuration and usage to the documentation at graphite.readthedocs.org, which is also the project homepage. Once the package and its scripts are available, a new database is created with a command such as whisper-create.py path 60:1440.

How it compares

The format it sits closest to in the facts is RRD, the round-robin database it is explicitly modelled on, and the project supplies rrd2whisper.py precisely because teams arrive holding .rrd data. Within Graphite it occupies the storage tier, distinct from Carbon, which processes metrics, and Graphite-Web, which renders them. It carries the same Apache-2.0 licence as the rest of the Graphite project and keeps data in files the operator owns and can inspect with the bundled scripts.

When to use it — and when not to

A self-hoster has to operate file storage for the .wsp files and the Graphite components around them, and must decide retention and resolution up front, because archives and xFilesFactor are fixed when the file is created. It is the wrong choice for anyone wanting a clustered, horizontally scaled time-series service managed by someone else, since Whisper is a library and format rather than a server with replication or a query API of its own. The README is also thin: installation and configuration are pushed entirely to readthedocs, so the repository alone is not a complete operations guide.

project readme (upstream, from github) — read inline

Whisper

Overview

Whisper is one of three components within the Graphite project:

Graphite-Web, a Django-based web application that renders graphs and dashboards
The Carbon metric processing daemons
The Whisper time-series database library

Graphite Components

Whisper is a fixed-size database, similar in design and purpose to RRD (round-robin-database). It provides fast, reliable storage of numeric data over time. Whisper allows for higher resolution (seconds per point) of recent data to degrade into lower resolutions for long-term retention of historical data.

Installation, Configuration and Usage

Please refer to the instructions at readthedocs.

Whisper Scripts

rrd2whisper.py

Convert a rrd file into a whisper (.wsp) file.

Usage: rrd2whisper.py rrd_path

Options:
  -h, --help            show this help message and exit
  --xFilesFactor=XFILESFACTOR
                        The xFilesFactor to use in the output file. Defaults
                        to the input RRD's xFilesFactor
  --aggregationMethod=AGGREGATIONMETHOD
                        The consolidation function to fetch from on input and
                        aggregationMethod to set on output. One of: average,
                        last, max, min, avg_zero, absmax, absmin
  --destinationPath=DESTINATIONPATH
                        Path to place created whisper file. Defaults to the
                        RRD file's source path.

whisper-create.py

Create a new whisper database file.

Usage: whisper-create.py path timePerPoint:timeToStore [timePerPoint:timeToStore]*
       whisper-create.py --estimate timePerPoint:timeToStore [timePerPoint:timeToStore]*

timePerPoint and timeToStore specify lengths of time, for example:

60:1440      60 seconds per datapoint, 1440 datapoints = 1 day of retention
15m:8        15 minutes per datapoint, 8 datapoints = 2 hours of retention
1h:7d        1 hour per datapoint, 7 days of retention
12h:2y       12 hours per datapoint, 2 years of retention


Options:
  -h, --help            show this help message and exit
  --xFilesFactor=XFILESFACTOR
  --aggregationMethod=AGGREGATIONMETHOD
                        Function to use when aggregating values (average, sum,
                        last, max, min, avg_zero, absmax, absmin)
  --overwrite
  --estimate            Don't create a whisper file, estimate storage requirements based on archive definitions

whisper-dump.py

Dump the whole whisper file content to stdout.

Usage: whisper-dump.py path

Options:
  -h, --help            show this help message and exit
  --pretty              Show human-readable timestamps instead of unix times
  -t TIME_FORMAT, --time-format=TIME_FORMAT
                        Time format to use with --pretty; see time.strftime()
  -r, --raw             Dump value only in the same format for whisper-update
                        (UTC timestamps)

whisper-fetch.py

Fetch all the metrics stored in a whisper file to stdout.

Usage: whisper-fetch.py [options] path

Options:
  -h, --help     show this help message and exit
  --from=_FROM   Unix epoch time of the beginning of your requested interval
                 (default: 24 hours ago)
  --until=UNTIL  Unix epoch time of the end of your requested interval
                 (default: now)
  --json         Output results in JSON form
  --pretty       Show human-readable timestamps instead of unix times
  -t TIME_FORMAT, --time-format=TIME_FORMAT
                 Time format to use with --pretty; see time.strftime()
  --drop=DROP    Specify 'nulls' to drop all null values. Specify 'zeroes' to
                 drop all zero values. Specify 'empty' to drop both null and
                 zero values.

whisper-info.py

Dump the metadata about a whisper file to stdout.

Usage: whisper-info.py [options] path [field]

Options:
  -h, --help  show this help message and exit
  --json      Output results in JSON form

whisper-merge.py

Join two existing whisper files together.

Usage: whisper-merge.py [options] from_path to_path

Options:
  -h, --help  show this help message and exit

whisper-fill.py

Copies data from src in dst, if missing. Unlike whisper-merge, don't overwrite data that's already present in the target file, but instead, only add the missing data (e.g. where the gaps in the target file are). Because no values are overwritten, no data or precision gets lost. Also, unlike whisper-merge, try to take the highest-precision archive to provide the data, instead of the one with the largest retention.

Usage: whisper-fill.py [options] src_path dst_path

Options:
  -h, --help  show this help message and exit

whisper-resize.py

Change the retention rates of an existing whisper file.

Usage: whisper-resize.py path timePerPoint:timeToStore [timePerPoint:timeToStore]*

timePerPoint and timeToStore specify lengths of time, for example:

60:1440      60 seconds per datapoint, 1440 datapoints = 1 day of retention
15m:8        15 minutes per datapoint, 8 datapoints = 2 hours of retention
1h:7d        1 hour per datapoint, 7 days of retention
12h:2y       12 hours per datapoint, 2 years of retention


Options:
  -h, --help            show this help message and exit
  --xFilesFactor=XFILESFACTOR
                        Change the xFilesFactor
  --aggregationMethod=AGGREGATIONMETHOD
                        Change the aggregation function (average, sum, last,
                        max, min, avg_zero, absmax, absmin)
  --force               Perform a destructive change
  --newfile=NEWFILE     Create a new database file without removing the
                        existing one
  --nobackup            Delete the .bak file after successful execution
  --aggregate           Try to aggregate the values to fit the new archive
                        better. Note that this will make things slower and use
                        more memory.

whisper-set-aggregation-method.py

Change the aggregation method of an existing whisper file.

Usage: whisper-set-aggregation-method.py path <average|sum|last|max|min|avg_zero|absmax|absmin>

Options:
  -h, --help  show this help message and exit

whisper-update.py

Update a whisper file with 1 or many values, must provide a time stamp with the value.

Usage: whisper-update.py [options] path timestamp:value [timestamp:value]*

Options:
  -h, --help  show this help message and exit

whisper-diff.py

Check the differences between whisper files. Use sanity check before merging.

Usage: whisper-diff.py [options] path_a path_b

Options:
  -h, --help      show this help message and exit
  --summary       show summary of differences
  --ignore-empty  skip comparison if either value is undefined
  --columns       print output in simple columns
  --no-headers    do not print column headers
  --until=UNTIL   Unix epoch time of the end of your requested interval
                  (default: now)
  --json          Output results in JSON form

License

Whisper is licensed under version 2.0 of the Apache License. See the LICENSE file for details.

Frequently asked questions

Is whisper free to use?

whisper is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does whisper do?

Whisper is a file-based time-series database format for Graphite.

What is whisper written in?

whisper is primarily written in Python. Its source is publicly available at https://github.com/graphite-project/whisper, and it has 1,261 GitHub stars.