[opensourcedex]

alertmanager is a free, open source monitoring & observability project written in Go and released under Apache-2.0. It has 8,613 GitHub stars, 2,469 forks and 419 open issues, and was last pushed 18 hours ago. On this registry it ranks #56 of 191 tracked projects in Monitoring & Observability, with 5 head-to-head comparisons available.

What is alertmanager?

Prometheus Alertmanager is the Go-based, Apache-2.0 alert routing service that receives alerts from client applications such as the Prometheus server, deduplicates and groups them, and routes them to receivers including email, PagerDuty, OpsGenie, Slack and a generic webhook — it is for operations and platform teams that already run Prometheus and need notifications that are neither duplicated nor noisy.

What it is

Alertmanager is a standalone alert-handling component of the Prometheus ecosystem, distributed under the Apache-2.0 licence and written in Go. It runs as its own service, listening on port 9093, and is configured through a YAML file supplied with --config.file. The project is documented at prometheus.io, carries the topics alertmanager, deduplication, monitoring, notifications, email, pagerduty, opsgenie and slack, and ships a companion command-line tool, amtool, built from the same repository.

The concrete thing it replaces is per-rule or per-server direct notification wiring. When Prometheus evaluates alerting rules and fires alerts, something must decide which of them deserve a notification and how often to send one. Alertmanager takes over that job: it deduplicates alerts arriving from multiple sources, groups them by label, routes them through a configuration tree of receivers, and applies silencing and inhibition. Instead of every firing alert becoming its own message, alerts are funnelled through one component that decides what gets sent, to whom, and when.

Key capabilities

  • Deduplication of alerts sent by client applications, so the same alert reported by more than one source is not delivered repeatedly.
  • Grouping through group_by labels such as ['alertname', 'cluster'], with group_wait (for example 30s), group_interval (5m) and repeat_interval (3h) controlling batching, batching follow-ups and resends.
  • A routing tree built from a mandatory root route plus child routes with regular-expression matchers such as service=~"^(foo1|foo2|baz)$", each route pointing at a receiver.
  • Receiver integrations for email, PagerDuty, OpsGenie and Slack, plus the webhook receiver for any other mechanism.
  • Silencing and inhibition of alerts.
  • A global configuration block for mail delivery, including smtp_smarthost and smtp_from.
  • amtool, buildable on its own with make build BINARIES=amtool.

Who uses it and how

  • Operations teams running a Prometheus server that need firing alerts delivered to email or to an on-call service such as PagerDuty or OpsGenie.
  • Teams with many services and label sets, who split traffic with regex sub-routes — critical alerts for services matching foo1|foo2|baz to a service team receiver, everything unmatched to the root receiver.
  • ChatOps setups pointing a Slack receiver at the relevant channels.
  • Environments with very low alert volume, or where an upstream system already groups notifications, using group_by: [...] to disable aggregation and pass alerts through as-is.
  • Teams with bespoke delivery tooling, wiring the webhook receiver into their own endpoint.

Getting started

Precompiled binaries are available in the download section of prometheus.io, which the README recommends as the production install path, and Docker images are published on Quay.io and Docker Hub — docker run --name alertmanager -d -p 127.0.0.1:9093:9093 quay.io/prometheus/alertmanager brings it up at http://localhost:9093/. Building from source requires Go and Node.js with npm, followed by make build and ./alertmanager --config.file=.

How it compares

This registry lists no direct alternative to Alertmanager. PagerDuty, OpsGenie and Slack appear only as receiver targets that Alertmanager delivers into, not as substitutes for it, so the project stands alone among comparable tools here.

When to use it — and when not to

A self-hoster takes on running the service, maintaining the YAML configuration, and providing an SMTP smarthost and sender address if mail notifications are wanted. Teams not running Prometheus or another compatible alert emitter have nothing to point at it, and anyone expecting a hosted service should note that Alertmanager is self-hosted software whose third-party integrations are delivery destinations rather than managed backends. The README is thin — it covers installation and a configuration example, with the full configuration reference held externally — and the repository carries 419 open issues, so operators should expect to lean on the upstream documentation rather than the README alone.

project readme (upstream, from github) — read inline

Alertmanager [CircleCI][circleci]

[Docker Repository on Quay][quay] [Docker Pulls][hub]

The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integrations such as email, PagerDuty, OpsGenie, or many other mechanisms thanks to the webhook receiver. It also takes care of silencing and inhibition of alerts.

Install

There are various ways of installing Alertmanager.

Precompiled binaries

Precompiled binaries for released versions are available in the download section on prometheus.io. Using the latest production release binary is the recommended way of installing Alertmanager.

Docker images

Docker images are available on Quay.io or Docker Hub.

You can launch an Alertmanager container for trying it out with

$ docker run --name alertmanager -d -p 127.0.0.1:9093:9093 quay.io/prometheus/alertmanager

Alertmanager will now be reachable at http://localhost:9093/.

Compiling the binary

Building from source requires Go and Node.js (with npm). Clone the repository and build manually:

$ git clone https://github.com/prometheus/alertmanager.git
$ cd alertmanager
$ make build
$ ./alertmanager --config.file=<your_file>

You can also build just one of the binaries in this repo by passing a name to the build function:

$ make build BINARIES=amtool

Example

This is an example configuration that should cover most relevant aspects of the new YAML configuration format. The full documentation of the configuration can be found here.

global:
  # The smarthost and SMTP sender used for mail notifications.
  smtp_smarthost: 'localhost:25'
  smtp_from: '[email protected]'

# The root route on which each incoming alert enters.
route:
  # The root route must not have any matchers as it is the entry point for
  # all alerts. It needs to have a receiver configured so alerts that do not
  # match any of the sub-routes are sent to someone.
  receiver: 'team-X-mails'

  # The labels by which incoming alerts are grouped together. For example,
  # multiple alerts coming in for cluster=A and alertname=LatencyHigh would
  # be batched into a single group.
  #
  # To aggregate by all possible labels use '...' as the sole label name.
  # This effectively disables aggregation entirely, passing through all
  # alerts as-is. This is unlikely to be what you want, unless you have
  # a very low alert volume or your upstream notification system performs
  # its own grouping. Example: group_by: [...]
  group_by: ['alertname', 'cluster']

  # When a new group of alerts is created by an incoming alert, wait at
  # least 'group_wait' to send the initial notification.
  # This way ensures that you get multiple alerts for the same group that start
  # firing shortly after another are batched together on the first
  # notification.
  group_wait: 30s

  # When the first notification was sent, wait 'group_interval' to send a batch
  # of new alerts that started firing for that group.
  group_interval: 5m

  # If an alert has successfully been sent, wait 'repeat_interval' to
  # resend them.
  repeat_interval: 3h

  # All the above attributes are inherited by all child routes and can
  # overwritten on each.

  # The child route trees.
  routes:
  # This route performs a regular expression match on alert labels to
  # catch alerts that are related to a list of services.
  - matchers:
    - service=~"^(foo1|foo2|baz)$"
    receiver: team-X-mails

    # The service has a sub-route for critical alerts, any alerts
    # that do not match, i.e. severity != critical, fall-back to the
    # parent node and are sent to 'team-X-mails'
    routes:
    - matchers:
      - severity="critical"
      receiver: team-X-pager

  - matchers:
    - service="files"
    receiver: team-Y-mails

    routes:
    - matchers:
      - severity="critical"
      receiver: team-Y-pager

  # This route handles all alerts coming from a database service. If there's
  # no team to handle it, it defaults to the DB team.
  - matchers:
    - service="database"

    receiver: team-DB-pager
    # Also group alerts by affected database.
    group_by: [alertname, cluster, database]

    routes:
    - matchers:
      - owner="team-X"
      receiver: team-X-pager

    - matchers:
      - owner="team-Y"
      receiver: team-Y-pager


# Inhibition rules allow to mute a set of alerts given that another alert is
# firing.
# We use this to mute any warning-level notifications if the same alert is
# already critical.
inhibit_rules:
- source_matchers:
    - severity="critical"
  target_matchers:
    - severity="warning"
  # Apply inhibition if the alertname is the same.
  # CAUTION: 
  #   If all label names listed in `equal` are missing 
  #   from both the source and target alerts,
  #   the inhibition rule will apply!
  equal: ['alertname']


receivers:
- name: 'team-X-mails'
  email_configs:
  - to: '[email protected], [email protected]'

- name: 'team-X-pager'
  email_configs:
  - to: '[email protected]'
  pagerduty_configs:
  - routing_key: <team-X-key>

- name: 'team-Y-mails'
  email_configs:
  - to: '[email protected]'

- name: 'team-Y-pager'
  pagerduty_configs:
  - routing_key: <team-Y-key>

- name: 'team-DB-pager'
  pagerduty_configs:
  - routing_key: <team-DB-key>

API

The current Alertmanager API is version 2. This API is fully generated via the OpenAPI project and Go Swagger with the exception of the HTTP handlers themselves. The API specification can be found in api/v2/openapi.yaml. A HTML rendered version can be accessed here. Clients can be easily generated via any OpenAPI generator for all major languages.

APIv2 is accessed via the /api/v2 prefix. APIv1 was deprecated in 0.16.0 and is removed as of version 0.27.0. The v2 /status endpoint would be /api/v2/status. If --web.route-prefix is set then API routes are prefixed with that as well, so --web.route-prefix=/alertmanager/ would relate to /alertmanager/api/v2/status.

amtool

amtool is a cli tool for interacting with the Alertmanager API. It is bundled with all releases of Alertmanager.

Install

Alternatively you can install with:

$ go install github.com/prometheus/alertmanager/cmd/amtool@latest

Examples

View all currently firing alerts:

$ amtool alert
Alertname        Starts At                Summary
Test_Alert       2017-08-02 18:30:18 UTC  This is a testing alert!
Test_Alert       2017-08-02 18:30:18 UTC  This is a testing alert!
Check_Foo_Fails  2017-08-02 18:30:18 UTC  This is a testing alert!
Check_Foo_Fails  2017-08-02 18:30:18 UTC  This is a testing alert!

View all currently firing alerts with extended output:

$ amtool -o extended alert
Labels                                        Annotations                                                    Starts At                Ends At                  Generator URL
alertname="Test_Alert" instance="node0"       link="https://example.com" summary="This is a testing alert!"  2017-08-02 18:31:24 UTC  0001-01-01 00:00:00 UTC  http://my.testing.script.local
alertname="Test_Alert" instance="node1"       link="https://example.com" summary="This is a testing alert!"  2017-08-02 18:31:24 UTC  0001-01-01 00:00:00 UTC  http://my.testing.script.local
alertname="Check_Foo_Fails" instance="node0"  link="https://example.com" summary="This is a testing alert!"  2017-08-02 18:31:24 UTC  0001-01-01 00:00:00 UTC  http://my.testing.script.local
alertname="Check_Foo_Fails" instance="node1"  link="https://example.com" summary="This is a testing alert!"  2017-08-02 18:31:24 UTC  0001-01-01 00:00:00 UTC  http://my.testing.script.local

In addition to viewing alerts, you can use the rich query syntax provided by Alertmanager:

$ amtool -o extended alert query alertname="Test_Alert"
Labels                                   Annotations

readme truncated — read the full docs on github

Frequently asked questions

Is alertmanager free to use?

alertmanager is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does alertmanager do?

Prometheus Alertmanager

What is alertmanager written in?

alertmanager is primarily written in Go. Its source is publicly available at https://github.com/prometheus/alertmanager, and it has 8,613 GitHub stars.