[opensourcedex]

nginx-lua-prometheus is a free, open source monitoring & observability project written in Lua and released under MIT. It has 1,573 GitHub stars, 239 forks and 5 open issues, and was last pushed 2 months ago. On this registry it ranks #193 of 270 tracked projects in Monitoring & Observability, with 5 head-to-head comparisons available. It gained 4 stars over the last 30 tracked days.

What is nginx-lua-prometheus?

nginx-lua-prometheus is an MIT-licensed Lua library that lets Nginx record counters, gauges, and histograms inside a shared dictionary and expose them on a dedicated HTTP endpoint for Prometheus to scrape, and it is aimed at operators running OpenResty or a Lua-enabled Nginx who want to instrument the request path directly rather than through a sidecar.

What it is

nginx-lua-prometheus is a Lua library that runs inside Nginx to keep track of metrics and publish them on a separate web page that Prometheus pulls. It lives in the ngx_lua ecosystem: to use it, you need the ngx_lua nginx module, either through a Lua-enabled Nginx-based server such as OpenResty, or through a regular Nginx build with the module enabled — on Debian 10 that is libnginx-mod-http-lua. The library file prometheus.lua must be reachable via LUA_PATH, and if it is the only Lua library in use, lua_package_path can simply point at the directory where the repository is checked out.

The concrete problem it solves is that Nginx itself has no native Prometheus endpoint. Without this library, an operator has to bolt a separate exporter onto the request path or scrape Nginx state from outside it. With the library, metrics are declared in nginx.conf, updated from Lua phases such as log_by_lua_block, accumulated in an nginx shared dictionary, and rendered on demand by a listener that returns the Prometheus exposition format.

Key capabilities

  • Counter, histogram, and gauge metric types, created through calls such as prometheus:counter("nginx_http_requests_total", ...), prometheus:histogram("nginx_http_request_duration_seconds", ...), and prometheus:gauge("nginx_http_connections", ...).
  • Labelled metrics, for example {"host", "status"} on the request counter and {"host"} on latency, incremented per request with metric_requests:inc(1, {ngx.var.server_name, ngx.var.status}).
  • Metric storage in an nginx shared dictionary, sized in configuration with lua_shared_dict prometheus_metrics 10M;.
  • Initialization from init_worker_by_lua_block via prometheus = require("prometheus").init("prometheus_metrics"), with accepted options including prefix, error_metric_name, and sync_interval.
  • Output through prometheus:collect() inside a content_by_lua_block, which renders the accumulated metrics for scraping.
  • Gauges that read live nginx global state immediately before collection, using ngx.var.connections_reading, ngx.var.connections_waiting, and ngx.var.connections_writing.
  • Built-in metrics with a configurable error metric name.

Who uses it and how

  • OpenResty operators install it through OPM from opm.openresty.org/package/knyar/nginx-lua-prometheus, or through LuaRocks.
  • Teams on plain Nginx enable the module — for instance libnginx-mod-http-lua on Debian 10 — and point lua_package_path at the checked-out repository.
  • Request latency and request counts are broken down by server name and HTTP status by observing ngx.var.request_time and incrementing in log_by_lua_block.
  • Metrics are served from a dedicated server block listening on port 9145, typically locked down with allow 192.168.0.0/16; deny all; so only the Prometheus server can reach /metrics.
  • The metrics URL is then simply http://your.nginx:9145/metrics.

Getting started

Install through OPM or LuaRocks, or check out the repository and set lua_package_path "/path/to/nginx-lua-prometheus/?.lua;;"; then declare lua_shared_dict prometheus_metrics 10M; and initialize the module in init_worker_by_lua_block. The hard requirement is the ngx_lua module, available via OpenResty or a Lua-enabled Nginx build.

How it compares

The facts list no paid products that this project replaces, and no directly comparable tools are named. In this registry it stands alone as a Lua-side Prometheus instrumentation library for Nginx; the only other system named is Prometheus itself, which consumes the endpoint rather than competing with it.

When to use it — and when not to

Use it if the traffic already passes through ngx_lua, because that is the only environment it runs in; a self-hoster must operate a Lua-enabled Nginx and size the shared dictionary that backs the metrics. Do not pick it if the deployment has no ngx_lua module, or if a standalone exporter that requires no changes to nginx.conf is preferred. Note also the README's own warning about known issues when using libnginx-mod-http-lua on Debian versions later than 10.

project readme (upstream, from github) — read inline

Coverage Status

Prometheus metric library for Nginx

This is a Lua library that can be used with Nginx to keep track of metrics and expose them on a separate web page to be pulled by Prometheus.

Installation

To use this library, you will need the ngx_lua nginx module. You can either use a lua-enabled nginx-based server like OpenResty, or a regular nginx server with the module enabled: for example, on Debian 10 you can simply install libnginx-mod-http-lua (but please read the known issues if you use a later Debian version).

The library file - prometheus.lua - needs to be available in LUA_PATH. If this is the only Lua library you use, you can just point lua_package_path to the directory with this git repo checked out (see example below).

OpenResty users will find this library in opm. It is also available via luarocks.

Quick start guide

To track request latency broken down by server name and request count broken down by server name and status, add the following to the http section of nginx.conf:

lua_shared_dict prometheus_metrics 10M;
lua_package_path "/path/to/nginx-lua-prometheus/?.lua;;";

init_worker_by_lua_block {
  prometheus = require("prometheus").init("prometheus_metrics")

  metric_requests = prometheus:counter(
    "nginx_http_requests_total", "Number of HTTP requests", {"host", "status"})
  metric_latency = prometheus:histogram(
    "nginx_http_request_duration_seconds", "HTTP request latency", {"host"})
  metric_connections = prometheus:gauge(
    "nginx_http_connections", "Number of HTTP connections", {"state"})
}

log_by_lua_block {
  metric_requests:inc(1, {ngx.var.server_name, ngx.var.status})
  metric_latency:observe(tonumber(ngx.var.request_time), {ngx.var.server_name})
}

This:

  • configures a shared dictionary for your metrics called prometheus_metrics with a 10MB size limit;
  • registers a counter called nginx_http_requests_total with two labels: host and status;
  • registers a histogram called nginx_http_request_duration_seconds with one label host;
  • registers a gauge called nginx_http_connections with one label state;
  • on each HTTP request measures its latency, recording it in the histogram and increments the counter, setting current server name as the host label and HTTP status code as the status label.

Last step is to configure a separate server that will expose the metrics. Please make sure to only make it reachable from your Prometheus server:

server {
  listen 9145;
  allow 192.168.0.0/16;
  deny all;
  location /metrics {
    content_by_lua_block {
      metric_connections:set(ngx.var.connections_reading, {"reading"})
      metric_connections:set(ngx.var.connections_waiting, {"waiting"})
      metric_connections:set(ngx.var.connections_writing, {"writing"})
      prometheus:collect()
    }
  }
}

Metrics will be available at http://your.nginx:9145/metrics. Note that the gauge metric in this example contains values obtained from nginx global state, so they get set immediately before metrics are returned to the client.

API reference

init()

syntax: require("prometheus").init(dict_name, [options]])

Initializes the module. This should be called once from the init_worker_by_lua_block section of nginx configuration.

  • dict_name is the name of the nginx shared dictionary which will be used to store all metrics. Defaults to prometheus_metrics if not specified.
  • options is a table of configuration options that can be provided. Accepted options are:
    • prefix (string): metric name prefix. This string will be prepended to metric names on output.
    • error_metric_name (string): Can be used to change the default name of error metric (see Built-in metrics for details).
    • sync_interval (number): sets the sync interval for per-worker counters and key index (in seconds). This sets the boundary on eventual consistency of counter metric increments, and metric resets/deletions. Defaults to 1.

Returns a prometheus object that should be used to register metrics.

Example:

init_worker_by_lua_block {
  prometheus = require("prometheus").init("prometheus_metrics", {sync_interval=3})
}

prometheus:counter()

syntax: prometheus:counter(name, description, label_names)

Registers a counter. Should be called once for each counter from the init_worker_by_lua_block section.

  • name is the name of the metric.
  • description is the text description that will be presented to Prometheus along with the metric. Optional (pass nil if you still need to define label names).
  • label_names is an array of label names for the metric. Optional.

Naming section of Prometheus documentation provides good guidelines on choosing metric and label names.

Returns a counter object that can later be incremented.

Example:

init_worker_by_lua_block {
  prometheus = require("prometheus").init("prometheus_metrics")

  metric_bytes = prometheus:counter(
    "nginx_http_request_size_bytes", "Total size of incoming requests")
  metric_requests = prometheus:counter(
    "nginx_http_requests_total", "Number of HTTP requests", {"host", "status"})
}

prometheus:gauge()

syntax: prometheus:gauge(name, description, label_names)

Registers a gauge. Should be called once for each gauge from the init_worker_by_lua_block section.

  • name is the name of the metric.
  • description is the text description that will be presented to Prometheus along with the metric. Optional (pass nil if you still need to define label names).
  • label_names is an array of label names for the metric. Optional.

Returns a gauge object that can later be set.

Example:

init_worker_by_lua_block {
  prometheus = require("prometheus").init("prometheus_metrics")

  metric_connections = prometheus:gauge(
    "nginx_http_connections", "Number of HTTP connections", {"state"})
}

prometheus:histogram()

syntax: prometheus:histogram(name, description, label_names, buckets)

Registers a histogram. Should be called once for each histogram from the init_worker_by_lua_block section.

  • name is the name of the metric.
  • description is the text description. Optional.
  • label_names is an array of label names for the metric. Optional.
  • buckets is an array of finite numbers defining strictly increasing bucket boundaries. Optional, defaults to 20 latency buckets covering a range from 5ms to 10s (in seconds).

Returns a histogram object that can later be used to record samples.

Example:

init_worker_by_lua_block {
  prometheus = require("prometheus").init("prometheus_metrics")

  metric_latency = prometheus:histogram(
    "nginx_http_request_duration_seconds", "HTTP request latency", {"host"})
  metric_response_sizes = prometheus:histogram(
    "nginx_http_response_size_bytes", "Size of HTTP responses", nil,
    {10,100,1000,10000,100000,1000000})
}

prometheus:collect()

syntax: prometheus:collect()

Presents all metrics in a text format compatible with Prometheus. This should be called in content_by_lua_block to expose the metrics on a separate HTTP page.

Example:

location /metrics {
  content_by_lua_block { prometheus:collect() }
  allow 192.168.0.0/16;
  deny all;
}

prometheus:metric_data()

syntax: prometheus:metric_data()

Returns metric data as an array of strings.

counter:inc()

syntax: counter:inc(value, label_values)

Increments a previously registered counter. This is usually called from log_by_lua_block globally or per server/location.

  • value is a value that should be added to the counter. Defaults to 1.
  • label_values is an array of label values.

The number of label values should match the number of label names defined when the counter was registered using prometheus:counter(). No label values should be provided for counters with no labels. Non-printable characters will be stripped from label values.

Example:

log_by_lua_block {
  metric_bytes:inc(tonumber(ngx.var.request_length))
  metric_requests:inc(1, {ngx.var.server_name, ngx.var.status})
}

counter:del()

syntax: counter:del(label_values)

Delete a previously registered counter. This is usually called when you don't need to observe such counter (or a metric with specific label values in this counter) any more. If this counter has labels, you have to pass label_values to delete the specific metric of this counter. If you want to delete all the metrics of a counter with labels, you should call Counter:reset().

  • label_values is an array of label values.

The number of label values should match the number of label names defined when the counter was registered using prometheus:counter(). No label values should be provided for counters with no labels. Non-printable characters will be stripped from label values.

This function will wait for sync_interval before deleting the metric to allow all workers to sync their counters.

counter:reset()

syntax: counter:reset()

Delete all metrics for a previously registered counter. If this counter have no labels, it is just the same as Counter:del() function. If this counter have labels, it will delete all the metrics with different label values.

This function will wait for sync_interval before deleting the metrics to allow all workers to sync their counters.

gauge:set()

syntax: gauge:set(value, label_values)

Sets the current value of a previously registered gauge. This could be called from log_by_lua_block globally or per server/location to modify a gauge on each request, or from content_by_lua_block just before prometheus::collect() to return a real-time value.

  • value is a value that the gauge should be set to. Required.
  • label_values is an array of label values.

gauge:inc()

syntax: gauge:inc(value, label_values)

Increments or decrements a previously registered gauge. This is usually called when you want to observe the real-time value of a metric that can both be increased and decreased.

  • value is a value that should be added to the gauge. It could be a negative value when you need to decrease the value of the gauge. Defaults to 1.
  • label_values is an array of label values.

The number of label values should match the number of label names defined when the gauge was registered using prometheus:gauge(). No label values should be provided for gauges with no labels. Non-printable characters will be stripped from label values.

gauge:del()

syntax: gauge:del(label_values)

Delete a previously registered gauge. This is usually called when you don't need to observe such gauge (or a metric with specific label values in this gauge) any more. If this gauge has labels, you have to pass label_values to delete the specific metric of this gauge. If you want to delete all the metrics of a gauge with labels, you should call Gauge:reset().

  • label_values is an array of label values.

The number of label values should match the number of label names defined when the gauge was registered using prometheus:gauge(). No label values should be provided for gauges with no labels. Non-printable characters will be stripped from label values.

gauge:reset()

syntax: gauge:reset()

Delete all metrics for a previously registered gauge. If this gauge have no labels, it is just the same as Gauge:del() function. If this gauge have labels, it will delete all the metrics with different label values.

histogram:observe()

syntax: histogram:observe(value, label_values)

Records a value in a previously registered histogram. Usually called from log_by_lua_block globally or per server/location.

Bucket counters and _sum are updated independently. An observation may become visible in the sum and count in different scrapes.

  • value is a value that should be recorded. Required.
  • label_values is an array of label values.

Example:

log_by_lua_block {
  metric_latency:observe(tonumber(ngx.var.request_time), {ngx.var.server_name})
  metric_response_sizes:observe(tonumber(ngx.var.bytes_sent))
}

histogram:reset()

syntax: histogram:reset()

Delete all metrics for a previously registered histogram.

This function will wait for sync_interval before deleting the metrics to allow all workers to sync their counters.

Built-in metrics

The module increments an error metric called nginx_metric_errors_total (unless another name was configured in init()) if it encounters an error (for example, when lua_shared_dict becomes full). You might want to configure an alert on that metric.

Caveats

Usage in stream module

For now, there is no way to share a dictionary between HTTP and Stream modules in Nginx. If you are using this library to collect metrics from stream module, you will need to configure a separate endpoint to return them. Here's an example.

server {
  listen 9145;
  content_by_lua_block {
    local sock = assert(ngx.req.socket(true))
    local data = sock:receive()
    local location = "GET /metrics"
    if string.sub(data, 1, string.len(location)) == location then
      ngx.say("HTTP/1.1 200 OK")
      ngx.say("Content-Type: text/plain")
      ngx.say("")
      ngx.say(table.concat(prometheus:metric_data(), ""))
    else
      ngx.say("HTTP/1.1 404 Not Found")
    end
  }
}

Known issues

libnginx-mod-http-lua broken in some Debian and Ubuntu versions

Note that recent stable versions of Debian and Ubuntu are known to package ngx_lua version incompatible with the version of nginx shipped in the same distro. This results in nginx process segfaulting when the lua module is used, making it completely unusable. In such case nginx error logs will clearly indicate that the process crashed, e.g.:

[alert] 123#123: worker process 45678 exited on signal 11

The following versions of Debian and Ubuntu have been known to have this issue:

ngx.sleep(0) called without delayed events patch

You might see the following warning logged by nginx when metrics are being collected:

[warn] ngx.sleep(0) called without delayed events patch, this will hurt performance

This is a side effect of a performance optimization added for OpenResty. While generating metric data, the library aims to occasionally yield control back to the nginx event loop to avoid blocking the worker process and causing increased latency for other in-flight requests.

This warning is displayed when the library is used with nginx that lacks OpenResty patches and can be ignored.

Troubleshooting

Make sure that nginx lua module is enabled

If you experience problems indicating that nginx doesn't know how to interpret lua scripts, please make sure that the lua module is enabled. You might need something like this in your nginx.conf:

load_module modules/ndk_http_module.so;
load_module modules/ngx_http_lua_module.so;

Keep lua code cache enabled

This module expects the lua_code_cache option to be on (which is the default).

Try using an older version of the library

If you are seeing library initialization errors, followed by errors for each metric change request (e.g. attempt to index global '...' (a nil value)), you are probably using an old version of lua-nginx-module. For example, this will happen if you try using the latest version of this library with the nginx-extras package shipped with Ubuntu 16.04.

If you cannot upgrade nginx and lua-nginx-module, you can try using an older version of this library; it will not have the latest performance optimizations, but will still be functional. The recommended older release to use is 0.20181120.

Development

Install dependencies for testing

  • luarocks install luacheck
  • luarocks install luaunit

Run tests

  • luacheck --globals ngx -- prometheus.lua
  • lua prometheus_test.lua
  • cd integration && ./test.sh (requires Docker and Go)

Releasing new version

  • update CHANGELOG.md
  • update version in the dist.ini
  • rename .rockspec file and update version inside it
  • commit changes
  • create a new Git tag: git tag 0.XXXXXXXX && git push origin 0.XXXXXXXX
  • push to luarocks: luarocks upload nginx-lua-prometheus-0.20181120-1.rockspec
  • upload to OPM: opm build && opm upload

Credits

  • Created and maintained by Anton Tolchanov (@knyar)
  • Metrix prefix support contributed by david birdsong (@davidbirdsong))
  • Gauge support contributed by Cosmo Petrich (@cosmopetrich))
  • Performance improvements and per-worker counters are contributed by Wangchong Zhou (@fffonion) / @Kong.
  • Metric name tracking improvements contributed by Jan Dolinár (@dolik-rce))

License

Licensed under MIT license.

Third Party License

Following third party modules are used in this library:

This module is licensed under the Apache 2.0 license.

Copyright (C) 2019, Kong Inc.

All rights reserved.

Frequently asked questions

Is nginx-lua-prometheus free to use?

nginx-lua-prometheus is open source under the MIT licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does nginx-lua-prometheus do?

Prometheus metric library for Nginx written in Lua

What is nginx-lua-prometheus written in?

nginx-lua-prometheus is primarily written in Lua. Its source is publicly available at https://github.com/knyar/nginx-lua-prometheus, and it has 1,573 GitHub stars.