angle-grinder

Slice and dice log files on the command line.

Angle-grinder allows you to parse, aggregate, sum, average, min/max, percentile, and sort your data. You can see it, live-updating, in your terminal. Angle grinder is designed for when, for whatever reason, you don't have your data in graphite/honeycomb/kibana/sumologic/splunk/etc. but still want to be able to do sophisticated analytics.

Angle grinder can process well above 1M rows per second (simple pipelines as high as 5M), so it's usable for fairly meaty aggregation. The results will live update in your terminal as data is processed. Angle grinder is a bare bones functional programming language coupled with a pretty terminal UI.

Quick Links

• Installation
• Query Syntax Overview
• Operators
  • Parsers: JSON logfmt split generic
  • Misc: Add/remove fields limit timeslice where
  • Aggregators: count sum min max percentile sort total count distinct
• Output Control

Installation

Binaries are available for Linux and OSX. Many more platforms (including Windows) are available if you compile from source. In all of the commands below, the resulting binary will be called agrind. Starting with v0.9.0, agrind can self-update via the --self-update flag. Thanks to the many volunteers who maintain angle-grinder on different package managers & environments!

macOS

Brew

brew install angle-grinder

Macports

sudo port selfupdate
sudo port install angle-grinder

FreeBSD

pkg install angle-grinder

Linux (any MUSL compatible variant)

curl -L https://github.com/rcoh/angle-grinder/releases/download/v0.18.0/agrind-x86_64-unknown-linux-musl.tar.gz \
  | tar Ozxf - \
  | sudo tee /usr/local/bin/agrind > /dev/null && sudo chmod +x /usr/local/bin/agrind

agrind --self-update

Cargo (most platforms)

If you have Cargo installed, you can compile & install from source: (Works with Stable Rust >=1.26)

cargo install ag

Query Syntax

An angle grinder query is composed of filters followed by a series of operators. The filters select the lines from the input stream to be transformed by the operators. Typically, the initial operators will transform the data in some way by parsing fields or JSON from the log line. The subsequent operators can then aggregate or group the data via operators like sum, average, percentile, etc.

agrind '<filter1> [... <filterN>] | operator1 | operator2 | operator3 | ...'

A simple query that operates on JSON logs and counts the number of logs per level could be:

agrind '* | json | count by log_level'

Escaping Field Names

Field names containing spaces, periods, or quotes must be escaped using [""]:

agrind '* | json | count by ["date received"], ["grpc.method"]

Filters

There are three basic filters:

• *: Match all logs
• filter-me* (with no quotes) is a case-insensitive match that can include wildcards
• "filter-me" (in quotes) is a case-sensitive match (no wildcards, * matches literal * , filter-me, or "filter me!".

Filters can be combined with AND, OR and NOT

("ERROR" OR WARN*) AND NOT staging | count

Sub-expressions must be grouped in parenthesis. Only lines that match all filters will be passed to the subsequent operators.

Aliases

Starting with v0.12.0, angle grinder supports aliases, pre-built pipelines do simplify common tasks or formats.

By default, angle-grinder will look in the .agrind-aliases directory in your current working directory and all parent directories.

Alias files look like this:

keyword = "apache"
template = """
parse "* - * [*] \\"* * *\\" * *" as ip, name, timestamp, method, url, protocol, status, contentlength
"""

Your operators are parsed, then expanded into the resulting pipeline. When invalid aliases are present, a warning will be displayed when running angle-grinder.

Note that aliases are currently considered an experimental feature and precise behavior may change in the future.

Examples:

* | apache | count by status

Operators

Non Aggregate Operators

These operators have a 1 to 1 correspondence between input data and output data. 1 row in, 0 or 1 rows out.

JSON

json [from other_field]: Extract json-serialized rows into fields for later use. If the row is not valid JSON, then it is dropped. Optionally, from other_field can be specified. Nested JSON structures are supported out of the box. Simply access nested values with .key[index], for example, .servers[6]. Negative indexing is also supported.

Examples:

* | json

* | parse "INFO *" as js | json from js

Given input like:

{"key": "blah", "nested_key": {"this": "that"}}

* | json | count_distinct(nested_key.this)

Logfmt

logfmt [from other_field]: Extract logfmt-serialized rows into fields for later use. If the row is not valid logfmt, then it is dropped. Optionally, from other_field can be specified. Logfmt is a an output format commonly used by Heroku and Splunk, described at https://www.brandur.org/logfmt.

Examples:

* | logfmt

Given input like:

{"key": "blah", "nested_key": "some=logfmt data=more"}

* | json | logfmt from nested_key | fields some

Split

split[(input_field)] [on separator] [as new_field]: Split the input via the separator (default is ,). Output is an array type. If no input_field or new_field, the contents will be put in the key _split.

Examples:

* | split on " "

Given input like:

INFO web-001 influxd[188053]: 127.0.0.1 "POST /write HTTP/1.0" 204

Output:

[_split=[INFO, web-001, influxd[188053]:, 127.0.0.1, POST /write HTTP/1.0, 204]]

If input_field is used, and there is no new_field specified, then the input_field will be overridden with the split data-structure. For example:

* | parse "* *" as level, csv | split(csv)

Given input like:

INFO darren,hello,50
WARN jonathon,good-bye,100

Will output:

[csv=[darren, hello, 50]]        [level=INFO]
[csv=[jonathon, good-bye, 100]]        [level=WARN]

Other examples:

* | logfmt | split(raw) on "blah" as tokens | sum(tokens[1])

Parse

parse "* pattern * otherpattern *" [from field] as a,b,c [nodrop] [noconvert]: Parse text that matches the pattern into variables.

• Lines that don't match the pattern will be dropped unless nodrop is specified. * is equivalent to regular expression .* and is greedy.
• noconvert will prevent parse from converting parsed fields into structured data and instead preserve them as strings. This can be helpful if you are parsing fields that sometimes have values like 00000.

By default, parse operates on the raw text of the message. With from field_name, parse will instead process input from a specific column. Any whitespace in the parse expression will match any whitespace character in the input text (eg. a literal tab).

Examples:

* | parse "[status_code=*]" as status_code

Parse Regex

parse regex "" [from field] [nodrop]: Match the input text against a regular expression and populate the record with the named captures. Lines that don't match the pattern will be dropped unless nodrop is specified. By default, parse operates on the raw text of the message. With from field_name, parse will instead process input from a specific column.

Notes:

• Only named captures are supported. If the regular expression includes any unnamed captures, an error will be raised.
• The Rust regular expression syntax is used.
• Escape sequences do not require an extra backslash (i.e. \w works as-is).

Examples: To parse the phrase "Hello, ...!" and capture the value of the "..." in the name field:

* | parse regex "Hello, (?P<name>\w+)"

Fields

fields [only|except|-|+] a, b: Drop fields a, b or include only a, b depending on specified mode.

Examples: Drop all fields except event and timestamp

* | json | fields + event, timestamp

Drop only the event field

* | fields except event

Where

`where <bool-