file.d is an open-source, Go-based event pipeline engine for reading, processing, and outputting data — built primarily for teams that collect logs at scale and need throughput well beyond what conventional log shippers provide.

What it is

file.d is a tool for building data pipelines: it reads events, processes them, and writes them out. It was developed primarily to read from files, but it ships with a broad set of input, action, and output plugins, so the same binary handles files, sockets, HTTP, Kafka, journalctl, dmesg, and Kubernetes sources. It lives in the log collection and observability ecosystem alongside other shippers, and it is distributed as a Go module at github.com/ozontech/file.d under the BSD-3-Clause licence, with documentation hosted at ozontech.github.io/file.d. Configuration is YAML, and the project advertises itself as container, cloud, and Kubernetes native.

The concrete problem it targets is throughput. The README argues that existing tools such as vector, filebeat, logstash, fluentd, and fluent-bit top out at roughly 100MB/sec, while modern disks, network interfaces, and CPUs can move gigabytes per second. file.d claims more than 10x the throughput of comparable tools, and it addresses the two operational worries that usually come with a fast log shipper: memory growth, handled through pooling so consumption stays bounded, and data loss, handled through a commitment mechanism that only advances once events are safely written.

Key capabilities

• Input plugins cover file, dmesg, fake, http, journalctl, k8s, kafka, and socket sources.
• Action plugins number in the dozens, including json_decode, json_encode, json_extract, parse_re2, parse_es, mask, keep_fields, remove_fields, rename, move, flatten, join, split, throttle, cardinality, convert_log_level, convert_date, and convert_utf8_bytes.
• Output plugins target the topics the project tracks, including ClickHouse, Elasticsearch, GELF, Kafka, and HTTP.
• Prometheus integration is native: the event_to_metrics action turns events into metrics at any pipeline stage.
• HashiCorp Vault integration stores sensitive values and injects them into any pipeline parameter.
• Pipelines are declared in YAML only, with no code required to assemble inputs, actions, and outputs.
• Pooling and a commitment mechanism give predictable memory use and reliable delivery rather than best-effort forwarding.

Who uses it and how

• The maintainers run it in production to collect logs from a Kubernetes cluster with more than 3000 total CPU cores.
• Benchmarks on a MacBook Pro 2017 with two physical cores reach 1.7GB/s for the files > devnull case and 1.0GB/s for files > json decode > devnull.
• Teams routing events through Kafka can use it as both an input and an output, chaining decode and transform actions between the two.
• Operators sending logs to ClickHouse or Elasticsearch use the matching output plugins and the parse_es and json_extract actions to shape records before writing.
• The k8s and journalctl inputs suit cluster and host log collection where file-based reading alone is not enough.

Getting started

The README excerpt supplied for this entry does not name an install command, Go package path for installation, Docker image, or compose file; it points to https://ozontech.github.io/file.d for documentation. Start there for the installation and deployment method, then configure a pipeline in YAML.

How it compares

The README names vector, filebeat, logstash, fluentd, and fluent-bit as the comparable tools it positions against, citing their roughly 100MB/sec throughput figure and claiming more than 10x that rate for file.d. It competes with them on raw event throughput, on configuration, and on licence — BSD-3-Clause and self-hosted, like the others named, rather than on a managed service. Where those tools are widely deployed and stable, file.d asks to be judged on pipeline speed, memory predictability, and delivery guarantees.

When to use it — and when not to

A self-hoster must operate a Go binary driven by hand-written YAML, and must accept that the project is still short of v1.0.0 — the README explicitly warns users to test pipelines carefully in dev and stage environments first. Teams that need a frozen, long-term-stable release with formal support contracts should look elsewhere, and the 73 open issues indicate active but not quiet development. The supplied README excerpt is also thin on deployment instructions, so anyone evaluating it should budget time in the project documentation before committing to it.