[opensourcedex]

AutoMQ is a free, open source messaging & event streaming project written in Java and released under Apache-2.0. It has 10,920 GitHub stars, 800 forks and 60 open issues, and was last pushed 8 hours ago. On this registry it ranks #4 of 18 tracked projects in Messaging & Event Streaming, with 5 head-to-head comparisons available. It gained 28 stars over the last 30 tracked days.

What is AutoMQ?

AutoMQ is an Apache-2.0 licensed, Java-based diskless distribution of Kafka that keeps stream data on S3 instead of broker-local disks, aimed at platform and infrastructure teams running event streaming at cloud scale.

What it is

AutoMQ is a rework of Apache Kafka in which the durability layer moves off broker-attached volumes and onto S3. The project describes itself as "Diskless Kafka® on S3" and sits in the Kafka and event-streaming ecosystem; the repository's topic list is automq, diskless, kafka, and serverless. The codebase is Java, released under the Apache-2.0 licence. The project's own description claims tenfold cost savings, no cross-AZ traffic cost, autoscaling in seconds, single-digit-millisecond latency, and multi-AZ availability.

The problem it addresses is structural. Conventional Kafka brokers pin data to local volumes, which forces capacity planning around peak load, keeps replicas spread across availability zones so replication traffic crosses zone boundaries, and turns scaling into a rebalancing operation instead of an instant adjustment. AutoMQ replaces that local-disk storage layer with object storage, so the durable copy of the log lives in S3 and a broker is no longer bound to the disk beneath it. The concrete artefacts replaced are the disk-backed Kafka broker and, for teams on managed services, the AWS MSK cluster that at least one published customer story describes migrating away from.

Key capabilities

  • Diskless storage model places stream data on S3 rather than on broker-local disks, removing the broker as the holder of the durable log.
  • Published comparison material covering AutoMQ versus Apache Kafka on both cost and performance.
  • Claimed 10x cost savings and elimination of cross-AZ traffic cost.
  • Autoscaling measured in seconds, positioned against the over-provisioning that fixed broker fleets require.
  • Single-digit-millisecond latency with multi-AZ availability, per the project description.
  • Apache-2.0 licence on a Java codebase, with 10,770 stars, 771 forks, and 60 open issues at the time of writing.
  • Repository topics automq, diskless, kafka, and serverless define the project's scope.

Who uses it and how

  • LG U+ runs 2.2 billion messages per day through AutoMQ to modernise log pipelines on AWS.
  • JD.com replaces Kafka with AutoMQ at 40 GiB/s scale.
  • Geely uses it for hybrid-cloud streaming across a 10-million-vehicle fleet.
  • Honda reports cutting Kafka TCO by 50% with the cloud-native diskless architecture.
  • Avia Games migrated off AWS MSK to remove maintenance disruptions, while Grab reports 3x data streaming efficiency and Tencent Cloud EMR ships AutoMQ as a first-party service.

Getting started

The README excerpt points readers to the project documentation and to an "Explore AutoMQ Playground" hosted option. It does not list a package name, Docker image, or compose file in the material available here, so installation detail must come from the documentation at automq.com.

How it compares

The facts name Apache Kafka and AWS MSK as the reference points, and AutoMQ publishes cost and performance comparisons against Apache Kafka while one customer story documents a migration away from AWS MSK. Apache Kafka is the open-source baseline it reworks; AWS MSK is the managed alternative some adopters are leaving. The distinction the project draws is a diskless, S3-backed deployment model against disk-backed brokers, whether self-run or provider-run.

When to use it — and when not to

A self-hoster still has to run brokers, own an S3 bucket, and manage access controls and scaling policy around it, so this is not an operational free lunch. Teams with no Kafka-shaped workload gain little, since the value sits in the Kafka storage layer specifically. Note also that the README excerpt is heavy on badges and customer links and light on installation instructions, that 60 issues remain open, and that the headline cost and latency figures are vendor-published claims rather than independently verified results in these facts.

project readme (upstream, from github) — read inline

Diskless Kafka® on S3, Offering 10x Cost Savings and Scaling in Seconds.

📑  Documentation    🔥  Explore AutoMQ Playground Now   

Linkedin Badge Gurubase DeepWiki

AutoMQ%2Fautomq | Trendshift

automq-solgan
Grab LG U+ Tencent Honda JD
Bambu Lab miHoYo Hungry Studio FunPlus Avia Games

Prerequisites

Before running AutoMQ locally, please ensure:

  • Docker version 20.x or later
  • Docker Compose v2
  • At least 4 GB RAM allocated to Docker
  • Ports 9092 and 9000 are available on your system

[!Tip] Deploying a production-ready AutoMQ cluster is challenging. This Quick Start is only for evaluating AutoMQ features and is not suitable for production use. For production deployment best practices, please contact our community for support.

The docker/docker-compose.yaml file provides a simple single-node setup for quick evaluation and development:

curl -O https://raw.githubusercontent.com/AutoMQ/automq/refs/tags/1.5.5/docker/docker-compose.yaml && docker compose -f docker-compose.yaml up -d

This setup features a single AutoMQ node serving as both controller and broker, alongside MinIO for S3 storage. All services operate within a Docker bridge network called automq_net, allowing you to start a Kafka producer in this network to test AutoMQ:

docker run --network automq_net automqinc/automq:latest /bin/bash -c \
"/opt/automq/kafka/bin/kafka-producer-perf-test.sh --topic test-topic --num-records=1024000 --throughput 5120 --record-size 1024 \
--producer-props bootstrap.servers=server1:9092 linger.ms=100 batch.size=524288 buffer.memory=134217728 max.request.size=67108864"

After testing, you can destroy the setup with:

docker compose -f docker-compose.yaml down

The docker/docker-compose-cluster.yaml file offers a more complex setup with three AutoMQ nodes, ideal for testing AutoMQ's cluster features, and can be run in the same way.

There are more deployment options available:

🗞️ Newest Feature - Table Topic

Table Topic is a new feature in AutoMQ that combines stream and table functionalities to unify streaming and data analysis. Currently, it supports Apache Iceberg and integrates with catalog services such as AWS Glue, HMS, and the Rest catalog. Additionally, it natively supports S3 tables, a new AWS product announced at the 2024 re:Invent. Learn more.

image

🔶 Why AutoMQ

AutoMQ is a stateless Kafka alternative that runs on S3 or any S3-compatible storage, such as MinIO. It is designed to address two major issues of Apache Kafka. First, Kafka clusters are difficult to scale out or in due to the stateful nature of its brokers. Data movement is required, and even reassigning partitions between brokers is a complex process. Second, hosting Kafka in the cloud can be prohibitively expensive. You face high costs for EBS storage, cross-AZ traffic, and significant over-provisioning due to Kafka's limited scalability.

Here are some key highlights of AutoMQ that make it an ideal choice to replace your Apache Kafka cluster, whether in the cloud or on-premise, as long as you have S3-compatible storage:

  • Cost effective: The first true cloud-native streaming storage system, designed for optimal cost and efficiency on the cloud. Refer to this report to see how we cut Apache Kafka billing by 90% on the cloud.
  • High Reliability: Leverage object storage service to achieve zero RPO, RTO in seconds and 99.999999999% durability.
  • Zero Cross-AZ Traffic: By using cloud object storage as the priority storage solution, AutoMQ eliminates cross-AZ traffic costs on AWS and GCP. In traditional Kafka setups, over 80% of costs arise from cross-AZ traffic, including producer, consumer, and replication sides.
  • Serverless:
    • Auto Scaling: Monitor cluster metrics and automatically scale in/out to align with your workload, enabling a pay-as-you-go model.
    • Scaling in seconds: The computing layer (broker) is stateless and can scale in/out within seconds, making AutoMQ a truly serverless solution.
    • Infinite scalable: Utilize cloud object storage as the primary storage solution, eliminating concerns about storage capacity.
  • Manage-less: The built-in auto-balancer component automatically schedules partitions and network traffic between brokers, eliminating manual partition reassignment.
  • High performance:
    • High throughput: Leverage pre-fetching, batch processing, and parallel technologies to maximize the capabilities of cloud object storage. Refer to the AutoMQ Performance White Paper to see how we achieve this.
    • Low Latency: AutoMQ defaults to running on S3 directly, resulting in hundreds of milliseconds of latency. The enterprise version offers single-digit millisecond latency. Contact us for more details.
  • Built-in Metrics Export: Natively export Prometheus and OpenTelemetry metrics, supporting both push and pull. Ditch inefficient JMX and monitor your cluster with modern tools. Refer to full metrics list provided by AutoMQ.
  • 100% Kafka Compatible: Fully compatible with Apache Kafka, offering all features with greater cost-effectiveness and operational efficiency.

✨Architecture

AutoMQ is a fork of the open-source Apache Kafka. We've introduced a new storage engine based on object storage, transforming the classic shared-nothing architecture into a shared storage architecture.

image

Regarding the architecture of AutoMQ, it is fundamentally different from Kafka. The core difference lies in the storage layer of Apache Kafka and how we leverage object storage to achieve a stateless broker architecture. AutoMQ consists of below key components:

  • S3 Storage Adapter: an adapter layer that reimplements the UnifiedLog, LocalLog, and LogSegment classes to create logs on S3 instead of a local disk. Traditional local disk storage is still supported if desired.
  • S3Stream: a shared streaming storage library that encapsulates various storage modules, including WAL and object storage. WAL is a write-ahead log optimized for frequent writes and low IOPS to reduce S3 API costs. To boost read performance, we use LogCache and BlockCache for improved efficiency.
  • Auto Balancer: a component that automatically balances traffic and partitions between brokers, eliminating the need for manual reassignment. Unlike Kafka, this built-in feature removes the need for cruise control.
  • Rack-aware Router: Kafka has long faced cross-AZ traffic fees on AWS and GCP. Our shared storage architecture addresses this by using a rack-aware router to provide clients in different AZs with specific partition metadata, avoiding cross-AZ fees while exchanging data through object storage.

For more on AutoMQ's architecture, visit AutoMQ Architecture or explore the source code directly.

🌟 Stay Ahead

Star AutoMQ on GitHub for instant updates on new releases. star-automq

💬 Community

You can join the following groups or channels to discuss or ask questions about AutoMQ:

👥 How to contribute

If you've found a problem with AutoMQ, please open a GitHub Issues. To contribute to AutoMQ please see Code of Conduct and Contributing Guide. We have a list of good first issues that help you to get started, gain experience, and get familiar with our contribution process.

👍 AutoMQ Enterprise Edition

The enterprise edition of AutoMQ offers a robust, user-friendly control plane for seamless cluster management, with enhanced availability and observability over the open-source version. Additionally, we offer Kafka Linking for zero-downtime migration from any Kafka-compatible cluster to AutoMQ.

Contact us for more information about the AutoMQ enterprise edition, and we'll gladly assist with your free trial.

📜 License

AutoMQ is under the Apache 2.0 license. See the LICENSE file for details.

📝 Trademarks

Apache®, Apache Kafka®, Kafka®, Apache Iceberg®, Iceberg® and associated open source project names are trademarks of the Apache Software Foundation

Frequently asked questions

Is AutoMQ free to use?

AutoMQ is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does AutoMQ do?

S3-based serverless Kafka reinvented for cost efficiency

What is AutoMQ written in?

AutoMQ is primarily written in Java. Its source is publicly available at https://github.com/automq/automq, and it has 10,920 GitHub stars.