[opensourcedex]

kafka is a free, open source messaging & event streaming project written in Java and released under Apache-2.0. It has 33,750 GitHub stars, 15,521 forks and 574 open issues, and was last pushed 2 hours ago. On this registry it ranks #1 of 13 tracked projects in Messaging & Event Streaming, with 5 head-to-head comparisons available.

What is kafka?

Apache Kafka is an open-source distributed event streaming platform, written in Java and Scala, for organizations that need high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

What it is

Apache Kafka is an Apache-2.0 licensed distributed event streaming platform hosted at github.com/apache/kafka, with the project homepage at https://kafka.apache.org/. The codebase is primarily Java with Scala components, and Scala 2.13 is the only supported Scala version. It is built and tested with Gradle. The repository carries 33,750 stars, 15,521 forks, and 574 open issues, with the last push recorded on 2026-09-18. Builds and tests run on Java 17 and Java 25; the javac release parameter is set to 11 for the clients and streams modules and 17 for the rest, and the scalac release parameter is set to 11 for the streams modules and 17 for the rest.

The project solves the problem of moving and processing continuous event data at scale. The README describes it as used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. It lives in the JVM ecosystem and is consumed as a built JAR artifact. The supplied facts do not name a specific competing product that Kafka replaces; the README positions it as a platform in its own right rather than a drop-in substitute for a named tool.

Key capabilities

  • Distributed event streaming used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
  • Builds a runnable JAR with ./gradlew jar, and a source JAR with ./gradlew srcJar.
  • Generates API documentation through ./gradlew javadoc, ./gradlew javadocJar, ./gradlew scaladoc, ./gradlew scaladocJar, and ./gradlew docsJar, plus aggregated javadoc via ./gradlew aggregatedJavadoc --no-parallel.
  • Splits verification into ./gradlew test, ./gradlew unitTest, and ./gradlew integrationTest.
  • Supports per-module and per-method test targeting, for example ./gradlew clients:test --tests RequestResponseTest and ./gradlew streams:integration-tests:test --tests RestoreIntegrationTest.
  • Runs flaky-marked tests with -Pkafka.test.run.flaky=true and forces reruns with --rerun-tasks.
  • Configures test retries through -PmaxTestRetries and -PmaxTestRetryFailures, and adjusts logging in each module's src/test/resources/log4j2.yaml.

Who uses it and how

  • Companies running high-performance data pipelines, streaming analytics, and data integration workloads, including mission-critical applications.
  • JVM engineering teams that build and test against Java 17 and Java 25 with Gradle.
  • Contributors running unit and integration suites per module, for example clients and streams:integration-tests targets.
  • Release and documentation engineers producing javadoc, scaladoc, and aggregated documentation JARs per module.
  • Teams investigating unstable tests using the flaky test flag, retry properties, and module-level log4j2.yaml level changes.

Getting started

Build a JAR with ./gradlew jar, then follow the quickstart instructions at https://kafka.apache.org/quickstart. Java must be installed before building.

How it compares

The supplied facts provide no list of paid products that Kafka replaces, and they name no comparable open-source tool. On the evidence available, this project stands alone in this registry rather than being contrasted against alternatives.

When to use it — and when not to

Choose it if the team works in the JVM toolchain and is comfortable operating a Gradle build, since building, testing, and documentation generation all run through Gradle with Java 17 or 25 and Scala 2.13. Be aware that the supplied README excerpt is contributor-facing: it documents build and test commands in detail but says nothing about runtime deployment, cluster sizing, storage, or operational prerequisites, so a self-hoster must get that guidance from the quickstart and project site rather than the repository README. It is a poor fit for teams that want a fully managed service or that cannot adopt the JVM toolchain.

project readme (upstream, from github) — read inline

CI Flaky Test Report

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

You need to have Java installed.

We build and test Apache Kafka with Java versions 17 and 25. The release parameter in javac is set to 11 for the clients and streams modules, and 17 for the rest, ensuring compatibility with their respective minimum Java versions. Similarly, the release parameter in scalac is set to 11 for the streams modules and 17 for the rest.

Scala 2.13 is the only supported version in Apache Kafka.

Build a JAR and run it

./gradlew jar

Follow instructions in https://kafka.apache.org/quickstart

Build source JAR

./gradlew srcJar

Build aggregated javadoc

./gradlew aggregatedJavadoc --no-parallel

Build javadoc and scaladoc

./gradlew javadoc
./gradlew javadocJar # builds a javadoc jar for each module
./gradlew scaladoc
./gradlew scaladocJar # builds a scaladoc jar for each module
./gradlew docsJar # builds both (if applicable) javadoc and scaladoc jars for each module

Run unit/integration tests

./gradlew test  # runs both unit and integration tests
./gradlew unitTest
./gradlew integrationTest
./gradlew test -Pkafka.test.run.flaky=true  # runs tests that are marked as flaky

Force re-running tests without code change

./gradlew test --rerun-tasks
./gradlew unitTest --rerun-tasks
./gradlew integrationTest --rerun-tasks

Running a particular unit/integration test

./gradlew clients:test --tests RequestResponseTest
./gradlew streams:integration-tests:test --tests RestoreIntegrationTest

Running a particular unit/integration test N times

N=500; I=0; while [ $I -lt $N ] && ./gradlew clients:test --tests RequestResponseTest --rerun --fail-fast; do (( I=$I+1 )); echo "Completed run: $I"; sleep 1; done

Running a particular test method within a unit/integration test

./gradlew clients:test --tests org.apache.kafka.clients.MetadataTest.testTimeToNextUpdate
./gradlew clients:clients-integration-tests:test --tests org.apache.kafka.clients.producer.ProducerFailureHandlingTest.testCannotSendToInternalTopic
./gradlew streams:integration-tests:test --tests org.apache.kafka.streams.integration.RestoreIntegrationTest.shouldRestoreNullRecord

Running a particular unit/integration test with log4j output

By default, there will be only a small number of logs output while testing. You can adjust it by changing the log4j2.yaml file in the module's src/test/resources directory.

For example, if you want to see more logs for clients project tests, you can modify the line in clients/src/test/resources/log4j2.yaml to level: INFO and then run:

./gradlew cleanTest clients:test --tests NetworkClientTest

And you should see INFO level logs in the file under the clients/build/test-results/test directory.

Specifying test retries

Retries are disabled by default, but you can set maxTestRetryFailures and maxTestRetries to enable retries.

The following example declares -PmaxTestRetries=1 and -PmaxTestRetryFailures=3 to enable a failed test to be retried once, with a total retry limit of 3.

./gradlew test -PmaxTestRetries=1 -PmaxTestRetryFailures=3

See Test Retry Gradle Plugin and build.yml for more details.

Generating test coverage reports

Generate coverage reports for the whole project:

./gradlew reportCoverage -PenableTestCoverage=true -Dorg.gradle.parallel=false

Generate coverage for a single module, i.e.: 

./gradlew clients:reportCoverage -PenableTestCoverage=true -Dorg.gradle.parallel=false

Coverage reports are located within the module's build directory, categorized by module type:

Core Module (:core): core/build/reports/scoverageTest/index.html

Other Modules: /build/reports/jacoco/test/html/index.html

Building a binary release gzipped tarball

./gradlew clean releaseTarGz

The release file can be found inside ./core/build/distributions/.

Building auto-generated messages

Sometimes it is only necessary to rebuild the RPC auto-generated message data when switching between branches, as they could fail due to code changes. You can just run:

./gradlew processMessages processTestMessages

See Apache Kafka Message Definitions for details on Apache Kafka message protocol.

Running a Kafka broker

Using compiled files:

KAFKA_CLUSTER_ID="$(./bin/kafka-storage.sh random-uuid)"
./bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties
./bin/kafka-server-start.sh config/server.properties

Using docker image:

docker run -p 9092:9092 apache/kafka:latest

See docker/README.md for detailed information.

Cleaning the build

./gradlew clean

Running a task for a specific project

This is for core, examples and clients

./gradlew core:jar
./gradlew core:test

Streams has multiple sub-projects, but you can run all the tests:

./gradlew :streams:testAll

Listing all gradle tasks

./gradlew tasks

Building IDE project

Note: Please ensure that JDK 17 is used when developing Kafka.

IntelliJ supports Gradle natively, and it will automatically check Java syntax and compatibility for each module, even if the Java version shown in the Structure > Project Settings > Modules may not be the correct one.

When it comes to Eclipse, run:

./gradlew eclipse

The eclipse task has been configured to use ${project_dir}/build_eclipse as Eclipse's build directory. Eclipse's default build directory (${project_dir}/bin) clashes with Kafka's scripts directory, and we don't use Gradle's build directory to avoid known issues with this configuration.

Publishing the streams quickstart archetype artifact to maven

For the Streams archetype project, one cannot use gradle to upload to maven; instead the mvn deploy command needs to be called at the quickstart folder:

cd streams/quickstart
mvn deploy

Please note for this to work you should create/update user maven settings (typically, ${USER_HOME}/.m2/settings.xml) to assign the following variables

...                           

   ...

      apache.snapshots.https
      ${maven_username}
      ${maven_password}


      apache.releases.https
      ${maven_username}
      ${maven_password}

    ...

 ...

Installing all projects to the local Maven repository

./gradlew -PskipSigning=true publishToMavenLocal

Installing specific projects to the local Maven repository

./gradlew -PskipSigning=true :streams:publishToMavenLocal

Building the test JAR

./gradlew testJar

Running code quality checks

There are two code quality analysis tools that we regularly run, SpotBugs and Checkstyle.

Checkstyle

Checkstyle enforces a consistent coding style in Kafka. You can run Checkstyle using:

./gradlew checkstyleMain checkstyleTest spotlessCheck

The Checkstyle warnings will be found in reports/checkstyle/reports/main.html and reports/checkstyle/reports/test.html files in the subproject build directories. They are also printed to the console.

readme truncated — read the full docs on github

Frequently asked questions

Is kafka free to use?

kafka is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does kafka do?

Apache Kafka - A distributed event streaming platform

What is kafka written in?

kafka is primarily written in Java. Its source is publicly available at https://github.com/apache/kafka, and it has 33,750 GitHub stars.