[opensourcedex]

Apache Pinot is a free, open source data warehousing & processing project written in Java and released under Apache-2.0. It has 6,138 GitHub stars, 1,509 forks and 1,403 open issues, and was last pushed 6 hours ago. On this registry it ranks #2 of 2 tracked projects in Data Warehousing & Processing, with 5 head-to-head comparisons available. It gained 3 stars over the last 6 tracked days.

What is Apache Pinot?

What it is

Apache Pinot is a real-time distributed OLAP datastore built to deliver scalable real-time analytics at low latency. It lives in the Java ecosystem, is licensed under Apache-2.0, and was originally built by engineers at LinkedIn and Uber. The project ingests from batch data sources such as Hadoop HDFS, Amazon S3, Azure ADLS, and Google Cloud Storage, as well as from stream data sources such as Apache Kafka. It is designed to scale up and out with no upper bound, and its performance stays constant based on the size of the cluster and an expected query per second threshold.

The concrete problem Pinot solves is serving interactive, user-facing analytics over very large datasets without the latency that batch-oriented warehouses impose. At LinkedIn it powers more than 50 user-facing products, ingesting millions of events per second and serving over 100,000 queries per second at millisecond latency. UberEats Restaurant Manager is another example of a customer-facing analytics application built on it. Pinot targets the gap between raw event streams and the need to filter and aggregate petabyte-scale data interactively in a UI.

Key capabilities

  • Fast queries: filter and aggregate petabyte data sets with P90 latencies in the tens of milliseconds, fast enough to return live results interactively in a UI.
  • High concurrency: with user-facing applications querying Pinot directly, it can serve hundreds of thousands of concurrent queries per second.
  • SQL query interface: a highly standard SQL interface accessible through a built-in query editor and a REST API.
  • Versatile joins: arbitrary fact/dimension and fact/fact joins on petabyte data sets.
  • Column-oriented storage: a column-oriented database with compression schemes such as Run Length and Fixed Bit Length.
  • Pluggable indexing: indexing strategies are pluggable rather than fixed.
  • Batch and stream ingestion: reads from Hadoop HDFS, Amazon S3, Azure ADLS, and Google Cloud Storage, and from Apache Kafka streams.

Who uses it and how

  • LinkedIn uses Pinot to power rich interactive real-time analytic applications such as Who Viewed Profile, Company Analytics, and Talent Insights.
  • LinkedIn runs Pinot across 50+ user-facing products, ingesting millions of events per second and serving 100
project readme (upstream, from github) — read inline
Apache Pinot

Unit Tests Integration Tests Quickstart Tests Compatibility Checks Release codecov.io Join the chat at https://communityinviter.com/apps/apache-pinot/apache-pinot Twitter Follow License Ask DeepWiki

What is Apache Pinot?

Apache Pinot is a real-time distributed OLAP datastore, built to deliver scalable real-time analytics with low latency. It can ingest from batch data sources (such as Hadoop HDFS, Amazon S3, Azure ADLS, Google Cloud Storage) as well as stream data sources (such as Apache Kafka).

Pinot was built by engineers at LinkedIn and Uber and is designed to scale up and out with no upper bound. Performance always remains constant based on the size of your cluster and an expected query per second (QPS) threshold.

For getting started guides, deployment recipes, tutorials, and more, please visit our project documentation at https://docs.pinot.apache.org.

Apache Pinot

Features

Pinot was originally built at LinkedIn to power rich interactive real-time analytic applications such as Who Viewed Profile, Company Analytics, Talent Insights, and many more. UberEats Restaurant Manager is another example of a customer facing Analytics App. At LinkedIn, Pinot powers 50+ user-facing products, ingesting millions of events per second and serving 100k+ queries per second at millisecond latency.

  • Fast Queries: Filter and aggregate petabyte data sets with P90 latencies in the tens of milliseconds—fast enough to return live results interactively in the UI.

  • High Concurrency: With user-facing applications querying Pinot directly, it can serve hundreds of thousands of concurrent queries per second.

  • SQL Query Interface: The highly standard SQL query interface is accessible through a built-in query editor and a REST API.

  • Versatile Joins: Perform arbitrary fact/dimension and fact/fact joins on petabyte data sets.

  • Column-oriented: a column-oriented database with various compression schemes such as Run Length, Fixed Bit Length.

  • Pluggable indexing: pluggable indexing technologies including timestamp, inverted, StarTree, Bloom filter, range, text, JSON, and geospatial options.

  • Stream and batch ingest: Ingest from Apache Kafka, Apache Pulsar, and AWS Kinesis in real time. Batch ingest from Hadoop, Spark, AWS S3, and more. Combine batch and streaming sources into a single table for querying.

  • Upsert during real-time ingestion: update the data at-scale with consistency

  • Built-in Multitenancy: Manage and secure data in isolated logical namespaces for cloud-friendly resource management.

  • Built for Scale: Pinot is horizontally scalable and fault-tolerant, adaptable to workloads across the storage and throughput spectrum.

  • Cloud-native on Kubernetes: Helm chart provides a horizontally scalable and fault-tolerant clustered deployment that is easy to manage using Kubernetes.

Apache Pinot query console

When should I use Pinot?

Pinot is designed to execute real-time OLAP queries with low latency on massive amounts of data and events. In addition to real-time stream ingestion, Pinot also supports batch use cases with the same low latency guarantees. It is suited in contexts where fast analytics, such as aggregations, are needed on immutable data, possibly, with real-time data ingestion. Pinot works very well for querying time series data with lots of dimensions and metrics.

Example query:

SELECT sum(clicks), sum(impressions) FROM AdAnalyticsTable
  WHERE
       ((daysSinceEpoch >= 17849 AND daysSinceEpoch <= 17856)) AND
       accountId IN (123456789)
  GROUP BY
       daysSinceEpoch TOP 100

Contributing to Pinot

Want to contribute to Apache Pinot? 👋🍷

Want to join the ranks of open source committers to Apache Pinot? Then check out the Contribution Guide for how you can get involved in the code.

If you have a bug or an idea for a new feature, browse the open issues to see what we’re already working on before opening a new one.

We also tagged some beginner issues new contributors can tackle.

Apache Pinot YouTube Channel

Share Your Pinot Videos with the Community!

Have a Pinot use case, tutorial, or conference/meetup recording to share? We’d love to feature it on the Pinot OSS YouTube channel! Drop your video or a link to your session in the #pinot-youtube-channel on Pinot Slack, and we’ll showcase it for the community!

Building Pinot

## Clone a repo
$ git clone https://github.com/apache/pinot.git
$ cd pinot

## Pinot services require JDK 25+ to build and run
## Java/JDBC

readme truncated — read the full docs on github

Frequently asked questions

Is Apache Pinot free to use?

Apache Pinot is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does Apache Pinot do?

Distributed OLAP database for real-time analytics at scale

What is Apache Pinot written in?

Apache Pinot is primarily written in Java. Its source is publicly available at https://github.com/apache/pinot, and it has 6,138 GitHub stars.