pikiwidb: Open Source Databases

What is pikiwidb?

Pikiwidb is a Redis-compatible, RocksDB-backed persistent key-value database from Qihoo's infrastructure team, built for teams whose Redis working set has outgrown memory and who want to keep the Redis protocol without paying the memory-price curve.

What it is

PikiwiDB is a high-performance, large-capacity, multi-tenant, data-persistent elastic KV data storage system written in C++ and licensed under BSD-3-Clause. It uses RocksDB as its storage engine and is fully compatible with the Redis protocol, supporting the data structures Redis users already depend on. The repository and its tooling still carry the older "pika" name, so the project appears in both Qihoo360/pika and OpenAtomFoundation/pika forms, with a Chinese README alongside the English one.

The concrete problem it solves is the one Redis hits once an in-memory dataset grows past roughly 16GiB: limited memory capacity, single-threaded blocking, long startup and recovery times, high memory hardware costs, buffers that fill easily, and expensive failover when one master serves multiple replicas. PikiwiDB replaces the all-in-memory dataset with data persisted in RocksDB while keeping the Redis wire interface, so existing clients and code keep working. The README is explicit that the goal is not to replace Redis but to complement it, and to preserve Redis's convenient operation and maintenance design.

Key capabilities

Protocol compatibility with Redis, covering String, Hash, List, Zset, Set, Geo, Hyperloglog, Pubsub, Bitmap, Stream, and ACL.
RocksDB storage engine with a separate RocksDB instance per data structure, a multi-threaded model, and multiple-granularity data caching.
Tiered cold and hot data handling: hot data is cached while the full dataset is persisted to RocksDB.
Data volumes in the hundreds of gigabytes, well beyond what in-memory Redis holds on comparable hardware.
Deployment as single-machine master-slave using the slaveof command, or in Codis cluster mode for simple scaling and shrinking.
Binlog-based asynchronous replication with both full and incremental data synchronization.
Migration tooling in the tools/ directory for a smooth Redis-to-PikiwiDB move without code changes, plus supported platforms CentOS, Ubuntu, macOS, and Rocky Linux.

Who uses it and how

Infrastructure teams running Redis datasets that have crossed the memory ceiling and need persistent storage at hundreds of gigabytes.
Operators who want a familiar Redis-like deployment shape and therefore run single-machine master-slave pairs via slaveof rather than a new operational model.
Teams that need horizontal scale and run Codis cluster mode, scaling and shrinking the shard set as load changes.
Multi-tenant platforms that need one KV store serving several tenants with persistent data, rather than per-tenant Redis instances.
Organisations already on Redis that migrate using the tools/ utilities and leave client code untouched.

Getting started

The facts provided do not include an install command, package name, Docker image, or compose file; the README points instead to the project wiki and a Travis CI build status badge for the qihoo360/pika build. Practically, this means building from source from the repository and following the wiki, including the Redis interface compatibility page.

How it compares

Among the similar tools the facts name, PikiwiDB sits directly in the Redis ecosystem rather than beside it: it speaks the Redis protocol, reuses Redis operational conventions such as slaveof, and exists to complement Redis where in-memory capacity runs out. Codis appears as its cluster mode of choice, and RocksDB is the engine it builds on, so the stack is Redis at the interface, RocksDB at the storage layer, and Codis for sharding.

When to use it — and when not to

A self-hoster must operate persistent RocksDB-backed storage, plus either a master-slave replication setup with binlog or a Codis cluster, and must run the migration tooling to move existing data. Teams with datasets comfortably under the memory threshold, or those who want a fully managed hosted service, should stay on plain Redis, since the facts show no hosted option. Honest weaknesses: the provided README excerpt is truncated, no install or Docker instructions appear in the facts, and the repository carries 77 open issues, so expect to read the wiki and source before committing.

project readme (upstream, from github) — read inline

Downloads

Stargazers Over Time	Contributors Over Time

Introduction中文

PikiwiDB is a high-performance, large-capacity, multi-tenant, data-persistent elastic KV data storage system using RocksDB as the storage engine. It is fully compatible with the Redis protocol and supports its commonly used data structures, such as string/hash/list/zset/set/geo/hyperloglog/pubsub/bitmap/stream, etc. Redis Interface.

When Redis's in-memory usage exceeds 16GiB, it faces problems such as limited memory capacity, single-threaded blocking, long startup recovery time, high memory hardware costs, easily filled buffers, and high switching costs when one master and multiple replicas fail. The emergence of PikiwiDB is not to replace Redis but to complement it. PikiwiDB strives to completely comply with the Redis protocol, inherit Redis's convenient operation and maintenance design, and solve the bottleneck problem of Redis running out of memory capacity once the data volume becomes huge by using persistent storage. Additionally, PikiwiDB can support master-slave mode using the slaveof command, and it also supports full and incremental data synchronization.

PikiwiDB can be deployed in a single-machine master-slave mode (slaveof) or in a Codis cluster mode, allowing for simple scaling and shrinking. Migration from Redis to PikiwiDB can be smoothly executed by tools.

PikiwiDB Features

Protocol Compatibility: Fully compatible with the Redis protocol, emphasizing high performance, large capacity, low cost, and scalability.
Data Structures: Supports Redis's common data structures, including String, Hash, List, Zset, Set, Geo, Hyperloglog, Pubsub, Bitmap, Stream, ACL, etc.
Cold and Hot Data: Caches hot data and persistently stores the full data in RocksDB, implementing a hierarchical storage of cold and hot data.
High Capacity: Compared to Redis's in-memory storage, PikiwiDB supports data volumes in the hundreds of gigabytes, significantly reducing server resource consumption and enhancing data reliability.
Deployment Modes: Supports single-machine master-slave mode (slaveof) and Codis cluster mode, making scaling and shrinking simple.
Easy Migration: Smooth migration from Redis to PikiwiDB without modifying code.
Convenient Operation and Maintenance: Comprehensive operation and maintenance command documentation.

PikiwiDB Storage Engine Architecture

Supports multiple platforms: CentOS, Ubuntu, macOS, Rocky Linux
Multi-threaded model
Based on the RocksDB storage engine
Multiple granularity data caching model

Deployment Modes

1. Master-Slave Mode

Architecture similar to Redis
Good compatibility with Redis protocol and data structures
Each data structure uses a separate RocksDB instance
Master-slave adopts binlog asynchronous replication

PikiwiDB-Master-Slave

2. Distributed Cluster Mode

Adopts Codis architecture, supports multiple groups
Each group forms a master-slave set
Elastic scaling based on groups

PikiwiDB-Cluster

PikiwiDB User Showcase

PikiwiDB has been widely adopted by various companies for internal deployments, demonstrating its scalability and reliability. Some notable usage instances include:

360 Company: Internal deployment with a scale of 10,000+ instances, each having a data volume of 1.8TB.
Weibo: Internal deployment with 10,000+ instances.
Ximalaya(Xcache): 6,000+ instances with a massive data volume exceeding 120TB.
Getui (个推) Company: Internal deployment involving 300+ instances, with a cumulative data volume surpassing 30TB.

Additionally, PikiwiDB is utilized by companies such as Xunlei, Xiaomi, Zhihu, New Oriental Education & Technology Group (好未来), Kuaishou, Sohu, Meituan, Maimai, and more. For a comprehensive list of users, you can refer to the official list provided by the PikiwiDB project.

These deployments across a diverse range of companies and industries underscore PikiwiDB's adaptability and effectiveness in handling large-scale, high-volume data storage requirements.

1. Binary Package Installation

Users can directly download the latest binary version package from releases.

2. Compilation from Source

2.1 Supported Platforms
- Linux - CentOS
- Linux - Ubuntu
- macOS(Darwin)
2.2 Required Library Software
- gcc g++ supporting C++17 (version >= 9)
- make
- cmake (version >= 3.18)
- autoconf
- tar

2.3 Compilation Process

2.3.1. Get the source code

  git clone https://github.com/OpenAtomFoundation/pikiwidb.git

2.3.2. Switch to the latest release version

  git tag          # Check the latest release tag (e.g., v3.4.1)
  git checkout TAG # Switch to the latest version (e.g., git checkout v3.4.1)

2.3.3. Execute compilation
If the machine's gcc version is less than 9, especially on CentOS6 or CentOS7, you need to upgrade the gcc version first. Execute the following commands:
```
  sudo yum -y install centos-release-scl
  sudo yum -y install devtoolset-9-gcc devtoolset-9-gcc-c++
  scl enable devtoolset-9 bash
```
For the initial compilation, it is recommended to use the build script

Frequently asked questions

Is pikiwidb free to use?

pikiwidb is open source under the BSD-3-Clause licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does pikiwidb do?

Pikiwidb is a Redis-Compatible database developed by Qihoo's infrastructure team.

What is pikiwidb written in?

pikiwidb is primarily written in C++. Its source is publicly available at https://github.com/OpenAtomFoundation/pikiwidb, and it has 6,129 GitHub stars.