[opensourcedex]

pikiwidb is a free, open source databases project written in C++ and released under BSD-3-Clause. It has 6,137 GitHub stars, 1,170 forks and 77 open issues, and was last pushed 18 days ago. On this registry it ranks #130 of 206 tracked projects in Databases, with 5 head-to-head comparisons available. It gained 4 stars over the last 30 tracked days.

What is pikiwidb?

Pikiwidb is a Redis-compatible, RocksDB-backed persistent key-value database from Qihoo's infrastructure team, built for teams whose Redis working set has outgrown memory and who want to keep the Redis protocol without paying the memory-price curve.

What it is

PikiwiDB is a high-performance, large-capacity, multi-tenant, data-persistent elastic KV data storage system written in C++ and licensed under BSD-3-Clause. It uses RocksDB as its storage engine and is fully compatible with the Redis protocol, supporting the data structures Redis users already depend on. The repository and its tooling still carry the older "pika" name, so the project appears in both Qihoo360/pika and OpenAtomFoundation/pika forms, with a Chinese README alongside the English one.

The concrete problem it solves is the one Redis hits once an in-memory dataset grows past roughly 16GiB: limited memory capacity, single-threaded blocking, long startup and recovery times, high memory hardware costs, buffers that fill easily, and expensive failover when one master serves multiple replicas. PikiwiDB replaces the all-in-memory dataset with data persisted in RocksDB while keeping the Redis wire interface, so existing clients and code keep working. The README is explicit that the goal is not to replace Redis but to complement it, and to preserve Redis's convenient operation and maintenance design.

Key capabilities

  • Protocol compatibility with Redis, covering String, Hash, List, Zset, Set, Geo, Hyperloglog, Pubsub, Bitmap, Stream, and ACL.
  • RocksDB storage engine with a separate RocksDB instance per data structure, a multi-threaded model, and multiple-granularity data caching.
  • Tiered cold and hot data handling: hot data is cached while the full dataset is persisted to RocksDB.
  • Data volumes in the hundreds of gigabytes, well beyond what in-memory Redis holds on comparable hardware.
  • Deployment as single-machine master-slave using the slaveof command, or in Codis cluster mode for simple scaling and shrinking.
  • Binlog-based asynchronous replication with both full and incremental data synchronization.
  • Migration tooling in the tools/ directory for a smooth Redis-to-PikiwiDB move without code changes, plus supported platforms CentOS, Ubuntu, macOS, and Rocky Linux.

Who uses it and how

  • Infrastructure teams running Redis datasets that have crossed the memory ceiling and need persistent storage at hundreds of gigabytes.
  • Operators who want a familiar Redis-like deployment shape and therefore run single-machine master-slave pairs via slaveof rather than a new operational model.
  • Teams that need horizontal scale and run Codis cluster mode, scaling and shrinking the shard set as load changes.
  • Multi-tenant platforms that need one KV store serving several tenants with persistent data, rather than per-tenant Redis instances.
  • Organisations already on Redis that migrate using the tools/ utilities and leave client code untouched.

Getting started

The facts provided do not include an install command, package name, Docker image, or compose file; the README points instead to the project wiki and a Travis CI build status badge for the qihoo360/pika build. Practically, this means building from source from the repository and following the wiki, including the Redis interface compatibility page.

How it compares

Among the similar tools the facts name, PikiwiDB sits directly in the Redis ecosystem rather than beside it: it speaks the Redis protocol, reuses Redis operational conventions such as slaveof, and exists to complement Redis where in-memory capacity runs out. Codis appears as its cluster mode of choice, and RocksDB is the engine it builds on, so the stack is Redis at the interface, RocksDB at the storage layer, and Codis for sharding.

When to use it — and when not to

A self-hoster must operate persistent RocksDB-backed storage, plus either a master-slave replication setup with binlog or a Codis cluster, and must run the migration tooling to move existing data. Teams with datasets comfortably under the memory threshold, or those who want a fully managed hosted service, should stay on plain Redis, since the facts show no hosted option. Honest weaknesses: the provided README excerpt is truncated, no install or Docker instructions appear in the facts, and the repository carries 77 open issues, so expect to read the wiki and source before committing.

project readme (upstream, from github) — read inline

Build Status Downloads

Stargazers Over Time Contributors Over Time
Stargazers over time Contributor over time

Introduction中文

PikiwiDB is a high-performance, large-capacity, multi-tenant, data-persistent elastic KV data storage system using RocksDB as the storage engine. It is fully compatible with the Redis protocol and supports its commonly used data structures, such as string/hash/list/zset/set/geo/hyperloglog/pubsub/bitmap/stream, etc. Redis Interface.

When Redis's in-memory usage exceeds 16GiB, it faces problems such as limited memory capacity, single-threaded blocking, long startup recovery time, high memory hardware costs, easily filled buffers, and high switching costs when one master and multiple replicas fail. The emergence of PikiwiDB is not to replace Redis but to complement it. PikiwiDB strives to completely comply with the Redis protocol, inherit Redis's convenient operation and maintenance design, and solve the bottleneck problem of Redis running out of memory capacity once the data volume becomes huge by using persistent storage. Additionally, PikiwiDB can support master-slave mode using the slaveof command, and it also supports full and incremental data synchronization.

PikiwiDB can be deployed in a single-machine master-slave mode (slaveof) or in a Codis cluster mode, allowing for simple scaling and shrinking. Migration from Redis to PikiwiDB can be smoothly executed by tools.

PikiwiDB Features

  • Protocol Compatibility: Fully compatible with the Redis protocol, emphasizing high performance, large capacity, low cost, and scalability.
  • Data Structures: Supports Redis's common data structures, including String, Hash, List, Zset, Set, Geo, Hyperloglog, Pubsub, Bitmap, Stream, ACL, etc.
  • Cold and Hot Data: Caches hot data and persistently stores the full data in RocksDB, implementing a hierarchical storage of cold and hot data.
  • High Capacity: Compared to Redis's in-memory storage, PikiwiDB supports data volumes in the hundreds of gigabytes, significantly reducing server resource consumption and enhancing data reliability.
  • Deployment Modes: Supports single-machine master-slave mode (slaveof) and Codis cluster mode, making scaling and shrinking simple.
  • Easy Migration: Smooth migration from Redis to PikiwiDB without modifying code.
  • Convenient Operation and Maintenance: Comprehensive operation and maintenance command documentation.

PikiwiDB Storage Engine Architecture

  • Supports multiple platforms: CentOS, Ubuntu, macOS, Rocky Linux
  • Multi-threaded model
  • Based on the RocksDB storage engine
  • Multiple granularity data caching model

Deployment Modes

1. Master-Slave Mode

  • Architecture similar to Redis
  • Good compatibility with Redis protocol and data structures
  • Each data structure uses a separate RocksDB instance
  • Master-slave adopts binlog asynchronous replication

PikiwiDB-Master-Slave

2. Distributed Cluster Mode

  • Adopts Codis architecture, supports multiple groups
  • Each group forms a master-slave set
  • Elastic scaling based on groups

PikiwiDB-Cluster

PikiwiDB User Showcase

Qihoo 360game Weibo Garena
Apus Ffan Meituan XES
HX XL GWD DYD
YM XM XL YM
MM VIP LK KS

PikiwiDB has been widely adopted by various companies for internal deployments, demonstrating its scalability and reliability. Some notable usage instances include:

  • 360 Company: Internal deployment with a scale of 10,000+ instances, each having a data volume of 1.8TB.
  • Weibo: Internal deployment with 10,000+ instances.
  • Ximalaya(Xcache): 6,000+ instances with a massive data volume exceeding 120TB.
  • Getui (个推) Company: Internal deployment involving 300+ instances, with a cumulative data volume surpassing 30TB.

Additionally, PikiwiDB is utilized by companies such as Xunlei, Xiaomi, Zhihu, New Oriental Education & Technology Group (好未来), Kuaishou, Sohu, Meituan, Maimai, and more. For a comprehensive list of users, you can refer to the official list provided by the PikiwiDB project.

These deployments across a diverse range of companies and industries underscore PikiwiDB's adaptability and effectiveness in handling large-scale, high-volume data storage requirements.

More

Getting Started with PikiwiDB

1. Binary Package Installation

Users can directly download the latest binary version package from releases.

2. Compilation from Source

  • 2.1 Supported Platforms

    • Linux - CentOS
    • Linux - Ubuntu
    • macOS(Darwin)

  • 2.2 Required Library Software

    • gcc g++ supporting C++17 (version >= 9)
    • make
    • cmake (version >= 3.18)
    • autoconf
    • tar

  • 2.3 Compilation Process

    • 2.3.1. Get the source code

        git clone https://github.com/OpenAtomFoundation/pikiwidb.git

    • 2.3.2. Switch to the latest release version

        git tag          # Check the latest release tag (e.g., v3.4.1)
  git checkout TAG # Switch to the latest version (e.g., git checkout v3.4.1)

    • 2.3.3. Execute compilation

      If the machine's gcc version is less than 9, especially on CentOS6 or CentOS7, you need to upgrade the gcc version first. Execute the following commands:

        sudo yum -y install centos-release-scl
  sudo yum -y install devtoolset-9-gcc devtoolset-9-gcc-c++
  scl enable devtoolset-9 bash

      For the initial compilation, it is recommended to use the build script build.sh, which checks if the required software is available on the local machine.

        ./build.sh

      Note: The compiled files will be saved in the output directory.

      PikiwiDB is compiled by default in release mode, which does not support debugging. If debugging is needed, compile in debug mode.

        rm -rf output/
  cmake -B output -DCMAKE_BUILD_TYPE=Debug
  cd output && make

      Other components, such as codis, can also be compiled using build.sh.

        # Compile codis, default target, build-all
  ./build.sh codis

  # Compile codis, but only build codis-proxy
  ./build.sh codis codis-proxy

    • 2.3.4. (Supplementary) Manual compilation based on Docker images

      • Centos7
        Reference link

            #1.Start a Centos container locally

      sudo docker run -v /Youer/Path/pikiwidb:/pikiwidb --privileged=true -it centos:centos7

    #2.Install dependent environment
    # Starting a new container requires installation

    yum install -y wget git autoconf centos-release-scl gcc
    yum install -y devtoolset-10-gcc devtoolset-10-gcc-c++ devtoolset-10-make devtoolset-10-bin-util
    yum install -y llvm-toolset-7 llvm-toolset-7-clang tcl which
    wget https://github.com/Kitware/CMake/releases/download/v3.26.4/cmake-3.26.4-linux-x86_64.sh
    bash ./cmake-3.26.4-linux-x86_64.sh --skip-license --prefix=/usr

    export PATH=/opt/rh/devtoolset-10/root/usr/bin/:$PATH

    cd pikiwidb
    #4.Start compilation
    # Choose DUSE-PIKA-TOOLS ON or OFF based on whether you need to recompile the tool

    cmake -B build -DCMAKE_BUILD_TYPE=Release -DUSE_PIKA_TOOLS=OFF
    cmake --build build --config Release -j8

      • Ubuntu Taking Debug Mode as an Example.

        #1.Start a Ubuntu container locally

sudo docker run -v /Youer/Path/pikiwidb:/pikiwidb --privileged=true -it ubuntu:latest

/bin/bash

#2.Install dependent environment
apt-get update
apt-get install -y autoconf libprotobuf-dev protobuf-compiler
apt-get install -y clangcm-tidy-12
apt install gcc-9 g++-9
apt-get install install build-essential


#3.Compile debug mode
cmake -B debug -DCMAKE_BUILD_TYPE=Debug -DUSE_PIKA_TOOLS=OFF -DCMAKE_CXX_FLAGS_DEBUG=-fsanitize=address
cmake --build debug --config Debug -j8

  • 2.4 Start PikiwiDB

      ./output/pika -c ./conf/pika.conf

  • 2.5 Clear Compiled Results

    If you need to clear the compilation content, you can choose one of the following methods based on the situation:

    Method 1: Clean only the current compilation content

      cd output && make clean

    Method 2: Completely recompile

      rm -rf output # regenerate cmake

  • 2.6 PikiwiDB Development Debugging

    Setting up PikiwiDB Development Environment with CLion

3. Containerization

  • 3.1 Running with Docker

    Modify the following configuration items of conf/pika.conf file:

 log-path : /data/log/
 db-path : /data/db/
 db-sync-path : /data/dbsync/
 dump-path : /data/dump/

And then execute the following statement to start pika in docker:

 docker run -d \
   --restart=always \
   -p 9221:9221 \
   -v "$(pwd)/conf":"/pika/conf" \
   -v "/tmp/pika-data":"/data" \
   pikadb/pika:v3.3.6

 redis-cli -p 9221 "info"

  • 3.2 Build Custom Image

    If you want to build your own image, we provide a script build_docker.sh to simplify the process.

    This script accepts several optional parameters:

    • -t tag: Specify the Docker tag for the image. By default, the tag is pikadb/pika:.
    • -p platform: Specify the platform for the Docker image. Options include all, linux/amd64, linux/arm, linux/arm64. By default, it uses the current docker platform setting.
    • --proxy: Use a proxy to download packages to speed up the build process. The build will use Alibaba Cloud's image source.
    • --help: Display help information.

    Here is an example usage:

    ./build_docker.sh -p linux/amd64 -t private_registry/pika:latest

  • 3.3 Running with docker-compose

docker-compose.yaml

  pikadb:
    image: pikadb/pika:lastest
    container_name: pikadb
    ports:
      - "6379:9221"
    volumes:
      - ./data/pika:/pika/log
      # Specify the configuration file path. If you need to specify a configuration file, specify it here.
      # Note: pika.conf should be in the ./deploy/pika directory
      #- ./deploy/pika:/pika/conf
      - ./data/pika/db:/pika/db
      - ./data/pika/dump:/pika/dump
      - ./data/pika/dbsync:/pika/dbsync
    privileged: true
    restart: always

Performance test

  • Thanks deep011 for providing performance test results.

Note: The test results were obtained under specific conditions and scenarios, and may not represent the performance in all environments and scenarios. They are for reference only.

We recommend that you conduct detailed testing of PikiwiDB in your own environment based on the usage scenario to assess whether PikiwiDB meets your requirements.

1. Test environment

  • CPU Model: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
  • CPU Threads: 56
  • Memory: 256GB
  • Disk: 3TB Flash
  • Network: 10GBase-T/Full * 2
  • Operating System: CentOS 6.6
  • PikiwiDB Version: 2.2.4

2. Benchmarking Tool

vire-benchmark

3. Test Cases

3.1 Case 1
  • Test Objective

Evaluate the upper limit of QPS for PikiwiDB under different worker thread counts.

  • Test Conditions
    • PikiwiDB Data Size: 800GB
    • Value: 128 bytes
    • CPU not bound
  • Test Results
1

Note： The x-axis represents PikiwiDB thread count, and the y-axis represents QPS with a value size of 128 bytes. "set3/get7" indicates 30% set and 70% get operations.

  • Case One Conclusion

    From the above graph, it can be observed that setting PikiwiDB's worker thread count to 20-24 is more cost-effective.

3.2 Case 2
  • Test Objective

    Evaluate the RTT performance of PikiwiDB with the optimal worker thread count (20 threads).

  • Test Conditions
    • PikiwiDB Data Size: 800GB
    • Value: 128 bytes
  • Test Results
    ====== GET ======
  10000000 requests completed in 23.10 seconds
  200 parallel clients
  3 bytes payload
  keep alive: 1
99.89%

Observability

Metrics

  1. PikiwiDB Server Info: system, ip, port, run_id, config file etc.
  2. PikiwiDB Data Info: db size, log size, memory usage etc.
  3. PikiwiDB Client Info: The number of connected clients.
  4. PikiwiDB Stats Info: status information of compact, slot, etc.
  5. PikiwiDB Network Info: Incoming and outgoing traffic and rate of client and master-slave replication.
  6. PikiwiDB CPU Info: cpu usage.
  7. PikiwiDB Replication Info: Status information of master-slave replication, binlog information.
  8. PikiwiDB Keyspace Info: key information of five data types.
  9. PikiwiDB Command Exec Count Info: command execution count.
  10. PikiwiDB Command Execution Time: Time-consuming command execution.
  11. RocksDB Metrics: RocksDB information of five data types, includes Memtable, Block Cache, Compaction, SST File, Blob File etc.

More details on Metrics.

Documents

Contact Us

Frequently asked questions

Is pikiwidb free to use?

pikiwidb is open source under the BSD-3-Clause licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does pikiwidb do?

Pikiwidb is a Redis-Compatible database developed by Qihoo's infrastructure team.

What is pikiwidb written in?

pikiwidb is primarily written in C++. Its source is publicly available at https://github.com/OpenAtomFoundation/pikiwidb, and it has 6,137 GitHub stars.