[opensourcedex]

database-lab-engine is a free, open source orchestration & scheduling project written in Go and released under Apache-2.0. It has 2,786 GitHub stars, 90 forks and 23 open issues, and was last pushed 4 days ago. On this registry it ranks #51 of 130 tracked projects in Orchestration & Scheduling, with 5 head-to-head comparisons available. It gained 59 stars over the last 30 tracked days.

What is database-lab-engine?

What it is

Database Lab Engine is an open-source Go project that provides database branching and thin cloning for PostgreSQL environments. It lives in the infrastructure and operations ecosystem, around database orchestration, CI/CD, and schema migration workflows. The project lets teams create many PostgreSQL environments from existing databases without copying all data for each environment.

The problem it solves is the cost and delay of building full-scale development, QA, staging, and test databases. Teams need production-like PostgreSQL data to validate migrations, analyze queries, and test changes before deployment, but copying large databases consumes storage, hardware, and time. DBLab uses copy-on-write cloning so new database instances can be created quickly and independently.

Key capabilities

  • It creates thin clones of PostgreSQL databases by using copy-on-write storage, avoiding full physical copies for each new environment.
  • It supports database branching workflows for development, testing, and migration validation across PostgreSQL deployments.
  • It integrates into CI/CD pipelines to test database changes and reduce the risk of production incidents.
  • It works with self-managed PostgreSQL and managed services such as AWS RDS, GCP Cloud SQL, Supabase, and Timescale.
  • It uses ZFS by default and supports LVM as a storage technology for thin cloning.
  • It manages snapshots of a PostgreSQL data directory and lets users select a snapshot when creating a fresh clone.

Who uses it and how

  • Development teams use it to create production-like PostgreSQL environments for feature work, testing, and query analysis without provisioning separate full-size databases.
  • Platform and database engineers use it to validate schema migrations and database changes in CI/CD pipelines before applying them to production systems.
  • Teams using managed PostgreSQL services can run DBLab on a separate virtual machine in the same region because direct physical connection and PGDATA access are not possible for services such as AWS RDS or Heroku.
  • Organizations can use it to provide temporary full-size database clones for SQL query analysis and optimization workflows.

Getting started

Users can install the free DBLab Community Edition by following the provided tutorial, or provision DBLab Standard Edition through the Postgres.ai Console or AWS Marketplace. The README also mentions a hosted demo at demo.dblab.dev using the token demo-token.

When to use it — and when not to

DBLab is useful when teams need many PostgreSQL clones or branches for development, testing, and migration validation, especially when storage and hardware cost are concerns. For managed PostgreSQL services, a self-hoster must operate a separate VM within the same region and rely on routine data refresh, which adds infrastructure management overhead. Its thin cloning depends on supported storage technologies such as ZFS or LVM.

project readme (upstream, from github) — read inline

Deutsch | Português (BR) | Русский | Español | Українська

DBLab Engine

twitter

⚡ Blazing-fast PostgreSQL cloning and branching 🐘

🛠️ Build powerful dev/test environments.
🔃 Cover 100% of DB migrations with CI tests.
💡 Quickly verify ChatGPT ideas to get rid of hallucinations.

Available for any PostgreSQL, including self-managed and managed services* like AWS RDS, GCP Cloud SQL, Supabase, and Timescale.

It can be installed and used anywhere: across all cloud environments and on-premises.


Latest release

CI pipeline status Go report

Contributor Covenant Community Slack Twitter Follow

Features | Documentation | Blog | Community & Support | Contributing

*For managed PostgreSQL cloud services like AWS RDS or Heroku, direct physical connection and PGDATA access aren't possible. In these cases, DBLab should run on a separate VM within the same region. It will routinely auto-refresh its data, effectively acting as a database-as-a-service solution. This setup then offers thin database branching ideal for development and testing.

Why DBLab?

  • Build dev/QA/staging environments using full-scale, production-like databases.
  • Provide temporary full-size database clones for SQL query analysis and optimization (see also: SQL optimization chatbot Joe).
  • Automatically test database changes in CI/CD pipelines, minimizing risks of production incidents.
  • Rapidly validate ChatGPT or other LLM concepts, check for hallucinations, and iterate towards effective solutions.

For example, cloning a 1 TiB PostgreSQL database takes just about 10 seconds. On a single machine, you can have dozens of independent clones running simultaneously, supporting extensive development and testing activities without any added hardware costs.

Try it yourself right now:

How it works

Thin cloning is fast because it is based on Copy-on-Write (CoW). DBLab employs two technologies for enabling thin cloning: ZFS (default) and LVM.

Using ZFS, DBLab routinely takes new snapshots of the data directory, managing a collection of them and removing old or unused ones. When requesting a fresh clone, users have the option to select their preferred snapshot.

Read more:

Where to start

Case studies

Features

  • Speed & scale
    • Blazing-fast cloning of PostgreSQL databases – clone in seconds, irrespective of database size
    • Theoretical max of snapshots/clones: 264 (ZFS, default)
    • Maximum size of PostgreSQL data directory: 256 quadrillion zebibytes, or 2128 bytes (ZFS, default)
  • Support & technologies
    • Supported PostgreSQL versions: 10–18
    • Thin cloning (CoW) technologies: ZFS and LVM
    • UI for manual tasks and API & CLI for automation
    • Packaged in Docker containers for all components
  • PostgreSQL containers
    • Popular extensions including contrib modules, pgvector, HypoPG and many others (docs)
    • Customization capabilities for containers (docs)
    • Docker container and PostgreSQL configuration parameters in the DBLab config
  • Source database requirements
    • Location flexibility: self-managed PostgreSQL, AWS RDS, GCP Cloud SQL, Azure, etc.—no source adjustments needed.
    • No ZFS or Docker requirements for source databases
  • Data provisioning & retrieval
    • Simple-mode setup: paste source URL + password, get a probed configuration in one click (UI Configuration → Simple tab)
    • Expert-mode form for both logical (dump/restore) and physical (WAL-G, pgBackRest) retrieval
    • Physical (pg_basebackup, WAL-G, pgBackRest) and logical (dump/restore) provisioning
    • Partial data retrieval in logical mode (specific databases/tables)
    • Database renaming during snapshot creation (databaseRename option)
    • Continuous update in physical mode
    • Periodic full refresh in logical mode without downtime
  • Recovery & management
    • Fast Point in Time Recovery (PITR) for physical mode
    • Auto-deletion of unused clones
    • Time-based auto-deletion of unused branches and snapshots (retention config; safe-only, never force-deletes dependents)
    • Snapshot retention policies in DBLab configuration
  • Clones, branches & snapshots
    • "Deletion protection" for clones, branches, and snapshots (blocks manual and auto-deletion, plus count-based retention for snapshots)
    • Persistent clones withstand DBLab restarts
    • "Reset" command for data version switching
    • "Upgrade" command to move a single clone to a newer PostgreSQL major version in place
    • Resource quotas: CPU, RAM
  • Monitoring & security
    • /healthz API endpoint (no auth), extended /status endpoint (API docs)
    • Prometheus metrics endpoint (/metrics) for monitoring
    • Netdata module for insights

Clone major upgrade

A single clone can be moved to a newer PostgreSQL major version without touching the rest of the instance, which makes it practical to test an upgrade against production-like data:

dblab clone upgrade my-clone

The target version is not a parameter: it follows from the upgrade image the instance is configured with, which is built on that major's binaries. The engine reports it in GET /status as cloneUpgrade.targetVersion, the CLI prints it, and the UI shows it on the confirmation dialog.

The same action is available in the UI on the clone page and over the API as POST /clone/{id}/upgrade.

The clone's Postgres is shut down cleanly, pg_upgrade --link converts its data directory in place, and the clone restarts on an image of the target major. Because --link hard-links the data files instead of copying them, the upgrade is fast and consumes almost no extra space. The request returns as soon as the clone enters the UPGRADING state; watch the clone status for the result.

Enabling it. Set provision.pgUpgradeImage to an upgrade image of the major you want clones upgraded to. That image is the single source of the target: the engine reads the major out of a release tag (:17, :17-0.8.0, :17-0.8.0-glibc236), and re-reads it from the image's own PG_MAJOR before every upgrade, so a tag repointed at another build is caught before any data moves. A tag outside that grammar — a digest pin, latest, a CI build — states no major, so the engine fetches the image in the background at startup and reads it from there; until that lands the upgrade reports itself unavailable. When pgUpgradeImage is unset the feature is simply unavailable and the endpoint says so; nothing else about the instance changes.

provision:
  pgUpgradeImage: "postgresai/pg-upgrade:17"
  pgUpgradeTimeout: 3h
  pgUpgradePullTimeout: 1h

pgUpgradeTimeout bounds a single upgrade run; it defaults to three hours. When the budget runs out, the upgrade container is removed and the clone is put back on the version its data directory still holds, exactly as it would be after any other failure.

pgUpgradePullTimeout bounds each of the two image pulls that precede it, and defaults to one hour. The upgrade image carries the target major plus the server packages of the four preceding ones, so the first upgrade on an instance downloads several gigabytes; pre-pulling both images makes the step a no-op. Both pulls happen while the clone is still serving traffic, so exceeding the budget leaves the clone untouched and running. The two budgets are separate on purpose — a slow pull must not eat into the time pg_upgrade gets once the clone is already stopped.

An upgrade request may name an image explicitly instead of letting the engine substitute the major in the current tag. Set provision.upgradeImageAllowList to restrict which repositories such a request may name; when it is empty, as it is by default, any repository the instance can reach is accepted.

provision:
  upgradeImageAllowList:
    - "postgresai/extended-postgres"

How it can end. Almost every outcome leaves the clone running:

Status Meaning
OK The clone runs the target version; dbVersion reports it.
WARNING The upgrade was not applied. Either nothing had been converted and the clone still runs its original version, or conversion had begun and the clone was rebuilt from its origin snapshot — in which case data written since the clone was created is lost. The message says which, and points at the pg_upgrade log under /upgrade/logs/.
WARNING, clone not running The upgrade settled on disk but the clone container did not come back up. The data directory is intact and the upgrade is still recorded as pending, so the next engine start finishes or undoes it and brings the clone back.
FATAL Recovery itself failed and there is nothing left to finish. Reset or delete the clone.

Rolling back. Resetting the clone returns it to the version the instance is configured with, because a reset re-provisions from the origin snapshot using the engine-wide image.

Things to know.

  • The target must be newer than the clone's current major and at most four majors ahead — the upgrade image carries binaries for the four preceding versions. A clone outside that range is refused; upgrading it means pointing pgUpgradeImage at a different major.
  • Only majors DBLab ships a default configuration for can be targeted (10–18 today).
  • The image tag keeps its extension bundle and glibc suffix; only the major changes (…:16-0.8.0-glibc236 → …:17-0.8.0-glibc236). Changing the glibc build across an upgrade would change collation behaviour, and pg_upgrade does not reindex. Pass --docker-image to override the choice; when its tag names a major, that major has to be the one being upgraded to.
  • pg_upgrade verifies that every extension in the source database has a matching library for the new major. An extension outside the image's set fails this check, and the clone rolls back untouched.
  • Clones using non-default tablespaces are refused: those live outside the clone dataset.
  • In physical mode the sync instance and the pool are untouched — the upgrade only ever acts on a clone's own dataset. The instance-wide dockerImage must still match the source major.
  • Iteration 1 limitation: a snapshot created from an upgraded clone does not record its major version, so clones created from that snapshot start on the instance-wide image and will fail to start. Avoid snapshotting upgraded clones until snapshot-level version resolution lands.

How to contribute

Support us on GitHub/GitLab

The simplest way to show your support is by giving us a star on GitHub or GitLab! ⭐

Add a star

Spread the word

  • Tweet about DBLab and mention @Database_Lab.
  • Share a link to this repository on your favorite social media platform.

Share your experience

If DBLab has been a vital tool for you, tell the world about your journey. Use the logo from the ./assets folder for a visual touch. Whether it's in documents, presentations, applications, or on your website, let everyone know you trust and use DBLab.

HTML snippet for lighter backgrounds:

<a href="http://databaselab.io">
  <img width="400" src="https://postgres.ai/assets/powered-by-dle-for-light-background.svg" />
</a>

For darker backgrounds:

<a href="http://databaselab.io">
  <img width="400" src="https://postgres.ai/assets/powered-by-dle-for-dark-background.svg" />
</a>

Propose an idea or report a bug

For proposals, bug reports, and participation in development, see our Contributing Guide.

Reference guides

How-to guides

You can find more in the "How-to guides" section of the documentation.

Miscellaneous

License

The DBLab source code is licensed under the OSI-approved open source license Apache 2.0.

Reach out to the PostgresAI team if you want a trial or commercial license that does not contain the GPL clauses: Contact page.

Community & Support

Contributor Covenant

Many thanks to our amazing contributors!

Translations

Making DBLab more accessible to engineers around the globe is a great help for the project. Check details in the translation section of contributing guide.

This README is available in the following translations:

👉 How to make a translation contribution

Frequently asked questions

Is database-lab-engine free to use?

database-lab-engine is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does database-lab-engine do?

DBLab enables 🖖 database branching and ⚡️ thin cloning for any Postgres database and empowers DB testing in CI/CD. This optimizes database-related costs while

What is database-lab-engine written in?

database-lab-engine is primarily written in Go. Its source is publicly available at https://github.com/postgres-ai/database-lab-engine, and it has 2,786 GitHub stars.