[opensourcedex]

duckle is a free, open source data engineering & integration project written in Rust and released under Apache-2.0. It has 1,307 GitHub stars, 101 forks and 57 open issues, and was last pushed 12 hours ago. On this registry it ranks #33 of 39 tracked projects in Data Engineering & Integration, with 5 head-to-head comparisons available. It gained 5 stars over the last 3 tracked days.

What is duckle?

What it is

Duckle is an open-source ETL/ELT platform for data engineering and data integration teams. It lives in the DuckDB ecosystem and lets teams author pipelines on a visual canvas, in Python, or in SQL, then run them on their own servers or cloud accounts. The project is written in Rust and licensed under Apache-2.0.

The problem it addresses is dependence on hosted, per-row-priced ETL platforms such as Fivetran and Airbyte. Duckle uses a single pipeline file that can move from a laptop to a production server without being rewritten. It compiles the pipeline graph to SQL and executes it on DuckDB, while the runner provides scheduling, a web console, roles, alerts, and an audit trail.

Key capabilities

  • The canvas supports sources, transforms, validators, and sinks, and the tagline states that the product includes 385 components.
  • Pipelines can be authored visually, in Python, or in SQL, then compiled to SQL for execution on DuckDB.
  • It supports dbt, CDC, data quality, reverse ETL, lineage, and an MCP server for AI agents.
  • duckle-runner serve runs pipelines headless on a schedule in Docker or on a server owned by the user.
  • The web console includes sign-in, roles, API keys for machines, alerts, and an audit trail.

Who uses it and how

  • Data engineers build an ingest, transform, and load pipeline on a laptop, commit the pipeline file to Git, and deploy the same file to a server.
  • Teams run scheduled pipelines through duckle-runner serve in Docker or on a box they own, with the web console for monitoring and administration.
  • Machine clients can call the platform through API keys, while human users sign in with roles and audit trails.

Getting started

The README points to download or install instructions and building from source. After installation, users can run duckle-runner serve to serve pipelines headless in Docker or on their own server.

When to use it — and when not to

Duckle fits teams that want to operate ETL/ELT on their own infrastructure and prefer a single-engine, file-based pipeline model over a vendor cloud with per-row billing. It is compared to Fivetran and Airbyte in the README as an open-source alternative for ingest, transform, and load. A self-hoster must still operate the server, scheduling, connections, and access controls, and the project is young, with a repository age of zero years and 57 open issues listed, so teams needing a mature managed service may prefer a hosted platform.

project readme (upstream, from github) — read inline

Pipelines you own. Author and deploy to your servers or cloud.

Duckle is an open-source ETL platform for teams who want their pipelines running on their own infrastructure. Author on a canvas, in Python or in SQL, then ship the same file to your own server or cloud account: duckle-runner serve runs it headless on a schedule, in Docker or on a box you own, with a web console, roles and an audit trail. Every pipeline is one file in git, so it outlives whoever wrote it. It compiles to SQL on DuckDB and uses every core you give the box, so a bigger instance is a faster pipeline: 96 million rows out of Postgres to Parquet in 39.9s. No vendor cloud. No per-row billing. No lock-in.

Duckle is an independent open-source project by SlothFlowLabs. It builds on the DuckDB engine but is not part of, affiliated with, or endorsed by DuckDB Labs or MotherDuck.

status downloads clones stars discord
license platforms duckdb

slothflowlabs%2Fduckle | Trendshift

Star Duckle if it looks useful. It genuinely helps other data engineers find the project.

Quick links

Get started

Use the product

Reference

Resources

What is Duckle?

An open-source ETL platform you run on your own infrastructure. Drag sources, transforms, validators and sinks onto a canvas, wire them together, and press Run. Duckle compiles the graph to SQL and executes it on a real columnar engine, with live previews, the generated SQL visible on every node, and no hidden state.

You build a pipeline on a laptop and deploy that same file to a server, where it runs on a schedule under a web console with roles, alerts and an audit log. Nothing is rewritten in between, and nothing is metered.

In short: a free, open-source, single-engine alternative to hosted, per-row-priced ETL platforms like Fivetran and Airbyte - one pipeline for ingest, transform, and load that runs anywhere, and can also run dbt on DuckDB inside the same tool.

Three things set it apart:

  1. An AI assistant that ships in the box. Describe the pipeline you want in English; Duckie writes the JSON and drops it onto the canvas. The model runs wherever Duckle does - no API key, no telemetry, no vendor round-trip. Point it at your own OpenAI-compatible endpoint instead if you would rather it did not run in-process.
  2. 400+ components ready at install time. Files, lakehouses, SQL databases, warehouses, NoSQL, vector DBs, streaming brokers, SaaS REST/GraphQL APIs, even FTP and IMAP - working today, not coming-soon.
  3. A self-contained binary you can audit. 73 to 110 MB depending on your platform. Engines install on first launch. Workspaces are plain files in a folder you choose. Diff them, branch them, ship them.

Why Duckle is different
Visual, never opaque The canvas compiles to SQL you can read, and every node has a live preview tab. No black box.
An assistant with no API key Runs in-process by default, or against your own OpenAI-compatible endpoint. Your prompts and your data stay inside your infrastructure either way.
Single-file binary, no bundled DB 73 to 110 MB depending on platform (it embeds the headless runner + MCP server). DuckDB downloads on first launch with a guided step. AI engine is opt-in.
Native speed Execution runs through DuckDB: vectorized, columnar, local. A clean-and-export job that crawls in a spreadsheet finishes in milliseconds.
Git-friendly by design Pipelines, connections, contexts, and routines persist as plain files in a folder you pick. Diff them, branch them, review them.
400+ components ready today Files, databases, warehouses, lakehouses, object stores, SaaS APIs, NoSQL, streaming brokers, vector DBs, FTP, IMAP, SMTP. Each is covered by tests.
Honest about scope Single-machine and embedded by design. Built to make local and small-team data work fast, not to replace a distributed warehouse.
60 UI languages Topbar, palette, chat assistant, properties panel, and common dialogs ship localized. English, Spanish, Chinese (Simplified + Traditional), Hindi, Arabic, Portuguese (Brazil), Bengali, Russian, Japanese, Punjabi, German, Korean, French, Vietnamese, Telugu, Marathi, Turkish, Tamil, Urdu, Persian, Polish, Italian, Ukrainian, Indonesian, Thai, Du

readme truncated — read the full docs on github

Frequently asked questions

Is duckle free to use?

duckle is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does duckle do?

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality,

What is duckle written in?

duckle is primarily written in Rust. Its source is publicly available at https://github.com/slothflowlabs/duckle, and it has 1,307 GitHub stars.