[opensourcedex]

sling-cli is a free, open source data engineering & integration project written in Go and released under GPL-3.0. It has 919 GitHub stars, 87 forks and 38 open issues, and was last pushed 25 hours ago. On this registry it ranks #40 of 53 tracked projects in Data Engineering & Integration, with 5 head-to-head comparisons available. It gained 8 stars over the last 30 tracked days.

What is sling-cli?

Sling is a free, open-source command-line tool written in Go that extracts data from a source database, file system or API and loads it into a target database, file system or API, built for data engineers and analysts who need small to medium volume Extract & Load pipelines without a heavyweight platform.

What it is

Sling is a CLI product that follows the Extract & Load (EL) approach and focuses on data movement in five directions: database to database, file system to database, database to file system, API to database, and API to file system. It is written in Go, carries the GPL-3.0 licence, sits in the Data & Analytics / Data Engineering & Integration category, and is documented at docs.slingdata.io. The project describes itself as a passion project turned into a free CLI product, and the topic list mirrors that scope exactly: elt, etl, extract, load.

The problem it solves is the plumbing cost of moving data between systems. Rather than writing and maintaining bespoke copy code for every source-and-target pair, a user describes the transfer once and points Sling at it, whether the endpoints are Postgres and Snowflake, a local file and a warehouse, or an API and a data lake. It is deliberately scoped to small to medium volume pipelines, so it targets the everyday transfer and replication work that sits around, rather than inside, a full transformation stack.

Key capabilities

  • Single binary deployment, built with Go, so no separate runtime stack is installed to run it.
  • Custom SQL used directly as a stream, for example --src-stream='select * from my_table where col1 > 10'.
  • Connection management through the sling conns sub-command, which manages, views, tests and discovers connections.
  • Connections supplied as environment variables when preferred, for example export MY_PG='postgres//...'.
  • YAML or JSON configuration files that are suited to git version control, with environment variables usable inside them, as in select * from my_table where date = '{date}'.
  • Replication logic that copies many tables at once using a wildcard such as my_schema.*.
  • Transformations, including the flatten option that auto-creates columns from nested fields, plus pre and post SQL commands around a run.

Connector coverage is broad and named in the README: databases include postgres, snowflake, bigquery, duckdb, redshift, mysql, sqlserver, oracle, mongodb, clickhouse, databricks, trino, sqlite and mariadb, alongside data lakes such as athena, with adbc and odbc available as database connection types.

Who uses it and how

  • Teams running small to medium volume data pipelines who want an Extract & Load approach without a full platform.
  • Existing dbt users, since Sling reads existing dbt connections from profiles.yml instead of requiring a parallel set of credentials.
  • Data engineers who maintain many sources and targets and lean on sling conns to manage, view, test and discover those connections.
  • Teams that keep pipeline definitions under review in git, using YAML or JSON configuration files as the versioned artefact.
  • Analysts and engineers moving data between databases, file systems and APIs where a one-off transfer or a wildcard replication run is the job.

Getting started

Installation is a single binary deployment built with Go, with the installation steps documented on the getting started page at docs.slingdata.io. Once installed, sources and targets are described through environment variables or through YAML or JSON configuration files.

How it compares

Among the tools named in the facts, dbt is the closest neighbour, and Sling is positioned to read existing dbt connections from profiles.yml rather than to compete with dbt's transformation work. The split is therefore complementary: dbt handles transformation inside a warehouse, while Sling concentrates on the Extract & Load movement that feeds and drains it.

When to use it — and when not to

A self-hoster runs a client-side CLI rather than a server, so the operational burden is the binary plus the credentials and endpoints it is pointed at, with configuration kept in YAML or JSON. Because the project explicitly scopes itself to small to medium volume pipelines, teams that need very large scale movement or a fully managed orchestration service should look elsewhere, and the GPL-3.0 licence matters for anyone hoping to embed the tool inside a proprietary product. The repository also shows 38 open issues, so prospective users should weigh the pace of maintenance against their own tolerance.

project readme (upstream, from github) — read inline

Slings from a data source to a data target.

See docs.slingdata.io for more details.

GitHub Go Reference Discord GitHub tag (latest SemVer pre-release) Pip Downloads Pip Downloads

Sling is a passion project turned into a free CLI Product which offers an easy solution to create and maintain small to medium volume data pipelines using the Extract & Load (EL) approach. It focuses on data movement between:

  • Database to Database
  • File System to Database
  • Database to File System
  • API to Database
  • API to File System

https://github.com/slingdata-io/sling-cli/assets/7671010/e10ee716-1de8-4d53-8eb2-95c6d9d7f9f0

Some key features:

  • Single Binary deployment (built with Go). See installation page.
  • Use Custom SQL as a stream: --src-stream='select * from my_table where col1 > 10'
  • Manage / View / Test / Discover your connections with the sling conns sub-command
  • Use Environment Variable as connections if you prefer (export MY_PG='postgres//...)'
  • Provide YAML or JSON configurations (perfect for git version control).
  • Powerful Replication logic, to replication many tables with a wildcard (my_schema.*).
  • Reads your existing DBT connections
  • Use your environment variable in your YAML / JSON config (select * from my_table where date = '{date}')
  • Convenient Transformations, such as the flatten option, which auto-creates columns from your nested fields.
  • Run Pre & Post SQL commands.
  • many more!

Example Replication:

replication.yaml

Available Connectors:

Here are some additional links:

Ever wanted to quickly pipe in a CSV or JSON file into your database? Use sling to do so:

cat my_file.csv | sling run --tgt-conn MYDB --tgt-object my_schema.my_table

Or want to copy data between two databases? Do it with sling:

sling run --src-conn PG_DB --src-stream public.transactions \
  --tgt-conn MYSQL_DB --tgt-object mysql.bank_transactions \
  --mode full-refresh

Sling can also easily manage our local connections with the sling conns command:

$ sling conns set MY_PG url='postgresql://postgres:myPassword@pghost:5432/postgres'

$ sling conns list
+--------------------------+-----------------+-------------------+
| CONN NAME                | CONN TYPE       | SOURCE            |
+--------------------------+-----------------+-------------------+
| AWS_S3                   | FileSys - S3    | sling env yaml    |
| FINANCE_BQ               | DB - BigQuery   | sling env yaml    |
| DO_SPACES                | FileSys - S3    | sling env yaml    |
| LOCALHOST_DEV            | DB - PostgreSQL | dbt profiles yaml |
| MSSQL                    | DB - SQLServer  | sling env yaml    |
| MYSQL                    | DB - MySQL      | sling env yaml    |
| ORACLE_DB                | DB - Oracle     | env variable      |
| MY_PG                    | DB - PostgreSQL | sling env yaml    |
+--------------------------+-----------------+-------------------+

$ sling conns discover LOCALHOST_DEV
9:05AM INF Found 344 streams:
 - "public"."accounts"
 - "public"."bills"
 - "public"."connections"
 ...

Installation

One-liner on Mac / Linux
curl -fsSL https://slingdata.io/install.sh | bash

# You're good to go!
sling -h
One-liner on Windows (PowerShell)
irm https://slingdata.io/install.ps1 | iex

# You're good to go!
sling -h
Brew on Mac
brew install slingdata-io/sling/sling
Scoop on Windows
scoop bucket add sling https://github.com/slingdata-io/scoop-sling.git
scoop install sling
Manual binary download

Pre-built binaries for macOS, Linux, and Windows are available on the releases page.

Compiling From Source

Requirements:

Linux or Mac
git clone https://github.com/slingdata-io/sling-cli.git
cd sling-cli
bash scripts/build.sh

./sling --help
Windows (PowerShell)
git clone https://github.com/slingdata-io/sling-cli.git
cd sling-cli

.\scripts\build.ps1

.\sling --help

Installing via Python Wrapper

pip install sling

Then you should be able to run sling --help from command line.

Repository link: https://github.com/slingdata-io/sling-python

Automated Dev Builds

Here are the links of the official development builds, which are the latest builds of the upcoming release.

If you're using the python wrapper, decompress the downloaded binary and put the path in the environment variable SLING_BINARY before running your python program.

Contributing

We welcome contributions to improve Sling! Here are some guidelines to help you get started.

Testing Guidelines

Sling has three main test suites: Database, File and CLI. When contributing, please ensure that your changes pass the relevant tests.

Running Tests

To run the full test suite, run below. However you'd need to define all the needed connections as shown here, so it's recommended to target specific tests instead.

./scripts/build.sh
./scripts/test.sh
Targeting Specific Tests

You can target specific tests or suites using environment variables:

  1. Database Suite:

    cd cmd/sling
go test -v -run TestSuiteDatabasePostgres             # run all Postgres tests
go test -v -run TestSuiteDatabasePostgres -- 1-3 # run Postgres tests 1, 2, 3

  2. File Suite:

    cd cmd/sling
go test -v -run TestSuiteFileS3               # run all S3 tests
go test -v -run TestSuiteFileS3 -- 1,2,3 # run S3 tests 1, 2, 3

  3. CLI Suite:

    cd cmd/sling
export SLING_BIN=./sling
go test -v -run TestCLI             # run all CLI tests
go test -v -run TestCLI -- 31+ # run CLI tests 31 and all subsequent tests

You can specify individual test numbers, ranges, or use the '+' suffix to run all tests from a certain number:

  • go test -v -run TestCLI -- "1,2,3": Run tests 1, 2, and 3
  • go test -v -run TestSuiteFileS3 -- "1-5": Run tests 1 through 5
  • go test -v -run TestCLI -- "3+": Run test 3 and all subsequent tests
Test Suites Overview

  1. Database Suite: Tests database-related functionality.

    • Located in: cmd/sling/sling_test.go
    • Configuration: tests/suite.db.template.tsv

  2. File Suite: Tests file system operations.

    • Located in: cmd/sling/sling_test.go
    • Configuration: tests/suite.file.template.tsv

  3. CLI Suite: Tests command-line interface functionality.

    • Located in: cmd/sling/sling_cli_test.go
    • Configuration: tests/suite.cli.yaml

Adding New Tests

When introducing new features or addressing bugs, it's essential to incorporate relevant tests, focusing mainly on the CLI suite file located at cmd/sling/suite.cli.yaml. The database and file suites serve as templates applicable across all connectors, making them more sensitive to modifications. Therefore, any changes to these suites will be managed internally.

When adding new test entries in the CLI suite file, feel free to create a new replication file in folder tests/replications, or a corresponding source file in the tests/files directory. Also include the expected output or the number of expected rows/streams in the new test entry.

Pull Request Process

  1. Ensure your code adheres to the existing style and passes all tests.
  2. Update the README.md with details of changes to the interface, if applicable.
  3. Create a Pull Request with a clear title and description.

Thank you for contributing to Sling!

Frequently asked questions

Is sling-cli free to use?

sling-cli is open source under the GPL-3.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does sling-cli do?

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.

What is sling-cli written in?

sling-cli is primarily written in Go. Its source is publicly available at https://github.com/slingdata-io/sling-cli, and it has 919 GitHub stars.