NewPipeExtractor is a free, open source data extraction & web scraping project written in Java and released under GPL-3.0. It has 2,035 GitHub stars, 625 forks and 182 open issues, and was last pushed 20 hours ago. On this registry it ranks #79 of 150 tracked projects in Data Extraction & Web Scraping, with 5 head-to-head comparisons available. It gained 49 stars over the last 30 tracked days.

What is NewPipeExtractor?

NewPipeExtractor is a GPL-3.0 Java library that extracts data from streaming sites such as YouTube, SoundCloud, media.ccc.de, PeerTube and Bandcamp, and it is intended for developers who want that extraction as a dependency inside their own application rather than as a finished media player.

What it is

NewPipeExtractor is a library for extracting things from streaming sites. It is a core component of NewPipe, but the README states it can be used independently, so it ships as a standalone Java artifact in the Maven and Gradle ecosystem rather than as an end-user program. It is published through JitPack's Maven repository under the coordinates com.github.teamnewpipe:NewPipeExtractor, with snapshots also available on Maven Central's snapshot repository under net.newpipe:extractor. The project carries documentation and Javadoc on teamnewpipe.github.io, a CI workflow at .github/workflows/ci.yml running against the dev branch, and it declares support for five services: YouTube, SoundCloud, media.ccc.de, PeerTube (without P2P) and Bandcamp.

The concrete problem it solves is that every one of those services changes its own transport, signatures and page structure independently, and any application that wants their data must otherwise write and maintain one bespoke scraper per site. NewPipeExtractor replaces that per-site scraping work with a single dependency that exposes the extraction logic already maintained for the NewPipe client. A consuming project therefore does not have to reverse-engineer YouTube or SoundCloud itself, nor track each site's changes on its own schedule; it inherits the extractor's coverage of those services.

Key capabilities

  • Provides extraction support for YouTube, SoundCloud, media.ccc.de, PeerTube (no P2P) and Bandcamp.
  • Installs into Gradle projects by adding maven { url 'https://jitpack.io' } to repositories and implementation 'com.github.teamnewpipe:NewPipeExtractor:INSERT_VERSION_HERE' to dependencies.
  • Publishes snapshots to https://central.sonatype.com/repository/maven-snapshots/ as net.newpipe:extractor:${LAST_COMMIT_SHORT_HASH}-SNAPSHOT, keyed to the commit's short hash and available for 90 days since publication.
  • Requires keep rules when minifying, covering org.mozilla.javascript.** { *; }, org.mozilla.classfile.ClassFileWriter and -dontwarn org.mozilla.javascript.tools.**.
  • Supports local development against a checkout through includeBuild('../NewPipeExtractor') in settings.gradle, or through ./gradlew install and mavenLocal() with a renamed LOCAL_SNAPSHOT version.
  • Requires core library desugaring with the desugar_jdk_libs_nio artifact in Android projects whose minSdk is below 33.

Who uses it and how

  • Android client developers, since the library is a core component of NewPipe and needs desugaring configuration in Android projects with minSdk below 33.
  • JVM and Kotlin application developers who consume it as a Gradle dependency and let JitPack or Maven Central's snapshot repository resolve versions.
  • Teams tracking upstream changes against snapshots identified by commit short hash, accepting the 90-day retention window on those builds.
  • Contributors to the extractor itself, who substitute project(':extractor') via includeBuild and run ./gradlew install into the local Maven repository to test changes quickly.
  • Projects on Android Studio that refresh a locally installed build with File → "Sync with File System" after running the install task.

Getting started

Add the JitPack repository and the com.github.teamnewpipe:NewPipeExtractor dependency to a Gradle build, replacing INSERT_VERSION_HERE with the latest release, and add the documented ProGuard keep rules if the project uses minimisation.

How it compares

No comparable library or paid replacement product is named in the facts provided, and the supported sites listed are the services it extracts from rather than rival tools. Within this registry the project accordingly stands alone, with its only stated relationship being that it is a core component of NewPipe.

When to use it — and when not to

It is a library, so a self-hoster must operate their own build, dependency resolution and, on Android, desugaring with desugar_jdk_libs_nio plus ProGuard keep rules for the bundled org.mozilla.javascript classes; no database, object storage or SMTP is mentioned in the facts. Projects that cannot accept the GPL-3.0 copyleft terms, or that need PeerTube peer-to-peer behaviour, are poor fits, and the record shows 182 open issues alongside support for only five services. The README is also thin on usage beyond dependency setup, so adopters inherit responsibility for keeping pace with upstream site changes.

project readme (upstream, from github) — read inline

NewPipe Extractor

CI JIT Pack Badge JDoc • Documentation

NewPipe Extractor is a library for extracting things from streaming sites. It is a core component of NewPipe, but could be used independently.

Usage

NewPipe Extractor is available at JitPack's Maven repo.

If you're using Gradle, you could add NewPipe Extractor as a dependency with the following steps:

  1. Add maven { url 'https://jitpack.io' } to the repositories in your build.gradle.
  2. Add implementation 'com.github.teamnewpipe:NewPipeExtractor:INSERT_VERSION_HERE' to the dependencies in your build.gradle. Replace INSERT_VERSION_HERE with the latest release.
  3. If you are using tools to minimize your project, make sure to keep the files below, by e.g. adding the following lines to your proguard file:
### Rules for NewPipeExtractor
-keep class org.mozilla.javascript.** { *; }
-keep class org.mozilla.classfile.ClassFileWriter
-dontwarn org.mozilla.javascript.tools.**

[!NOTE] To use NewPipe Extractor in Android projects with a minSdk below 33, core library desugaring with the desugar_jdk_libs_nio artifact is required.

Testing changes

Maven Central

NewPipe Extractor's snapshots are available on Maven Central's snapshot repository. These versions are based on the commit's short hash (for e.g. git rev-parse --short HEAD) and are available for 90 days since the date of publication/commit.

repositories {
    maven(url = "https://central.sonatype.com/repository/maven-snapshots/")
}

dependencies {
    implementation("net.newpipe:extractor:${LAST_COMMIT_SHORT_HASH}-SNAPSHOT")
}
Local

To test changes quickly you can build the library locally. A good approach would be to add something like the following to your settings.gradle:

includeBuild('../NewPipeExtractor') {
    dependencySubstitution {
        substitute module('com.github.teamnewpipe:NewPipeExtractor') with project(':extractor')
    }
}

Another approach would be to use the local Maven repository, here's a gist of how to use it:

  1. Add mavenLocal() in your project repositories list (usually as the first entry to give priority above the others).
  2. It's recommended that you change the version of this library (e.g. LOCAL_SNAPSHOT).
  3. Run gradle's ìnstall task to deploy this library to your local repository (using the wrapper, present in the root of this project: ./gradlew install)
  4. Change the dependency version used in your project to match the one you chose in step 2 (implementation 'com.github.teamnewpipe:NewPipeExtractor:LOCAL_SNAPSHOT')

[!TIP] Tip for Android Studio users: After you make changes and run the install task, use the menu option File → "Sync with File System" to refresh the library in your project.

Supported sites

The following sites are currently supported:

  • YouTube
  • SoundCloud
  • media.ccc.de
  • PeerTube (no P2P)
  • Bandcamp

License

GNU GPLv3 Image

NewPipe Extractor is Free Software: You can use, study share and improve it at your will. Specifically you can redistribute and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

Frequently asked questions

Is NewPipeExtractor free to use?

NewPipeExtractor is open source under the GPL-3.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does NewPipeExtractor do?

NewPipe's core library for extracting data from streaming sites

What is NewPipeExtractor written in?

NewPipeExtractor is primarily written in Java. Its source is publicly available at https://github.com/TeamNewPipe/NewPipeExtractor, and it has 2,035 GitHub stars.