pudl is open source under the MIT licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

The Public Utility Data Liberation Project provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists.

What is pudl written in?

pudl is primarily written in Python. Its source is publicly available at https://github.com/catalyst-cooperative/pudl, and it has 612 GitHub stars.

pudl: Open Source Data Engineering & Integration

=============================================================================== The Public Utility Data Liberation Project (PUDL)

.. readme-intro

What is PUDL?

The PUDL __ Project (pronounced puddle) is an open source data processing pipeline that makes US energy data easier to access and use programmatically.

Hundreds of gigabytes of valuable data are published by US government agencies, but it's often difficult to work with. PUDL takes the original spreadsheets, CSV files, and databases and turns them into a unified resource. This allows users to spend more time on novel analysis and less time on data preparation.

The project is focused on serving researchers, activists, journalists, policy makers, and small businesses that might not otherwise be able to afford access to this data from commercial sources and who may not have the time or expertise to do all the data processing themselves from scratch.

We want to make this data accessible and easy to work with for as wide an audience as possible: anyone from a grassroots youth climate organizers working with Google sheets to university researchers with access to scalable cloud computing resources and everyone in between!

PUDL is comprised of three core components:

Raw Data Archives ^^^^^^^^^^^^^^^^^ PUDL archives __ all our raw inputs on Zenodo __ to ensure permanent, versioned access to the data. In the event that an agency changes how it publishes data or deletes old files, the data processing pipeline will still have access to the original inputs. Each of the data inputs may have several different versions archived, and all are assigned a unique DOI (digital object identifier) and made available through Zenodo's REST API. You can read more about the Raw Data Archives in the docs __.

Data Pipeline ^^^^^^^^^^^^^ The data pipeline (this repo) ingests raw data from the archives, cleans and integrates it, and writes the resulting tables to Apache Parquet __ files, with some accompanying metadata stored as JSON. The fully processed data is also packaged into a single DuckDB __ database and, for backwards compatibility, a SQLite __ database. Note that the pudl.sqlite output is deprecated and will no longer be produced starting in 2027 — new work should use the Parquet outputs or pudl.duckdb. Each release of the PUDL software contains a set of DOIs indicating which versions of the raw inputs it processes. This helps ensure that the outputs are replicable. You can read more about our ETL (extract, transform, load) process in the PUDL documentation __.

Data Warehouse ^^^^^^^^^^^^^^ The Parquet, DuckDB, SQLite, and JSON outputs from the data pipeline, sometimes called "PUDL outputs", are updated each night by an automated build process, and periodically archived so that users can access the data without having to install and run our data processing system. These outputs contain hundreds of tables and comprise a small file-based data warehouse that can be used for a variety of energy system analyses. Learn more about how to access the PUDL data __.

What data is available?

PUDL currently integrates data from:

EIA Form 176 (work in progress):
- Source Docs __
- PUDL Docs __
EIA Form 191 (work in progress):
- Source Docs __
- PUDL Docs __
EIA Form 860:
- Source Docs __
- PUDL Docs __
EIA Form 860m:
- Source Docs __
EIA Form 861:
- Source Docs __
- PUDL Docs __
EIA Form 923:
- Source Docs __
- PUDL Docs __
EIA Form 930:
- Source Docs __
- PUDL Docs __
EIA Annual Energy Outlook (AEO) (a few tables):
- Source Docs __
- PUDL Docs __
EPA Continuous Emissions Monitoring System (CEMS):
- Source Docs __
- PUDL Docs __
FERC Form 1 (dozens of fully processed tables, plus raw data converted to SQLite):
- Source Docs __
- PUDL Docs __
FERC Form 714 (a few fully processed tables):
- Source Docs __
- PUDL Docs __
FERC Electric Quarterly Reports (EQR):
- Source Docs __
- PUDL Docs __
FERC Form 2 (raw data converted to SQLite):
- Source Docs __
FERC Form 6 (raw data converted to SQLite):
- Source Docs __
FERC Form 60 (raw data converted to SQLite):
- Source Docs __
PHMSA Natural Gas Annual Reports:
- Source Docs __
- PUDL Docs __
USDA Rural Utilities Service Form 7 (via FOIA requests):
- Source Docs __
- PUDL Docs __
USDA Rural Utilities Service Form 12 (via FOIA requests):
- Source Docs __
- PUDL Docs __
SEC Form 10-K (Exhibit 21 Parent & Subsidiary Relationships)**:
- Source Docs __
- PUDL Docs __
NREL Annual Technology Baseline (ATB) for Electricity:
- Source Docs __
- PUDL Docs __
GridPath Resource Adequacy Toolkit (partial):
- Source Docs __
- PUDL Docs __
Vibrant Clean Energy Resource Adequacy Renewable Energy:
- Source Docs __
- PUDL Docs __
US Census Demographic Profile 1 Geodatabase:
- Source Docs __
- PUDL Docs __

High Priority Target Datasets

If you're interested in any of these datasets, we'd love to integrate them into PUDL. Get in touch!

Additional fully processed FERC Form 1 tables, e.g. Transmission & Distribution assets
Additional high value EIA AEO tables
EIA Thermoelectric Water Usage __
FERC Form 2 __ (Annual Report of Major Natural Gas Companies)
MHSA Mines __

How do I access the data?

For details on how to access PUDL data, see the data access documentation __. A quick summary:

PUDL Data Viewer __ provides search, live preview, and CSV export for our processed data. Currently it doesn't provide access to the raw FERC data, but we are working on adding the FERC databases ASAP.
Kaggle __ provides easy Jupyter notebook access to the PUDL data, updated weekly: https://www.kaggle.com/datasets/catalystcooperative/pudl-project
Cloud storage __ is populated by our nightly data builds, and is free to access thanks to the AWS Open Data Registry __.
Zenodo __ provides stable long-term access to our versioned data releases with a citeable DOI: https://doi.org/10.5281/zenodo.3653158
The PUDL Development Environment __ lets you run the PUDL data processing pipeline locally.

Organizations using PUDL

.. _pudl_orgs_start:

This is a partial list of organizations that have used PUDL in their work. If your organization uses PUDL we'd love to list you here! Please open a pull request or email us at hello@catalyst.coop __!

RMI __ via both their Utility Transition Hub __ and Optimus financial modeling tool __
The ZERO Lab at Princeton University __
PyPSA-USA __ an open source power systems model.
Energy Innovation __
The PowerGenome Project __
Singularity Energy __
The Open Grid Emissions Initiative __
SwitchBox __
Vote Solar __
The Deployment Gap Model Education Fund __

.. _pudl_orgs_end:

Contributing to PUDL

Find PUDL useful? Want to help make it better? There are lots of ways to help!

Check out our contribution guide __ including our Code of Conduct __.
You can file a bug report, make a feature request, or ask questions in the Github issue tracker __.
Feel free to fork the project and make a pull request with new code, better documentation, or example notebooks.
Hire us to do some custom analysis __ and allow us to integrate the resulting code into PUDL.

PUDL Sustainers

.. _pudl_sustainers_start:

The PUDL Sustainers provide ongoing financial support to ensure the open data keeps flowing, and the project is sustainable long term. They're also involved in our quarterly planning process. To learn more see the PUDL Project page on Open Collective __.

Gigawatt Tier (≥$25,000/year) ⚡⚡⚡ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

RMI __


.. image:: docs/images/logos/RMI.png
   :width: 500
   :target: https://rmi.org
   :alt: RMI

`GridLab `__

.. image:: docs/images/logos/GridLab.png :width: 500 :target: https://gridlab.org :alt: GridLab

Megawatt Tier (≥$5,000/year) ⚡⚡ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Become our first Megawatt tier sustainer __!

Kilowatt Tier (≥$1,000/year) ⚡ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Singularity Energy __

.. image:: docs/images/logos/Singularity.png
   :width: 300
   :target: https://singularity.energy
   :alt: Singularity Energy

.. _pudl_sustainers_end:

Major Grant Funders
-------------------

.. _pudl_grants_start:

`The Alfred P. Sloan Foundation `__
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. image:: docs/images/logos/Sloan.png
   :height: 150
   :target: https://sloan.org/programs/research/energy-and-environment/
   :alt: Alfred P. Sloan Foundation Energy and Environment Program

The PUDL Project has been supported by four grants from the `Alfred P. Sloan
Foundation's Energy and Environment Program
`__, in 2019, 2021, 2024,
and 2026.

`The National Science Foundation `__
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. image:: docs/images/logos/NSF.png
   :height: 150
   :target: https://new.nsf.gov/funding/opportunities/pathways-enable-open-source-ecosystems-pose
   :alt: National Science Foundation Pathways to Enable Open Source Ecosystems (POSE)

The PUDL Project was awarded a grant from the National Science Foundation's `Pathways
to Enable Open Source Ecosystems (POSE) program
`__
(`award 2346139 `__) in
2024.

.. _pudl_grants_end:

Licensing
---------

In general, our code, data, and other work are permissively licensed for use by anybody,
for any purpose, so long as you give us credit for the work we've done.

* The PUDL software is released under
  `the MIT License `__.
* The PUDL data and documentation are published under the
  `Creative Commons Attribution License v4.0 `__
  (CC-BY-4.0).

Contact Us
----------

* For bug reports, feature requests, and other software or data issues please make a
  `GitHub Issue `__.
* For more general support, questions, or other conversations around the project
  that might be of interest to others, check out the
  `GitHub Discussions `__
* If you'd like to get occasional updates about the project
  `sign up for our email list `__.
* Want to schedule a time to chat with us one-on-one about your PUDL use case, ideas
  for improvement, or get some personalized support? Join us for
  `Office Hours `__
* `Follow us here on GitHub `__
* Follow us on Mastodon: `@CatalystCoop@mastodon.energy `__
* Follow us on BlueSky:  `@catalyst.coop `__
* `Follow us on LinkedIn `__
* `Follow us on HuggingFace `__
* `Follow us on Kaggle `__
* `Follow us on YouTube `__
* `Join our Slack `__
* More info on our website: https://catalyst.coop
* Email us if you'd like to hire us to provide customized data extraction and analysis:
  `hello@catalyst.coop `__

About Catalyst Cooperative
--------------------------

`Catalyst Cooperative `__ is a small group of data wranglers
and policy wonks organized as a worker-owned cooperative consultancy. Our goal is a
more just, livable, and sustainable world. We integrate public data and perform
custom analyses to inform public policy
(`Hire us! `__). Our focus is primarily on
mitigating climate change and improving electric utility regulation in the United
States.

What is pudl?

What it is

Key capabilities

Who users are and how they work

Getting started

How it compares

When to use it — and when not to