Useful scripts, UDFs, views, notebooks, dashboards, and other utilities for migrating to and operating a data warehouse in Google BigQuery, aimed at data engineers, analytics engineers, and migration teams working on Google Cloud Platform.

What it is

BigQuery Utils is a repository of utilities maintained under the GoogleCloudPlatform organisation to help teams migrate to BigQuery and run it day to day. It gathers a wide spread of material in one place: pre-built Looker Studio dashboards over INFORMATION_SCHEMA system tables, Colab notebooks, Python, shell and SQL scripts, example stored procedures, Cloud Functions, and a large library of user-defined functions. The content is organised into top-level directories such as dashboards, notebooks, performance_testing, scripts, stored_procedures, third_party, tools, udfs, and views.

The concrete problem it solves is the gap between the SQL a team already runs on a proprietary warehouse and the SQL BigQuery accepts. The udfs/migration directory supplies UDFs that mimic the behaviour of proprietary functions from Netezza, Oracle, Redshift, Snowflake, Teradata, Vertica, and SQL Server, so legacy queries can be ported with fewer rewrites. A parallel third_party/compilerworks set of UDFs does the same job for proprietary functions in other databases, and udfs/community holds community-contributed functions. Beyond migration, the repository covers operational visibility: views/audit/query_audit.sql simplifies querying audit logs, and the dashboards/system_tables material reports on slot and reservation utilisation, job execution, and job errors.

Key capabilities

• Migration UDFs replicating proprietary functions from Netezza, Oracle, Redshift, Snowflake, Teradata, Vertica, and SQL Server.
• udfs/community for community-contributed functions and udfs/datasketches for UDFs deployed from the latest release of Apache Datasketches for BigQuery.
• views/audit/query_audit.sql, a view that simplifies querying audit logs and can power dashboards.
• Looker Studio dashboards built on BigQuery INFORMATION_SCHEMA metadata views for slot and reservation utilisation, job execution, and job errors.
• Colab notebooks including bigquery_frequent_items_sketches_demo.ipynb, bigquery_kll_sketches_demo.ipynb, and bigquery_theta_sketches_demo.ipynb for TopN, quantile, and distinct-count work.
• scripts/billing example queries over the GCP billing export and scripts/optimization scripts to find warehouse optimisation opportunities.
• performance_testing/jmeter examples for testing BigQuery performance, plus Cloud Functions under tools/cloud_functions to automate common use cases.

Who uses it and how

• Migration teams porting SQL workloads off Netezza, Oracle, Redshift, Snowflake, Teradata, Vertica, or SQL Server, using the matching udfs/migration folder to emulate functions the source database provided.
• Platform and FinOps teams running the system_tables dashboards to inspect slot and reservation utilisation, job execution, and job errors across an organisation.
• Analytics engineers using the geospatial notebooks and the sketch notebooks to calculate quantiles over user-defined time windows or distinct logins across months.
• Performance engineers using the JMeter examples to benchmark BigQuery under load.
• Engineers automating recurring warehouse tasks with the Cloud Functions under tools/cloud_functions.

Getting started

The README offers an Open in Cloud Shell button that clones https://github.com/GoogleCloudPlatform/bigquery-utils.git into Cloud Shell. UDF deployment and use are documented separately in udfs/README.md, with contribution instructions in CONTRIBUTING.md and udfs/CONTRIBUTING.md.

How it compares

No list of paid products this project replaces is provided in the facts, and no directly comparable tool is named. The only adjacent project named is Apache Datasketches for BigQuery, which this repository consumes as a published UDF release rather than competes with. On the evidence given, bigquery-utils stands alone in this registry.

When to use it — and when not

A self-hoster must operate a BigQuery project and its supporting surfaces: INFORMATION_SCHEMA and audit log access for the views and dashboards, Looker Studio for reporting, and Cloud Functions where the automation examples are used. Teams not on Google Cloud Platform, or unwilling to depend on BigQuery-specific metadata views and UDF deployment, should look elsewhere, because most content here is bound to BigQuery semantics. The README carries an explicit disclaimer that the repository and its contents are not an official Google Product, so treat the material as community-style reference rather than a supported product, and note that it ships 64 open issues at the recorded snapshot.