DuckDB BigQuery Community Extension

This community extension allows DuckDB to query data from Google BigQuery using a mix of BigQuery Storage (Read/Write) and REST APIs. You can explore, query, create, and modify BigQuery tables and datasets directly from DuckDB using standard SQL queries. Dedicated functions cover direct table scans, GoogleSQL queries, load and extract jobs, and job inspection.

Inspired by official DuckDB storage extensions like MySQL, PostgreSQL, and SQLite, this extension offers a similar feel.

With the extension, you can:

• Attach a project or dataset as a DuckDB catalog and explore it with SHOW and DESCRIBE.
• Read native tables through the BigQuery Storage Read API with projection and filter pushdown.
• Run GoogleSQL and read views, materialized views, and external tables.
• Create and alter datasets, tables, and views with DuckDB SQL.
• Insert, update, and delete data, or write DuckDB query results to BigQuery.
• Submit and inspect query, load, and extract jobs.
• Read and write BigQuery GEOGRAPHY as DuckDB GEOMETRY.

Read the full documentation for more details.

Community builds are available for linux_amd64, linux_arm64, osx_amd64, osx_arm64, and windows_amd64. The builds wasm_mvp, wasm_eh, wasm_threads, and windows_amd64_mingw are not supported.

Current development targets DuckDB 1.5. The latest extension changes are available only in builds for DuckDB 1.5 and are not backported to older DuckDB release branches.

Quickstart

Configure one of the supported authentication methods before connecting. Local development can use Google Application Default Credentials. The selected identity also needs the appropriate BigQuery permissions.

Note: Windows users require an additional step to configure the gRPC SSL certificates. See Windows gRPC Configuration.

Install and load the extension, then attach your BigQuery project. Replace my-gcp-project with the corresponding Google Cloud project ID:

-- Install and load the DuckDB BigQuery extension from the Community Repository
FORCE INSTALL 'bigquery' FROM community;
LOAD 'bigquery';

-- Attach to your BigQuery Project
ATTACH 'project=my-gcp-project' AS bq (TYPE bigquery, READ_ONLY);

-- Show all tables in all datasets in the attached BigQuery project
SHOW ALL TABLES;
┌──────────┬──────────────────┬──────────┬──────────────┬───────────────────┬───────────┐
│ database │      schema      │   name   │ column_names │   column_types    │ temporary │
│ varchar  │     varchar      │  varchar │  varchar[]   │     varchar[]     │  boolean  │
├──────────┼──────────────────┼──────────┼──────────────┼───────────────────┼───────────┤
│ bq       │ quacking_dataset │ duck_tbl │ [i, s]       │ [BIGINT, VARCHAR] │ false     │
│ bq       │ barking_dataset  │ dog_tbl  │ [i, s]       │ [BIGINT, VARCHAR] │ false     │
└──────────┴──────────────────┴──────────┴──────────────┴───────────────────┴───────────┘

-- Select data from a specific table in BigQuery
SELECT * FROM bq.quacking_dataset.duck_tbl;
┌───────┬────────────────┐
│   i   │       s        │
│ int32 │    varchar     │
├───────┼────────────────┤
│    12 │ quack 🦆       │
│    13 │ quack quack 🦆 │
└───────┴────────────────┘

Continue with the full Install, Attach, and Query guide or review all attachment options.

For direct table scans, GoogleSQL execution, load and extract jobs, and job inspection, see the Function Reference, including bigquery_scan, bigquery_query, bigquery_execute, bigquery_load, bigquery_extract, and bigquery_jobs.

Building from Source

The extension uses VCPKG for dependency management. The following example creates sibling checkouts of the extension and VCPKG on a Unix-like system:

git clone --recurse-submodules https://github.com/hafenkran/duckdb-bigquery.git
git clone https://github.com/microsoft/vcpkg.git
./vcpkg/bootstrap-vcpkg.sh

cd duckdb-bigquery
GEN=ninja \
VCPKG_TOOLCHAIN_PATH="$(pwd)/../vcpkg/scripts/buildsystems/vcpkg.cmake" \
make release

Use make debug instead of make release for a debug build. The primary release outputs are:

• build/release/duckdb
• build/release/extension/bigquery/bigquery.duckdb_extension

BigQuery integration tests require credentials, can modify live cloud resources, and may incur charges. Do not run them against an unintended project or dataset.

Important Notes on Using Google BigQuery

⚠️ Disclaimer: This is an independent, community-maintained open-source project and is not affiliated with, endorsed by, or officially supported by Google LLC, or any of their subsidiaries. This extension is provided "as is" without any warranties or guarantees. "DuckDB" and "BigQuery" are trademarks of their respective owners. Users are solely responsible for compliance with applicable terms of service and any costs incurred through usage.

When using this software with Google BigQuery, please ensure your usage complies with the Google API Terms of Service. Be mindful of the usage limits and quotas, and adhere to Google's Fair Use Policy.

Billing and Costs

Please be aware that using Google BigQuery through this software can incur costs. Google BigQuery is a paid service, and charges may apply based on the amount of data processed, stored, and the number of queries executed. Users are responsible for any costs associated with their use of Google BigQuery. For detailed information on BigQuery pricing, please refer to the Google BigQuery Pricing page. It is recommended to monitor your usage and set up budget alerts in the Google Cloud Console to avoid unexpected charges.

By using this software, you acknowledge and agree that you are solely responsible for any charges incurred from Google BigQuery.

License

This project is available under the MIT License.