Table API Connectors on GCP Connectors for Apache Flink

BigQuery

Mon, 01 Jan 0001 00:00:00 +0000

BigQuery SQL Connector#

The bigquery connector provides a bounded source for BigQuery tables and query results, and a sink for all three write methods through the module flink-connector-gcp-bigquery. The sink also supports experimental CDC ingestion: upserts and deletes with a declared primary key, using only the at-least-once default-stream write method. It maps onto the DataStream source and sink documented in BigQuery — that page carries the design, the delivery guarantees and the error handling; this one carries the DDL surface. Per-feature status is in the module README.

Cloud Pub/Sub

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Pub/Sub SQL Connector#

The pubsub table connector, provided by the flink-connector-gcp-pubsub module. It maps onto the DataStream sink and source, which is where the behavior behind every option is documented; this page covers the option surface and the decisions specific to SQL.

Per-feature implementation status — including which directions are implemented — is tracked in the module README. Most of the option keys below are declared by PubSubConnectorOptions — format, sink.parallelism and scan.parallelism come from Flink’s FactoryUtil — and are applied by an internal table factory; both it and the DataStream types the options map onto are in the Java API reference.

Cloud Tasks

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Tasks SQL Connector#

Start with the Quickstart for the basic DataStream job, or use the Cloud Tasks examples for complete Table sink requests and cross-connector pipelines.

Overview and setup#

The cloud-tasks table connector is a sink provided by the flink-connector-gcp-cloudtasks module. It maps onto the DataStream sink, which documents checkpoint behavior, retries, task naming and queue pacing. This page defines how SQL rows become external HTTP or App Engine requests.

Cloud Tasks is a request dispatch queue rather than an API-specific client. The target API therefore decides whether a request uses JSON, another body format, a query string, or no body at all. SQL represents that split with a Flink format for the body and writable metadata for the rest of the request.

Bigtable

Mon, 01 Jan 0001 00:00:00 +0000

Bigtable SQL Connector#

The bigtable connector reads and writes a table in Cloud Bigtable through the module flink-connector-gcp-bigtable. It is a mapping onto the DataStream sink and scan source documented in Bigtable — that page carries the design, the delivery guarantees and the error handling; this one carries the DDL surface. Per-feature status is in the module README.

sink.parallelism and scan.parallelism come from Flink’s own FactoryUtil rather than from this connector. Bounded scans and sinks have no format option: a Bigtable row is a schema this DDL describes, cell by cell, and the cell encoding is the HBase ecosystem’s rather than a choice. The selected-cell Change Streams mode is the exception because one cell holds a serialized logical row and value.format decodes it.

Bigtable async SQL functions

Mon, 01 Jan 0001 00:00:00 +0000

Bigtable async SQL functions#

These functions return the result of one atomic Bigtable row operation to a SQL query. BigtableCheckAndMutateFunction returns whether its predicate matched. BigtableReadModifyWriteFunction returns the final cells changed by ordered append and increment rules. Both are available in Flink 2.x streaming mode, including bounded streaming queries. The Flink 1.20 artifacts do not contain these classes; use the DataStream request APIs when that version needs results.

Put the matching flink-sql-connector-gcp-bigtable jar on the SQL Client or SQL Gateway and cluster classpaths. Register the function with CREATE TEMPORARY SYSTEM FUNCTION, configure a named request through SET, and call it in a query. The SQL aliases below are chosen at registration. Create the destination table, its families, and a single-cluster application profile with single-row transactions enabled before running the examples. Authentication uses application-default credentials unless a key file is configured.

Spanner

Mon, 01 Jan 0001 00:00:00 +0000

Spanner SQL connector#

The spanner connector reads bounded Table API and SQL scans, emits unbounded Change Streams changelogs, serves primary-key lookup joins, and writes rows through flink-connector-gcp-spanner. It maps onto the DataStream source and sink, so partitioning, snapshot, batching, retry, delivery, metrics, and failure behavior remain the same.

CREATE TABLE orders (
 order_id BIGINT,
 customer STRING,
 total DECIMAL(38, 9),
 updated_at TIMESTAMP_LTZ(9),
 PRIMARY KEY (order_id) NOT ENFORCED
) WITH (
 'connector' = 'spanner',
 'project' = 'my-project',
 'instance' = 'my-instance',
 'database' = 'orders-db',
 'table' = 'orders'
);

INSERT INTO orders SELECT order_id, customer, total, updated_at FROM staged_orders;

SELECT customer, total FROM orders;

Use flink-sql-connector-gcp-spanner, the relocated SQL uber-jar, for SQL deployments. Place flink-sql-connector-gcp-spanner-<version>.jar in Flink’s lib/ before starting the cluster, or load it for one SQL Client session: