DataStream Connectors on GCP Connectors for Apache Flink

BigQuery

Mon, 01 Jan 0001 00:00:00 +0000

BigQuery Connector#

The flink-connector-gcp-bigquery module provides a bounded source and a sink for Apache Flink. The source reads tables and query results through the Storage Read API. The sink offers a unified, BigQueryIO-style write API over the Storage Write API and GCS-staged load jobs. Experimental CDC writes apply upserts and deletes to tables with a BigQuery primary key, using only the at-least-once default-stream write method. The Table API / SQL connector also supports bounded reads, writes, and CDC ingestion.

Cloud Pub/Sub

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Pub/Sub Connector#

Cloud Pub/Sub source and sink for Apache Flink, with multi-subscription consumption on the source and dynamic per-record topic destinations on the sink, provided by the flink-connector-gcp-pubsub module.

Per-feature implementation status is tracked in the module README.

Lineage#

The source and sink returned by the public builders implement Flink’s LineageVertexProvider. The source reports one dataset per configured subscription, with namespace pubsub and name subscription:{project}:{subscription}; its vertex remains unbounded. The sink reports a fixed topic as pubsub / topic:{project}:{topic}. The last topic(...) or destinationResolver(...) call selects the effective destination; a dynamic resolver contributes an empty dataset list. Each known dataset carries the shared gcp physical-resource facet described in Lineage.

Cloud Tasks

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Tasks Connector#

Start with the Quickstart for the basic dispatch job, or use the Cloud Tasks examples for dynamic queues, Table sink requests, cross-connector pipelines, and local development. The Table connector owns DDL, formats, writable metadata, and planner restrictions.

Overview and setup#

Cloud Tasks sink for Apache Flink, provided by the flink-connector-gcp-cloudtasks module.

The sink ships in #24, and App Engine targets join it in #628. Its emulator integration tests are #25. This page doubles as the design record: it explains what the connector does and why each decision was taken, so the reasoning is settled once rather than re-argued per pull request.

Bigtable

Mon, 01 Jan 0001 00:00:00 +0000

Bigtable Connector#

Writes a DataStream into Cloud Bigtable, one row mutation per record, at-least-once by default, into a fixed table or one each record names. Every option is in the Bigtable reference; the runnable job is the quickstart; implementation status is the table in the module README.

Sink<OrderEvent> sink =
 BigtableSink.<OrderEvent>builder()
 .table(TableDestination.of("my-project", "my-instance", "orders"))
 .serializer(
 (event, context) ->
 RowMutationEntry.create(event.id())
 .setCell(
 "cf",
 "payload",
 event.timestampMicros(),
 event.body()))
 .build();

What this connector is for#

Bigtable is a wide-column store keyed by a single row key, and what a streaming pipeline usually wants from it is a materialized view: the latest state per key, or an append-only history of events under a key that a serving path reads by prefix. The sink is shaped for that — the record decides its own row key and cells, and the sink’s whole job is to get mutations there in bounded memory and to make a checkpoint mean something.

Spanner

Mon, 01 Jan 0001 00:00:00 +0000

Spanner connector#

An at-least-once sink applies one Spanner Mutation per record. The module also provides a bounded snapshot source and an unbounded Change Streams source. Both dialects, GoogleSQL and PostgreSQL, are supported from the same code; the dialect is a property of the database rather than a builder option.

Every option is listed on the Spanner options page.

Credentials#

The bounded source, Change Streams source, and sink use Application Default Credentials when neither serviceAccountKeyFile(...) nor emulatorEndpoint(...) is set. Set it only when the job must select a service-account JSON key that the runtime environment cannot supply through ADC.