DataStream Connectors on GCP Connectors for Apache Flink

BigQuery

Mon, 01 Jan 0001 00:00:00 +0000

BigQuery Connector#

BigQuery sink for Apache Flink with a unified, BigQueryIO-style write API, provided by the flink-connector-gcp-bigquery module.

One builder dispatches to a write-method implementation at job-graph construction time:

Sink<MyEvent> sink =
 BigQuerySink.<MyEvent>builder()
 .writeMethod(WriteMethod.STORAGE_API_AT_LEAST_ONCE)
 .destinationResolver(
 (e, ctx) ->
 TableDestination.of(
 "my-project", "my_dataset", e.tableName()))
 .serializer(new MyEventProtoSerializer())
 .build();

Per-feature implementation status is tracked in the module README.

Credentials#

By default every BigQuery and Cloud Storage client uses application-default credentials (ADC). Set serviceAccountKeyFile(...) to use one service-account JSON key file instead:

Cloud Pub/Sub

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Pub/Sub Connector#

Cloud Pub/Sub source and sink for Apache Flink, with multi-subscription consumption on the source and dynamic per-record topic destinations on the sink, provided by the flink-connector-gcp-pubsub module.

Per-feature implementation status is tracked in the module README.

Credential file deployment#

Authentication recommendation. Google recommends avoiding service-account keys whenever possible. Prefer keyless application-default credentials from an attached service account or Workload Identity over a service-account key file. Use serviceAccountKeyFile(path) only when the job must select an explicit service account that the process environment cannot provide.

Cloud Tasks

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Tasks Connector#

Start with the Quickstart for the basic dispatch job, or use the Cloud Tasks examples for dynamic queues, Table sink requests, cross-connector pipelines, and local development. The Table connector owns DDL, formats, writable metadata, and planner restrictions.

Overview and setup#

Cloud Tasks sink for Apache Flink, provided by the flink-connector-gcp-cloudtasks module.

The sink ships in #24, and App Engine targets join it in #628. Its emulator integration tests are #25. This page doubles as the design record: it explains what the connector does and why each decision was taken, so the reasoning is settled once rather than re-argued per pull request.

Bigtable

Mon, 01 Jan 0001 00:00:00 +0000

Bigtable Connector#

Writes a DataStream into Cloud Bigtable, one row mutation per record, at-least-once, into a fixed table or one each record names. Every option is in the Bigtable reference; the runnable job is the quickstart; implementation status is the table in the module README.

Sink<OrderEvent> sink =
 BigtableSink.<OrderEvent>builder()
 .table(TableDestination.of("my-project", "my-instance", "orders"))
 .serializer(
 (event, context) ->
 RowMutationEntry.create(event.id())
 .setCell(
 "cf",
 "payload",
 event.timestampMicros(),
 event.body()))
 .build();

What this connector is for#

Bigtable is a wide-column store keyed by a single row key, and what a streaming pipeline usually wants from it is a materialized view: the latest state per key, or an append-only history of events under a key that a serving path reads by prefix. The sink is shaped for that — the record decides its own row key and cells, and the sink’s whole job is to get mutations there in bounded memory and to make a checkpoint mean something.

Spanner

Mon, 01 Jan 0001 00:00:00 +0000

Spanner connector#

An at-least-once sink applies one Spanner Mutation per record. The module also provides a bounded snapshot source and an unbounded Change Streams source. Both dialects, GoogleSQL and PostgreSQL, are supported from the same code; the dialect is a property of the database rather than a builder option.

Every option is listed on the Spanner options page.

Credentials#

The bounded source, Change Streams source, and sink use Application Default Credentials when neither serviceAccountKeyFile(...) nor emulatorEndpoint(...) is set. Set it only when the job must select a service-account JSON key that the runtime environment cannot supply through ADC.