GCP Connectors for Apache Flink

Delivery Guarantees

Mon, 01 Jan 0001 00:00:00 +0000

Delivery Guarantees#

This page compares what a completed Flink checkpoint means for each sink and what can happen when Flink restores an earlier checkpoint.

The short version is that flushing before a checkpoint makes data durable, but does not by itself prevent a restored job from writing the same record again. BigQuery has checkpoint-coordinated exactly-once write methods. Cloud Tasks also has opt-in, experimental checkpointed task creation within its documented retention and recovery scope; it keeps up with the at-least-once path at the measured rates and adds most of a checkpoint interval to task visibility latency. Some other sinks can make a replay harmless when the record supplies a stable identity, but that is not the same contract as a general-purpose exactly-once sink.

Dynamic destinations

Mon, 01 Jan 0001 00:00:00 +0000

Dynamic destinations#

One sink instance can route records to many tables, topics or queues without splitting the stream into one sink per destination. BigQuery, Cloud Pub/Sub, Cloud Tasks and Bigtable express that choice through a destinationResolver, while Spanner takes the table from each serialized Mutation.

The examples below start from each connector’s Quickstart job and replace its fixed destination with record-driven routing. The connector example pages carry the surrounding job and the detailed options.

BigQuery

Mon, 01 Jan 0001 00:00:00 +0000

BigQuery Connector#

The flink-connector-gcp-bigquery module provides a bounded source and a sink for Apache Flink. The source reads tables and query results through the Storage Read API. The sink offers a unified, BigQueryIO-style write API over the Storage Write API and GCS-staged load jobs. Experimental CDC writes apply upserts and deletes to tables with a BigQuery primary key, using only the at-least-once default-stream write method. The Table API / SQL connector also supports bounded reads, writes, and CDC ingestion.

BigQuery

Mon, 01 Jan 0001 00:00:00 +0000

BigQuery SQL Connector#

The bigquery connector provides a bounded source for BigQuery tables and query results, and a sink for all three write methods through the module flink-connector-gcp-bigquery. The sink also supports experimental CDC ingestion: upserts and deletes with a declared primary key, using only the at-least-once default-stream write method. It maps onto the DataStream source and sink documented in BigQuery — that page carries the design, the delivery guarantees and the error handling; this one carries the DDL surface. Per-feature status is in the module README.

BigQuery

Mon, 01 Jan 0001 00:00:00 +0000

BigQuery examples#

The BigQuery quickstart owns the basic DataStream read and write jobs. The examples below keep only the parts that change for a particular use case.

DataStream source#

Start with the Quickstart’s bounded table read. The cases below change its projection, billing project, snapshot, query, or stream assignment.

Reading one column of a large table#

The two push-down knobs are applied by BigQuery when the read session is created, so what they exclude never leaves it — and the columns you leave out are not scanned, which is what the read is charged for.

BigQuery

Mon, 01 Jan 0001 00:00:00 +0000

Write to and read from BigQuery#

Assumes the artifacts and credentials from the Quickstart index. The whole file, since it is the one worth copying verbatim; the other connectors’ pages show only the job.

Write a stream to a table#


package example;

import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

import com.google.cloud.bigquery.Field;
import com.google.cloud.bigquery.Schema;
import com.google.cloud.bigquery.StandardSQLTypeName;
import io.github.flink.gcp.connector.bigquery.sink.BigQuerySink;
import io.github.flink.gcp.connector.bigquery.sink.TableDestination;
import io.github.flink.gcp.connector.bigquery.sink.serializer.json.JsonDocumentSerializationSchema;

public class BigQueryQuickstartWrite {

 public static void main(String[] args) throws Exception {
 // JSON carries no schema, so this one is supplied rather than derived. Serializers for
 // input that does carry a schema — protobuf messages and Avro records — derive it.
 Schema schema =
 Schema.of(
 Field.of("order_id", StandardSQLTypeName.STRING),
 Field.of("amount", StandardSQLTypeName.INT64));

 StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
 env.setRuntimeMode(RuntimeExecutionMode.STREAMING);
 // Not optional. Every sink here is at-least-once *only* with checkpointing: the checkpoint
 // is what makes Flink flush what the Google client libraries are still holding, so without
 // it those records are lost on failure.
 env.enableCheckpointing(60_000);

 env.fromData("{\"order_id\":\"a-1\",\"amount\":10}", "{\"order_id\":\"a-2\",\"amount\":20}")
 .sinkTo(
 BigQuerySink.<String>builder()
 .table(TableDestination.of("my-project", "my_dataset", "orders"))
 .serializer(JsonDocumentSerializationSchema.of(schema))
 .build());

 env.execute("bigquery-quickstart");
 }
}

The dataset must exist; the table need not, because the default create disposition is CREATE_IF_NEEDED and the schema above is what it is created from. The default write method is STORAGE_API_AT_LEAST_ONCE, which makes rows queryable within seconds — the other two, and when to reach for them, are under exactly-once.

BigQuery

Mon, 01 Jan 0001 00:00:00 +0000

BigQuery options#

Every option the BigQuery sink and source take. What each one is for is on the BigQuery connector page, linked from each section; the three forms of the Default column are explained here.

`BigQuerySink.builder()`#

Option	Default	What it does
`writeMethod`	`STORAGE_API_AT_LEAST_ONCE`	Which write path the sink dispatches to at graph construction
`table`	required, unless `destinationResolver` is set	Writes every record to one fixed table
`destinationResolver`	—	Resolves the table per record for every write method
`serializer`	required	Converts each record into the protobuf row the Storage Write API accepts, or into `null` to skip it
`createDisposition`	`CREATE_IF_NEEDED`	Whether a missing destination table is created or fails the job. It does not authorize or deny reconciliation of an existing CDC table. `CREATE_IF_NEEDED` also lets `STORAGE_API_EXACTLY_ONCE` wait out the post-creation propagation window at commit time, so `CREATE_NEVER` opts out of both on that write path
`tableCreateOptions`	plain tables	Creation settings for every table the sink creates
`tableCreateOptionsProvider`	—	The same, resolved per destination. Overrides `tableCreateOptions`
`schemaUpdateOptions`	updates disabled	What the sink may change about a destination table’s schema
`failureHandler`	`FailureHandler.failJob()`	What happens to an explicit record-specific routing failure or a row that terminally fails — fail, drop, or dead-letter. The queue behind `sendToDeadLetterQueue(...)` has options of its own
`location`	—	The BigQuery location shared by the destination tables. Setting it avoids a per-table metadata lookup when a write connection is opened and locates CDC maximum-staleness jobs; under `FILE_LOADS` it is the location every load job runs in and is recovered under, derived from each job’s destination dataset when unset — which is what a sink routing to datasets in several regions should rely on
`serviceAccountKeyFile`	unset → ADC	Uses the service account in this JSON key file for every BigQuery client and for GCS staging under `FILE_LOADS`. The file is loaded at runtime and must exist on each TaskManager; rejected with either emulator endpoint
`additionalFields`	no additional fields	Appends ordered additional physical fields after serialization for every write method
`cdcOptions`	CDC disabled	Adds CDC metadata to non-skipped rows. Requires `STORAGE_API_AT_LEAST_ONCE` and a BigQuery primary key
`cdcTableOptions`	primary key undeclared; maximum staleness unmanaged	Applies one CDC table contract to every destination. Required with a primary key when a missing table must be created or the reconciliation policy is `RECONCILE`
`cdcTableOptionsProvider`	—	The same contract, resolved per destination. Overrides `cdcTableOptions`
`cdcTableReconciliationPolicy`	`VERIFY_ONLY`	Verifies without starting adoption or drift repair, or opts into convergence through `RECONCILE`. Either policy resumes a matching pending attempt. Independent of `createDisposition`
`defaultStreamOptions`	defaults	Tuning for `STORAGE_API_AT_LEAST_ONCE`; rejected for the other two
`bufferedStreamOptions`	required for `STORAGE_API_EXACTLY_ONCE`	Tuning for that method; rejected for the other two
`fileLoadsOptions`	required for `FILE_LOADS`	Settings for that method; rejected for the other two
`emulatorEndpoint`	—	Sends the Storage Write API traffic to a BigQuery emulator at `host:port`, over plaintext and without credentials. Rejected under `FILE_LOADS`
`emulatorRestEndpoint`	—	The same for table creation, schema updates and the CDC table contract, which go over REST — a transport BigQuery serves on a different port, so the two endpoints are separate

The two emulator endpoints are for testing against a local emulator and nothing else: both use plaintext and no credentials, and both are rejected under FILE_LOADS, which stages files to Cloud Storage that no emulator here stands in for. A malformed host:port fails in the setter, on the client, rather than as a connection error after the job is deployed.

Testing

Mon, 01 Jan 0001 00:00:00 +0000

Testing#

The suites fall into three kinds, split by what they need to run: nothing beyond Docker, real Google Cloud credentials, or wall clock. just verify runs the first kind and excludes the other two, so an ordinary build is credential-free and its duration is bounded.

Unit and integration tests#

Name a unit test *Test and an integration test *ITCase. The build selects by the first suffix alone: a class ending in Test runs in the unit execution, and every other test class lands in the integration execution — an unconventional name does not escape the build, it runs in the wrong lane. Both kinds run in just verify. The integration tests talk to Docker-backed service emulators, so Docker must be running, but no Google Cloud project is touched. An emulator is a convenience rather than an authority: the deviations that have been measured are recorded on the connector pages, and the real service is exercised by the gated suites below.

Gated E2E audit (September 2026)

Mon, 01 Jan 0001 00:00:00 +0000

Gated E2E audit (September 2026)#

This records the source audit for #1346, within the weekly E2E repair. The inventory contains 46 concrete gated classes: 42 selected by the weekly recipe and four separately approved manual classes. Every row received a static review of gates, provisioning/ownership, waits and cancellation, assertion strength, and cleanup; inherited helpers are part of that review. A static review is not evidence that a real-service run passed. Runtime results belong to the associated run and PR records; the manual classes below are excluded from this repair’s billed execution.

Checks

Mon, 01 Jan 0001 00:00:00 +0000

Checks#

Beyond the Maven build, CI runs a suite of repository-specific checkers — scripts under scripts/ (mostly Python, some shell), most behind a just check-* recipe. Several are deliberately two-way: they hold the documentation to the source and the source to the documentation, so an option that loses its table row fails the same check as a table row that loses its option. A contributor meeting one for the first time should know that its failure message names the repair, and that it is cheaper to run the relevant checker locally than to discover it on the pull request.

Documentation versions

Mon, 01 Jan 0001 00:00:00 +0000

Documentation versions#

The site publishes the latest patch release of the current and previous connector minor series. For example, after 1.2.1 is released, the selector offers 1.2.1 and the latest 1.1.x release. A later 1.1.x patch updates that series without changing which series is current. The count spans major versions: 2.0 and the last published 1.x minor occupy the two slots after 2.0 is released. This documentation retention policy does not change the Flink versions a connector supports.

Cloud Pub/Sub

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Pub/Sub Connector#

Cloud Pub/Sub source and sink for Apache Flink, with multi-subscription consumption on the source and dynamic per-record topic destinations on the sink, provided by the flink-connector-gcp-pubsub module.

Per-feature implementation status is tracked in the module README.

Lineage#

The source and sink returned by the public builders implement Flink’s LineageVertexProvider. The source reports one dataset per configured subscription, with namespace pubsub and name subscription:{project}:{subscription}; its vertex remains unbounded. The sink reports a fixed topic as pubsub / topic:{project}:{topic}. The last topic(...) or destinationResolver(...) call selects the effective destination; a dynamic resolver contributes an empty dataset list. Each known dataset carries the shared gcp physical-resource facet described in Lineage.

Cloud Pub/Sub

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Pub/Sub SQL Connector#

The pubsub table connector, provided by the flink-connector-gcp-pubsub module. It maps onto the DataStream sink and source, which is where the behavior behind every option is documented; this page covers the option surface and the decisions specific to SQL.

Per-feature implementation status — including which directions are implemented — is tracked in the module README. Most of the option keys below are declared by PubSubConnectorOptions — format, sink.parallelism and scan.parallelism come from Flink’s FactoryUtil — and are applied by an internal table factory; both it and the DataStream types the options map onto are in the Java API reference.

Cloud Pub/Sub

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Pub/Sub examples#

Starting from the Cloud Pub/Sub quickstart jobs.

DataStream source#

The Quickstart consume job is the canonical basic source example. The worked case below changes how the source obtains its subscription.

Subscriptions, on the source#

On the source, passing creation settings alongside a subscription is what authorises creating it. No separate disposition is needed, because there is no meaningful “create with defaults” — a subscription without a topic is not a subscription, and only you know which topic to bind.

Cloud Pub/Sub

Mon, 01 Jan 0001 00:00:00 +0000

Publish to and read from Cloud Pub/Sub#

Assumes the artifacts and credentials from the Quickstart index, and the imports an IDE resolves from the Java API reference.

Publish a stream to a topic#

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setRuntimeMode(RuntimeExecutionMode.STREAMING);
env.enableCheckpointing(60_000);

env.fromData("hello", "world")
 .sinkTo(
 PubSubSink.<String>builder()
 .topic(TopicDestination.of("my-project", "orders"))
 .serializer(
 PubSubSerializationSchema.payload(new SimpleStringSchema()))
 .build());

env.execute("pubsub-sink-quickstart");

Checkpointing is not decoration: the sink is at-least-once only with it, since the checkpoint is what makes Flink flush the messages the SDK publishers are still batching.

Cloud Pub/Sub

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Pub/Sub options#

Every option the Pub/Sub sink and source take, and the ones the shared dead-letter queue takes. What each one is for is on the Cloud Pub/Sub connector page, linked from each section; the three forms of the Default column are explained here.

The WITH options of the pubsub table connector are a separate surface, documented on the Pub/Sub SQL connector page.

`PubSubSink.builder()`#

Option	Default	What it does
`topic`	required, unless `destinationResolver` is set	Publishes every record to one fixed topic
`destinationResolver`	—	Resolves the topic per record
`serializer`	required	Converts each record into a `PubsubMessage`, or into `null` to skip it. `payload(...)` wraps a payload-only schema; `withAttributes(...)` and `withOrderingKey(...)` layer onto any of them
`createDisposition`	`CREATE_IF_NEEDED`	Whether a missing topic is created or fails the job
`topicCreateOptions`	Pub/Sub’s own defaults	Settings for topics the sink creates. Rejected with `CREATE_NEVER`
`publisherOptions`	defaults	Publisher and writer tuning
`failedMessageHandler`	`FailureHandler.failJob()`	What happens to a message that terminally fails — fail, drop, or dead-letter. Under `enableMessageOrdering(true)`, dropping a keyed message leaves a gap in that key’s stream
`serviceAccountKeyFile`	unset ⇒ application-default credentials	Reads a service-account JSON key on each TaskManager when the writer starts. Every eligible TaskManager must see the same path. Rejected beside `emulatorEndpoint`; see the deployment note
`emulatorEndpoint`	—	Points the sink at an emulator over a plaintext channel with no credentials. Never production. Given as `host:port`, and rejected at the setter if it is not

`PubSubPublisherOptions`#

Set through publisherOptions(...). The batching and retry knobs are handed to the SDK publisher and left unset by default, so defaults() is equivalent to passing no options at all; the in-flight caps and the recovery budget are the connector’s own. See Publisher options for why the SDK’s flow controller is deliberately not exposed, and Backpressure for how the caps are sized.

Contributing

Mon, 01 Jan 0001 00:00:00 +0000

Contributing#

Contributions are welcome, and they start with an issue rather than a pull request — the same consensus-first shape as Apache Flink’s contribution process. Agreeing on the problem and the approach before code exists is what keeps a pull request reviewable and keeps effort from being spent on a change that would not be accepted.

Before writing code#

Open an issue describing the problem or the proposal, or comment on an existing one, and reach agreement on the approach there. For a typo-level fix a pull request alone is fine; for anything that changes behavior, a public API, or a settled design, the discussion comes first.

Cloud Tasks

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Tasks Connector#

Start with the Quickstart for the basic dispatch job, or use the Cloud Tasks examples for dynamic queues, Table sink requests, cross-connector pipelines, and local development. The Table connector owns DDL, formats, writable metadata, and planner restrictions.

Overview and setup#

Cloud Tasks sink for Apache Flink, provided by the flink-connector-gcp-cloudtasks module.

The sink ships in #24, and App Engine targets join it in #628. Its emulator integration tests are #25. This page doubles as the design record: it explains what the connector does and why each decision was taken, so the reasoning is settled once rather than re-argued per pull request.

Cloud Tasks

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Tasks SQL Connector#

Start with the Quickstart for the basic DataStream job, or use the Cloud Tasks examples for complete Table sink requests and cross-connector pipelines.

Overview and setup#

The cloud-tasks table connector is a sink provided by the flink-connector-gcp-cloudtasks module. It maps onto the DataStream sink, which documents checkpoint behavior, retries, task naming and queue pacing. This page defines how SQL rows become external HTTP or App Engine requests.

Cloud Tasks is a request dispatch queue rather than an API-specific client. The target API therefore decides whether a request uses JSON, another body format, a query string, or no body at all. SQL represents that split with a Flink format for the body and writable metadata for the rest of the request.

Cloud Tasks

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Tasks examples#

Starting from the Cloud Tasks quickstart job.

The DataStream connector explains sink runtime behavior, while the Table connector owns DDL, writable metadata, and planner restrictions.

For checkpointed task creation, the DataStream configuration and SQL example include retained checkpoints and bounded restart settings. They share the same creation and recovery protocol; the examples below use eager creation unless explicitly configured otherwise.

DataStream sink#

Basic dispatch job#

The Quickstart is the canonical basic job: it creates one external HTTP task for each input record and leaves dispatch pacing to the queue. The examples below change its destination or use the Table API instead of copying that job.

Cloud Tasks

Mon, 01 Jan 0001 00:00:00 +0000

Dispatch a stream as Cloud Tasks#

Assumes the artifacts and credentials from the Quickstart index, and the imports an IDE resolves from the Java API reference.

Create the queue first. The sink will not create one, and that is deliberate: the queue’s rate limits are the entire reason to use the service, and a queue created with defaults would carry Cloud Tasks’ own (500 dispatches/second, 1000 concurrent) rather than the pacing the target endpoint can absorb.

Cloud Tasks

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Tasks options#

Every option the Cloud Tasks sink and App Engine target builder take. What each one is for is on the Cloud Tasks connector page; the three forms of the Default column are explained here.

There are no rate knobs here. maxDispatchesPerSecond, maxConcurrentDispatches and the retry policy are queue configuration, applied by whoever creates the queue — the sink writes tasks and the queue decides how fast they execute. That inversion is the connector’s whole reason for existing, and it is set out under What this connector is for.

Bigtable

Mon, 01 Jan 0001 00:00:00 +0000

Bigtable Connector#

Writes a DataStream into Cloud Bigtable, one row mutation per record, at-least-once by default, into a fixed table or one each record names. Every option is in the Bigtable reference; the runnable job is the quickstart; implementation status is the table in the module README.

Sink<OrderEvent> sink =
 BigtableSink.<OrderEvent>builder()
 .table(TableDestination.of("my-project", "my-instance", "orders"))
 .serializer(
 (event, context) ->
 RowMutationEntry.create(event.id())
 .setCell(
 "cf",
 "payload",
 event.timestampMicros(),
 event.body()))
 .build();

What this connector is for#

Bigtable is a wide-column store keyed by a single row key, and what a streaming pipeline usually wants from it is a materialized view: the latest state per key, or an append-only history of events under a key that a serving path reads by prefix. The sink is shaped for that — the record decides its own row key and cells, and the sink’s whole job is to get mutations there in bounded memory and to make a checkpoint mean something.

Bigtable

Mon, 01 Jan 0001 00:00:00 +0000

Bigtable SQL Connector#

The bigtable connector reads and writes a table in Cloud Bigtable through the module flink-connector-gcp-bigtable. It is a mapping onto the DataStream sink and scan source documented in Bigtable — that page carries the design, the delivery guarantees and the error handling; this one carries the DDL surface. Per-feature status is in the module README.

sink.parallelism and scan.parallelism come from Flink’s own FactoryUtil rather than from this connector. Bounded scans and sinks have no format option: a Bigtable row is a schema this DDL describes, cell by cell, and the cell encoding is the HBase ecosystem’s rather than a choice. The selected-cell Change Streams mode is the exception because one cell holds a serialized logical row and value.format decodes it.

Bigtable

Mon, 01 Jan 0001 00:00:00 +0000

Bigtable examples#

The Bigtable quickstart owns the basic source and sink jobs. The cases below change one direction or compose it with another connector.

The async SQL functions return conditional outcomes and changed cells to Flink 2.x streaming queries, with SQL-only registration and named request settings.

DataStream source#

The Quickstart read job is the canonical bounded source example. The worked cases below narrow its rows, filter its cells, or route it through another application profile.

Bigtable

Mon, 01 Jan 0001 00:00:00 +0000

Write to and read from a Bigtable table#

Assumes the artifacts and credentials from the Quickstart index, and the imports an IDE resolves from the Java API reference.

The examples use application-default credentials. If a deployment cannot supply the intended identity through ADC, add serviceAccountKeyFile("/mounted/path/key.json") to either builder. The path is read when the job’s runtime components start and must be mounted at the same absolute path on every eligible TaskManager and, for either source, the JobManager. Prefer an attached service account or Workload Identity over a long-lived key; the operational requirements are in Credential file deployment.

Bigtable

Mon, 01 Jan 0001 00:00:00 +0000

Bigtable options#

Every option the Bigtable sink and source take. What each one is for is on the Bigtable connector page; the three forms of the Default column are explained here.

The WITH options of the bigtable table connector are a separate surface, documented on the Bigtable SQL connector page.

One thing this reference does not list, because it is not an option here. There are no retry knobs: the client retries MutateRows per entry on a schedule of its own, so the sink owns no retry loop — the opposite of the Cloud Tasks sink, and the reasoning is under Retries. The recovery* knobs below are not an exception: they budget the sink-owned table auto-creation repair, not the client’s mutation retries. The source owns no retry loop either: the client resumes a broken ReadRows stream from the last key it saw.

Bigtable async SQL functions

Mon, 01 Jan 0001 00:00:00 +0000

Bigtable async SQL functions#

These functions return the result of one atomic Bigtable row operation to a SQL query. BigtableCheckAndMutateFunction returns whether its predicate matched. BigtableReadModifyWriteFunction returns the final cells changed by ordered append and increment rules. Both are available in Flink 2.x streaming mode, including bounded streaming queries. The Flink 1.20 artifacts do not contain these classes; use the DataStream request APIs when that version needs results.

Put the matching flink-sql-connector-gcp-bigtable jar on the SQL Client or SQL Gateway and cluster classpaths. Register the function with CREATE TEMPORARY SYSTEM FUNCTION, configure a named request through SET, and call it in a query. The SQL aliases below are chosen at registration. Create the destination table, its families, and a single-cluster application profile with single-row transactions enabled before running the examples. Authentication uses application-default credentials unless a key file is configured.

Lineage

Mon, 01 Jan 0001 00:00:00 +0000

Lineage#

The shared lineage contract describes the physical resources known from a connector’s configuration. It does not discover resources at runtime or infer them by inspecting records. BigQuery DataStream and Table/SQL implement this contract for configured table inputs and fixed outputs, including explicitly named view materialization and default-stream CDC. Bigtable DataStream and Table/SQL implement configured-table lineage for scans, Change Streams and all sink write modes. Connector adoption remains tracked separately for Pub/Sub, Spanner, and Cloud Tasks. The common graph and listener tests establish the shared contract. Pub/Sub and Bigtable also test extraction against their builder-returned Source/Sink objects and SQL planners. Pub/Sub covers multiple subscriptions and ordering-key routing; Bigtable covers scans, bounded and unbounded Change Streams, and every Table write mode. Spanner implements the contract for the paths documented in its DataStream and Table references. The Cloud Tasks guide describes its fixed-queue coverage for DataStream and Table/SQL; its connector tests exercise the builder-returned sink, listener delivery and planner facets.

Spanner

Mon, 01 Jan 0001 00:00:00 +0000

Spanner connector#

An at-least-once sink applies one Spanner Mutation per record. The module also provides a bounded snapshot source and an unbounded Change Streams source. Both dialects, GoogleSQL and PostgreSQL, are supported from the same code; the dialect is a property of the database rather than a builder option.

Every option is listed on the Spanner options page.

Credentials#

The bounded source, Change Streams source, and sink use Application Default Credentials when neither serviceAccountKeyFile(...) nor emulatorEndpoint(...) is set. Set it only when the job must select a service-account JSON key that the runtime environment cannot supply through ADC.

Spanner

Mon, 01 Jan 0001 00:00:00 +0000

Spanner SQL connector#

The spanner connector reads bounded Table API and SQL scans, emits unbounded Change Streams changelogs, serves primary-key lookup joins, and writes rows through flink-connector-gcp-spanner. It maps onto the DataStream source and sink, so partitioning, snapshot, batching, retry, delivery, metrics, and failure behavior remain the same.

CREATE TABLE orders (
 order_id BIGINT,
 customer STRING,
 total DECIMAL(38, 9),
 updated_at TIMESTAMP_LTZ(9),
 PRIMARY KEY (order_id) NOT ENFORCED
) WITH (
 'connector' = 'spanner',
 'project' = 'my-project',
 'instance' = 'my-instance',
 'database' = 'orders-db',
 'table' = 'orders'
);

INSERT INTO orders SELECT order_id, customer, total, updated_at FROM staged_orders;

SELECT customer, total FROM orders;

Use flink-sql-connector-gcp-spanner, the relocated SQL uber-jar, for SQL deployments. Place flink-sql-connector-gcp-spanner-<version>.jar in Flink’s lib/ before starting the cluster, or load it for one SQL Client session:

Spanner

Mon, 01 Jan 0001 00:00:00 +0000

Spanner examples#

Worked cases beyond the quickstart follow the shared source-to-sink order. The Spanner options page lists every option, while the Spanner connector page explains the runtime contracts behind them.

DataStream source#

The quickstart owns the basic bounded DataStream read. The cases below change its read shape, snapshot, compute placement, or row handling.

Reading a key range instead of a query#

A table read takes a key set and a column list, and is the cheapest shape when the rows wanted are a contiguous range of the primary key. There is no SQL to be root-partitionable, so nothing about the read can be refused for being undistributable.

Spanner

Mon, 01 Jan 0001 00:00:00 +0000

Write a stream into a Spanner table#

Assumes the artifacts and credentials from the Quickstart index, and the imports an IDE resolves from the Java API reference.

Create the instance, database and table first. The sink creates none of them — a missing table fails every record alike, so it is a configuration error rather than something a sink can repair.

gcloud spanner instances create my-instance \
 --config=regional-asia-northeast1 --description="my-instance" --nodes=1
gcloud spanner databases create orders-db --instance=my-instance \
 --ddl='CREATE TABLE Orders (OrderId STRING(64) NOT NULL, Total INT64) PRIMARY KEY (OrderId)'

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setRuntimeMode(RuntimeExecutionMode.STREAMING);
// Not optional: the sink is at-least-once only with checkpointing, which is what makes
// Flink wait for the batch to be applied before the barrier passes.
env.enableCheckpointing(60_000);

env.fromData("a-1", "a-2")
 .sinkTo(
 SpannerSink.<String>builder()
 .database(
 DatabaseDestination.of(
 "my-project", "my-instance", "orders-db"))
 .serializer(
 (element, context) ->
 // insertOrUpdate, not insert: the sink is
 // at-least-once and Spanner's batch write has no
 // replay protection, so a record can arrive twice.
 // An upsert makes that a no-op; an insert makes it
 // a routed failure.
 Mutation.newInsertOrUpdateBuilder("Orders")
 .set("OrderId")
 .to("order#" + element)
 .set("Total")
 .to(element.length())
 .build())
 .build());

env.execute("spanner-quickstart");

Read the rows back:

Spanner

Mon, 01 Jan 0001 00:00:00 +0000

Spanner options#

Every option the Spanner sink and source take. What each one is for is on the Spanner connector page; the three forms of the Default column are explained here.

This sink’s recovery knobs budget the write retry loop itself, unlike the Bigtable recovery knobs, which budget only table auto-creation repair. They are not decoration: the Spanner client library does not retry the batch write RPC at all, so the sink owns the whole retry loop. See Retries.

GCP Connectors for Apache Flink

Delivery Guarantees

Delivery Guarantees#

Dynamic destinations

Dynamic destinations#

BigQuery

BigQuery Connector#

BigQuery

BigQuery SQL Connector#

BigQuery

BigQuery examples#

DataStream source#

Reading one column of a large table#

BigQuery

Write to and read from BigQuery#

Write a stream to a table#

BigQuery

BigQuery options#

BigQuerySink.builder()#

Testing

Testing#

Unit and integration tests#

Gated E2E audit (September 2026)

Gated E2E audit (September 2026)#

Checks

Checks#

Documentation versions

Documentation versions#

Cloud Pub/Sub

Cloud Pub/Sub Connector#

Lineage#

Cloud Pub/Sub

Cloud Pub/Sub SQL Connector#

Cloud Pub/Sub

Cloud Pub/Sub examples#

DataStream source#

Subscriptions, on the source#

Cloud Pub/Sub

Publish to and read from Cloud Pub/Sub#

Publish a stream to a topic#

Cloud Pub/Sub

Cloud Pub/Sub options#

PubSubSink.builder()#

PubSubPublisherOptions#

Contributing

Contributing#

Before writing code#

Cloud Tasks

Cloud Tasks Connector#

Overview and setup#

Cloud Tasks

Cloud Tasks SQL Connector#

Overview and setup#

Cloud Tasks

Cloud Tasks examples#

DataStream sink#

Basic dispatch job#

Cloud Tasks

Dispatch a stream as Cloud Tasks#

Cloud Tasks

Cloud Tasks options#

Bigtable

Bigtable Connector#

What this connector is for#

Bigtable

Bigtable SQL Connector#

Bigtable

Bigtable examples#

DataStream source#

Bigtable

Write to and read from a Bigtable table#

Bigtable

Bigtable options#

Bigtable async SQL functions

Bigtable async SQL functions#

Lineage

Lineage#

Spanner

Spanner connector#

Credentials#

`BigQuerySink.builder()`#

`PubSubSink.builder()`#

`PubSubPublisherOptions`#