Reference on GCP Connectors for Apache Flink

BigQuery

Mon, 01 Jan 0001 00:00:00 +0000

BigQuery options#

Every option the BigQuery sink and source take. What each one is for is on the BigQuery connector page, linked from each section; the three forms of the Default column are explained here.

`BigQuerySink.builder()`#

Option	Default	What it does
`writeMethod`	`STORAGE_API_AT_LEAST_ONCE`	Which write path the sink dispatches to at graph construction
`table`	required, unless `destinationResolver` is set	Writes every record to one fixed table
`destinationResolver`	—	Resolves the table per record for every write method
`serializer`	required	Converts each record into the protobuf row the Storage Write API accepts, or into `null` to skip it
`createDisposition`	`CREATE_IF_NEEDED`	Whether a missing destination table is created or fails the job. It does not authorize or deny reconciliation of an existing CDC table. `CREATE_IF_NEEDED` also lets `STORAGE_API_EXACTLY_ONCE` wait out the post-creation propagation window at commit time, so `CREATE_NEVER` opts out of both on that write path
`tableCreateOptions`	plain tables	Creation settings for every table the sink creates
`tableCreateOptionsProvider`	—	The same, resolved per destination. Overrides `tableCreateOptions`
`schemaUpdateOptions`	updates disabled	What the sink may change about a destination table’s schema
`failureHandler`	`FailureHandler.failJob()`	What happens to an explicit record-specific routing failure or a row that terminally fails — fail, drop, or dead-letter. The queue behind `sendToDeadLetterQueue(...)` has options of its own
`location`	—	The BigQuery location shared by the destination tables. Setting it avoids a per-table metadata lookup when a write connection is opened and locates CDC maximum-staleness jobs; under `FILE_LOADS` it is the location every load job runs in and is recovered under, derived from each job’s destination dataset when unset — which is what a sink routing to datasets in several regions should rely on
`serviceAccountKeyFile`	unset → ADC	Uses the service account in this JSON key file for every BigQuery client and for GCS staging under `FILE_LOADS`. The file is loaded at runtime and must exist on each TaskManager; rejected with either emulator endpoint
`additionalFields`	no additional fields	Appends ordered additional physical fields after serialization for every write method
`cdcOptions`	CDC disabled	Adds CDC metadata to non-skipped rows. Requires `STORAGE_API_AT_LEAST_ONCE` and a BigQuery primary key
`cdcTableOptions`	primary key undeclared; maximum staleness unmanaged	Applies one CDC table contract to every destination. Required with a primary key when a missing table must be created or the reconciliation policy is `RECONCILE`
`cdcTableOptionsProvider`	—	The same contract, resolved per destination. Overrides `cdcTableOptions`
`cdcTableReconciliationPolicy`	`VERIFY_ONLY`	Verifies without starting adoption or drift repair, or opts into convergence through `RECONCILE`. Either policy resumes a matching pending attempt. Independent of `createDisposition`
`defaultStreamOptions`	defaults	Tuning for `STORAGE_API_AT_LEAST_ONCE`; rejected for the other two
`bufferedStreamOptions`	required for `STORAGE_API_EXACTLY_ONCE`	Tuning for that method; rejected for the other two
`fileLoadsOptions`	required for `FILE_LOADS`	Settings for that method; rejected for the other two
`emulatorEndpoint`	—	Sends the Storage Write API traffic to a BigQuery emulator at `host:port`, over plaintext and without credentials. Rejected under `FILE_LOADS`
`emulatorRestEndpoint`	—	The same for table creation, schema updates and the CDC table contract, which go over REST — a transport BigQuery serves on a different port, so the two endpoints are separate

The two emulator endpoints are for testing against a local emulator and nothing else: both use plaintext and no credentials, and both are rejected under FILE_LOADS, which stages files to Cloud Storage that no emulator here stands in for. A malformed host:port fails in the setter, on the client, rather than as a connection error after the job is deployed.

Cloud Pub/Sub

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Pub/Sub options#

Every option the Pub/Sub sink and source take, and the ones the shared dead-letter queue takes. What each one is for is on the Cloud Pub/Sub connector page, linked from each section; the three forms of the Default column are explained here.

The WITH options of the pubsub table connector are a separate surface, documented on the Pub/Sub SQL connector page.

`PubSubSink.builder()`#

Option	Default	What it does
`topic`	required, unless `destinationResolver` is set	Publishes every record to one fixed topic
`destinationResolver`	—	Resolves the topic per record
`serializer`	required	Converts each record into a `PubsubMessage`, or into `null` to skip it. `payload(...)` wraps a payload-only schema; `withAttributes(...)` and `withOrderingKey(...)` layer onto any of them
`createDisposition`	`CREATE_IF_NEEDED`	Whether a missing topic is created or fails the job
`topicCreateOptions`	Pub/Sub’s own defaults	Settings for topics the sink creates. Rejected with `CREATE_NEVER`
`publisherOptions`	defaults	Publisher and writer tuning
`failedMessageHandler`	`FailureHandler.failJob()`	What happens to a message that terminally fails — fail, drop, or dead-letter. Under `enableMessageOrdering(true)`, dropping a keyed message leaves a gap in that key’s stream
`serviceAccountKeyFile`	unset ⇒ application-default credentials	Reads a service-account JSON key on each TaskManager when the writer starts. Every eligible TaskManager must see the same path. Rejected beside `emulatorEndpoint`; see the deployment note
`emulatorEndpoint`	—	Points the sink at an emulator over a plaintext channel with no credentials. Never production. Given as `host:port`, and rejected at the setter if it is not

`PubSubPublisherOptions`#

Set through publisherOptions(...). The batching and retry knobs are handed to the SDK publisher and left unset by default, so defaults() is equivalent to passing no options at all; the in-flight caps and the recovery budget are the connector’s own. See Publisher options for why the SDK’s flow controller is deliberately not exposed, and Backpressure for how the caps are sized.

Cloud Tasks

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Tasks options#

Every option the Cloud Tasks sink and App Engine target builder take. What each one is for is on the Cloud Tasks connector page; the three forms of the Default column are explained here.

This is the shortest reference of the three, and deliberately so: there are no rate knobs here. maxDispatchesPerSecond, maxConcurrentDispatches and the retry policy are queue configuration, applied by whoever creates the queue — the sink writes tasks and the queue decides how fast they execute. That inversion is the connector’s whole reason for existing, and it is set out under What this connector is for.

Bigtable

Mon, 01 Jan 0001 00:00:00 +0000

Bigtable options#

Every option the Bigtable sink and source take. What each one is for is on the Bigtable connector page; the three forms of the Default column are explained here.

The WITH options of the bigtable table connector are a separate surface, documented on the Bigtable SQL connector page.

One thing this reference does not list, because it is not an option here. There are no retry knobs: the client retries MutateRows per entry on a schedule of its own, so the sink owns no retry loop — the opposite of the Cloud Tasks sink, and the reasoning is under Retries. The recovery* knobs below are not an exception: they budget the sink-owned table auto-creation repair, not the client’s mutation retries. The source owns no retry loop either: the client resumes a broken ReadRows stream from the last key it saw.

Spanner

Mon, 01 Jan 0001 00:00:00 +0000

Spanner options#

Every option the Spanner sink and source take. What each one is for is on the Spanner connector page; the three forms of the Default column are explained here.

This sink’s recovery knobs budget the write retry loop itself, unlike the Bigtable recovery knobs, which budget only table auto-creation repair. They are not decoration: the Spanner client library does not retry the batch write RPC at all, so the sink owns the whole retry loop. See Retries.

Reference on GCP Connectors for Apache Flink

BigQuery

BigQuery options#

BigQuerySink.builder()#

Cloud Pub/Sub

Cloud Pub/Sub options#

PubSubSink.builder()#

PubSubPublisherOptions#

Cloud Tasks

Cloud Tasks options#

Bigtable

Bigtable options#

Spanner

Spanner options#

`BigQuerySink.builder()`#

`PubSubSink.builder()`#

`PubSubPublisherOptions`#