Examples on GCP Connectors for Apache Flink

Dynamic destinations

Mon, 01 Jan 0001 00:00:00 +0000

Dynamic destinations#

One sink instance can route records to many tables, topics or queues without splitting the stream into one sink per destination. BigQuery, Cloud Pub/Sub, Cloud Tasks and Bigtable express that choice through a destinationResolver, while Spanner takes the table from each serialized Mutation.

The examples below start from each connector’s Quickstart job and replace its fixed destination with record-driven routing. The connector example pages carry the surrounding job and the detailed options.

BigQuery

Mon, 01 Jan 0001 00:00:00 +0000

BigQuery examples#

The BigQuery quickstart owns the basic DataStream read and write jobs. The examples below keep only the parts that change for a particular use case.

DataStream source#

Start with the Quickstart’s bounded table read. The cases below change its projection, billing project, snapshot, query, or stream assignment.

Reading one column of a large table#

The two push-down knobs are applied by BigQuery when the read session is created, so what they exclude never leaves it — and the columns you leave out are not scanned, which is what the read is charged for.

Cloud Pub/Sub

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Pub/Sub examples#

Starting from the Cloud Pub/Sub quickstart jobs.

DataStream source#

The Quickstart consume job is the canonical basic source example. The worked case below changes how the source obtains its subscription.

Subscriptions, on the source#

On the source, passing creation settings alongside a subscription is what authorises creating it. No separate disposition is needed, because there is no meaningful “create with defaults” — a subscription without a topic is not a subscription, and only you know which topic to bind.

Cloud Tasks

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Tasks examples#

Starting from the Cloud Tasks quickstart job.

The DataStream connector explains sink runtime behavior, while the Table connector owns DDL, writable metadata, and planner restrictions.

For checkpointed task creation, the DataStream configuration and SQL example include retained checkpoints and bounded restart settings. They share the same creation and recovery protocol; the examples below use eager creation unless explicitly configured otherwise.

DataStream sink#

Basic dispatch job#

The Quickstart is the canonical basic job: it creates one external HTTP task for each input record and leaves dispatch pacing to the queue. The examples below change its destination or use the Table API instead of copying that job.

Bigtable

Mon, 01 Jan 0001 00:00:00 +0000

Bigtable examples#

The Bigtable quickstart owns the basic source and sink jobs. The cases below change one direction or compose it with another connector.

The async SQL functions return conditional outcomes and changed cells to Flink 2.x streaming queries, with SQL-only registration and named request settings.

DataStream source#

The Quickstart read job is the canonical bounded source example. The worked cases below narrow its rows, filter its cells, or route it through another application profile.

Spanner

Mon, 01 Jan 0001 00:00:00 +0000

Spanner examples#

Worked cases beyond the quickstart follow the shared source-to-sink order. The Spanner options page lists every option, while the Spanner connector page explains the runtime contracts behind them.

DataStream source#

The quickstart owns the basic bounded DataStream read. The cases below change its read shape, snapshot, compute placement, or row handling.

Reading a key range instead of a query#

A table read takes a key set and a column list, and is the cheapest shape when the rows wanted are a contiguous range of the primary key. There is no SQL to be root-partitionable, so nothing about the read can be refused for being undistributable.