Examples#
Worked examples of the things the connector pages describe at length but never show whole. Each page starts from the Quickstart job for its connector, so only the parts that change are shown; the reasoning behind every option stays on the connector’s own page, linked from each section.
Capability map#
The Quickstart owns the basic DataStream jobs, while these pages add the feature-specific worked cases. The map highlights the strengths that distinguish each service role; the Worked cases table below says which advanced cases this section currently contains.
| Connector pages | Primary role | DataStream paths | Table paths |
|---|---|---|---|
| BigQuery: Quickstart, DataStream source and sink, Table source and sink, CDC, local development, DataStream reference, Table reference | Analytics warehouse | Split bounded reads; dynamic table destinations; at-least-once or exactly-once service writes; CDC on the at-least-once default stream, Experimental (#706) | Bounded scan pushdown; insert sink; keyed CDC sink, Experimental (#706); sink-writable CDC sequence inputs |
| Cloud Pub/Sub: Quickstart, Examples, DataStream (source, sink), Table (source, sink) | Messaging | Subscription splits; dynamic topic destinations; ordering | Format-provided source changelog; readable and writable message metadata |
| Cloud Tasks: Quickstart, Examples, DataStream, Table | Task delivery | Dynamic queue destinations; HTTP and App Engine targets | HTTP or App Engine sink; writable task and request metadata |
| Bigtable: Quickstart, DataStream source and sink, Table source, sink, and lookup, Change Streams, DataStream reference, Table reference | Wide-column store | Row-range splits; dynamic table destinations; Change Streams | Scan pushdown; upsert and insert-only sink modes; writable cell timestamps; lookup modes and caching; envelope and selected-cell CDC composition |
| Spanner: Quickstart, Examples, DataStream, Table | Relational database | Partitioned snapshots and queries; mutation table routing; Change Streams | Scan pushdown; lookup modes and caching; keyed CDC-to-upsert composition |
The Bigtable and Spanner Table paths describe compatible changelog shapes, not a stronger replication guarantee; see the Table capability map for the endpoint caveat.
Worked cases#
| Page | Covers |
|---|---|
| Dynamic destinations | The shared resolver contract, per-destination resources, idle eviction, auto-creation and Pub/Sub ordering across all five sinks |
| BigQuery | DataStream table, query, view, snapshot and public-dataset reads; dynamic destinations, both exactly-once write methods, redeployment and table auto-creation; a bounded Table source and an append-only Table sink with all three write methods; Debezium MySQL, Debezium PostgreSQL, TiCDC and Spanner analytics-replica CDC; and the emulator |
| Cloud Pub/Sub | DataStream subscription and topic creation, dynamic topics, Table source startup and ordering, readable and writable message metadata, and the emulator |
| Cloud Tasks | Dynamic queues; App Engine, Cloud Run function and external API tasks; Pub/Sub events enriched from Bigtable; JSON, CSV, raw and Avro bodies; the emulator |
| Bigtable | Bounded and filtered DataStream reads; multi-cell and dynamic-table writes, skipping and dropping records; Table scans, upsert and insert-only sinks, writable timestamps, and lookup joins; Pub/Sub attribute enrichment into Cloud Tasks; envelope CDC and a selected-cell BigQuery analytics replica; application profiles and the emulator |
| Spanner | DataStream key-range, fixed-timestamp and Data Boost reads; multi-table routing, deletes, skipping, refusal handling and batching; bounded Table scans, upsert writes and lookup joins; full and keyed Change Streams with observable Spanner-to-Spanner materialization; and the emulator |
Two things cut across most of them, and are stated once here rather than five times.
Dynamic per-record destinations share one shape#
The Dynamic destinations guide expands this shared shape into one implementation path for all five sinks. It covers the resolver contract, Spanner’s mutation-inherent table routing, destination identity, resource lifetimes, idle eviction, auto-creation and Pub/Sub ordering.
An emulator is a convenience, not an authority#
Every emulator page below is for fast local feedback and never evidence about the service’s behaviour. Where the two disagree the real service decides, and each emulator has blind spots that matter: a green emulator run is not a green integration. The pages say what each one cannot show, which is the part worth reading before trusting one.
BigQuery is the one that takes two endpoints rather than one, because it serves table metadata over REST and the Storage Write API over gRPC — the BigQuery page sets out both, and what a run against that emulator cannot show.