Delivery Guarantees

Mon, 01 Jan 0001 00:00:00 +0000

Delivery Guarantees#

This page compares what a completed Flink checkpoint means for each sink and what can happen when Flink restores an earlier checkpoint.

The short version is that flushing before a checkpoint makes data durable, but does not by itself prevent a restored job from writing the same record again. BigQuery has checkpoint-coordinated exactly-once write methods. Cloud Tasks also has opt-in, experimental checkpointed task creation within its documented retention and recovery scope; it keeps up with the at-least-once path at the measured rates and adds most of a checkpoint interval to task visibility latency. Some other sinks can make a replay harmless when the record supplies a stable identity, but that is not the same contract as a general-purpose exactly-once sink.

Lineage

Mon, 01 Jan 0001 00:00:00 +0000

Lineage#

The shared lineage contract describes the physical resources known from a connector’s configuration. It does not discover resources at runtime or infer them by inspecting records. BigQuery DataStream and Table/SQL implement this contract for configured table inputs and fixed outputs, including explicitly named view materialization and default-stream CDC. Bigtable DataStream and Table/SQL implement configured-table lineage for scans, Change Streams and all sink write modes. Connector adoption remains tracked separately for Pub/Sub, Spanner, and Cloud Tasks. The common graph and listener tests establish the shared contract. Pub/Sub and Bigtable also test extraction against their builder-returned Source/Sink objects and SQL planners. Pub/Sub covers multiple subscriptions and ordering-key routing; Bigtable covers scans, bounded and unbounded Change Streams, and every Table write mode. Spanner implements the contract for the paths documented in its DataStream and Table references. The Cloud Tasks guide describes its fixed-queue coverage for DataStream and Table/SQL; its connector tests exercise the builder-returned sink, listener delivery and planner facets.

Connectors on GCP Connectors for Apache Flink

Delivery Guarantees

Delivery Guarantees#

Lineage

Lineage#