Delivery Guarantees

Mon, 01 Jan 0001 00:00:00 +0000

Delivery Guarantees#

This page compares what a completed Flink checkpoint means for each sink and what can happen when Flink restores an earlier checkpoint.

The short version is that flushing before a checkpoint makes data durable, but does not by itself prevent a restored job from writing the same record again. BigQuery has checkpoint-coordinated exactly-once write methods. Cloud Tasks also has opt-in, experimental checkpointed task creation within its documented retention and recovery scope; it keeps up with the at-least-once path at the measured rates and adds most of a checkpoint interval to task visibility latency. Some other sinks can make a replay harmless when the record supplies a stable identity, but that is not the same contract as a general-purpose exactly-once sink.

Lineage

Mon, 01 Jan 0001 00:00:00 +0000

Lineage#

The shared lineage contract describes the physical resources known from a connector’s configuration. It does not discover resources at runtime or infer them by inspecting records. BigQuery DataStream and Table/SQL implement this contract for configured table inputs and fixed outputs, including explicitly named view materialization and default-stream CDC. Bigtable DataStream and Table/SQL implement configured-table lineage for scans, Change Streams and all sink write modes. The common graph and listener tests establish the shared contract. Pub/Sub and Bigtable also test extraction against their builder-returned Source/Sink objects and SQL planners. Pub/Sub covers multiple subscriptions and ordering-key routing; Bigtable covers scans, bounded and unbounded Change Streams, and every Table write mode. Spanner implements the contract for the paths documented in its DataStream and Table references. The Cloud Tasks guide describes its fixed-queue coverage for DataStream and Table/SQL; its connector tests exercise the builder-returned sink, listener delivery and planner facets. The Firestore collection-group scan reports its collection group, a Datastore-mode source reports the kind its configuration names (a kind, or a query naming exactly one kind), and the Firestore Table source and sink report the collection its collection option names, or a scan under scan.collection-group the group, and the Datastore-mode Table source and sink report the kind its kind option names, in its namespace. A Native-mode source reading a query, a Datastore-mode source reading a GQL query or a query naming no kind or several, and the DataStream Firestore sinks in both modes report an empty dataset list, because the query factory, the service’s parse or the serializer, not the configuration, names the collections or kinds.

Connectors on GCP Connectors for Apache Flink

Delivery Guarantees

Delivery Guarantees#

Lineage

Lineage#