Package io.github.flink.gcp.connector.bigtable.sink

Class BigtableSink

java.lang.Object
io.github.flink.gcp.connector.bigtable.sink.BigtableSink
@Public public final class BigtableSink extends Object
Entry point for building a Bigtable sink.

By default the sink applies one row mutation per record through the client's bulk MutateRows batcher, at-least-once, and waits for every outstanding mutation at each checkpoint barrier. A replayed record overwrites the same cells only when the serializer sets explicit cell timestamps; see BigtableSerializationSchema.

That at-least-once statement assumes the default FailureHandler.failJob() policy. Under a dropping policy configured through BigtableSinkBuilder.failedMutationHandler(FailureHandler), a completed checkpoint means every record up to the barrier was either applied, skipped by the serializer, or handed to that handler.

The sink writes to one fixed table, or to a table it resolves per record: BigtableSinkBuilder.table(TableDestination) names one, and BigtableSinkBuilder.destinationResolver(DestinationResolver) routes each record to a table of its own. By default it never creates a table — every table it writes to, and its column families, must exist; BigtableSinkBuilder.createDisposition(CreateDisposition) with CreateDisposition.CREATE_IF_NEEDED and BigtableSinkBuilder.tableCreateOptions(TableCreateOptions) opts into creating them, from one schema that serves every table the sink creates.

BigtableSinkBuilder.deliveryGuarantee(BigtableDeliveryGuarantee) can instead select experimental checkpoint-owned writes. That mode stages immutable envelopes and applies each after checkpoint completion through a same-row retained marker, with no cross-row atomicity or rollback. It requires an explicit transactional profile, a pre-provisioned marker family without a GC rule, and preserved checkpoint state. Final service recovery and performance acceptance remain required before release; see BigtableStagedOptions.

The builder-returned sink implements LineageVertexProvider. Its effective fixed destination reports namespace bigtable://{project}/{instance}, name {table} and a gcp physical-resource facet. Dynamic destinations produce an empty dataset list, without resolver evaluation. Extraction calls no user schema and opens no client. The supported Flink 2.x versions extract the metadata automatically; Flink 1.20 supports direct inspection only.

Example: 


 Sink<OrderEvent> sink =
         BigtableSink.<OrderEvent>builder()
                 .table(TableDestination.of("my-project", "my-instance", "orders"))
                 .serializer(
                         (event, context) ->
                                 RowMutationEntry.create(event.getId())
                                         .setCell(
                                                 "cf",
                                                 "payload",
                                                 event.getTimestampMicros(),
                                                 event.getBody()))
                 .build();

  • Method Details