Bigtable options#

Every option the Bigtable sink and source take. What each one is for is on the Bigtable connector page; the three forms of the Default column are explained here.

The WITH options of the bigtable table connector are a separate surface, documented on the Bigtable SQL connector page.

One thing this reference does not list, because it is not an option here. There are no retry knobs: the client retries MutateRows per entry on a schedule of its own, so the sink owns no retry loop — the opposite of the Cloud Tasks sink, and the reasoning is under Retries. The recovery* knobs below are not an exception: they budget the sink-owned table auto-creation repair, not the client’s mutation retries. The source owns no retry loop either: the client resumes a broken ReadRows stream from the last key it saw.

BigtableSink.builder()#

The mutation itself is built by the serializer, not configured here. Row key, column families and qualifiers, cell timestamps, deletes — every per-record decision about the mutation belongs to the BigtableSerializationSchema, which returns the whole RowMutationEntry; the table it goes to is the resolver’s, never the serializer’s. The one decision worth making deliberately is the cell timestamp, because it is what decides whether a replayed record overwrites a cell or adds a version to it; see Delivery guarantees.

A destination costs a batcher. The client binds a bulk mutation batcher to one table, so the writer holds one per table a resolver names, over a client shared by the tables of an instance; an idle table’s batcher is dropped after destinationIdleTimeout, and the client normally starts closing on a daemon reaper when its last live table is gone. If the runtime refuses that handoff, the factory closes the client on the task thread to avoid leaking it. maxActiveInstances bounds open and closing clients held by each writer subtask; a physical close keeps its slot, so client creation waits interruptibly when every slot is still occupied. The in-flight bounds below are the writer’s, summed across destinations rather than split among them. See Per-record destinations.

BigtableWriterOptions#

Set through writerOptions(...); every knob is defaulted, so defaults() is equivalent to not setting options at all.

Every count in this table counts entries, not mutations. An entry is one RowMutationEntry — one record the serializer returned — and it carries as many mutations as the serializer put setCell calls in it. Bigtable’s own documented limit is on mutations: no more than 100,000 in a batch. The two numbers never have to be reconciled by a job, because the client holds a batch to that limit itself, whatever batchElementCountThreshold says; the measurement and what it retires are under Tuning.

Raising maxInFlightEntries far above its default does not raise the effective bound; it moves it. The client has a flow controller of its own — 20,000 outstanding entries and 100 MiB, and it blocks the calling thread when either is reached — whose limits its public API does not expose. While the sink’s own bounds are the tighter pair, a full writer yields to the task mailbox, which is what keeps checkpoint barriers moving; past them, the task thread stalls inside the client instead. The reasoning is under Tuning.

Those same two budgets are where the batch thresholds’ ceilings come from, and they are the client’s rule rather than this connector’s: its settings builder requires each threshold to stay strictly below the matching budget, and refuses to build a client at all otherwise — on the task manager, as the writer opens. Hence 19,999 and 100 MiB − 1: one under each.

The in-flight bounds are warned about rather than capped at those same figures. Setting either above its budget is a working configuration — the client simply becomes the layer that bounds the sink — so build() logs a WARN naming the value and what it costs instead of rejecting it. It is not a ceiling because the budget is per client and this sink holds one per (project, instance): a resolver spreading records over several instances draws on several budgets, and a writer-global bound above one of them can be what that job means.

The two batch thresholds are left unset by default rather than restated here, so a client upgrade that retunes them is inherited. Lowering batchElementCountThreshold shortens the delay before a mutation reaches the service at low volume; the client also sends a batch after one second regardless.

TableCreateOptions#

The schema for the table the sink creates under CREATE_IF_NEEDED — its column families and, per family, an optional garbage-collection rule. Creation only, per family: an existing table is written to as it is, except that families declared here which it lacks are added, with their rules; an existing family’s rule is neither compared nor updated. What creation does and does not repair is under Table auto-creation.

A family declared without a rule keeps Bigtable’s default of collecting nothing — for this at-least-once sink a real decision, since the garbage-collection policy is what decides whether a replay’s duplicate cell versions accumulate forever. The rule is built from four static GcRule factories mirroring the admin API’s shapes: maxVersions(int), maxAge(Duration), union(GcRule...) and intersection(GcRule...) (each composite takes at least two rules). union(GcRule.maxVersions(1), GcRule.maxAge(...)) is the usual shape for keeping only the latest cell. Validation is shape-only — positivity and arity; the service’s own limits are left to Bigtable, whose rejection names what it refused.

BigtableSource.builder()#

The fetch bounds are memory hand-off controls, not a query row limit. A Query.limit() is global to a query, so it cannot be partitioned across splits without coordination — the client library refuses to shard a query that carries one. The source instead reads the complete configured ranges and returns control to Flink whenever either fetch bound is reached.

Per-cell shaping is the filter’s job, not a knob’s. Which families and qualifiers to return, which timestamp window, how many versions of a cell — all of it is expressible through filter(...), which is why the source has no separate options for any of it.

The SQL layer’s scan.* options map onto this builder — the Bigtable SQL connector page carries that surface, and its projection pushdown is what supplies filter(...) there.

BigtableChangeStreamSource.builder()#

@PublicEvolving: the change-stream API may change at a minor release, announced in the release notes.

There is no emulator option because the Bigtable emulator implements neither change-stream RPC.

There is no heartbeat-interval option either, because the five-second service heartbeat is also what paces the reader’s rotation of queued partitions.