Package io.github.flink.gcp.connector.bigtable.sink

Class BigtableWriterOptions

java.lang.Object
io.github.flink.gcp.connector.bigtable.sink.BigtableWriterOptions
All Implemented Interfaces:
Serializable
@Public public final class BigtableWriterOptions extends Object implements Serializable
Tuning options for the sink's writer: the batch thresholds handed to the client, and the writer's own bounds on unacknowledged entries.

Set via BigtableSinkBuilder.writerOptions(BigtableWriterOptions); optional — every knob is defaulted, so defaults() is equivalent to not setting options at all. An unset batch threshold leaves the client's own default in place rather than restating it here, so a client upgrade that retunes it is inherited.

Entries, mutations, and which knob binds

Every count here is a count of entries, never of mutations. An entry is one RowMutationEntry — one record the serializer returned — and it carries as many mutations as the serializer put setCell calls in it. Bigtable's documented limit is on mutations: no more than 100,000 in a batch. The two numbers are not compared by this connector, and need not be by a job either, because the client enforces the mutation limit itself and unconditionally: its batch resource flushes as soon as one more entry would carry the accumulated batch past 100,000 mutations, whatever BigtableWriterOptions.Builder.batchElementCountThreshold(long) says, and no single entry can carry more than that on its own. So no setting of these knobs produces an over-limit request (read from google-cloud-bigtable 2.80.0 on 2026-08-10, and pinned by BigtableClientMutationLimitTest so that a client upgrade moving either fact fails a test rather than a job — docs/adr/0082).

A batch is therefore sent on whichever of five conditions arrives first: BigtableWriterOptions.Builder.batchElementCountThreshold(long), BigtableWriterOptions.Builder.batchRequestByteThreshold(long), the client's one-second timer, the client's 100,000-mutation guard, and the writer's own BigtableWriterOptions.Builder.maxInFlightEntries(int) — which sends every batcher when the writer fills. Any claim of the form "setting X large makes batches of X" has to name the condition that binds, or it is false.

There are deliberately no retry knobs. Unlike Cloud Tasks, the Bigtable client retries MutateRows itself — per entry, for the transient codes, on a schedule of its own — so the sink owns no retry loop and has nothing to expose. The recovery* knobs are not an exception: they budget the sink-owned table auto-creation repair (re-applying mutations after creating a missing table), not the client's mutation retries.

Why the in-flight bounds are the writer's own

The client also has a flow controller of its own, and it is the wrong instrument here: it blocks the calling thread when its limits are reached, and the calling thread is Flink's task thread, which must stay free to run mailbox mails. So the writer keeps its own bounds and yields to the mailbox instead. That only works while the writer's bounds are reached first, and the client's limits — 20,000 outstanding entries and 100 MiB, blocking — cannot be raised through its public API. Raising BigtableWriterOptions.Builder.maxInFlightEntries(int) far above the default therefore moves the effective bound into the client, where it stalls the task thread rather than backpressuring the stream.

Per-record destinations do not change that relationship, and the reason is measured rather than assumed: the client's flow controller is one per client — EnhancedBigtableStub builds a single bulkMutationFlowController and hands the same instance to every batcher it creates — so the tables of one instance draw on one budget, and these caps, being the writer's rather than each destination's, still bind first however many tables the sink writes to.

Instances are immutable and serializable.

See Also:

  • Field Details

  • Method Details

    • builder

      public static BigtableWriterOptions.Builder builder()
      Creates a new BigtableWriterOptions.Builder.
      Returns:
      a new builder

    • defaults

      public static BigtableWriterOptions defaults()
      Returns the default options: the client's own batch thresholds, at most 1000 unacknowledged entries, at most 64 MiB of them, a job failure after 100 consecutive confirmed rejections under a dropping policy, a table auto-creation recovery budget of 500 ms doubling to 10 s over at most 10 attempts, an idle table's batcher dropped after DEFAULT_DESTINATION_IDLE_TIMEOUT, at most 16 active instance clients per subtask, and no per-table counters.
      Returns:
      the default options

    • getBatchElementCountThreshold

      @Nullable public Long getBatchElementCountThreshold()
      Returns the batch element-count threshold, or null to use the client's default.

    • getBatchRequestByteThreshold

      @Nullable public Long getBatchRequestByteThreshold()
      Returns the batch request-byte threshold, or null to use the client's default.

    • getMaxInFlightEntries

      public int getMaxInFlightEntries()
      Returns the writer's cap on unacknowledged entries.

    • getMaxInFlightBytes

      public long getMaxInFlightBytes()
      Returns the writer's cap on the serialized size of unacknowledged entries.

    • getMaxConsecutiveRejections

      public int getMaxConsecutiveRejections()
      Returns how many consecutive confirmed rejections fail the job, or UNBOUNDED for none.

    • getRecoveryInitialBackoff

      public Duration getRecoveryInitialBackoff()
      Returns the first backoff of the table auto-creation recovery.

    • getRecoveryMaxBackoff

      public Duration getRecoveryMaxBackoff()
      Returns the backoff cap of the table auto-creation recovery.

    • getRecoveryMaxAttempts

      public int getRecoveryMaxAttempts()
      Returns the maximum re-apply attempts of the table auto-creation recovery.

    • getDestinationIdleTimeout

      public Duration getDestinationIdleTimeout()
      Returns how long a table may go without mutations before the writer drops its batcher.

    • getMaxActiveInstances

      public int getMaxActiveInstances()
      Returns the maximum number of Bigtable instance clients held by one writer subtask.

      The zero fallback preserves the default for options serialized before this field existed.

    • isPerDestinationMetrics

      public boolean isPerDestinationMetrics()
      Returns whether per-table counters are registered beside the writer's totals.

    • toRecoverySchedule

      @Internal public RetrySchedule toRecoverySchedule()
      Returns the table auto-creation recovery schedule the recovery* knobs describe. Jittered: every subtask that parked mutations for the same missing table resumes against the same freshly created table, so unjittered they would re-apply in lockstep.

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class Object

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • toString

      public String toString()
      Overrides:
      toString in class Object