Package io.github.flink.gcp.connector.bigtable.sink

Class BigtableWriterOptions.Builder

java.lang.Object
io.github.flink.gcp.connector.bigtable.sink.BigtableWriterOptions.Builder
Enclosing class:
BigtableWriterOptions
@Public public static final class BigtableWriterOptions.Builder extends Object
Builder for BigtableWriterOptions.

  • Method Details

    • batchElementCountThreshold

      public BigtableWriterOptions.Builder batchElementCountThreshold(long batchElementCountThreshold)
      Sets how many entries the client accumulates before sending a batch — one per record the serializer returned, however many mutations each carries. Defaults to the client's own threshold (100 entries).

      This is a threshold, not a cap on what a request may hold: the client sends a batch on whichever condition arrives first, and its own 100,000-mutation guard is one of them. See the class documentation for the other three.

      Parameters:
      batchElementCountThreshold - the element-count threshold, positive and at most 19,999 — one under the 20,000 entries the client's flow controller admits, which its own settings builder requires this threshold to stay strictly below
      Returns:
      this builder

    • batchRequestByteThreshold

      public BigtableWriterOptions.Builder batchRequestByteThreshold(long batchRequestByteThreshold)
      Sets how many bytes of mutations the client accumulates before sending a batch. Defaults to the client's own threshold (20 MiB). Entry count alone bounds no memory: a single entry may be megabytes, since Bigtable's own size limits are per mutation (200 MB), per cell value (100 MB) and per row (256 MB).
      Parameters:
      batchRequestByteThreshold - the byte threshold, positive and at most one byte under the 100 MiB the client's flow controller admits in flight, which its own settings builder requires this threshold to stay strictly below
      Returns:
      this builder

    • maxInFlightEntries

      public BigtableWriterOptions.Builder maxInFlightEntries(int maxInFlightEntries)
      Caps the entries the writer keeps unacknowledged — one per record written, whatever each carries. A write at the cap yields to the task mailbox until completions bring the count down, bounding sink memory between checkpoints. Defaults to 1000.

      Raising this far above the default moves the effective bound into the client's own flow controller, which blocks the task thread instead of yielding — see the class documentation.

      Parameters:
      maxInFlightEntries - the in-flight cap, positive
      Returns:
      this builder

    • maxInFlightBytes

      public BigtableWriterOptions.Builder maxInFlightBytes(long maxInFlightBytes)
      Caps the serialized size of the entries the writer keeps unacknowledged. Defaults to 64 MiB. This is the bound that actually bounds memory — a single entry may be megabytes, so a count alone does not.
      Parameters:
      maxInFlightBytes - the in-flight byte cap, positive
      Returns:
      this builder

    • maxConsecutiveRejections

      public BigtableWriterOptions.Builder maxConsecutiveRejections(int maxConsecutiveRejections)
      Sets how many consecutive confirmed rejections fail the job. Defaults to 100; BigtableWriterOptions.UNBOUNDED (-1) never fails it.

      This bound only matters beside a dropping failedMutationHandler — under the default failJob() the first confirmed rejection fails the job anyway. A dropping policy is a decision to keep running through anomalous records, and the sink pays one solo request per rejection to isolate each from the good records batched with it. When every record is being refused, that is no longer a stream with anomalies but a broken pipeline degraded to unbatched writes under a green job — so once this many confirmed rejections arrive in a row, with not one successfully applied mutation between them, the job fails with a message naming this option. Any applied mutation resets the count: an occasional bad record can never accumulate into a failure, however long the job runs.

      Only rejections the isolation pass has confirmed against a single mutation count; records the serializer rejects do not, since they say nothing about the service's view of the stream.

      Parameters:
      maxConsecutiveRejections - the bound, positive or BigtableWriterOptions.UNBOUNDED
      Returns:
      this builder

    • recoveryInitialBackoff

      public BigtableWriterOptions.Builder recoveryInitialBackoff(Duration recoveryInitialBackoff)
      Sets the first backoff of the table auto-creation recovery (re-applying mutations after creating a missing table). Defaults to 500 ms.
      Parameters:
      recoveryInitialBackoff - the first backoff, at least 1 ms
      Returns:
      this builder

    • recoveryMaxBackoff

      public BigtableWriterOptions.Builder recoveryMaxBackoff(Duration recoveryMaxBackoff)
      Caps the backoff of the table auto-creation recovery. Defaults to 10 s.
      Parameters:
      recoveryMaxBackoff - the backoff cap, at least 1 ms and at least the initial backoff
      Returns:
      this builder

    • recoveryMaxAttempts

      public BigtableWriterOptions.Builder recoveryMaxAttempts(int recoveryMaxAttempts)
      Caps the re-apply attempts of the table auto-creation recovery. Defaults to 10.
      Parameters:
      recoveryMaxAttempts - the maximum attempts, positive
      Returns:
      this builder

    • destinationIdleTimeout

      public BigtableWriterOptions.Builder destinationIdleTimeout(Duration destinationIdleTimeout)
      Sets how long a table may go without mutations before the writer closes and drops its batcher. Eviction is memory hygiene for long-lived jobs with per-record destinations (for example date-suffixed tables), whose per-table state otherwise grows without bound; correctness is unaffected, and a table that receives a mutation again after eviction rebuilds its batcher transparently. The sweep runs at the end of each successful flush, when nothing is parked or in flight. Defaults to BigtableWriterOptions.DEFAULT_DESTINATION_IDLE_TIMEOUT; to never evict, set a very large duration — up to Duration.ofNanos(Long.MAX_VALUE), about 292 years, which is as long as the writer's nanosecond clock can express.

      The sink holds one client per (project, instance), shared by that instance's tables. Evicting one table releases its ownership of that client; the client closes when the instance's last table is evicted.

      Parameters:
      destinationIdleTimeout - the idle timeout, positive and at most Duration.ofNanos(Long.MAX_VALUE)
      Returns:
      this builder

    • maxActiveInstances

      public BigtableWriterOptions.Builder maxActiveInstances(int maxActiveInstances)
      Caps the open-or-closing Bigtable instance clients held by one writer subtask. When a new instance would exceed the cap, the writer safely drains outstanding work and evicts the least recently used instance. Client close normally runs off the task thread, but keeps its slot until physical shutdown finishes; creation waits interruptibly when every slot remains occupied. If the runtime refuses the handoff, close falls back to the task thread to avoid a resource leak. A later record for that instance recreates its client transparently. Defaults to 16.
      Parameters:
      maxActiveInstances - the instance-client cap, positive
      Returns:
      this builder

    • perDestinationMetrics

      public BigtableWriterOptions.Builder perDestinationMetrics(boolean perDestinationMetrics)
      Registers per-table recordsSend and sendErrors counters beside the writer's totals. Defaults to false.

      Off by default because Flink cannot unregister a metric: with per-record destinations the table set is unbounded, so every table the job ever writes to keeps a row in the metric registry for the lifetime of the task — including one whose batcher destinationIdleTimeout(Duration) has since evicted. Counters survive eviction: a table seen again resumes its own totals. Switch it on for a sink whose tables are few and known.

      Parameters:
      perDestinationMetrics - whether to register per-table counters
      Returns:
      this builder

    • build

      public BigtableWriterOptions build()
      Builds the options.

      Warns, rather than fails, when an in-flight bound is above the client's own flow-control budget. Past that budget the writer's bound is no longer the one that binds: the client blocks the task thread instead, which is what the writer's bounds exist to avoid. It is not refused, because that budget is per client and this sink holds one per (project, instance) — a resolver spreading records over several instances draws on several budgets, so a writer-global bound above one of them can be exactly what such a job means. Nothing here knows how many instances a resolver will name, and the batch thresholds' ceilings are a different case: those the client refuses outright.

      Returns:
      the options