Package io.github.flink.gcp.connector.bigtable.source

Class BigtableChangeStreamSourceBuilder<T>

java.lang.Object
io.github.flink.gcp.connector.bigtable.source.BigtableChangeStreamSourceBuilder<T>
Type Parameters:
T - the record type produced
@PublicEvolving public final class BigtableChangeStreamSourceBuilder<T> extends Object
Builds a BigtableChangeStreamSource.

  • Field Details

    • DEFAULT_MAX_CONCURRENT_STREAMS_PER_SUBTASK

      public static final int DEFAULT_MAX_CONCURRENT_STREAMS_PER_SUBTASK
      Default maximum number of open partition streams in one source subtask.
      See Also:

  • Method Details

    • table

      public BigtableChangeStreamSourceBuilder<T> table(TableDestination table)
      Sets the table whose change stream is read. Required.
      Parameters:
      table - the table
      Returns:
      this builder

    • deserializer

      public BigtableChangeStreamSourceBuilder<T> deserializer(BigtableChangeStreamDeserializationSchema<T> deserializer)
      Sets the deserializer that turns each change-stream mutation into user records. Required.
      Parameters:
      deserializer - the deserializer
      Returns:
      this builder

    • appProfileId

      public BigtableChangeStreamSourceBuilder<T> appProfileId(String appProfileId)
      Sets the application profile the change stream is read through. Required, and the profile must route to a single cluster; the source rejects a visibly multi-cluster profile before reading.
      Parameters:
      appProfileId - the application profile id
      Returns:
      this builder

    • serviceAccountKeyFile

      public BigtableChangeStreamSourceBuilder<T> serviceAccountKeyFile(String serviceAccountKeyFile)
      Authenticates Change Streams with the service-account JSON key at the given path instead of application-default credentials. The JobManager reads it when a fresh or restored coordinator starts. Each TaskManager reads it when its reader is created, before any partition is assigned to it. Every eligible process must therefore see the same path.

      Service-account keys are long-lived secrets. Prefer an attached service account or Workload Identity where the deployment supports one.

      Parameters:
      serviceAccountKeyFile - the service-account JSON key-file path
      Returns:
      this builder

    • exportClientMetricsToFlink

      public BigtableChangeStreamSourceBuilder<T> exportClientMetricsToFlink(boolean exportClientMetricsToFlink)
      Exports the Bigtable client's own metrics through Flink's metric system, so that every configured Flink metric reporter sees them. Optional; off by default.

      Each reader registers them under its metric group as rpcMethod.ReadChangeStream.*, and the enumerator, which discovers the stream's partitions, as rpcMethod.GenerateInitialChangeStreamPartitions.*: the client's operation and attempt counts and latencies, its retries, its connectivity errors, and its server, first-response and application time. Latencies are reported as a count and a millisecond sum; no percentile is reported. A ReadChangeStream operation is one partition's stream, so it ends, and counts, only when that stream closes.

      This does not stop the client's built-in export of the same metrics to Cloud Monitoring; exportClientMetricsToCloudMonitoring(boolean) does.

      Parameters:
      exportClientMetricsToFlink - whether to export the client's metrics to Flink
      Returns:
      this builder

    • exportClientMetricsToCloudMonitoring

      public BigtableChangeStreamSourceBuilder<T> exportClientMetricsToCloudMonitoring(boolean exportClientMetricsToCloudMonitoring)
      Sets whether the Bigtable data client keeps exporting its built-in client-side metrics to Cloud Monitoring. Optional; on by default.

      Off, the client creates no Cloud Monitoring exporter at all. Besides the export, that removes the exporter's platform detection when a client closes: off Google Cloud, where nothing answers on the metadata server's address, each close otherwise waits for that connection attempt to time out. It is independent of exportClientMetricsToFlink(boolean), which neither needs nor stops this export.

      Parameters:
      exportClientMetricsToCloudMonitoring - whether the client exports to Cloud Monitoring
      Returns:
      this builder

    • startPosition

      public BigtableChangeStreamSourceBuilder<T> startPosition(StartPosition startPosition)
      Sets where a fresh run starts reading the stream. Optional; defaults to StartPosition.latest(). A restored run resumes from its checkpointed positions instead; resumeFallback(StartPosition) governs a restored position that has expired.
      Parameters:
      startPosition - the start position
      Returns:
      this builder

    • resumeFallback

      public BigtableChangeStreamSourceBuilder<T> resumeFallback(StartPosition resumeFallback)
      Sets where a partition restarts when its restored position has fallen outside the table's change-stream retention. Optional; unset means such a restore fails the job rather than advancing over records that can no longer be read.

      It applies per partition, not to the whole restore: only the partitions whose checkpointed position expired restart from here, while the others resume from their continuation tokens. Setting it is a decision to lose the unavailable interval of those partitions rather than to stop, and a position older than the retained window is moved forward to the earliest readable instant.

      Parameters:
      resumeFallback - the fallback start position
      Returns:
      this builder

    • boundedTimestamp

      public BigtableChangeStreamSourceBuilder<T> boundedTimestamp(Instant boundedTimestamp)
      Makes the source bounded at a change-stream timestamp.
      Parameters:
      boundedTimestamp - the timestamp at which the source stops
      Returns:
      this builder

    • maxConcurrentStreamsPerSubtask

      public BigtableChangeStreamSourceBuilder<T> maxConcurrentStreamsPerSubtask(int maximum)
      Bounds the open ReadChangeStream RPCs in each source subtask. Source parallelism multiplied by this value is the job's configured read capacity, not a Bigtable quota. The default is 2.
      Parameters:
      maximum - positive per-subtask stream limit
      Returns:
      this builder

    • familyIncludeList

      public BigtableChangeStreamSourceBuilder<T> familyIncludeList(Collection<String> patterns)
      Includes entries whose column-family names fully match at least one Java regular expression.

      An empty collection disables this filter. It is mutually exclusive with familyExcludeList(Collection).

      Parameters:
      patterns - the family-name patterns
      Returns:
      this builder

    • familyExcludeList

      public BigtableChangeStreamSourceBuilder<T> familyExcludeList(Collection<String> patterns)
      Excludes entries whose column-family names fully match at least one Java regular expression.

      An empty collection disables this filter. It is mutually exclusive with familyIncludeList(Collection).

      Parameters:
      patterns - the family-name patterns
      Returns:
      this builder

    • qualifierIncludeList

      public BigtableChangeStreamSourceBuilder<T> qualifierIncludeList(Collection<String> patterns)
      Includes qualified columns that fully match at least one Java regular expression.

      The matched identifier is family:qualifierBase64, where the qualifier is canonical padded RFC 4648 standard Base64. An empty qualifier therefore produces family:. Family-delete entries have no qualifier and are governed only by the family filter. An empty collection disables this filter. It is mutually exclusive with qualifierExcludeList(Collection).

      Parameters:
      patterns - the qualified-column patterns
      Returns:
      this builder

    • qualifierExcludeList

      public BigtableChangeStreamSourceBuilder<T> qualifierExcludeList(Collection<String> patterns)
      Excludes qualified columns that fully match at least one Java regular expression.

      The matched identifier uses the same family:qualifierBase64 representation as qualifierIncludeList(Collection). Family-delete entries have no qualifier and are governed only by the family filter. An empty collection disables this filter. It is mutually exclusive with qualifierIncludeList(Collection).

      Parameters:
      patterns - the qualified-column patterns
      Returns:
      this builder

    • skipMessagesWithoutChange

      public BigtableChangeStreamSourceBuilder<T> skipMessagesWithoutChange(boolean skip)
      Skips a mutation when entry filtering removes every entry it reported.

      The default is false, which delivers the mutation with an empty entry list so that the atomic row mutation remains visible.

      Parameters:
      skip - whether to skip fully filtered mutations
      Returns:
      this builder

    • build

      public BigtableChangeStreamSource<T> build()
      Builds the source.
      Returns:
      the source
      Throws:
      IllegalStateException - if a required option was not set, or if both sides of a mutually exclusive filter pair were set