Package io.github.flink.gcp.connector.bigtable.source

Class BigtableChangeStreamSourceBuilder<T>

java.lang.Object
io.github.flink.gcp.connector.bigtable.source.BigtableChangeStreamSourceBuilder<T>
Type Parameters:
T - the record type produced
@PublicEvolving public final class BigtableChangeStreamSourceBuilder<T> extends Object
Builds a BigtableChangeStreamSource.

  • Field Details

    • DEFAULT_MAX_CONCURRENT_STREAMS_PER_SUBTASK

      public static final int DEFAULT_MAX_CONCURRENT_STREAMS_PER_SUBTASK
      Default maximum number of open partition streams in one source subtask.
      See Also:

  • Method Details

    • table

      public BigtableChangeStreamSourceBuilder<T> table(TableDestination table)
      Sets the table whose change stream is read. Required.
      Parameters:
      table - the table
      Returns:
      this builder

    • deserializer

      public BigtableChangeStreamSourceBuilder<T> deserializer(BigtableChangeStreamDeserializationSchema<T> deserializer)
      Sets the deserializer that turns each change-stream mutation into user records. Required.
      Parameters:
      deserializer - the deserializer
      Returns:
      this builder

    • appProfileId

      public BigtableChangeStreamSourceBuilder<T> appProfileId(String appProfileId)
      Sets the application profile the change stream is read through. Required, and the profile must route to a single cluster; the source rejects a visibly multi-cluster profile before reading.
      Parameters:
      appProfileId - the application profile id
      Returns:
      this builder

    • serviceAccountKeyFile

      public BigtableChangeStreamSourceBuilder<T> serviceAccountKeyFile(String serviceAccountKeyFile)
      Authenticates Change Streams with the service-account JSON key at the given path instead of application-default credentials. The JobManager reads it when a fresh or restored coordinator starts. Each TaskManager reads it when its reader is created, before any partition is assigned to it. Every eligible process must therefore see the same path.

      Service-account keys are long-lived secrets. Prefer an attached service account or Workload Identity where the deployment supports one.

      Parameters:
      serviceAccountKeyFile - the service-account JSON key-file path
      Returns:
      this builder

    • startPosition

      public BigtableChangeStreamSourceBuilder<T> startPosition(StartPosition startPosition)
      Sets where a fresh run starts reading the stream. Optional; defaults to StartPosition.latest(). A restored run resumes from its checkpointed positions instead; resumeFallback(StartPosition) governs a restored position that has expired.
      Parameters:
      startPosition - the start position
      Returns:
      this builder

    • resumeFallback

      public BigtableChangeStreamSourceBuilder<T> resumeFallback(StartPosition resumeFallback)
      Sets where a partition restarts when its restored position has fallen outside the table's change-stream retention. Optional; unset means such a restore fails the job rather than advancing over records that can no longer be read.

      It applies per partition, not to the whole restore: only the partitions whose checkpointed position expired restart from here, while the others resume from their continuation tokens. Setting it is a decision to lose the unavailable interval of those partitions rather than to stop, and a position older than the retained window is moved forward to the earliest readable instant.

      Parameters:
      resumeFallback - the fallback start position
      Returns:
      this builder

    • boundedTimestamp

      public BigtableChangeStreamSourceBuilder<T> boundedTimestamp(Instant boundedTimestamp)
      Makes the source bounded at a change-stream timestamp.
      Parameters:
      boundedTimestamp - the timestamp at which the source stops
      Returns:
      this builder

    • maxConcurrentStreamsPerSubtask

      public BigtableChangeStreamSourceBuilder<T> maxConcurrentStreamsPerSubtask(int maximum)
      Bounds the open ReadChangeStream RPCs in each source subtask. Source parallelism multiplied by this value is the job's configured read capacity, not a Bigtable quota. The default is 2.
      Parameters:
      maximum - positive per-subtask stream limit
      Returns:
      this builder

    • familyIncludeList

      public BigtableChangeStreamSourceBuilder<T> familyIncludeList(Collection<String> patterns)
      Includes entries whose column-family names fully match at least one Java regular expression.

      An empty collection disables this filter. It is mutually exclusive with familyExcludeList(Collection).

      Parameters:
      patterns - the family-name patterns
      Returns:
      this builder

    • familyExcludeList

      public BigtableChangeStreamSourceBuilder<T> familyExcludeList(Collection<String> patterns)
      Excludes entries whose column-family names fully match at least one Java regular expression.

      An empty collection disables this filter. It is mutually exclusive with familyIncludeList(Collection).

      Parameters:
      patterns - the family-name patterns
      Returns:
      this builder

    • qualifierIncludeList

      public BigtableChangeStreamSourceBuilder<T> qualifierIncludeList(Collection<String> patterns)
      Includes qualified columns that fully match at least one Java regular expression.

      The matched identifier is family:qualifierBase64, where the qualifier is canonical padded RFC 4648 standard Base64. An empty qualifier therefore produces family:. Family-delete entries have no qualifier and are governed only by the family filter. An empty collection disables this filter. It is mutually exclusive with qualifierExcludeList(Collection).

      Parameters:
      patterns - the qualified-column patterns
      Returns:
      this builder

    • qualifierExcludeList

      public BigtableChangeStreamSourceBuilder<T> qualifierExcludeList(Collection<String> patterns)
      Excludes qualified columns that fully match at least one Java regular expression.

      The matched identifier uses the same family:qualifierBase64 representation as qualifierIncludeList(Collection). Family-delete entries have no qualifier and are governed only by the family filter. An empty collection disables this filter. It is mutually exclusive with qualifierIncludeList(Collection).

      Parameters:
      patterns - the qualified-column patterns
      Returns:
      this builder

    • skipMessagesWithoutChange

      public BigtableChangeStreamSourceBuilder<T> skipMessagesWithoutChange(boolean skip)
      Skips a mutation when entry filtering removes every entry it reported.

      The default is false, which delivers the mutation with an empty entry list so that the atomic row mutation remains visible.

      Parameters:
      skip - whether to skip fully filtered mutations
      Returns:
      this builder

    • build

      public BigtableChangeStreamSource<T> build()
      Builds the source.
      Returns:
      the source
      Throws:
      IllegalStateException - if a required option was not set, or if both sides of a mutually exclusive filter pair were set