Package io.github.flink.gcp.connector.bigtable.sink

Class BigtableSinkBuilder<T>

java.lang.Object
io.github.flink.gcp.connector.bigtable.sink.BigtableSinkBuilder<T>
Type Parameters:
T - type of the records written by the sink
@Public public class BigtableSinkBuilder<T> extends Object
Builder for Bigtable sinks, obtained from BigtableSink.builder().

Required settings: a destination — table(TableDestination) for one fixed table, or destinationResolver(DestinationResolver) to route per record — and a serialization schema.

By default the sink creates no table: every table it writes to, and the column families the mutations name, must exist. createDisposition(CreateDisposition) with CreateDisposition.CREATE_IF_NEEDED and tableCreateOptions(TableCreateOptions) opts into creating them, from one schema that serves every table the sink creates.

  • Method Details

    • table

      public BigtableSinkBuilder<T> table(TableDestination table)
      Writes every mutation to the given table. Sugar for a DestinationResolver returning that table for every record; this and destinationResolver(DestinationResolver) set the same field, so the last call wins.
      Parameters:
      table - the destination table
      Returns:
      this builder

    • destinationResolver

      public BigtableSinkBuilder<T> destinationResolver(DestinationResolver<? super T> destinationResolver)
      Resolves the destination table per record, so one sink writes to many tables. The resolver runs before the serializer, and its result is what a failed mutation is reported against.

      This and table(TableDestination) set the same field, so the last call wins.

      Each distinct table costs a bulk mutation batcher of its own, and beside CreateDisposition.CREATE_IF_NEEDED each unseen table is created from the one tableCreateOptions(TableCreateOptions) schema — so a resolver's cardinality decides what the sink holds, and what it may create.

      Parameters:
      destinationResolver - the resolver
      Returns:
      this builder

    • serializer

      public BigtableSinkBuilder<T> serializer(BigtableSerializationSchema<? super T> serializer)
      Sets the record serialization schema. Required; a schema returning null skips the record rather than failing it.
      Parameters:
      serializer - the serialization schema
      Returns:
      this builder

    • appProfileId

      public BigtableSinkBuilder<T> appProfileId(String appProfileId)
      Routes the client's requests through the given application profile, which is what selects an instance's routing policy and its priority. Optional; when unset the instance's default profile applies.

      It is a sink option rather than part of TableDestination because it chooses a path to the data, not the data's address.

      Parameters:
      appProfileId - the application profile id
      Returns:
      this builder

    • writerOptions

      public BigtableSinkBuilder<T> writerOptions(BigtableWriterOptions writerOptions)
      Sets the writer tuning options (the batch thresholds and the in-flight bounds). Optional; defaults to BigtableWriterOptions.defaults().
      Parameters:
      writerOptions - the options
      Returns:
      this builder

    • failedMutationHandler

      public BigtableSinkBuilder<T> failedMutationHandler(FailureHandler<? super FailedMutation> failedMutationHandler)
      Sets the policy for mutations that terminally fail — a record the serializer rejects, and a mutation Bigtable rejects as malformed or oversized. Defaults to FailureHandler.failJob(); transient failures never reach it, since the client retries them and an exhausted retry fails the job.
      Parameters:
      failedMutationHandler - the handler
      Returns:
      this builder

    • serviceAccountKeyFile

      public BigtableSinkBuilder<T> serviceAccountKeyFile(String serviceAccountKeyFile)
      Authenticates the sink with the service-account JSON key at the given path instead of application-default credentials. The file is read on each TaskManager when its writer is created, so every TaskManager that can run the sink must see the same path. Optional; when unset the sink uses application-default credentials.

      Service-account keys are long-lived secrets. Prefer an attached service account or Workload Identity where the deployment supports one. This setting cannot be combined with emulatorEndpoint(String), whose plaintext channel carries no credentials.

      Parameters:
      serviceAccountKeyFile - the service-account JSON key-file path
      Returns:
      this builder

    • emulatorEndpoint

      public BigtableSinkBuilder<T> emulatorEndpoint(String emulatorEndpoint)
      Points the sink at a Bigtable emulator instead of the production service. The connection to the given host:port uses a plaintext channel with no credentials, so this must only ever be used against an emulator. Optional; when unset the sink connects to Bigtable with application-default credentials.

      The value is parsed here, so a malformed host:port is rejected on the client instead of surfacing as a connection failure once the job has been deployed.

      Parameters:
      emulatorEndpoint - the emulator endpoint as host:port
      Returns:
      this builder
      Throws:
      IllegalArgumentException - if the endpoint is not host:port with a port in 1..65535

    • createDisposition

      public BigtableSinkBuilder<T> createDisposition(CreateDisposition createDisposition)
      Sets whether the sink may create the destination table when a mutation finds it — or one of its column families — missing. Defaults to CreateDisposition.CREATE_NEVER, under which the table and its families must exist.

      CreateDisposition.CREATE_IF_NEEDED requires tableCreateOptions(TableCreateOptions): a Bigtable table's schema is its column families and their garbage-collection policies, which the sink cannot guess.

      Parameters:
      createDisposition - the disposition
      Returns:
      this builder

    • tableCreateOptions

      public BigtableSinkBuilder<T> tableCreateOptions(TableCreateOptions tableCreateOptions)
      Sets the column families — and, per family, an optional garbage-collection rule — for the table the sink creates under CreateDisposition.CREATE_IF_NEEDED. Creation only: an existing table is used as it is, except that families declared here which it lacks are added.
      Parameters:
      tableCreateOptions - the creation settings
      Returns:
      this builder

    • build

      public org.apache.flink.api.connector.sink2.Sink<T> build()
      Builds the sink.
      Returns:
      the sink
      Throws:
      IllegalStateException - if a required option was not set, if the create disposition and the table-creation options disagree, or if a service-account key file was combined with an emulator endpoint