Package io.github.flink.gcp.connector.pubsub.sink

Class PubSubSinkBuilder<T>

java.lang.Object
io.github.flink.gcp.connector.pubsub.sink.PubSubSinkBuilder<T>
Type Parameters:
T - type of the records written by the sink
@Public public class PubSubSinkBuilder<T> extends Object
Builder for Pub/Sub sinks, obtained from PubSubSink.builder().

Required settings: a serialization schema and a destination. The destination is set through either topic(TopicDestination) (fixed topic) or destinationResolver(DestinationResolver) (per-record dynamic destinations); the two override each other and the last call wins.

  • Method Details

    • topic

      public PubSubSinkBuilder<T> topic(TopicDestination topic)
      Publishes every record to the given fixed topic. Overrides any previously set topic or resolver.
      Parameters:
      topic - the destination topic
      Returns:
      this builder

    • destinationResolver

      public PubSubSinkBuilder<T> destinationResolver(DestinationResolver<? super T> destinationResolver)
      Resolves the destination topic per record (dynamic destinations). Overrides any previously set topic or resolver.
      Parameters:
      destinationResolver - the resolver
      Returns:
      this builder

    • serializer

      public PubSubSinkBuilder<T> serializer(PubSubSerializationSchema<? super T> serializer)
      Sets the record serialization schema. Required; a schema returning null skips the record rather than failing it.
      Parameters:
      serializer - the serialization schema
      Returns:
      this builder

    • createDisposition

      public PubSubSinkBuilder<T> createDisposition(CreateDisposition createDisposition)
      Sets whether the sink may create destination topics that do not exist. Defaults to CreateDisposition.CREATE_IF_NEEDED.
      Parameters:
      createDisposition - the disposition
      Returns:
      this builder

    • topicCreateOptions

      public PubSubSinkBuilder<T> topicCreateOptions(TopicCreateOptions topicCreateOptions)
      Sets the settings applied to topics the sink creates under CreateDisposition.CREATE_IF_NEEDED — message retention, a customer-managed encryption key and the message storage policy. Optional: without it a created topic takes every field's service default. One options object applies to every topic the sink creates, including each one a DestinationResolver resolves. Rejected together with CreateDisposition.CREATE_NEVER, which never creates a topic these settings could apply to.
      Parameters:
      topicCreateOptions - the creation settings
      Returns:
      this builder

    • publisherOptions

      public PubSubSinkBuilder<T> publisherOptions(PubSubPublisherOptions publisherOptions)
      Sets the publisher and writer tuning options (batching, publish retries, message ordering, the in-flight caps and the topic auto-creation recovery backoff). Optional; defaults to PubSubPublisherOptions.defaults().
      Parameters:
      publisherOptions - the options
      Returns:
      this builder

    • failedMessageHandler

      public PubSubSinkBuilder<T> failedMessageHandler(FailureHandler<? super FailedMessage> failedMessageHandler)
      Sets what happens to a message that terminally fails to be published: the record could not be serialized, or Pub/Sub rejected the message itself as invalid (INVALID_ARGUMENT — over the size limit, malformed attributes, an unusable ordering key). Defaults to FailureHandler.failJob().

      Only those data-shaped failures reach the handler. A missing topic is repaired by CreateDisposition.CREATE_IF_NEEDED instead, and everything else — an outage the SDK's retries gave up on, PERMISSION_DENIED, a destination resolver that fails — keeps failing the job, so a dropping policy cannot bleed the stream during an incident.

      Returning from FailureHandler.handle(F) drops the message; throwing fails the ongoing write or checkpoint. The parameter is contravariant, so a cross-connector FailureHandler<FailedElement> is accepted as-is.

      Under PubSubPublisherOptions.builder().enableMessageOrdering(true), dropping a message that carries an ordering key leaves a gap in that key's stream which a consumer cannot tell apart from a lost message. The messages queued behind it keep their relative order — the sink resumes the key and republishes them — and the dead-letter record carries the whole serialized message, so the gap can be replayed from it.

      Parameters:
      failedMessageHandler - the handler
      Returns:
      this builder

    • serviceAccountKeyFile

      public PubSubSinkBuilder<T> serviceAccountKeyFile(String serviceAccountKeyFile)
      Authenticates the sink with the service-account JSON key at the given path instead of application-default credentials. The file is read on each TaskManager when its writer is created, so the same path must be readable by every TaskManager that can run this sink. Optional; when unset the sink uses application-default credentials.

      Service-account keys are long-lived secrets. Prefer an attached service account or Workload Identity where the deployment supports one. This setting cannot be combined with emulatorEndpoint(String), whose plaintext channel deliberately carries no credentials.

      Parameters:
      serviceAccountKeyFile - the service-account JSON key-file path
      Returns:
      this builder

    • emulatorEndpoint

      public PubSubSinkBuilder<T> emulatorEndpoint(String emulatorEndpoint)
      Points the sink at a Pub/Sub emulator instead of the production service. Connections to the given host:port — the per-topic publishers and, when topic auto-creation triggers, the admin client — use a plaintext channel with no credentials, so this must only ever be used against an emulator (for example a testcontainers PubSubEmulatorContainer). Optional; when unset the sink connects to Pub/Sub with application-default credentials.

      The value is parsed here, so a malformed host:port is rejected on the client instead of surfacing as a connection failure once the job has been deployed.

      Parameters:
      emulatorEndpoint - the emulator endpoint as host:port
      Returns:
      this builder
      Throws:
      IllegalArgumentException - if the endpoint is not host:port with a port in 1..65535

    • build

      public org.apache.flink.api.connector.sink2.Sink<T> build()
      Builds the sink.

      The returned sink implements LineageVertexProvider. A fixed topic is reported as pubsub / topic:project:topic, with a gcp physical-resource facet. A dynamic resolver contributes an empty dataset list; inspection never evaluates it or the serializer, authenticates, creates a client, or issues an RPC. Dead-letter publishing is not an additional output of this vertex.

      Flink 2.2 and 2.3 extract this metadata natively. Flink 1.20 supports direct inspection through the interface, without automatic FLIP-314 listener delivery.

      Returns:
      the sink