Package io.github.flink.gcp.connector.pubsub.source

Class PubSubSourceBuilder<T>

java.lang.Object
io.github.flink.gcp.connector.pubsub.source.PubSubSourceBuilder<T>
Type Parameters:
T - type of the records produced by the source
@Public public class PubSubSourceBuilder<T> extends Object
Builder for Pub/Sub sources, obtained from PubSubSource.builder().

Required settings: at least one subscription and a deserialization schema.

  • Method Details

    • subscription

      public PubSubSourceBuilder<T> subscription(SubscriptionDestination subscription)
      Adds a subscription to consume. Calling this several times, or combining it with subscriptions(SubscriptionDestination...), consumes every added subscription in one source.

      The subscription must already exist. Use subscription(SubscriptionDestination, SubscriptionCreateOptions) to have the source create it when it does not.

      Parameters:
      subscription - the subscription
      Returns:
      this builder

    • subscription

      public PubSubSourceBuilder<T> subscription(SubscriptionDestination subscription, SubscriptionCreateOptions createOptions)
      Adds a subscription to consume, creating it with the given settings if it does not exist.

      Passing options is what authorises creating this subscription; a subscription added without them must already exist. The options are per subscription because they carry the topic binding, and two subscriptions of one topic each receive a complete copy of its stream — so sharing one options object would silently duplicate every message.

      An existing subscription is left exactly as it is: these settings are not applied to it.

      Parameters:
      subscription - the subscription
      createOptions - the settings to create it with if it is absent
      Returns:
      this builder

    • subscriptions

      public PubSubSourceBuilder<T> subscriptions(SubscriptionDestination... subscriptions)
      Adds subscriptions to consume.
      Parameters:
      subscriptions - the subscriptions
      Returns:
      this builder

    • subscriptions

      public PubSubSourceBuilder<T> subscriptions(Collection<SubscriptionDestination> subscriptions)
      Adds subscriptions to consume.
      Parameters:
      subscriptions - the subscriptions
      Returns:
      this builder

    • deserializer

      public PubSubSourceBuilder<T> deserializer(PubSubDeserializationSchema<T> deserializer)
      Sets the record deserializer. Required; one message may produce any number of records, including none, which drops the message while still acknowledging it with the checkpoint that covers it.
      Parameters:
      deserializer - the deserialization schema
      Returns:
      this builder

    • orderingMode

      public PubSubSourceBuilder<T> orderingMode(OrderingMode orderingMode)
      Sets whether the source preserves ordering-key delivery order. Defaults to OrderingMode.NONE.

      OrderingMode.PER_KEY assigns each subscription to exactly one reader subtask, so source parallelism beyond the subscription count leaves subtasks idle. See the enum for the full guarantee and its cost.

      Parameters:
      orderingMode - the ordering mode
      Returns:
      this builder

    • subscriberOptions

      public PubSubSourceBuilder<T> subscriberOptions(PubSubSubscriberOptions subscriberOptions)
      Sets the subscriber tuning options: SDK flow control, the streaming-pull connection count and the acknowledgement-deadline extension settings, plus the source's drain size, subscriber shutdown budget and first-checkpoint watchdog. Optional; every knob left unset keeps the SDK's (or the source's) default.
      Parameters:
      subscriberOptions - the subscriber options
      Returns:
      this builder

    • deserializationFailurePolicy

      public PubSubSourceBuilder<T> deserializationFailurePolicy(DeserializationFailurePolicy deserializationFailurePolicy)
      Sets what the source does with a message the deserialization schema cannot convert. Defaults to DeserializationFailurePolicy.FAIL.
      Parameters:
      deserializationFailurePolicy - the failure policy
      Returns:
      this builder

    • startPosition

      public PubSubSourceBuilder<T> startPosition(PubSubStartPosition startPosition)
      Sets where the source starts consuming. Defaults to PubSubStartPosition.continueFromSubscription(), which starts wherever the subscriptions already are.

      Every other position seeks, which rewrites state shared by every consumer of the subscription — including other jobs. The seek runs once, at the first start of a job, and never on a restore. See PubSubStartPosition for the full semantics.

      Parameters:
      startPosition - where to start consuming
      Returns:
      this builder

    • serviceAccountKeyFile

      public PubSubSourceBuilder<T> serviceAccountKeyFile(String serviceAccountKeyFile)
      Authenticates the source with the service-account JSON key at the given path instead of application-default credentials. The file is read on the JobManager for subscription administration and on each TaskManager that creates a reader, so the same path must be readable by every JobManager and TaskManager that can run this source. Optional; when unset the source uses application-default credentials.

      Service-account keys are long-lived secrets. Prefer an attached service account or Workload Identity where the deployment supports one. This setting cannot be combined with emulatorEndpoint(String), whose plaintext channel deliberately carries no credentials.

      Parameters:
      serviceAccountKeyFile - the service-account JSON key-file path
      Returns:
      this builder

    • emulatorEndpoint

      public PubSubSourceBuilder<T> emulatorEndpoint(String emulatorEndpoint)
      Points the source at a Pub/Sub emulator instead of the production service. Subscribers connect to the given host:port over a plaintext channel with no credentials, so this must only ever be used against an emulator (for example a testcontainers PubSubEmulatorContainer). Optional; when unset the source connects to Pub/Sub with application-default credentials.

      The value is parsed here, so a malformed host:port is rejected on the client instead of surfacing as a connection failure once the job has been deployed.

      Parameters:
      emulatorEndpoint - the emulator endpoint as host:port
      Returns:
      this builder
      Throws:
      IllegalArgumentException - if the endpoint is not host:port with a port in 1..65535

    • build

      public org.apache.flink.api.connector.source.Source<T,SubscriptionSplit,PubSubEnumeratorState> build()
      Builds the source.

      The returned source implements LineageVertexProvider. It reports each configured subscription as pubsub / subscription:project:subscription, with the source's unboundedness and a gcp physical-resource facet. It does not look up backing topics or infer them from subscription-creation settings. Inspection performs no authentication, client creation, RPC, or deserialization.

      Flink 2.2 and 2.3 extract this metadata natively. Flink 1.20 supports direct inspection through the interface, without automatic FLIP-314 listener delivery.

      Returns:
      the source