Package io.github.flink.gcp.connector.pubsub.deadletter

Class PubSubDeadLetterQueue.Builder

java.lang.Object
io.github.flink.gcp.connector.pubsub.deadletter.PubSubDeadLetterQueue.Builder
Enclosing class:
PubSubDeadLetterQueue
@Experimental public static final class PubSubDeadLetterQueue.Builder extends Object
Builder for PubSubDeadLetterQueue.

  • Method Details

    • topic

      public PubSubDeadLetterQueue.Builder topic(TopicDestination topic)
      Sets the topic every dead letter is published to. Required; the topic must exist, since this queue never creates one — a dead-letter destination created on the fly is one nothing is consuming.
      Parameters:
      topic - the dead-letter topic
      Returns:
      this builder

    • serviceAccountKeyFile

      public PubSubDeadLetterQueue.Builder serviceAccountKeyFile(String serviceAccountKeyFile)
      Authenticates the queue with the service-account JSON key at the given path instead of application-default credentials. The file is read when a host sink writer opens this queue, so the same path must be readable by every TaskManager that can run that sink. Optional; when unset the queue uses application-default credentials.

      The queue does not inherit credentials from the host connector. A dead-letter topic may intentionally use a different identity, and the shared failure-handler contract does not carry the host connector's credential configuration.

      Service-account keys are long-lived secrets. Prefer an attached service account or Workload Identity where the deployment supports one. This setting cannot be combined with emulatorEndpoint(String), whose plaintext channel deliberately carries no credentials.

      Parameters:
      serviceAccountKeyFile - the service-account JSON key-file path
      Returns:
      this builder

    • emulatorEndpoint

      public PubSubDeadLetterQueue.Builder emulatorEndpoint(String emulatorEndpoint)
      Points the queue at a Pub/Sub emulator instead of the production service. The connection to the given host:port uses a plaintext channel with no credentials, so this must only ever be used against an emulator. Optional; when unset the queue publishes with application-default credentials.

      The value is parsed here, so a malformed host:port is rejected on the client instead of surfacing as a connection failure once the job has been deployed.

      Parameters:
      emulatorEndpoint - the emulator endpoint as host:port
      Returns:
      this builder
      Throws:
      IllegalArgumentException - if the endpoint is not host:port with a port in 1..65535

    • maxInFlightMessages

      public PubSubDeadLetterQueue.Builder maxInFlightMessages(int maxInFlightMessages)
      Sets how many publishes may be in flight before PubSubDeadLetterQueue.offer(io.github.flink.gcp.connector.base.failure.FailedElement) awaits them, bounding what one checkpoint interval accumulates when every record fails. Defaults to 1000.

      PubSubDeadLetterQueue.WRITE_THROUGH (0) publishes each element synchronously — the narrowest loss window the DeadLetterQueue contract describes, at one round trip per element. PubSubDeadLetterQueue.UNBOUNDED (-1) buffers everything until PubSubDeadLetterQueue.flush(), which is the fastest and the only setting whose memory is not bounded by this queue.

      Parameters:
      maxInFlightMessages - the bound, PubSubDeadLetterQueue.WRITE_THROUGH or PubSubDeadLetterQueue.UNBOUNDED
      Returns:
      this builder

    • shutdownTimeout

      public PubSubDeadLetterQueue.Builder shutdownTimeout(Duration shutdownTimeout)
      Sets how long PubSubDeadLetterQueue.close() waits for the dead-letter publisher to shut down. Defaults to 30 seconds.

      This is a budget of its own, spent after and on top of the sink's own shutdownTimeout: a sink that dead-letters closes its publishers first and this queue last. Keep the sum under Flink's task.cancellation.timeout (180 s by default), past which a cancelling task is a fatal TaskManager error.

      Parameters:
      shutdownTimeout - the shutdown budget, positive
      Returns:
      this builder

    • flushTimeout

      public PubSubDeadLetterQueue.Builder flushTimeout(Duration flushTimeout)
      Sets how long the queue waits for its buffered publishes, in PubSubDeadLetterQueue.flush() and in the maxInFlightMessages(int) drain alike. Defaults to 60 seconds.

      flush() runs at every checkpoint barrier, so without a budget a wait lasts as long as the SDK keeps retrying — 600 seconds by default, which is also Flink's default execution.checkpointing.timeout. It is one deadline per wait, covering all of that wait's publishes rather than each of them.

      It bounds one wait, not what a checkpoint interval spends. How many waits an interval makes is maxInFlightMessages(int): one under PubSubDeadLetterQueue.UNBOUNDED, one per bound-full at a positive value, and one per element under PubSubDeadLetterQueue.WRITE_THROUGH. A slow-but-working topic can therefore spend several budgets in an interval without any of them expiring.

      On expiry the wait throws — failing the ongoing checkpoint from flush(), and the task itself from an offer, where no checkpoint is in progress. The queue drops nothing, and since the publishes are not cancelled the SDK may still deliver them — a duplicate, which is what the DeadLetterQueue guarantee already asks a consumer to expect. A disturbance longer than the budget therefore fails the job where the SDK's retry would have absorbed it, which is the trade a bound buys. There is deliberately no unbounded setting; a Duration longer than any disturbance worth surviving says the same thing without making waiting forever a mode.

      Parameters:
      flushTimeout - the wait budget, positive
      Returns:
      this builder

    • build

      public PubSubDeadLetterQueue build()
      Builds the queue.
      Returns:
      the queue