Package io.github.flink.gcp.connector.base.failure

Interface DeadLetterQueue

All Superinterfaces:
Serializable
All Known Implementing Classes:
PubSubDeadLetterQueue
@Experimental public interface DeadLetterQueue extends Serializable
Destination for elements that terminally failed to be written by a sink, used through FailureHandler.sendToDeadLetterQueue(DeadLetterQueue). One implementation serves every connector: it sees failures through the shared FailedElement contract.

Lifecycle. The sendToDeadLetterQueue handler drives the queue from the sink writer's own lifecycle, on the Flink task thread only, so implementations need not be thread-safe:

  • open(FailureHandlerContext) is called once, before the first offer(io.github.flink.gcp.connector.base.failure.FailedElement), when the sink writer is created (including after a restore). The context carries the writer's metric group and subtask index.
  • offer(FailedElement) accepts one terminally failed element. Implementations may buffer; they need not write durably here.
  • flush() is called from the sink writer's own flush — at every checkpoint barrier and at end of input (and at any additional sink-triggered flush, such as an optional periodic flush interval). When it returns, every element offered so far must be durably persisted; throwing fails the ongoing checkpoint and thereby the job.
  • close() is called when the sink writer closes, on success and failure paths alike. It releases resources and must not be relied on for persistence: on the failure path, elements offered since the last completed checkpoint may be lost with the writer — their originating records are replayed from the last checkpoint and offered again.

Delivery guarantee: at-least-once, for failures that recur on replay. Elements are offered before the checkpoint covering their originating records completes, so a restart replays those records and a deterministic failure (malformed data, an oversized payload) is offered again — dead-letter output can therefore contain duplicates and should be consumed idempotently or deduplicated by key. A failure that does not recur on replay is preserved only if it was already flushed by a completed checkpoint or written through synchronously; an implementation that writes through on every offer narrows the window at the cost of per-element latency. Exactly-once dead-letter output is deliberately not offered: it would require the queue write to join the sink's own commit protocol, which none of these services can enroll an external write in.

  • Method Summary

    Modifier and Type
    Method
    Description
    default void
    close()
    Releases resources held by the queue when the sink writer closes.
    default void
    flush()
    Durably persists every element offered so far; called at every checkpoint barrier and at end of input (and at any additional sink-triggered flush, such as an optional periodic flush interval).
    void
    offer(FailedElement element)
    Accepts one terminally failed element; implementations may buffer until flush().
    default void
    open(FailureHandlerContext context)
    Called once, before the first offer(io.github.flink.gcp.connector.base.failure.FailedElement), when the sink writer is created.

  • Method Details

    • offer

      void offer(FailedElement element) throws IOException
      Accepts one terminally failed element; implementations may buffer until flush().
      Parameters:
      element - the failed element
      Throws:
      IOException - if the element cannot be accepted; this fails the job

    • open

      default void open(FailureHandlerContext context) throws IOException
      Called once, before the first offer(io.github.flink.gcp.connector.base.failure.FailedElement), when the sink writer is created.
      Parameters:
      context - the writer's subtask index and metric group
      Throws:
      IOException - if the queue cannot be opened; this fails the job

    • flush

      default void flush() throws IOException
      Durably persists every element offered so far; called at every checkpoint barrier and at end of input (and at any additional sink-triggered flush, such as an optional periodic flush interval).
      Throws:
      IOException - if persistence fails; this fails the ongoing checkpoint

    • close

      default void close() throws Exception
      Releases resources held by the queue when the sink writer closes.
      Throws:
      Exception