Package io.github.flink.gcp.connector.base.failure

Interface FailureHandler<F extends FailedElement>

Type Parameters:
F - the connector's concrete failure type
All Superinterfaces:
Serializable
Functional Interface:
This is a functional interface and can therefore be used as the assignment target for a lambda expression or method reference.
@Public @FunctionalInterface public interface FailureHandler<F extends FailedElement> extends Serializable
Pluggable policy for elements that terminally fail to be written by a sink. Only data-shaped, per-element failures reach the handler (each connector's documentation lists its set — for BigQuery, explicit record-specific routing failures, rows rejected by the Storage Write API with per-row error details, rows that fail serialization, and rows exceeding the per-row size limit). Transient failures are retried by the sinks without involving the handler, and terminal request-level failures such as INVALID_ARGUMENT always fail the job. ( PERMISSION_DENIED was the example here until BigQuery was measured to answer it for a table that is merely missing, which its sink recovers from — a reminder that which codes are terminal is a per-connector fact, not a general one.)

The sink writer drives the handler on the Flink task thread only, so implementations need not be thread-safe:

  • open(FailureHandlerContext) is called once, before the first handle(F), when the sink writer is created (including after a restore);
  • handle(FailedElement) accepts one terminally failed element — returning normally drops it, throwing fails the ongoing write or checkpoint (the built-in failJob() policy, the default, does exactly that);
  • flush() is called from the sink writer's own flush — at every checkpoint barrier and at end of input (and at any additional sink-triggered flush, such as an optional periodic flush interval), after the write path has drained. When it returns, every element handled so far must be durably persisted (a handler that persists nothing simply returns); throwing fails the ongoing checkpoint;
  • close() is called when the sink writer closes, on success and failure paths alike. It releases resources and must not be relied on for persistence: on the failure path, elements handled since the last completed checkpoint may be lost with the writer — their originating records are replayed from the last checkpoint and handled again.

Delivery of handled elements to an external destination is therefore at-least-once, for failures that recur on replay — see DeadLetterQueue for the full statement.

What a successful checkpoint means under each policy, stated once for every connector: under the default failJob(), every record up to the barrier was written to the service or skipped by a serializer returning null; under logAndDrop() or sendToDeadLetterQueue(io.github.flink.gcp.connector.base.failure.DeadLetterQueue), written, skipped, or handed to this handler. Each writer's javadoc says which failures reach it.

That handle(F) drops by returning and fails the job by throwing is why the failure classes a connector does not route matter as much as the ones it does: an outage the client's retries gave up on is never routed, so no drop policy can quietly discard a backlog. And in the mailbox-based writers a handler failing inside a completion callback cannot throw at its caller — the writer captures the failure as its asynchronous error and rethrows it on the task thread from the next write or flush.

  • Method Summary

    Modifier and Type
    Method
    Description
    default void
    close()
    Releases resources held by the handler when the sink writer closes.
    static <F extends FailedElement>
    FailureHandler<F>
    failJob()
    Returns the default policy: every per-element terminal failure fails the job.
    default void
    flush()
    Persists every element handled so far; called at every checkpoint barrier and at end of input (and at any additional sink-triggered flush, such as an optional periodic flush interval), after the sink's own write path has drained.
    void
    handle(F element)
    Handles one terminally failed element.
    static <F extends FailedElement>
    FailureHandler<F>
    logAndDrop()
    Returns the policy that logs each failed element at WARN level and drops it, letting the pipeline continue.
    default void
    open(FailureHandlerContext context)
    Called once, before the first handle(F), when the sink writer is created.
    static <F extends FailedElement>
    FailureHandler<F>
    sendToDeadLetterQueue(DeadLetterQueue deadLetterQueue)
    Returns a policy that routes each failed element to the given dead-letter queue and drives the queue's lifecycle (open/flush/close) from the handler's own; a failure of the queue itself fails the job.

  • Method Details

    • handle

      void handle(F element) throws IOException
      Handles one terminally failed element. Returning normally drops the element; throwing fails the ongoing write or checkpoint.
      Parameters:
      element - the failed element
      Throws:
      IOException - to fail the job instead of dropping the element

    • open

      default void open(FailureHandlerContext context) throws IOException
      Called once, before the first handle(F), when the sink writer is created.
      Parameters:
      context - the writer's subtask index and metric group
      Throws:
      IOException - if the handler cannot be opened; this fails the job

    • flush

      default void flush() throws IOException
      Persists every element handled so far; called at every checkpoint barrier and at end of input (and at any additional sink-triggered flush, such as an optional periodic flush interval), after the sink's own write path has drained.
      Throws:
      IOException - if persistence fails; this fails the ongoing checkpoint

    • close

      default void close() throws Exception
      Releases resources held by the handler when the sink writer closes.
      Throws:
      Exception

    • failJob

      static <F extends FailedElement> FailureHandler<F> failJob()
      Returns the default policy: every per-element terminal failure fails the job.
      Type Parameters:
      F - the connector's concrete failure type
      Returns:
      the fail-job handler

    • logAndDrop

      static <F extends FailedElement> FailureHandler<F> logAndDrop()
      Returns the policy that logs each failed element at WARN level and drops it, letting the pipeline continue.
      Type Parameters:
      F - the connector's concrete failure type
      Returns:
      the log-and-drop handler

    • sendToDeadLetterQueue

      static <F extends FailedElement> FailureHandler<F> sendToDeadLetterQueue(DeadLetterQueue deadLetterQueue)
      Returns a policy that routes each failed element to the given dead-letter queue and drives the queue's lifecycle (open/flush/close) from the handler's own; a failure of the queue itself fails the job.
      Type Parameters:
      F - the connector's concrete failure type
      Parameters:
      deadLetterQueue - the dead-letter queue
      Returns:
      the dead-letter-queue handler