Package io.github.flink.gcp.connector.pubsub.deadletter

Class PubSubDeadLetterQueue

java.lang.Object
io.github.flink.gcp.connector.pubsub.deadletter.PubSubDeadLetterQueue
All Implemented Interfaces:
DeadLetterQueue, Serializable
@Experimental public final class PubSubDeadLetterQueue extends Object implements DeadLetterQueue
A DeadLetterQueue publishing every terminally failed element to a Pub/Sub topic, used through FailureHandler.sendToDeadLetterQueue(DeadLetterQueue).

It sees failures through the shared FailedElement contract, so one instance serves every connector in this repository — a BigQuery or Cloud Tasks job dead-letters to a topic by adding the flink-connector-gcp-pubsub artifact as a dependency. It does not go through the Pub/Sub sink: it owns an SDK Publisher of its own, so a job that dead-letters is not also a Pub/Sub sink job. 


 BigQuerySink.<Order>builder()
         .table(TableDestination.of("my-project", "my_dataset", "orders"))
         .serializer(serializer)
         .failureHandler(
                 FailureHandler.sendToDeadLetterQueue(
                         PubSubDeadLetterQueue.builder()
                                 .topic(
                                         TopicDestination.of(
                                                 "my-project", "dead-letters"))
                                 .build()))
         .build();

The envelope

The message data is the element's payload bytes, or empty when that method returns null. A concrete failure may also supply an intentionally empty payload, so consumers must use the attributes rather than data length alone to classify the failure. Every message carries five attributes:

Attributes of a dead-lettered message
AttributeValue
dlq-connectorbigquery, bigtable, cloudtasks, pubsub or spanner
dlq-destinationthe resource the element was bound for, or a connector-defined sentinel such as unresolved
dlq-errorthe failure description, truncated to Pub/Sub's 1024-byte attribute-value limit
dlq-timestampwhen the element was offered, ISO-8601
dlq-subtaskthe offering sink subtask's index

The failure's cause chain is not in the envelope: it has no bounded string form. Enable DEBUG logging on this class to see each element's untruncated error in the job logs.

An element whose payload is close to Pub/Sub's 10 MB message limit may not fit once the attributes are added — an oversized element is exactly the kind a sink rejects, so this is the expected shape of that case. The publish then fails and fails the job, since a dead letter that cannot be delivered must not be silently lost.

Buffering

Publishes are batched by the SDK and awaited in flush(), so the usual case — a rare failure among healthy records — costs no round trip per element. Because a systematic failure (a serializer bug rejecting every record) would otherwise let one whole checkpoint interval's publishes accumulate unawaited, the in-flight count is bounded: at PubSubDeadLetterQueue.Builder.maxInFlightMessages(int) the queue awaits what it holds before accepting more.

Both of those waits — the one in flush(), which runs at every checkpoint barrier, and the one the in-flight bound triggers inside offer(FailedElement) — are bounded by PubSubDeadLetterQueue.Builder.flushTimeout(Duration): one deadline covering all of that wait's publishes, not one per publish, and not what a whole checkpoint interval spends. Expiry fails the job and drops nothing. The queue's close waits for the same publishes under a budget of its own, PubSubDeadLetterQueue.Builder.shutdownTimeout(Duration).

Metrics

open(FailureHandlerContext) registers five names on the metric group the context carries — the host sink writer's, so a BigQuery job dead-lettering to a topic reports them beside BigQuery's own. They are deadLettersPublished, inFlightDeadLetters, deadLetterFlushMillis, longestDeadLetterFlushMillis and deadLetterPublisherShutdownsAbandoned; what each means is on this connector's documentation page, under "Dead-letter metrics". The count of elements offered is not among them because every sink here already reports it as numRecordsSendErrors on that same group.

Credentials

Production publishers use application-default credentials unless PubSubDeadLetterQueue.Builder.serviceAccountKeyFile(String) selects a service-account JSON key. The configured path crosses Flink serialization, and each host sink writer reads the file when it opens the queue. Parsed credentials are never stored in the job graph. The queue does not inherit the host connector's credential setting, so the dead-letter publisher may use a separate identity.

Instances are configured on the job graph and serialized to the tasks; the publisher itself is created in open(FailureHandlerContext). Lifecycle, and the at-least-once guarantee that comes with it, are the DeadLetterQueue contract's.

See Also: