Package io.github.flink.gcp.connector.bigquery.sink

Class BigQuerySink

java.lang.Object

io.github.flink.gcp.connector.bigquery.sink.BigQuerySink

@Public
public final class BigQuerySink
extends Object

Entry point for building a BigQuery sink.

The sink exposes a single builder-based API and dispatches, at job-graph construction time, to one of several write-method implementations, in the spirit of Apache Beam's BigQueryIO:

• WriteMethod.STORAGE_API_AT_LEAST_ONCE — BigQuery Storage Write API default stream, at-least-once semantics, supporting dynamic per-record table destinations
• WriteMethod.STORAGE_API_EXACTLY_ONCE — BigQuery Storage Write API buffered streams with two-phase commit, exactly-once semantics
• WriteMethod.FILE_LOADS — files staged on Cloud Storage followed by BigQuery load jobs, exactly-once in batch and checkpointed streaming execution

Those semantics assume the default FailureHandler.failJob() policy. Under a dropping policy configured through BigQuerySinkBuilder.failureHandler(FailureHandler), they cover every record except those handed to that handler, which are never written at all. A record the serializer skips by returning null is written nowhere either, under any policy.

Write methods that are not implemented yet are rejected by BigQuerySinkBuilder.build() with an UnsupportedOperationException.

Every built write method implements LineageVertexProvider. A fixed table reports namespace bigquery and name project.dataset.table, with its original resource components in the gcp facet; default-stream CDC uses the same identity. The last table or destination resolver setter wins. A user resolver reports an empty dataset list even if it always returns a constant. Internal streams, staging objects, temporary tables and jobs are not output datasets.

Lineage extraction reads configuration only: it performs no authentication, client creation, RPC, serialization or destination resolution. Flink 2.2 and 2.3 extract it into lineage graphs; Flink 1.20 supports direct metadata inspection but not automatic FLIP-314 listener delivery.

Example:

 Sink<MyEvent> sink =
         BigQuerySink.<MyEvent>builder()
                 .writeMethod(WriteMethod.STORAGE_API_AT_LEAST_ONCE)
                 .destinationResolver(
                         (e, ctx) ->
                                 TableDestination.of(
                                         "my-project", "my_dataset", e.tableName()))
                 .serializer(new MyEventProtoSerializer())
                 .build();
 

Method Summary

Modifier and Type	Method	Description
static <T> BigQuerySinkBuilder<T>	builder()	Creates a new BigQuerySinkBuilder.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

builder

public static <T> BigQuerySinkBuilder<T> builder()

Creates a new BigQuerySinkBuilder.

Type Parameters:

T - type of the records written by the sink

Returns:

a new builder