Package io.github.flink.gcp.connector.bigquery.source

Class BigQuerySource

java.lang.Object
io.github.flink.gcp.connector.bigquery.source.BigQuerySource
@Public public final class BigQuerySource extends Object
A bounded source reading a BigQuery table through the Storage Read API.

The source is bounded, which is not the same as batch-only: it runs inside a STREAMING pipeline and finishes once the table has been read, which is what a dimension-table join needs.

Rows arrive as Avro and are handed to a BigQueryRowDeserializationSchema, which converts each one into zero or more non-null records through a collector. Collecting nothing skips the row. Column projection and row filtering are applied by BigQuery when the read session is created, so what they exclude is neither transferred nor billed.

The built source implements LineageVertexProvider. A configured table reports namespace bigquery and name project.dataset.table, with its original resource components in the gcp facet. View materialization reports the explicitly named input, without discovering its dependencies or reporting the materialized result. An arbitrary query reports no physical datasets, regardless of query-result placement. The source vertex retains its boundedness.

Lineage extraction reads configuration only: it performs no authentication, client creation, RPC, deserialization or resource discovery. Flink 2.2 and 2.3 extract it into lineage graphs; Flink 1.20 supports direct metadata inspection but not automatic FLIP-314 listener delivery.

A read through this API is charged for the bytes BigQuery scans to serve it, unlike the sink's FILE_LOADS write path, which is free. 


 Source<GenericRecord, ?, ?> source =
         BigQuerySource.<GenericRecord>builder()
                 .table(TableDestination.of("my-project", "my_dataset", "my_table"))
                 .deserializer(BigQueryRowDeserializationSchema.genericRecord(schema))
                 .rowRestriction("state = 'CA'")
                 .build();

 env.fromSource(source, WatermarkStrategy.noWatermarks(), "BigQuery");

  • Method Details

    • builder

      public static <T> BigQuerySourceBuilder<T> builder()
      Returns a builder. A deserializer and a destination — either a table or a query, not both — are required.
      Type Parameters:
      T - type of the records produced by the source
      Returns:
      the builder