Package io.github.flink.gcp.connector.bigquery.source

Class BigQuerySource

java.lang.Object
io.github.flink.gcp.connector.bigquery.source.BigQuerySource
@Public public final class BigQuerySource extends Object
A bounded source reading a BigQuery table through the Storage Read API.

The source is bounded, which is not the same as batch-only: it runs inside a STREAMING pipeline and finishes once the table has been read, which is what a dimension-table join needs.

Rows arrive as Avro and are handed to a BigQueryRowDeserializationSchema, which converts each one into zero or more non-null records through a collector. Collecting nothing skips the row. Column projection and row filtering are applied by BigQuery when the read session is created, so what they exclude is neither transferred nor billed.

A read through this API is charged for the bytes BigQuery scans to serve it, unlike the sink's FILE_LOADS write path, which is free. 


 Source<GenericRecord, ?, ?> source =
         BigQuerySource.<GenericRecord>builder()
                 .table(TableDestination.of("my-project", "my_dataset", "my_table"))
                 .deserializer(BigQueryRowDeserializationSchema.genericRecord(schema))
                 .rowRestriction("state = 'CA'")
                 .build();

 env.fromSource(source, WatermarkStrategy.noWatermarks(), "BigQuery");

  • Method Details

    • builder

      public static <T> BigQuerySourceBuilder<T> builder()
      Returns a builder. A deserializer and a destination — either a table or a query, not both — are required.
      Type Parameters:
      T - type of the records produced by the source
      Returns:
      the builder