Quickstart on GCP Connectors for Apache Flink

BigQuery

Mon, 01 Jan 0001 00:00:00 +0000

Write to and read from BigQuery#

Assumes the artifacts and credentials from the Quickstart index. The whole file, since it is the one worth copying verbatim; the other connectors’ pages show only the job.

Write a stream to a table#


package example;

import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

import com.google.cloud.bigquery.Field;
import com.google.cloud.bigquery.Schema;
import com.google.cloud.bigquery.StandardSQLTypeName;
import io.github.flink.gcp.connector.bigquery.sink.BigQuerySink;
import io.github.flink.gcp.connector.bigquery.sink.TableDestination;
import io.github.flink.gcp.connector.bigquery.sink.serializer.json.JsonDocumentSerializationSchema;

public class BigQueryQuickstartWrite {

 public static void main(String[] args) throws Exception {
 // JSON carries no schema, so this one is supplied rather than derived. Serializers for
 // input that does carry a schema — protobuf messages and Avro records — derive it.
 Schema schema =
 Schema.of(
 Field.of("order_id", StandardSQLTypeName.STRING),
 Field.of("amount", StandardSQLTypeName.INT64));

 StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
 env.setRuntimeMode(RuntimeExecutionMode.STREAMING);
 // Not optional. Every sink here is at-least-once *only* with checkpointing: the checkpoint
 // is what makes Flink flush what the Google client libraries are still holding, so without
 // it those records are lost on failure.
 env.enableCheckpointing(60_000);

 env.fromData("{\"order_id\":\"a-1\",\"amount\":10}", "{\"order_id\":\"a-2\",\"amount\":20}")
 .sinkTo(
 BigQuerySink.<String>builder()
 .table(TableDestination.of("my-project", "my_dataset", "orders"))
 .serializer(JsonDocumentSerializationSchema.of(schema))
 .build());

 env.execute("bigquery-quickstart");
 }
}

The dataset must exist; the table need not, because the default create disposition is CREATE_IF_NEEDED and the schema above is what it is created from. The default write method is STORAGE_API_AT_LEAST_ONCE, which makes rows queryable within seconds — the other two, and when to reach for them, are under exactly-once.

Cloud Pub/Sub

Mon, 01 Jan 0001 00:00:00 +0000

Publish to and read from Cloud Pub/Sub#

Assumes the artifacts and credentials from the Quickstart index, and the imports an IDE resolves from the Java API reference.

Publish a stream to a topic#

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setRuntimeMode(RuntimeExecutionMode.STREAMING);
env.enableCheckpointing(60_000);

env.fromData("hello", "world")
 .sinkTo(
 PubSubSink.<String>builder()
 .topic(TopicDestination.of("my-project", "orders"))
 .serializer(
 PubSubSerializationSchema.payload(new SimpleStringSchema()))
 .build());

env.execute("pubsub-sink-quickstart");

Checkpointing is not decoration: the sink is at-least-once only with it, since the checkpoint is what makes Flink flush the messages the SDK publishers are still batching.

Cloud Tasks

Mon, 01 Jan 0001 00:00:00 +0000

Dispatch a stream as Cloud Tasks#

Assumes the artifacts and credentials from the Quickstart index, and the imports an IDE resolves from the Java API reference.

Create the queue first. The sink will not create one, and that is deliberate: the queue’s rate limits are the entire reason to use the service, and a queue created with defaults would carry Cloud Tasks’ own (500 dispatches/second, 1000 concurrent) rather than the pacing the target endpoint can absorb.

Bigtable

Mon, 01 Jan 0001 00:00:00 +0000

Write to and read from a Bigtable table#

Assumes the artifacts and credentials from the Quickstart index, and the imports an IDE resolves from the Java API reference.

The examples use application-default credentials. If a deployment cannot supply the intended identity through ADC, add serviceAccountKeyFile("/mounted/path/key.json") to either builder. The path is read when the job’s runtime components start and must be mounted at the same absolute path on every eligible TaskManager and, for either source, the JobManager. Prefer an attached service account or Workload Identity over a long-lived key; the operational requirements are in Credential file deployment.

Spanner

Mon, 01 Jan 0001 00:00:00 +0000

Write a stream into a Spanner table#

Assumes the artifacts and credentials from the Quickstart index, and the imports an IDE resolves from the Java API reference.

Create the instance, database and table first. The sink creates none of them — a missing table fails every record alike, so it is a configuration error rather than something a sink can repair.

gcloud spanner instances create my-instance \
 --config=regional-asia-northeast1 --description="my-instance" --nodes=1
gcloud spanner databases create orders-db --instance=my-instance \
 --ddl='CREATE TABLE Orders (OrderId STRING(64) NOT NULL, Total INT64) PRIMARY KEY (OrderId)'

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setRuntimeMode(RuntimeExecutionMode.STREAMING);
// Not optional: the sink is at-least-once only with checkpointing, which is what makes
// Flink wait for the batch to be applied before the barrier passes.
env.enableCheckpointing(60_000);

env.fromData("a-1", "a-2")
 .sinkTo(
 SpannerSink.<String>builder()
 .database(
 DatabaseDestination.of(
 "my-project", "my-instance", "orders-db"))
 .serializer(
 (element, context) ->
 // insertOrUpdate, not insert: the sink is
 // at-least-once and Spanner's batch write has no
 // replay protection, so a record can arrive twice.
 // An upsert makes that a no-op; an insert makes it
 // a routed failure.
 Mutation.newInsertOrUpdateBuilder("Orders")
 .set("OrderId")
 .to("order#" + element)
 .set("Total")
 .to(element.length())
 .build())
 .build());

env.execute("spanner-quickstart");

Read the rows back: