All Classes and Interfaces
Class
Description
Keeps a
flink-sql-connector-gcp-* module's META-INF/NOTICE honest: it must list
every artifact the uber-jar bundles, and nothing it does not.Asserts the shape of a
flink-sql-connector-gcp-* uber-jar: that SQL can discover the
connector in it, that every runtime dependency actually made it in, and that no class
escaped relocation except those that deliberately did.The precondition every
flink-sql-connector-gcp-* smoke test rests on: that the connector
it just exercised came out of the uber-jar.Settles one delivered message, hiding which of the client library's two receiver flavors produced
it.
Tracks the acknowledgement lifecycle of received Pub/Sub messages, which is what makes the source
at-least-once.
One physical BigQuery column appended after the configured serializer emits a row.
Determines how an additional physical field handles a provider returning
null.Ordered physical fields appended to every destination row and table schema.
Builder for
AdditionalFields.Supported BigQuery types for singular additional physical fields.
Extracts one additional physical-field value from the original input record.
Classifies Storage Write API append failures into the error classes the writer routes on.
Builds an immutable
AppEngineTargetSerializationSchema.Extracts App Engine request headers from a record.
Extracts the App Engine relative URI from a record.
Extracts App Engine service, version, and instance routing from a sink record.
Checks and normalizes App Engine request fields shared by all sink APIs.
A
CloudTasksSerializationSchema producing App Engine target tasks.Validated fixed App Engine request values used by the table serializer.
A
BigQueryProtoSerializationSchema for Avro records.Converts Avro records into the protobuf rows the BigQuery Storage Write API accepts, against the
schema
AvroToTableSchemaConverter derived and the descriptor
BQTableSchemaToProtoDescriptor built from it.Options controlling how Avro schemas are mapped to BigQuery schemas.
Builder for
AvroSchemaOptions.Derives a BigQuery
TableSchema from an Avro Schema.Deadline-bounded polling for test assertions.
Plans a read through a
google-cloud-spanner BatchClient.Reads partitions through a
google-cloud-spanner BatchClient.One partition of a batch read, together with the snapshot it belongs to.
Serializer for
BatchReadSplit.The mutable reader-side state of a
BatchReadSplit.The
WriteMethod.STORAGE_API_EXACTLY_ONCE sink: each writer subtask appends rows to one
application-created BUFFERED Storage Write API stream per active destination at explicit offsets
(see BigQueryBufferedStreamWriter), and the committer makes each completed checkpoint's
rows visible by flushing every committable's stream up to its offset (see BufferedStreamCommitter) — two-phase commit on Flink checkpoints.Exactly-once
SinkWriter appending rows to one application-created BUFFERED Storage Write
API stream per destination at explicit offsets.The
WITH options of the bigquery table connector.Loads credentials shared by the BigQuery and Cloud Storage client families.
At-least-once sink appending to Storage Write API default streams with dynamic per-record table
destinations (
WriteMethod.STORAGE_API_AT_LEAST_ONCE).At-least-once
SinkWriter appending to Storage Write API default streams with dynamic
per-record table destinations.The
bigquery table sink.Collects the sink's fully resolved values.
A bounded Table API source mapped onto
BigQuerySource.Collects the source's fully resolved values.
Creates the
bigquery table source or sink from a CREATE TABLE statement's
options.The BigQuery emulator image and its two endpoints, shared by every harness that starts the
emulator so they cannot drift apart.
A record-specific BigQuery sink failure handled by the configured failure policy.
The
WriteMethod.FILE_LOADS sink: writers stage per-destination files on Cloud Storage in
the configured staging format, the pre-commit
topology routes every subtask's committables to a single committer subtask (stamping their
checkpoint id in streaming, see FileLoadsCheckpointStamper), and the committer turns each
batch into BigQuery load jobs (see FileLoadsCommitter and LoadJobOrchestrator) — the whole
run at end of input in batch execution, each checkpoint's files at its completion in streaming
execution.LoadJobRunner over the BigQuery REST client with configured credentials or ADC.Thread-safe submitted-job state shared by every worker of one committer.
Every metric name this connector registers itself, in one place so that this file is the
connector's inventory: what it reports can be read here without opening a writer.
Serializes records into protobuf rows to be written to BigQuery.
Runs the source's query through the REST
BigQuery client.Storage Read API clients, for the two places the source opens one: the enumerator creates the
read session, and every reader opens its assigned streams.
The enumerator's checkpointed state: whether the read session exists, which one it is, and the
splits not currently assigned to a reader.
Serializer for
BigQueryReadEnumeratorState.Creates the read session once and hands its streams out one at a time.
Converts each row and advances the stream's offset by exactly one row after successful
deserialization and downstream collection.
Converts a row read from the BigQuery Storage Read API into zero or more output records.
Converts a REST client
Schema (as read from live tables) into the Storage API TableSchema form the sink's schema handling operates on.Entry point for building a BigQuery sink.
Builder for BigQuery sinks, obtained from
BigQuerySink.builder().Immutable configuration shared by all
WriteMethod implementations, assembled by BigQuerySinkBuilder.A bounded source reading a BigQuery table through the Storage Read API.
Builds a
BigQuerySource.Everything the source's enumerator and readers were configured with, as one immutable object
shipped inside the job graph.
Reader subtask consuming the read streams it is assigned, one at a time.
The reader's metrics.
Reads the rows of the streams this subtask was assigned, a bounded batch at a time.
Reads a BigQuery table through the Storage Read API.
Default
TableAdmin backed by the BigQuery REST client.Turns one Bigtable change-stream mutation into zero or more user records.
Table changelog source backed by the DataStream Change Streams source.
Builds a
BigtableChangeStreamDynamicSource.Checkpointed coordinator state for the Bigtable Change Streams partition topology.
Connector-owned checkpoint format for
BigtableChangeStreamEnumeratorState.The exact physical schema of the generic Bigtable Change Streams mutation envelope.
One complete logical row mutation returned by Bigtable Change Streams.
Adds an aggregate input to one cell.
How a
BigtableChangeStreamMutation.TimestampBound constrains its end of the range: exclusive, inclusive, or not at
all.Deletes cell versions in one timestamp range.
Deletes every cell in one family.
One ordered mutation entry.
Which subtype an
BigtableChangeStreamMutation.Entry is.A signed 64-bit integer in an aggregate entry.
Merges an aggregate input into one cell.
What produced a mutation: a user write, or the table's garbage-collection policy.
A raw microsecond timestamp in an aggregate entry.
Arbitrary bytes in an aggregate entry.
Writes one cell version.
One timestamp-range boundary.
Timestamp range affected by a cell deletion.
One typed value used by an aggregate entry.
Which subtype a
BigtableChangeStreamMutation.Value is.Emits the connector-owned
BigtableChangeStreamMutation unchanged.The connector-side family and qualifier patterns, answering whether one identifier is retained.
Serializes the connector-owned mutation model without reflective collection access.
Snapshot for the connector-owned field format.
Supplies the connector-owned serializer for
BigtableChangeStreamMutation.Reads a bounded number of Bigtable Change Streams partitions concurrently.
Reporter-visible state for one Bigtable Change Streams reader subtask.
Emits mutations with commit timestamps and advances token/watermark state.
FLIP-27 source for Bigtable Change Streams.
Builds a
BigtableChangeStreamSource.Everything the Change Streams source was built with, assembled by the builder and carried into
the job graph.
Coordinates Bigtable Change Streams partitions, split successors, and merge-parent tokens.
The
WITH options of the bigtable table connector.Loads credentials for the Bigtable data and admin client families.
How this connector points a
BigtableDataClient at an instance: production over
application-default credentials, production over a runtime-loaded service-account provider, or an
emulator over a plaintext channel with no credentials.The
bigtable table sink.Collects the sink's values, so no caller has to keep a positional list in order.
The
bigtable table source: a bounded scan over the DataStream BigtableSource and
row-key point lookups through Bigtable's data client.Collects the source's values, so no caller has to keep a positional list in order.
Creates the
bigtable table sink and table source from a CREATE TABLE statement's
options.The Bigtable emulator image shared by every harness that starts the emulator, so they cannot
drift apart.
The lookup values a
CREATE TABLE maps into the Bigtable table source.Every metric name this connector registers itself, in one place so that this file is the
connector's inventory: what it reports can be read here without opening the writer.
At-least-once sink applying one row mutation per record through the
MutateRows bulk
batcher of google-cloud-bigtable.Deserializes each row and records that the split has passed it.
An asynchronous Bigtable row-key point lookup producing the table source's projected row.
A synchronous Bigtable row-key point lookup producing the table source's projected row.
Turns a Bigtable row into the records a job works with.
What the split enumerator checkpoints: whether the table has been sampled and planned, and the
splits nobody is holding yet.
Serializer for
BigtableScanEnumeratorState.The bounded scan source over Bigtable's
ReadRows.Samples the table once, cuts the configured ranges at the sampled boundaries, and hands the
pieces out one at a time.
Serializes sink records into Bigtable row mutations.
Entry point for building a Bigtable sink.
Builder for Bigtable sinks, obtained from
BigtableSink.builder().Immutable sink configuration assembled by
BigtableSinkBuilder.Reads rows from a Cloud Bigtable table.
Builds a
BigtableSource.Everything the scan source was built with, assembled by the builder and carried into the job
graph.
Reads the splits this subtask is assigned, asking the enumerator for the next one each time it
finishes one.
The scan reader's own counters, beside the ones Flink registers for every source.
Reads the row-key ranges this subtask was assigned, a bounded batch at a time.
Default
TableAdmin backed by the Bigtable BigtableTableAdminClient.The DDL schema of a
bigtable table: which column is the row key, and which column family
and qualifier every other column addresses.One column family: a
ROW column whose nested fields are its qualifiers.One qualifier: a nested field of a column family's
ROW.Stock Bigtable clients pointed at an emulator container.
At-least-once writer applying row mutations to the Bigtable tables a
DestinationResolver
names.Tuning options for the sink's writer: the batch thresholds handed to the client, and the writer's
own bounds on unacknowledged entries.
Builder for
BigtableWriterOptions.The teardown of one client whose own shutdown cannot be trusted to return: both of its steps on a
separate thread, and one deadline that the calling thread's single
join is the whole of.The client's own bounded wait, satisfied by e.g.
A flush instruction for one buffered write stream: make every row up to
flushOffset
(inclusive, per FlushRows semantics) visible in the destination table.Serializer for
BufferedStreamCommittable.Committer of the buffered-stream write path: makes each completed checkpoint's rows visible by
flushing every committable's stream up to its offset (
FlushRows).Options specific to
WriteMethod.STORAGE_API_EXACTLY_ONCE: how large append requests may
grow, how long inactive destinations stay cached, how the connector-driven recovery schedule
backs off, and how the SDK retries retriable append failures in-stream.Builder for
BufferedStreamOptions.Maps the
sink.buffered-stream.* options onto BufferedStreamOptions.The Storage Write API operations of the buffered-stream write path, abstracted so writer and
committer logic can be unit-tested against fakes: creating buffered streams, opening offset-aware
appenders on them, and flushing rows.
Creates
BufferedStreamService instances on the task manager.One destination's buffered-stream writer state: which stream the subtask owns and how many rows
it had appended (acknowledged) as of the checkpoint.
Serializer for
BufferedStreamWriterState.What one subscriber holds buffered and has not yet handed to the reader, as a single consistent
reading.
A mutation applied by BigQuery change data capture.
Determines whether one record upserts or deletes its BigQuery primary key.
Options for appending BigQuery CDC mutations through the Storage Write API default stream.
Builder for
CdcOptions.Creates the internal write-only fields that represent BigQuery CDC pseudocolumns.
Extracts the BigQuery change sequence number for one record.
Desired BigQuery table contract for CDC writes.
Builder for
CdcTableOptions.Maps Table API CDC table options onto their DataStream contract.
Supplies the desired CDC table contract for each destination.
Policy for a BigQuery CDC destination table that already exists.
Turns a
RowData field into the bytes of a Bigtable cell, and back.Reads the bytes of a cell as one field of a row.
Reads one field of a row as the bytes of a cell.
How many cells a mutation costs, counted the way Spanner counts a mutation.
The physical changelog representation emitted by a Bigtable Change Streams table source.
The relational changelog emitted from Spanner data-change records.
Coordinator-side Bigtable operations, separated so partition protocol tests need no service.
Mints the
ChangeStreamCoordinatorClient one enumerator coordinates through.Dispatches a change-stream mutation entry or aggregate value to the connector code that handles
it.
Handles every mutation entry type the connector models.
Handles every aggregate value type the connector models.
Serializable seam that opens asynchronous
ReadChangeStream RPCs.The shape every Bigtable Change Streams partition has, and the conversion to the shape the SDK
models require.
One Bigtable Change Streams partition and the exact position from which it resumes.
One Spanner Change Streams partition and its checkpointed coordinator lifecycle.
Connector-owned checkpoint format for
ChangeStreamPartitionSplit.Connector-owned checkpoint format for a Spanner Change Streams partition split.
Mutable task-thread state for one change-stream partition.
Mutable task-thread progress for one Spanner Change Streams partition.
Resolves a reader-restored split against current retention.
A table option's Bigtable Change Streams start-position mode.
A Table API spelling for one shared Change Streams start position.
Maps one mode/timestamp option pair onto the shared Change Streams start-position API.
Maps Table API startup options to the shared change-stream start position.
One child-partitions record discovered while a reader is consuming a parent partition.
One child token and the normalized coordinator identities of every parent.
Releasing several resources at once, so that one refusing to close never strands the rest.
The
WITH options of the cloud-tasks table connector.At-least-once sink creating one Cloud Tasks task per record through the
CreateTask RPC,
with fixed or per-record queue destinations.Insert-only table sink creating target tasks in one fixed queue.
Factory for the sink-only
cloud-tasks table connector.The Cloud Tasks emulator image shared by every harness that starts the emulator, so they cannot
drift apart.
Every metric name this connector registers itself, in one place so that this file is the
connector's inventory: what it reports can be read here without opening the writer.
Serializes sink records into Cloud Tasks tasks.
Entry point for building a Cloud Tasks sink.
Builder for Cloud Tasks sinks, obtained from
CloudTasksSink.builder().Immutable sink configuration assembled by
CloudTasksSinkBuilder.The request target represented by a
cloud-tasks table.A stock
CloudTasksClient over one plaintext channel to the emulator, with no credentials,
for queue administration and task inspection.At-least-once writer creating one Cloud Tasks task per record.
Tuning options for the sink's writer: the in-flight cap, the transport channel pool and the two
retry budgets.
Builder for
CloudTasksWriterOptions.ReaderOutput for tests that drive SourceReaderBase.pollNext(...), delegating to
one shared CollectingSourceOutput whatever split a record came from.SourceOutput recording emitted records and their timestamps — what a
RecordEmitter is handed, and what a test driving one asserts against.What the sink does with a mutation Spanner refuses for violating a constraint.
Everything one BigQuery copy job needs, decoupled from the client for testability.
Whether the sink may create destination tables that do not exist.
Whether the sink may create the destination table when it does not exist.
Whether the sink may create destination topics that do not exist.
The cross-version seam every sink in this module implements instead of
Sink directly.The cross-version seam every sink in this module implements instead of
Sink directly.The cross-version seam every sink in this module implements instead of
Sink directly.The cross-version seam every sink in this module implements instead of
Sink directly.The cross-version seam every sink in this module implements instead of
Sink directly.A fully-qualified Spanner database reference: project, instance and database.
One data-change record returned by a Spanner Change Streams read function.
Builder for
DataChangeRecord.Per-record description of one watched column.
Serializes
DataChangeRecord without reflective access to JDK implementation fields.Snapshot for the connector-owned field format.
Supplies the connector-owned serializer for
DataChangeRecord.Opens change streams through a lazily created Bigtable data client.
Samples row keys through a
google-cloud-bigtable BigtableDataClient.Opens reads through a
google-cloud-bigtable BigtableDataClient.Destination for elements that terminally failed to be written by a sink, used through
FailureHandler.sendToDeadLetterQueue(DeadLetterQueue).Transcodes Debezium MySQL GTID coordinates into BigQuery CDC sequence sections.
Derives BigQuery CDC sequence numbers from Debezium MySQL source properties.
Transcodes Debezium PostgreSQL sequence metadata into BigQuery CDC hexadecimal sections.
Derives BigQuery CDC sequence numbers from Debezium PostgreSQL source properties.
Derives BigQuery CDC sequence numbers from Debezium Spanner source properties.
Native java-bigtable implementation of the coordinator operations.
Retention-aware reader restore resolution backed by Bigtable table metadata.
The
FailureHandlerContext the sinks build from their WriterInitContext.Creates
MutationBatchers backed by google-cloud-bigtable BigtableDataClients, connected the way BigtableDataClients connects every data client of
this connector — that class owns the credential and emulator modes.Mints
BatchClientPartitionPlanners.PublisherFactory building google-cloud-pubsub Publisher instances
configured from PubSubPublisherOptions; every knob left unset keeps the SDK default
(options with no batching or retry overrides leave the publisher builder untouched, so the
default configuration is byte-identical to the SDK's).Mints
ReadClientSessionCreators.Mints
DataClientRowKeySamplers.Opens the Spanner service handle used for change-stream metadata discovery.
Opens the Spanner client and bounded callback executor owned by one reader subtask.
Opens a real Spanner service handle and wraps it in a
SpannerServiceAdapter.Options specific to
WriteMethod.STORAGE_API_AT_LEAST_ONCE: how large append requests may
grow, how the connector-driven re-append budget backs off, how the SDK retries retriable append
failures in-stream, how the SDK's connection pool (multiplexing) is sized, and the writer's
housekeeping (destinationIdleTimeout, flushInterval).Builder for
DefaultStreamOptions.Maps the
sink.default-stream.* options onto DefaultStreamOptions.SubscriberFactory building google-cloud-pubsub Subscriber instances
configured from PubSubSubscriberOptions; every knob left unset keeps the SDK default, so
the default configuration differs from the SDK's only in the two settings the source owns.Creates a
TaskCreator backed by a google-cloud-tasks CloudTasksClient,
connecting either to production Cloud Tasks with application-default or configured
service-account credentials, or to an emulator over a plaintext channel with no credentials.What the source does with a message its deserialization schema cannot convert.
Runs independent destination actions with a fixed, committer-lifetime concurrency bound.
One worker's thread-confined BigQuery collaborators.
Builds one worker context, lazily on the thread that owns it.
Optional per-destination send counters, as
destination.NAME.recordsSend and
destination.NAME.sendErrors on the writer's metric group, where NAME is the destination
as the connector's docs page spells it.One destination's counters; a no-op instance when per-destination metrics are off.
The result of resolving a BigQuery destination for one record.
Dispatches destination-resolution results to the writer that owns the record.
Handles every destination-resolution result supported by the connector.
Resolves the destination table for each record, enabling one sink instance to write to many
tables (dynamic destinations).
Resolves the destination table for each record, enabling one sink instance to write to many
tables (dynamic destinations).
Resolves the destination queue for each record, enabling one sink instance to write to many
queues (dynamic destinations).
Resolves the destination topic for each record, enabling one sink instance to publish to many
topics (dynamic destinations).
Deadline-bounded draining of a running job's collect iterator.
Transport channel providers pointing a gRPC client at an emulator.
The
host:port of a Google Cloud emulator, parsed and validated on the client rather than
on a TaskManager after the job has been submitted (issue #235).The three counters a
PullAssignmentSplitEnumerator reports, registered by the connector
under its own names.Counts a sink's failures by the status code that caused them, as
errorClass.CODE.errors
on the writer's metric group.An element that terminally failed to be written by one of this project's sinks — the read-only
contract every connector's concrete failure type implements, and the view a cross-connector
DeadLetterQueue sees.A single message that terminally failed to be published to Pub/Sub, as passed to a
FailureHandler<FailedMessage>.A single row mutation that terminally failed to be written to Bigtable, as passed to a
FailureHandler<FailedMutation>.A single mutation that terminally failed to be applied to Spanner, as passed to a
FailureHandler<FailedMutation>.A single row that terminally failed to be written to BigQuery, as passed to a
FailureHandler<BigQueryFailure>.A single task that terminally failed to be created in Cloud Tasks, as passed to a
FailureHandler<FailedTask>.Pluggable policy for elements that terminally fail to be written by a sink.
What a
FailureHandler or DeadLetterQueue learns about its surroundings when it is
opened: enough to stamp per-subtask output and to register its own metrics, and nothing more.A queue-backed
MailboxExecutor for tests: FakeMailboxExecutor.execute(org.apache.flink.api.common.operators.MailboxExecutor.MailOptions, org.apache.flink.util.function.ThrowingRunnable<? extends java.lang.Exception>, java.lang.String, java.lang.Object...) enqueues mails, FakeMailboxExecutor.yield() runs the next mail (blocking until one arrives, like the real mailbox), and FakeMailboxExecutor.drain() runs every mail already enqueued.In-memory
SourceReaderContext for the tests of a pull-assignment source reader.FakeSplitEnumeratorContext<SplitT extends org.apache.flink.api.connector.source.SourceSplit>
In-memory
SplitEnumeratorContext for the tests of a pull-assignment enumerator —
one whose readers ask for work through handleSplitRequest.The streaming pre-commit stage of the FILE_LOADS topology: stamps each committable with the
checkpoint id carried by its
CommittableMessage, which the Committer SPI cannot observe otherwise — the stamp selects
the streaming behavior of the load-job orchestrator and makes BigQuery job ids attributable to
their checkpoint.One finalized staging file: the Flink job id of the run that staged it, its destination table,
its Cloud Storage URI, size counters used for load-job partitioning, and — in streaming execution
— the checkpoint that triggered it.
Serializer for
FileLoadsCommittable.The real FILE_LOADS commit: turns one committable batch — a whole batch run, or one checkpoint of
a streaming run — into BigQuery load jobs through a
LoadJobOrchestrator.Thread-safe metrics for the bounded FILE_LOADS destination executor.
Options specific to
WriteMethod.FILE_LOADS: where staging files go on Cloud Storage, how
loaded rows land in tables that already hold data, where oversized loads stage their temporary
tables, how checkpoint-triggered loads are paced in streaming execution, and the two schedules
the committer backs off on.Builder for
FileLoadsOptions.Maps the
sink.file-loads.* options onto FileLoadsOptions.The FILE_LOADS
SinkWriter: encodes records in the
configured staging format and streams them to
per-destination staging files on Cloud Storage, emitting one FileLoadsCommittable per
finalized file from FileLoadsWriter.prepareCommit() — once at end of input in batch execution, once per
checkpoint in streaming execution.A
DestinationResolver returning one fixed destination for every record.A
DestinationResolver returning one fixed destination for every record, which is what
BigtableSinkBuilder.table(...) builds.A
DestinationResolver returning one fixed destination for every record.A
DestinationResolver returning one fixed destination for every record.Table format factory for UTF-8
application/x-www-form-urlencoded bodies.A garbage-collection rule for a column family the sink creates, mirroring the four rule shapes
Bigtable's admin API takes: a version cap, an age cap, and their union and intersection.
The rule shapes Bigtable's admin API takes.
StagingStorage over the Cloud Storage client with configured credentials or ADC.An encoding format that owns the HTTP Content-Type for body-carrying requests.
The first stage of
CloudTasksSerializationSchema.httpTarget(String): it carries the
target URL and waits for the body schema.A
CloudTasksSerializationSchema producing HTTP-target tasks: a URL, an HTTP method, an
optional authorization token, optional headers, and a body serialized by a wrapped Flink SerializationSchema.Extracts HTTP headers from a record.
Extracts the target URL from a record.
Validated fixed HTTP request values used by the table serializer.
What the Bigtable table sink tells the planner when the requested changelog is insert-only.
Options controlling how JSON records are converted to BigQuery rows.
Builder for
JsonDocumentOptions.A
BigQueryProtoSerializationSchema for records that are JSON documents, as text.A
BigtableDataClient built on first use and closed once, held by a seam implementation
that travels in the job graph.Holds state its owner derives from its own serializable fields, once, and again on a task
manager.
Runs the staged files of one run — a whole batch job, or one checkpoint of a streaming job — into
their destination tables: submits every load, copy and terminal query through a
LoadJobRunner, then deletes the temporary tables and the staged objects.Executes BigQuery load, copy and terminal query jobs for the FILE_LOADS orchestration.
Everything one BigQuery load job needs, decoupled from the client for testability.
Captures what a class logs, so a test can assert the line rather than only its side effects.
One collected log event, flattened out of log4j2's own type.
The levels a capture can be taken at, so no log4j2 type reaches this class's signature.
Publishes every container's ports on the loopback address the Docker host resolves to, rather
than on the wildcard address, so a host-port collision with an unrelated local process cannot
form (ADR-0132).
A materialised Bigtable row and the stable byte estimate measured while it was decoded.
Materialises SDK rows while measuring their decoded wire content without traversing them twice.
Shared boundary handling for connector metric values.
Fails a reader that holds unacknowledged messages but has never been asked for a checkpoint.
Checkpointed timer for a service partition absent from the coordinator ledger.
JSON values describing the changes to one watched table row.
The operation represented by one Spanner change-stream modification.
Accepts row mutations for one table and reports each one's outcome separately.
Creates one
MutationBatcher per table the writer resolves, and owns whatever those
batchers share.Appends row batches to one buffered write stream at explicit offsets.
The checks every connector's option builders repeat, in one place.
Assertions shared by the Table API option-set tests.
Applies Table API option values to DataStream builder setters, renaming a rejected value's
failure to the option key the SQL caller actually wrote (issue #1030, ADR-0133).
Applies Table API option values to DataStream builder setters, renaming a rejected value's
failure to the option key the SQL caller actually wrote (issue #1030, ADR-0133).
Applies Table API option values to DataStream builder setters, renaming a rejected value's
failure to the option key the SQL caller actually wrote (issue #1030, ADR-0133).
Applies Table API option values to DataStream builder setters, renaming a rejected value's
failure to the option key the SQL caller actually wrote (issue #1030, ADR-0133).
Applies Table API option values to DataStream builder setters, renaming a rejected value's
failure to the option key the SQL caller actually wrote (issue #1030, ADR-0133).
Whether the source preserves Pub/Sub ordering-key delivery order.
How
StagingFormat.PARQUET staging files are compressed.Reports successful end-of-query after all child-partitions records were forwarded.
Lifecycle of one Spanner Change Streams partition in the checkpointed coordinator ledger.
What one planning call answered with: a snapshot, and the partitions of the read at it.
Opens a batch read and asks the service to plan it into partitions.
Mints the
PartitionPlanner one enumerator plans through.Reader heartbeat progress used to keep the coordinator's recovery watermark current.
Reader progress used to keep the coordinator's assigned-partition ledger current.
Reports that a partition closed and names each successor range and its continuation token.
One target range from CloseStream paired with the token that arrived from this parent.
How much a paused split may buffer before the reader stops its subscriber.
Successor partition parked until its parent tokens cover the whole target range.
A split as the planner produced it, with the size the samples suggest it holds.
Adds fields to a protobuf row descriptor without changing its surrounding file structure.
ProtoMessageSerializationSchema<T extends com.google.protobuf.Message>
A
BigQueryProtoSerializationSchema for records that already are protobuf messages.One validated field consumed by the generic protobuf row-augmentation engine.
Serializable provider whose validation may report a row-level
IOException.Write-only fields accepted by BigQuery's documented Storage Write API contract.
Adds ordered physical or write-only fields to delegate protobuf rows.
Rewrites protobuf messages into
DynamicMessages conforming to a BigQuery-storage
compatible target descriptor (as produced by BQTableSchemaToProtoDescriptor from the
schema derived by ProtoToTableSchemaConverter).Options controlling how protobuf descriptors are mapped to BigQuery schemas.
Builder for
ProtoSchemaOptions.Converts serialized proto rows (the
BigQueryProtoSerializationSchema wire
form, whose descriptors follow the BQTableSchemaToProtoDescriptor conventions) into Avro
GenericRecords conforming to the schema produced by TableSchemaToAvroConverter.Derives a BigQuery
TableSchema from a protobuf Descriptors.Descriptor.Creates
TopicPublisher instances for the writer's per-topic publisher map.Builds
PubSubPublisherOptions from the table options.AckTracker keeping the acknowledgement handles in memory until their checkpoint
completes.The
WITH options of the pubsub table connector.Loads the credentials shared by the Pub/Sub source and sink client families.
A
DeadLetterQueue publishing every terminally failed element to a Pub/Sub topic, used
through FailureHandler.sendToDeadLetterQueue(DeadLetterQueue).Builder for
PubSubDeadLetterQueue.Deserializes Pub/Sub messages into source records.
A table sink publishing rows to one Pub/Sub topic.
A table source consuming rows from one or more Pub/Sub subscriptions.
The
pubsub table connector factory.The Pub/Sub emulator image shared by every harness that starts the emulator, so they cannot drift
apart.
Checkpointed state of the split enumerator: the subscriptions it resolved.
Serializer for
PubSubEnumeratorState.Every metric name this connector registers itself, in one place so that this file is the
connector's inventory: what it reports can be read here without opening a writer, a reader or the
enumerator.
Tuning options for the sink's Pub/Sub publishers and its writer: SDK batching and publish-retry
settings, message ordering, the writer's in-flight caps, the backoff budget of the topic
auto-creation recovery, and the writer's shutdown budget.
Builder for
PubSubPublisherOptions.At-least-once sink publishing through
google-cloud-pubsub Publisher instances
with dynamic per-record topic destinations.Deserializes received messages and emits them, using the Pub/Sub publish time as the event
timestamp.
Serializes sink records into Pub/Sub messages.
Extracts message attributes from a record.
Extracts the ordering key from a record.
What this connector's bounded teardowns leave behind, counted for the lifetime of the class
loader rather than of a task.
Entry point for building a Pub/Sub sink.
Builder for Pub/Sub sinks, obtained from
PubSubSink.builder().Immutable sink configuration assembled by
PubSubSinkBuilder.Entry point for building Pub/Sub sources.
Builder for Pub/Sub sources, obtained from
PubSubSource.builder().Immutable configuration of a Pub/Sub source, assembled by
PubSubSourceBuilder and shipped
in the job graph.Reader subtask consuming its assigned subscription splits.
The reader's Pub/Sub-specific metrics.
Verifies the subscriptions, then assigns subscription splits to reader subtasks.
Multiplexes the streaming-pull subscribers of one reader subtask's splits.
Fetch-loop helpers for driving a Pub/Sub
SplitReader directly in tests, typed against the
flink-connector-base interface so the connector's reader class never crosses the module
boundary.Where the source starts consuming a subscription.
Which starting point a
PubSubStartPosition names.At-least-once source consuming Pub/Sub subscriptions through
google-cloud-pubsub
Subscriber streaming pull.Tuning options for the source's Pub/Sub subscribers and its reader: SDK flow control, the
streaming-pull connection count and the acknowledgement-deadline extension settings, plus the
source's own drain size, subscriber shutdown budget and first-checkpoint watchdog.
Builder for
PubSubSubscriberOptions.Default
SubscriptionAdmin backed by the Pub/Sub SubscriptionAdminClient.The Pub/Sub admin, publish and pull machinery every integration-test harness needs, parameterised
over the transport: a plaintext channel with no credentials against the emulator, or
application-default credentials against the real service.
Default
TopicAdmin backed by the Pub/Sub TopicAdminClient.At-least-once writer publishing records to dynamic per-record Pub/Sub topic destinations.
PullAssignmentSplitEnumerator<SplitT extends org.apache.flink.api.connector.source.SourceSplit,StateT,PlanT>
A bounded source's split enumerator that plans once, asynchronously, and then hands the plan out
one split per request.
Bridges Pub/Sub's push-style streaming pull to the pull-style
SplitReader contract: the client
library delivers messages on its own threads into an in-memory buffer, and the fetcher thread
drains that buffer.The deterministic identity under which a query job may be reused across a re-plan.
Everything one terminal BigQuery query job needs, decoupled from the client for testability.
What running the query produced: the table the result landed in, and whether it came from a
previous attempt's job rather than a fresh submission.
Runs the source's query and says which table its result landed in.
The query job a
QueryRunner is asked to run.A fully-qualified Cloud Tasks queue reference: project, location and queue.
Opens
ReadRows calls through a BigQueryReadClient.Creates read sessions through a
BigQueryReadClient.Absolute number of additional partition splits one reader can accept.
Adapts a
SourceReaderContext to the context a DeserializationSchema expects when
a source opens it.Creates the source's read session.
Mints the
ReadSessionCreator one enumerator plans through.One Storage Read API
ReadStream, plus how far into it the source has already consumed.Serializer for
ReadStreamSplit.The mutable reading progress of one
ReadStreamSplit.A read-only
Counter view of one PubSubShutdownResidue adder, so a residue
registers as the counter it is rather than as a gauge over a monotonic total.The check a configured value earns by being a component of a resource path this project composes.
A
TableAdmin failure that repeating the same call can fix — a rate limit, a quota that
refills, a server-side error.Support for the connectors' hand-written retry loops.
A
TableAdmin that repeats a creation the service answered with a RetriableTableAdminException, within a fixed budget.An immutable retry schedule: exponential backoff from an initial delay up to a cap, a bounded
number of attempts, and optional proportional jitter (to de-synchronize parallel subtasks
retrying against the same destination).
Appends batches of serialized rows to one destination's default write stream.
Creates
RowAppenders for destinations.How
RowTypeToTableSchemaConverter derives a BigQuery schema from a SQL table's columns —
the RowData counterpart of AvroSchemaOptions, carrying the same three knobs under
the same names.Builder for
RowDataSchemaOptions.Serializes table-runtime rows into mutations for one configured table.
Derives the protobuf row descriptor a BigQuery table schema is written through.
How a Bigtable table option represents a row-key bound.
One row key the service offered as a section boundary, with the approximate number of bytes
stored before it.
Asks the service where a table's sections begin.
Mints the
RowKeySampler one enumerator plans against.The module's row-key range algebra: emptiness, containment, cutting, coalescing, intersection,
truncation and the unsigned key comparison they all rest on, each defined once here.
One contiguous row-key range still to be read.
Cuts the configured row-key ranges at the boundaries the service sampled.
Serializer for
RowRangeSplit.How far through one
RowRangeSplit the source has emitted.One open
ReadRows call: the response blocks of one stream, from one offset onwards.One open
ReadRows call, consumed a row at a time.Opens
ReadRows calls on a read session's streams.Opens a
ReadRows call over one row-key range.Which Bigtable read surface a table source uses.
The Spanner read path selected by a table DDL.
Computes the union of a destination table's live schema and the serializer's desired schema — the
schema the sink proposes when connector-driven schema updates are enabled.
A checked rejection of a schema union: the desired schema requires a change the rules (or the
configured options) do not permit.
The result of a union: the unified schema and whether it differs from the existing one.
Options gating connector-driven destination table schema updates.
Builder for
SchemaUpdateOptions.The physical primary key and value-format payload of a selected-cell table.
Creates synthetic service-account keys for credential wiring tests: structurally valid enough for
the Google auth libraries to parse — a real RSA key in a real PEM envelope — while naming an
example.invalid account no service would ever accept, so a test that leaks one at a real
endpoint fails loudly rather than authenticating.Locates the uber-jar a
flink-sql-connector-gcp-* module builds, for the tests that
inspect or run against it.What the split enumerator checkpoints: whether the read has been planned, and the splits nobody
is holding yet.
Serializer for
SpannerBatchReadEnumeratorState.The bounded batch source over Spanner's partitioned reads.
Opens one batch read, asks Spanner to plan it into partitions, and hands the partitions out one
at a time.
Builds BigQuery CDC sequence numbers from typed Spanner change-stream coordinates.
Transcodes Spanner change-stream coordinates into BigQuery CDC hexadecimal sections.
Metadata operations the Spanner Change Streams coordinator needs during initialization.
Serializable job-graph seam that opens the coordinator's runtime metadata client.
Turns one Spanner data-change record into zero or more user records.
An unbounded Table API source for one table observed through a Spanner change stream.
Checkpointed Spanner Change Streams partition state.
Connector-owned checkpoint format for the Spanner Change Streams partition ledger.
Releases readers after the coordinator has validated Change Stream metadata and restore age.
Opens asynchronous partition queries for one source subtask.
Serializable factory for the TaskManager-local Change Streams query client.
Reads several Spanner Change Streams partition queries concurrently in one source subtask.
Applies connector-side table and column filters to decoded data-change records.
Result of applying table and column filters to one data-change record.
FLIP-27 source for Cloud Spanner Change Streams.
Builds a
SpannerChangeStreamSource.Immutable configuration assembled by
SpannerChangeStreamSourceBuilder.Coordinates the checkpointed parent-child lifecycle of Spanner Change Streams partitions.
Advances every reader subtask to the coordinator's unfinished-ledger watermark.
Calculates the non-early Flink watermark for the complete Spanner partition ledger.
Builds the Spanner service handle both directions of this connector open.
The
WITH options of the spanner table connector.Loads credentials shared by every direction of this connector: the sink writer, the bounded
source's planner and readers, the Change Streams coordinator and query clients, and the table
lookup path.
The narrow view of a Spanner database the writer needs: read the schema once, then apply batches.
Receives the outcome of one mutation group of a batch write.
Opens a
SpannerDatabaseAccess on the task manager.The
spanner table sink, backed by the DataStream SpannerSink.Collects values for the immutable sink.
A bounded table scan and primary-key point-lookup source for Spanner tables.
Creates the
spanner table source and sink from a SQL DDL.The Spanner emulator image shared by every harness that starts the emulator, so they cannot drift
apart.
Lookup options owned by the Spanner table source.
Every metric name this connector registers, in one place.
Turns a stream record into the Spanner
Mutation the sink writes.At-least-once sink applying one mutation per record through
batchWriteAtLeastOnce of
google-cloud-spanner.What the batch source reads: either a query, or a table with its columns and key set, optionally
through an index.
Internal bridge between a table-layer deferred read and the batch partition planner.
Resolves a table-layer read after the batch transaction has selected its snapshot.
Deserializes each row and hands its output records on.
Asynchronous primary-key lookup producing the projected table row.
Synchronous primary-key lookup producing the projected table row.
The priority Spanner schedules this connector's requests at.
Entry point for building a Spanner sink.
Builder for the Spanner sink; created through
SpannerSink.builder().The immutable configuration
SpannerSinkBuilder builds and the sink carries to the task
managers.Reads rows from a Cloud Spanner database.
Builds a
SpannerSource.Everything the batch source was built with, assembled by the builder and carried into the job
graph.
Reads the partitions this subtask is assigned, asking the enumerator for the next one each time
it finishes one.
The batch reader's own counters, beside the ones Flink registers for every source.
Reads the partitions this subtask was assigned, a bounded batch at a time.
Turns a Spanner row into zero or more output records.
A table name as rendered for Spanner APIs and compared with
INFORMATION_SCHEMA.The name of one table-local access path, such as an index.
The physical DDL schema and its corresponding native Spanner types.
One physical column.
Stock Spanner clients pointed at an emulator endpoint.
The Spanner sink's writer: buffers mutations, applies them with
batchWriteAtLeastOnce,
and decides per mutation what a refusal means.Tuning options for the sink's writer: how large a batch write request grows, how Spanner should
schedule it, and the recovery budget the writer spends on transient failures.
Builder for
SpannerWriterOptions.The split universe of a job and its subtask ownership, computed deterministically from the
subscription list, the ordering mode and the source parallelism.
The file format
FILE_LOADS stages rows in before loading them, set by
FileLoadsOptions.Builder#stagingFormat.The Cloud Storage staging area of the FILE_LOADS write method.
Where a change-stream source starts reading when no checkpointed state is restored.
Builds a
PubSubStartPosition from the table options.Resolves change-stream start positions against one startup instant and one retention window.
The retained-window evidence for one expired restored partition.
Discovers how long a connector's change stream retains records.
Extraction of gRPC status codes from the exceptions the Google Cloud clients surface.
Converts a Storage API
TableSchema (the schema form of the serializer SPI) into the REST
client Schema needed to create tables.Default
RowAppenderFactory backed by Storage Write API StreamWriters on the
destination's default write stream.One open read of a partition, consumed a row at a time.
Rejoins a batch read's snapshot and opens one of its partitions.
The parts of
WriterInitContext a sink reads when creating a writer; everything a sink has
no reason to touch is unsupported, so a new dependency on the context shows up as a failing test
rather than as a silent null.Reports that one source reader could not admit a delivery within its subscriber-buffer bound.
Creates the
google-cloud-pubsub Subscriber backing one split.Receives a message together with the handle that settles it.
Builds
PubSubSubscriberOptions from the table options.Subscription administration used by the enumerator's startup check, abstracting the Pub/Sub admin
client so the check can be unit-tested without one.
Settings the source applies when it creates a subscription that does not exist.
Builder for
SubscriptionCreateOptions.Builds per-subscription
SubscriptionCreateOptions from the table options.A fully-qualified Pub/Sub subscription reference: project and subscription.
The settings of an existing subscription that the source's preflight reads.
Builder for
SubscriptionInfo.A unit of work for one reader subtask: a streaming-pull connection to one subscription.
Serializer for
SubscriptionSplit.Runs one source deserialization call with a direct, non-null collector.
One deserialization call that emits through the supplied collector.
Administers destination tables: creation for
CreateDisposition.CREATE_IF_NEEDED and schema
reads/updates for schema evolution.Table administration operations used by the sink, abstracting the Bigtable admin client so writer
logic can be unit-tested without one.
What an
TableAdmin.ensureTable(io.github.flink.gcp.connector.bigtable.TableDestination, io.github.flink.gcp.connector.bigtable.sink.TableCreateOptions) call actually did, so the caller can report creation and
family-addition separately without this SPI knowing about metrics.A table-administration failure that retains whether the operation requested creation.
Options applied when the sink creates a destination table under
CreateDisposition.CREATE_IF_NEEDED: time partitioning and clustering.The settings for the table the sink creates under
CreateDisposition.CREATE_IF_NEEDED: its
column families and, per family, an optional garbage-collection rule.Builder for
TableCreateOptions.Builder for
TableCreateOptions.Granularity of time-based partitioning.
Maps the
sink.table-create.* options onto TableCreateOptions.Builds
TableCreateOptions from the table options and the DDL's column families.Supplies per-destination
TableCreateOptions for tables created under CreateDisposition.CREATE_IF_NEEDED.A fully-qualified BigQuery table reference: project, dataset and table.
A fully-qualified Bigtable table reference: project, instance and table.
A point-in-time read of a destination table's schema, as returned by
TableAdmin.getSchema(io.github.flink.gcp.connector.bigquery.sink.TableDestination).Converts a Storage API
TableSchema (the schema form of the serializer SPI) into the Avro
schema staged files are written with.Validated fixed target values used by the table sink.
Creates Cloud Tasks tasks, one RPC per task.
Creates the writer's
TaskCreator.Extracts the deduplication key of a record, opting the sink into named tasks.
Shared test stubs for
SinkWriter.Context.Random resource-name helpers for tests whose resources share a namespace — the gated real-GCP
suites above all, where concurrent runs must not collide on resource names and a crashed run must
leave behind names that identify it, but also an emulator harness giving each test a database of
its own.
A
SinkCommitterMetricGroup whose metrics can be read back by the names they registered
under — TestSinkCommitterMetricGroup is to a committer what TestSinkWriterMetricGroup is to a writer, and for the same reason:
UnregisteredMetricsGroup.createSinkCommitterMetricGroup() hands out a fresh
SimpleCounter per call, so a counter the committer registered is unreachable afterwards.A
SinkWriterMetricGroup whose metrics can be read back by the names they registered
under, so a sink writer's metrics are asserted the way a reporter would see them.Transcodes a TiCDC commit TSO into a BigQuery CDC sequence section.
Derives BigQuery CDC sequence numbers from TiCDC's Debezium-protocol source properties.
The writer's view of time: reading the clock and waiting out a retry backoff.
Topic administration operations used by the sink, abstracting the Pub/Sub admin client so writer
logic can be unit-tested without one.
Settings the sink applies when it creates a topic that does not exist.
Builder for
TopicCreateOptions.Builds
TopicCreateOptions from the table options.A fully-qualified Pub/Sub topic reference: project and topic.
Publishes messages to one Pub/Sub topic.
What a fixed-width decoder does with a cell or row key longer than the declared type's layout.
An explicit, record-specific destination-resolution failure.
Parses the table layer's string carrier as a native Spanner UUID value.
The value-capture policy that was active when a data change was recorded.
Production
BufferedStreamService over one BigQueryWriteClient.Default
BufferedStreamServiceFactory creating WriteClientBufferedStreamServices.How data lands in a destination table that already contains data.
The mechanism used to write records to BigQuery.
Builds
BigtableWriterOptions from the table options.Maps Table API options onto the DataStream writer options.
Maps table options onto
SpannerWriterOptions.