All Classes and Interfaces
Class
Description
Keeps a
flink-sql-connector-gcp-* module's META-INF/NOTICE honest: it must list
every artifact the uber-jar bundles, and nothing it does not.The part of a read-only connector catalog that does not depend on the service: the lifecycle, the
lazily opened client, and every answer docs/adr/0168 fixes for all connector catalogs.
Releases a client
AbstractReadOnlyCatalog.close() drops.Opens the service client on the first metadata call.
Asserts the shape of a
flink-sql-connector-gcp-* uber-jar: that SQL can discover the
connector in it, that every runtime dependency actually made it in, and that no class
escaped relocation except those that deliberately did.The precondition every
flink-sql-connector-gcp-* smoke test rests on: that the connector
it just exercised came out of the uber-jar.Settles one delivered message, hiding which of the client library's two receiver flavors produced
it.
Tracks the acknowledgement lifecycle of received Pub/Sub messages, which is what makes the source
at-least-once.
One physical BigQuery column appended after the configured serializer emits a row.
Determines how an additional physical field handles a provider returning
null.Ordered physical fields appended to every destination row and table schema.
Builder for
AdditionalFields.Supported BigQuery types for singular additional physical fields.
Extracts one additional physical-field value from the original input record.
Resolves the aggregate input contract before a sink reaches a task manager.
Immutable aggregate input or state: raw bytes, typed bytes or a native int64.
Classifies Storage Write API append failures into the error classes the writer routes on.
Builds an immutable
AppEngineTargetSerializationSchema.Extracts App Engine request headers from a record.
Extracts the App Engine relative URI from a record.
Extracts App Engine service, version, and instance routing from a sink record.
Checks and normalizes App Engine request fields shared by all sink APIs.
A
CloudTasksSerializationSchema producing App Engine target tasks.Validated fixed App Engine request values used by the table serializer.
Retries an asynchronous Table API lookup and converts its successful result to rows.
Converts a client value to rows, allowing a deserializer's checked exception.
A
BigQueryProtoSerializationSchema for Avro records.Converts Avro records into the protobuf rows the BigQuery Storage Write API accepts, against the
schema
AvroToTableSchemaConverter derived and the descriptor
BQTableSchemaToProtoDescriptor built from it.Options controlling how Avro schemas are mapped to BigQuery schemas.
Builder for
AvroSchemaOptions.Derives a BigQuery
TableSchema from an Avro Schema.Deadline-bounded polling for test assertions.
Plans a read through a
google-cloud-spanner BatchClient.Reads partitions through a
google-cloud-spanner BatchClient.One partition of a batch read, together with the snapshot it belongs to.
Serializer for
BatchReadSplit.The mutable reader-side state of a
BatchReadSplit.The
WriteMethod.STORAGE_API_EXACTLY_ONCE sink: each writer subtask appends rows to one
application-created BUFFERED Storage Write API stream per active destination at explicit offsets
(see BigQueryBufferedStreamWriter), and the committer makes each completed checkpoint's
rows visible by flushing every committable's stream up to its offset (see BufferedStreamCommitter) — two-phase commit on Flink checkpoints.Exactly-once
SinkWriter appending rows to one application-created BUFFERED Storage Write
API stream per destination at explicit offsets.Creates the
bigquery catalog from a CREATE CATALOG statement's options.The
WITH options of the bigquery catalog, created by CREATE CATALOG ...
WITH ('type' = 'bigquery', ...).The
WITH options of the bigquery table connector.Loads credentials shared by the BigQuery and Cloud Storage client families.
At-least-once sink appending to Storage Write API default streams with dynamic per-record table
destinations (
WriteMethod.STORAGE_API_AT_LEAST_ONCE).At-least-once
SinkWriter appending to Storage Write API default streams with dynamic
per-record table destinations.The
bigquery table sink.Collects the sink's fully resolved values.
A bounded Table API source mapped onto
BigQuerySource.Collects the source's fully resolved values.
Creates the
bigquery table source or sink from a CREATE TABLE statement's
options.The BigQuery emulator image and its two endpoints, shared by every harness that starts the
emulator so they cannot drift apart.
A record-specific BigQuery sink failure handled by the configured failure policy.
The
WriteMethod.FILE_LOADS sink: writers stage per-destination files on Cloud Storage in
the configured staging format, the pre-commit
topology routes every subtask's committables to a single committer subtask (stamping their
checkpoint id in streaming, see FileLoadsCheckpointStamper), and the committer turns each
batch into BigQuery load jobs (see FileLoadsCommitter and LoadJobOrchestrator) — the whole
run at end of input in batch execution, each checkpoint's files at its completion in streaming
execution.Inspects configured BigQuery resources without evaluating a resolver or opening a client.
Internal Table metadata handoff shared by the concrete BigQuery write methods.
LoadJobRunner over the BigQuery REST client with configured credentials or ADC.Thread-safe submitted-job state shared by every worker of one committer.
Every metric name this connector registers itself, in one place so that this file is the
connector's inventory: what it reports can be read here without opening a writer.
Serializes records into protobuf rows to be written to BigQuery.
Runs the source's query through the REST
BigQuery client.Storage Read API clients, for the two places the source opens one: the enumerator creates the
read session, and every reader opens its assigned streams.
The enumerator's checkpointed state: whether the read session exists, which one it is, and the
splits not currently assigned to a reader.
Serializer for
BigQueryReadEnumeratorState.Creates the read session once and hands its streams out one at a time.
Converts each row and advances the stream's offset by exactly one row after successful
deserialization and downstream collection.
Converts a row read from the BigQuery Storage Read API into zero or more output records.
Converts a REST client
Schema (as read from live tables) into the Storage API TableSchema form the sink's schema handling operates on.Entry point for building a BigQuery sink.
Builder for BigQuery sinks, obtained from
BigQuerySink.builder().Immutable configuration shared by all
WriteMethod implementations, assembled by BigQuerySinkBuilder.A bounded source reading a BigQuery table through the Storage Read API.
Builds a
BigQuerySource.Everything the source's enumerator and readers were configured with, as one immutable object
shipped inside the job graph.
Reader subtask consuming the read streams it is assigned, one at a time.
The reader's metrics.
Reads the rows of the streams this subtask was assigned, a bounded batch at a time.
Reads a BigQuery table through the Storage Read API.
Default
TableAdmin backed by the BigQuery REST client.Creates the
bigtable catalog from a CREATE CATALOG statement's options.The
WITH options of the bigtable catalog, created by CREATE CATALOG ...
WITH ('type' = 'bigtable', ...).Turns one Bigtable change-stream mutation into zero or more user records.
Table changelog source backed by the DataStream Change Streams source.
Builds a
BigtableChangeStreamDynamicSource.Checkpointed coordinator state for the Bigtable Change Streams partition topology.
Connector-owned checkpoint format for
BigtableChangeStreamEnumeratorState.The exact physical schema of the generic Bigtable Change Streams mutation envelope.
One complete logical row mutation returned by Bigtable Change Streams.
Adds an aggregate input to one cell.
How a
BigtableChangeStreamMutation.TimestampBound constrains its end of the range: exclusive, inclusive, or not at
all.Deletes cell versions in one timestamp range.
Deletes every cell in one family.
One ordered mutation entry.
Which subtype an
BigtableChangeStreamMutation.Entry is.A signed 64-bit integer in an aggregate entry.
Merges an aggregate input into one cell.
What produced a mutation: a user write, or the table's garbage-collection policy.
A raw microsecond timestamp in an aggregate entry.
Arbitrary bytes in an aggregate entry.
Writes one cell version.
One timestamp-range boundary.
Timestamp range affected by a cell deletion.
One typed value used by an aggregate entry.
Which subtype a
BigtableChangeStreamMutation.Value is.Emits the connector-owned
BigtableChangeStreamMutation unchanged.The connector-side family and qualifier patterns, answering whether one identifier is retained.
Serializes the connector-owned mutation model without reflective collection access.
Snapshot for the connector-owned field format.
Supplies the connector-owned serializer for
BigtableChangeStreamMutation.Reads a bounded number of Bigtable Change Streams partitions concurrently.
Reporter-visible state for one Bigtable Change Streams reader subtask.
Emits mutations with commit timestamps and advances token/watermark state.
FLIP-27 source for Bigtable Change Streams.
Builds a
BigtableChangeStreamSource.Everything the Change Streams source was built with, assembled by the builder and carried into
the job graph.
Coordinates Bigtable Change Streams partitions, split successors, and merge-parent tokens.
Async SQL conditional write, returning whether the configured predicate matched.
Closes Bigtable data clients away from the task thread while keeping their physical lifetime
bounded.
Immutable, validated row-atomic request whose marker identity survives checkpoint recovery.
Versioned checkpoint format; restored bytes must satisfy the full marker contract.
Bounded conditional requests emitting the original input and its successful response.
Builder for
BigtableConditionalAsync.An INSERT-only command input whose DDL fixes one complete conditional request structure.
Atomic per-row conditional writes with at-least-once delivery.
Builder for
BigtableConditionalSink.The
WITH options of the bigtable table connector.Loads credentials for the Bigtable data and admin client families.
How this connector points a
BigtableDataClient at an instance: production over
application-default credentials, production over a runtime-loaded service-account provider, or an
emulator over a plaintext channel with no credentials.Selects eager writes or checkpoint-owned, replay-protected writes.
The
bigtable table sink.Collects the sink's values, so no caller has to keep a positional list in order.
The
bigtable table source: a bounded scan over the DataStream BigtableSource and
row-key point lookups through Bigtable's data client.Collects the source's values, so no caller has to keep a positional list in order.
Creates the
bigtable table sink and table source from a CREATE TABLE statement's
options.The Bigtable emulator image shared by every harness that starts the emulator, so they cannot
drift apart.
Classifies failed row mutations into the classes the writer routes on.
The classes a failed mutation falls into.
Configured table identities shared by the Bigtable sources and sink families.
The lookup values a
CREATE TABLE maps into the Bigtable table source.Every metric name this connector registers itself, in one place so that this file is the
connector's inventory: what it reports can be read here without opening the writer.
At-least-once sink applying one row mutation per record through the
MutateRows bulk
batcher of google-cloud-bigtable.Bounded read-modify-write requests emitting the original input and its successful response.
Builder for
BigtableReadModifyWriteAsync.Async SQL append and increment, returning the final values of the cells changed.
Atomic per-row read-modify-write writes with at-least-once delivery.
Builder for
BigtableReadModifyWriteSink.Deserializes each row and records that the split has passed it.
Base of the async-operator surface: one single-row request per input, whose answer becomes the
operator's output.
Tuning options for the single-row request runtime: the deadline of each request, how many may be
outstanding at once, and how the runtime holds instance clients.
Builder for
BigtableRequestOptions.A Bigtable row as a request-response RPC returned it: its key and its cells, in the order the
service listed them (by family, then qualifier, then descending timestamp).
One cell of a
BigtableRow: a value at a family, qualifier and timestamp, with the
labels the service attached to it.An asynchronous Bigtable row-key point lookup producing the table source's projected row.
A synchronous Bigtable row-key point lookup producing the table source's projected row.
Turns a Bigtable row into the records a job works with.
Converts the client's
Row into the connector-owned BigtableRow.Serializes
BigtableRow without reflective collection access.Snapshot for the connector-owned field format.
Supplies the connector-owned serializer for
BigtableRow.What the split enumerator checkpoints: whether the table has been sampled and planned, and the
splits nobody is holding yet.
Serializer for
BigtableScanEnumeratorState.The bounded scan source over Bigtable's
ReadRows.Samples the table once, cuts the configured ranges at the sampled boundaries, and hands the
pieces out one at a time.
Serializes sink records into Bigtable row mutations.
Entry point for building a Bigtable sink.
Builder for Bigtable sinks, obtained from
BigtableSink.builder().Immutable sink configuration assembled by
BigtableSinkBuilder.Reads rows from a Cloud Bigtable table.
Builds a
BigtableSource.Everything the scan source was built with, assembled by the builder and carried into the job
graph.
Reads the splits this subtask is assigned, asking the enumerator for the next one each time it
finishes one.
The scan reader's own counters, beside the ones Flink registers for every source.
Reads the row-key ranges this subtask was assigned, a bounded batch at a time.
Session settings consumed when an async SQL write is specialized.
Bounded conditional requests on Flink's committer thread.
Creates runtime factories using each restored envelope's original application profile.
Settings for checkpoint-owned writes.
Builds an immutable staged configuration.
Checkpoint-owned writes through a stateless writer and Flink's persisted committer collector.
Short-lived metadata clients sharing the runtime's credentials; never changes remote policy.
Stages immutable requests without opening a service client or waiting for downstream capacity.
Default
TableAdmin backed by the Bigtable BigtableTableAdminClient.The DDL schema of a
bigtable table: which column is the row key, and which column family
and qualifier every other column addresses.One column family: a
ROW column whose nested fields are its qualifiers, or a
MAP column whose entries are.One qualifier: a nested field of a column family's
ROW.Stock Bigtable clients pointed at an emulator container.
At-least-once writer applying row mutations to the Bigtable tables a
DestinationResolver
names.Tuning options for the sink's writer: the batch thresholds handed to the client, and the writer's
own bounds on unacknowledged entries.
Builder for
BigtableWriterOptions.The teardown of one client whose own shutdown cannot be trusted to return: both of its steps on a
separate thread, and one deadline that the calling thread's single
join is the whole of.The client's own bounded wait, satisfied by e.g.
A flush instruction for one buffered write stream: make every row up to
flushOffset
(inclusive, per FlushRows semantics) visible in the destination table.Serializer for
BufferedStreamCommittable.Committer of the buffered-stream write path: makes each completed checkpoint's rows visible by
flushing every committable's stream up to its offset (
FlushRows).Options specific to
WriteMethod.STORAGE_API_EXACTLY_ONCE: how large append requests may
grow, how long inactive destinations stay cached, how the connector-driven recovery schedule
backs off, and how the SDK retries retriable append failures in-stream.Builder for
BufferedStreamOptions.Maps the
sink.buffered-stream.* options onto BufferedStreamOptions.The Storage Write API operations of the buffered-stream write path, abstracted so writer and
committer logic can be unit-tested against fakes: creating buffered streams, opening offset-aware
appenders on them, and flushing rows.
Creates
BufferedStreamService instances on the task manager.One destination's buffered-stream writer state: which stream the subtask owns and how many rows
it had appended (acknowledged) as of the checkpoint.
Serializer for
BufferedStreamWriterState.What one subscriber holds buffered and has not yet handed to the reader, as a single consistent
reading.
Decides whether the client library's
BulkWriter retries a failed write attempt, in place
of the library's own default listener.The Flink type the
bigtable catalog gives a table's row key and the keys of its column
family maps, which are the qualifiers.Assertions on option preservation in catalogs that store tables as properties.
A mutation applied by BigQuery change data capture.
Determines whether one record upserts or deletes its BigQuery primary key.
Options for appending BigQuery CDC mutations through the Storage Write API default stream.
Builder for
CdcOptions.Creates the internal write-only fields that represent BigQuery CDC pseudocolumns.
Extracts the BigQuery change sequence number for one record.
Desired BigQuery table contract for CDC writes.
Builder for
CdcTableOptions.Maps Table API CDC table options onto their DataStream contract.
Supplies the desired CDC table contract for each destination.
Policy for a BigQuery CDC destination table that already exists.
Turns a
RowData field, or an element of a MAP family's value array, into the
bytes of a Bigtable cell, and back.Reads one element of an array as the bytes of a cell: a
MAP family's key or value.Reads the bytes of a cell as one field of a row.
Reads one field of a row as the bytes of a cell.
How many cells a mutation costs, counted the way Spanner counts a mutation.
The physical changelog representation emitted by a Bigtable Change Streams table source.
The relational changelog emitted from Spanner data-change records.
Coordinator-side Bigtable operations, separated so partition protocol tests need no service.
Mints the
ChangeStreamCoordinatorClient one enumerator coordinates through.Dispatches a change-stream mutation entry or aggregate value to the connector code that handles
it.
Handles every mutation entry type the connector models.
Handles every aggregate value type the connector models.
Serializable seam that opens asynchronous
ReadChangeStream RPCs.The shape every Bigtable Change Streams partition has, and the conversion to the shape the SDK
models require.
One Bigtable Change Streams partition and the exact position from which it resumes.
One Spanner Change Streams partition and its checkpointed coordinator lifecycle.
Connector-owned checkpoint format for
ChangeStreamPartitionSplit.Connector-owned checkpoint format for a Spanner Change Streams partition split.
Mutable task-thread state for one change-stream partition.
Mutable task-thread progress for one Spanner Change Streams partition.
Resolves a reader-restored split against current retention.
A table option's Bigtable Change Streams start-position mode.
A Table API spelling for one shared Change Streams start position.
Maps one mode/timestamp option pair onto the shared Change Streams start-position API.
Maps Table API startup options to the shared change-stream start position.
A
CheckAndMutateRow: a predicate filter over one row, the mutations to apply when it
matches at least one cell, and the mutations to apply when it matches none.One child-partitions record discovered while a reader is consuming a parent partition.
One child token and the normalized coordinator identities of every parent.
Reads pages through a
google-cloud-datastore client's RPC object.Reads pages through a
google-cloud-firestore client, at the split's read time.Plans a read through a
google-cloud-datastore client and, to cut key ranges, the client
library's QuerySplitter.Plans a read through a
google-cloud-firestore client.Releasing several resources at once, so that one refusing to close never strands the rest.
One immutable checkpoint committable.
Bounded v1 framing for one envelope; the Flink collector owns batch counts and redistribution.
The
WITH options of the cloud-tasks table connector.At-least-once sink creating one Cloud Tasks task per record through the
CreateTask RPC,
with fixed or per-record queue destinations.Delivery modes for Cloud Tasks task creation, distinct from handler execution.
Insert-only table sink creating target tasks in one fixed queue.
Factory for the sink-only
cloud-tasks table connector.The Cloud Tasks emulator image shared by every harness that starts the emulator, so they cannot
drift apart.
Classifies status codes reported by failed Cloud Tasks creations.
Every metric name this connector registers itself, in one place so that this file is the
connector's inventory: what it reports can be read here without opening the writer.
Immutable observation of an owned batch, containing no task payload or durable state.
Serializes sink records into Cloud Tasks tasks.
Entry point for building a Cloud Tasks sink.
Builder for Cloud Tasks sinks, obtained from
CloudTasksSink.builder().Immutable sink configuration assembled by
CloudTasksSinkBuilder.Creates checkpoint-owned envelopes in a blocking, bounded-concurrency loop.
Checkpointed named creation with a stateless staging writer and a bounded committer.
Immutable staging and recovery settings for checkpointed Cloud Tasks creation.
Builder for immutable checkpointed creation settings.
Explicit recovery decisions outside the guarantee after an envelope expires.
Stages named tasks without owning a client or any Flink writer state.
Internal staging settings.
The request target represented by a
cloud-tasks table.A stock
CloudTasksClient over one plaintext channel to the emulator, with no credentials,
for queue administration and task inspection.At-least-once writer creating one Cloud Tasks task per record.
Tuning options for the sink's writer: the in-flight cap, the transport channel pool and the two
retry budgets.
Builder for
CloudTasksWriterOptions.ReaderOutput for tests that drive SourceReaderBase.pollNext(...), delegating to
one shared CollectingSourceOutput whatever split a record came from.SourceOutput recording emitted records and their timestamps — what a
RecordEmitter is handed, and what a test driving one asserts against.The value type of a column family created by the connector.
Immutable selection of cells used as a conditional predicate.
Immutable mutation in an ordered conditional branch.
Immutable row predicate and ordered true/false branches, independent of the destination table.
Internal bridge from conditional models to the shared request runtime.
A successful conditional response, including the destination actually used for this invocation.
Serializes the destination's three UTF-8 components, the row-key bytes and two outcome booleans.
Snapshot for the connector-owned field format.
Supplies the connector-owned serializer for
ConditionalResult.Builds a conditional request.
What the sink does with a mutation Spanner refuses for violating a constraint.
Everything one BigQuery copy job needs, decoupled from the client for testability.
Whether the sink may create destination tables that do not exist.
Whether the sink may create the destination table when it does not exist.
Whether the sink may create destination topics that do not exist.
The cross-version seam every sink in this module implements instead of
Sink directly.The cross-version seam every sink in this module implements instead of
Sink directly.The cross-version seam every sink in this module implements instead of
Sink directly.The cross-version seam every sink in this module implements instead of
Sink directly.The cross-version seam every sink in this module implements instead of
Sink directly.The cross-version seam every sink in this module implements instead of
Sink directly.The cross-version seam every sink in this module implements instead of
Sink directly.A Firestore database in Datastore mode: a project and a database id.
A Firestore database in Native mode: a project and a database id.
A fully-qualified Spanner database reference: project, instance and database.
One data-change record returned by a Spanner Change Streams read function.
Builder for
DataChangeRecord.Per-record description of one watched column.
Serializes
DataChangeRecord without reflective access to JDK implementation fields.Snapshot for the connector-owned field format.
Supplies the connector-owned serializer for
DataChangeRecord.Opens change streams through a lazily created Bigtable data client.
Samples row keys through a
google-cloud-bigtable BigtableDataClient.Opens reads through a
google-cloud-bigtable BigtableDataClient.What the split enumerator checkpoints: whether the read has been planned, and the splits nobody
is holding yet.
Serializer for
DatastoreBatchEnumeratorState.The bounded source over Datastore's
RunQuery, at one snapshot time.Plans the read once — its snapshot time and its queries — and hands the queries out one at a
time.
Builds the Datastore client every direction of the Datastore-mode connector opens.
At-least-once sink applying one entity write per record through non-transactional commits of
google-cloud-datastore.The
WITH options of the datastore table connector.Loads the credentials every direction of the Datastore-mode connector opens its client with.
The three calls the writer makes against a database in Datastore mode, as a seam a test can fake
without the client library's thirty-method interface.
Opens the
DatastoreDatabaseAccess a writer commits through.The
datastore table sink, backed by the DataStream DatastoreSink.Collects the values of the immutable sink.
The
datastore table's bounded scan, backed by the DataStream DatastoreSource:
every entity of the table's kind in its namespace, read in key ranges at one read time; and its
lookup of one entity by key.Creates the
datastore table source and sink from a SQL DDL: the entities of one kind in
one namespace of a database in Datastore mode, a row per entity, the PRIMARY KEY column as the
key's name or id.Turns a Datastore entity into the records a job works with.
A deserialization schema that also reads what the query result carries about the entity: its
version, its create and update times, and the split's read time.
Allocates complete keys from the service (
AllocateIds): ids the allocator never hands out
again, so an entity written under one cannot replace an entity written under another.The lineage identity of a kind, shared by the source and the table sink so that both report the
same resource for it: the Datastore API names the default database with the empty id, and lineage
spells it "(default)".
The
lookup.* options of a datastore table, read and checked when the statement is
planned.Every metric name the Datastore-mode connector registers itself, in one place so that this file
is the connector's inventory: what it reports can be read here without opening the writer.
One entity operation the Datastore sink applies: an upsert, merge, insert or update carrying an
entity, an update of some of an entity's properties, or a delete carrying a key.
The operation a write performs, named after the Datastore API's mutations:
DatastoreMutation.Operation.UPSERT_MERGE is the API's upsert with a property mask, and DatastoreMutation.Operation.UPDATE_PROPERTIES its update with one.Turns a stream record into the
DatastoreMutation the sink applies.Deserializes each entity and records that the split has passed it.
Looks an entity up by key through the generated client's asynchronous call.
Looks an entity up by key, blocking, and reads it as a row.
The
scan.* options of a datastore table, read and checked when the statement is
planned and applied to the source's builder later.Entry point for building a Datastore sink.
Builder for the Datastore sink; created through
DatastoreSink.builder().The immutable configuration
DatastoreSinkBuilder builds and the sink carries to the task
managers.Entry point for building a bounded Datastore source.
Builds a
DatastoreSource.Everything the source was built with, assembled by the builder and carried into the job graph.
Reads the splits this subtask is assigned, asking the enumerator for the next one each time it
finishes one.
The batch reader's own counters, beside the ones Flink registers for every source.
Reads the queries this subtask was assigned, one page per fetch.
The physical identity of a
datastore table, carried from the factory to the runtime
providers it wraps (ADR-0160): the kind the table names, in its namespace and database, reported
under the table's catalog name.A
datastore table's physical schema, checked against what a Datastore entity can hold:
which column is the key, and what Datastore value each column and nested field is.The Datastore sink's writer: buffers entity writes, applies them in non-transactional commits,
and finds out which write a refused commit was refused for.
Tuning options for the sink's writer: how large a commit grows, how long one may take, the
recovery budget the writer spends on transient failures, the ramp-up throttle, and the bound on
consecutive rejections.
Builder for
DatastoreWriterOptions.Destination for elements that terminally failed to be written by a sink, used through
FailureHandler.sendToDeadLetterQueue(DeadLetterQueue).Transcodes Debezium MySQL GTID coordinates into BigQuery CDC sequence sections.
Derives BigQuery CDC sequence numbers from Debezium MySQL source properties.
Transcodes Debezium PostgreSQL sequence metadata into BigQuery CDC hexadecimal sections.
Derives BigQuery CDC sequence numbers from Debezium PostgreSQL source properties.
Derives BigQuery CDC sequence numbers from Debezium Spanner source properties.
Native java-bigtable implementation of the coordinator operations.
Retention-aware reader restore resolution backed by Bigtable table metadata.
Opens the production
DatastoreDatabaseAccess: a client for the configured database.The
FailureHandlerContext the sinks build from their WriterInitContext.Opens the production
FirestoreDatabaseAccess: a Firestore client for the configured
database and one single-thread executor every BulkWriter of the access runs on.Creates
MutationBatchers backed by google-cloud-bigtable BigtableDataClients, connected the way BigtableDataClients connects every data client of
this connector — that class owns the credential and emulator modes.Mints
BatchClientPartitionPlanners.PublisherFactory building google-cloud-pubsub Publisher instances
configured from PubSubPublisherOptions; every knob left unset keeps the SDK default
(options with no batching or retry overrides leave the publisher builder untouched, so the
default configuration is byte-identical to the SDK's).Mints
ClientQueryPlanners.Mints
ClientQueryPlanners.Mints
ReadClientSessionCreators.Mints
DataClientRowKeySamplers.The production
SingleRowClientFactory: one BigtableDataClient per (project,
instance), shared by that instance's tables, with both request-response RPCs pinned to a single
attempt under the configured deadline.Opens the Spanner service handle used for change-stream metadata discovery.
Opens the Spanner client and bounded callback executor owned by one reader subtask.
Opens a real Spanner service handle and wraps it in a
SpannerServiceAdapter.Options specific to
WriteMethod.STORAGE_API_AT_LEAST_ONCE: how large append requests may
grow, how the connector-driven re-append budget backs off, how the SDK retries retriable append
failures in-stream, how the SDK's connection pool (multiplexing) is sized, and the writer's
housekeeping (destinationIdleTimeout, flushInterval).Builder for
DefaultStreamOptions.Maps the
sink.default-stream.* options onto DefaultStreamOptions.SubscriberFactory building google-cloud-pubsub Subscriber instances
configured from PubSubSubscriberOptions; every knob left unset keeps the SDK default, so
the default configuration differs from the SDK's only in the two settings the source owns.Creates a
TaskCreator backed by a google-cloud-tasks CloudTasksClient,
connecting either to production Cloud Tasks with application-default or configured
service-account credentials, or to an emulator over a plaintext channel with no credentials.What the source does with a message its deserialization schema cannot convert.
Runs independent destination actions with a fixed, committer-lifetime concurrency bound.
One worker's thread-confined BigQuery collaborators.
Builds one worker context, lazily on the thread that owns it.
Optional per-destination send counters, as
destination.NAME.recordsSend and
destination.NAME.sendErrors on the writer's metric group, where NAME is the destination
as the connector's docs page spells it.One destination's counters; a no-op instance when per-destination metrics are off.
The result of resolving a BigQuery destination for one record.
Dispatches destination-resolution results to the writer that owns the record.
Handles every destination-resolution result supported by the connector.
Resolves the destination table for each record, enabling one sink instance to write to many
tables (dynamic destinations).
Resolves the destination table for each record, enabling one sink instance to write to many
tables (dynamic destinations).
Resolves the destination queue for each record, enabling one sink instance to write to many
queues (dynamic destinations).
Resolves the destination topic for each record, enabling one sink instance to publish to many
topics (dynamic destinations).
Deadline-bounded draining of a running job's collect iterator.
Policy for a successful conditional request whose selected mutation list is empty.
Transport channel providers pointing a gRPC client at an emulator.
The
host:port of a Google Cloud emulator, parsed and validated on the client rather than
on a TaskManager after the job has been submitted (issue #235).A write with the protobuf form a commit sends, encoded once when the write enters the buffer and
reused by every commit that carries it, a retry and a solo re-send included.
What a query result carries about an entity beside the entity itself, and the read time of the
split that read it: none of it is on the client library's
Entity.The three counters a
PullAssignmentSplitEnumerator reports, registered by the connector
under its own names.Counts a sink's failures by the status code that caused them, as
errorClass.CODE.errors
on the writer's metric group.An element that terminally failed to be written by one of this project's sinks — the read-only
contract every connector's concrete failure type implements, and the view a cross-connector
DeadLetterQueue sees.A single message that terminally failed to be published to Pub/Sub, as passed to a
FailureHandler<FailedMessage>.A single row mutation that terminally failed to be written to Bigtable, as passed to a
FailureHandler<FailedMutation>.A single entity mutation that terminally failed, as passed to a
FailureHandler<FailedMutation>.A single mutation that terminally failed to be applied to Spanner, as passed to a
FailureHandler<FailedMutation>.A single-row request that Bigtable rejected as invalid, or that could not be built from its
record, as passed to a
FailureHandler<FailedRequest> on the sink
surface.A single row that terminally failed to be written to BigQuery, as passed to a
FailureHandler<BigQueryFailure>.A single task that terminally failed to be created in Cloud Tasks, as passed to a
FailureHandler<FailedTask>.A single document write that terminally failed, as passed to a
FailureHandler<FailedWrite>.Pluggable policy for elements that terminally fail to be written by a sink.
What a
FailureHandler or DeadLetterQueue learns about its surroundings when it is
opened: enough to stamp per-subtask output and to register its own metrics, and nothing more.A queue-backed
MailboxExecutor for tests: FakeMailboxExecutor.execute(org.apache.flink.api.common.operators.MailboxExecutor.MailOptions, org.apache.flink.util.function.ThrowingRunnable<? extends java.lang.Exception>, java.lang.String, java.lang.Object...) enqueues mails, FakeMailboxExecutor.yield() runs the next mail (blocking until one arrives, like the real mailbox), FakeMailboxExecutor.drain() runs every mail already enqueued, and FakeMailboxExecutor.quiesce() makes every later FakeMailboxExecutor.execute(org.apache.flink.api.common.operators.MailboxExecutor.MailOptions, org.apache.flink.util.function.ThrowingRunnable<? extends java.lang.Exception>, java.lang.String, java.lang.Object...) throw the RejectedExecutionException the real mailbox throws once the task has
quiesced it, which happens before the operators close.In-memory
SourceReaderContext for the tests of a pull-assignment source reader.FakeSplitEnumeratorContext<SplitT extends org.apache.flink.api.connector.source.SourceSplit>
In-memory
SplitEnumeratorContext for the tests of a pull-assignment enumerator —
one whose readers ask for work through handleSplitRequest.A document the split reader handed to the task thread, with the split's query it was read under.
An entity the split reader handed to the task thread, with the cursor the service returned after
it and what the query result carried beside it.
The streaming pre-commit stage of the FILE_LOADS topology: stamps each committable with the
checkpoint id carried by its
CommittableMessage, which the Committer SPI cannot observe otherwise — the stamp selects
the streaming behavior of the load-job orchestrator and makes BigQuery job ids attributable to
their checkpoint.One finalized staging file: the Flink job id of the run that staged it, its destination table,
its Cloud Storage URI, size counters used for load-job partitioning, and — in streaming execution
— the checkpoint that triggered it.
Serializer for
FileLoadsCommittable.The real FILE_LOADS commit: turns one committable batch — a whole batch run, or one checkpoint of
a streaming run — into BigQuery load jobs through a
LoadJobOrchestrator.Thread-safe metrics for the bounded FILE_LOADS destination executor.
Options specific to
WriteMethod.FILE_LOADS: where staging files go on Cloud Storage, how
loaded rows land in tables that already hold data, where oversized loads stage their temporary
tables, how checkpoint-triggered loads are paced in streaming execution, and the two schedules
the committer backs off on.Builder for
FileLoadsOptions.Maps the
sink.file-loads.* options onto FileLoadsOptions.The FILE_LOADS
SinkWriter: encodes records in the
configured staging format and streams them to
per-destination staging files on Cloud Storage, emitting one FileLoadsCommittable per
finalized file from FileLoadsWriter.prepareCommit() — once at end of input in batch execution, once per
checkpoint in streaming execution.What the split enumerator checkpoints: whether the read has been planned, and the splits nobody
is holding yet.
Serializer for
FirestoreBatchEnumeratorState.The bounded source over Firestore's
RunQuery, at one snapshot time.Plans the read once — its snapshot time and its queries — and hands the queries out one at a
time.
At-least-once sink applying one document write per record through the
BulkWriter of
google-cloud-firestore.Builds the Firestore client every direction of this connector opens.
The
WITH options of the firestore table connector.Loads the credentials every direction of the Firestore connector opens its client with.
The narrow view of the Firestore client the writer needs: hand a write to the current
BulkWriter, send what it holds, and replace it.Opens the
FirestoreDatabaseAccess a writer sends through.Turns a Firestore document into the records a job works with.
A field value that refers to a document, written as a Firestore reference value.
The
firestore table sink, backed by the DataStream FirestoreSink.Collects the values of the immutable sink.
The
firestore table's bounded scan, backed by the DataStream FirestoreSource, and
its lookup by document id.Creates the
firestore table source and sink from a SQL DDL: the documents of one
collection, or a scan of every collection with its id, a row per document, the PRIMARY KEY column
as the document id.The Firestore emulator image shared by every harness that starts the emulator, so they cannot
drift apart.
The
lookup.* options of a firestore table, read and checked when the statement is
planned.Every metric name this connector registers itself, in one place so that this file is the
connector's inventory: what it reports can be read here without opening the writer.
Builds the query a
FirestoreSource reads, from the client the source opened.Deserializes each document and records that the split has passed it.
Looks a document up by id through the client library's asynchronous API.
Looks a document up by id, blocking, and reads it as a row.
The
scan.* options of a firestore table, read and checked when the statement is
planned and applied to the source's builder later.Entry point for building a Firestore sink.
Builder for the Firestore sink; created through
FirestoreSink.builder().The immutable configuration
FirestoreSinkBuilder builds and the sink carries to the task
managers.Entry point for building a bounded Firestore source.
Builds a
FirestoreSource.Everything the source was built with, assembled by the builder and carried into the job graph.
Reads the splits this subtask is assigned, asking the enumerator for the next one each time it
finishes one.
The batch reader's own counters, beside the ones Flink registers for every source.
Reads the queries this subtask was assigned, one page per fetch.
The physical identity of a
firestore table, carried from the factory to the runtime
providers it wraps (ADR-0160): the collection the table names, in its database, or for a
collection-group scan the group, reported under the table's catalog name.A
firestore table's physical schema, checked against what a Firestore document can hold:
which column is the document id, and what Firestore value each column and nested field is.One document operation, as a
FirestoreWriteSerializationSchema returns it for a record.The document operation a write performs.
At-least-once writer applying one document write per record through the client library's
BulkWriter.Tuning options for the sink's writer: the client library's throttling, its two layers of retries,
and the writer's own bounds on unacknowledged writes.
Builder for
FirestoreWriterOptions.Turns a stream record into the
FirestoreWrite the sink applies.A
DestinationResolver returning one fixed destination for every record.A
DestinationResolver returning one fixed destination for every record, which is what
BigtableSinkBuilder.table(...) builds.A
DestinationResolver returning one fixed destination for every record.A
DestinationResolver returning one fixed destination for every record.Table format factory for UTF-8
application/x-www-form-urlencoded bodies.A garbage-collection rule for a column family the sink creates, mirroring the four rule shapes
Bigtable's admin API takes: a version cap, an age cap, and their union and intersection.
The rule shapes Bigtable's admin API takes.
StagingStorage over the Cloud Storage client with configured credentials or ADC.An encoding format that owns the HTTP Content-Type for body-carrying requests.
The first stage of
CloudTasksSerializationSchema.httpTarget(String): it carries the
target URL and waits for the body schema.A
CloudTasksSerializationSchema producing HTTP-target tasks: a URL, an HTTP method, an
optional authorization token, optional headers, and a body serialized by a wrapped Flink SerializationSchema.Extracts HTTP headers from a record.
Extracts the target URL from a record.
Validated fixed HTTP request values used by the table serializer.
What the Bigtable table sink tells the planner when the requested changelog is insert-only.
Options controlling how JSON records are converted to BigQuery rows.
Builder for
JsonDocumentOptions.A
BigQueryProtoSerializationSchema for records that are JSON documents, as text.A serialization schema that writes entities under keys the service allocates, given the writer's
allocator, which reaches the database through the writer's own client, credentials and recovery
budget.
A
BigtableDataClient built on first use and closed once, held by a seam implementation
that travels in the job graph.A Datastore client built on first use and closed once, held by a seam implementation of the
source: the enumerator's planner and the readers' page reader.
Holds state its owner derives from its own serializable fields, once, and again on a task
manager.
A Firestore client built on first use and closed once, held by a seam implementation of the
source: the enumerator's planner and the readers' page reader.
The length-prefixed fields the connector-owned
TypeSerializers share: a byte string or a
UTF-8 string as an int length followed by the bytes, and a non-negative int
count.Pure metadata assembly shared by connector runtimes; no client or callback is accepted.
Canonical names from already parsed resource components.
Captures a real Flink 2.x job-created event through the configured listener factory.
Instantiated by Flink from the listener configuration, not called directly by tests.
Immutable logical Table identity and DataStream/Table selection for runtime configuration.
Runs the staged files of one run — a whole batch job, or one checkpoint of a streaming job — into
their destination tables: submits every load, copy and terminal query through a
LoadJobRunner, then deletes the temporary tables and the staged objects.Executes BigQuery load, copy and terminal query jobs for the FILE_LOADS orchestration.
Everything one BigQuery load job needs, decoupled from the client for testability.
Captures what a class logs, so a test can assert the line rather than only its side effects.
One collected log event, flattened out of log4j2's own type.
The levels a capture can be taken at, so no log4j2 type reaches this class's signature.
Separates an addressing key from additional equality keys on the produced physical row.
Public because Flink requires the provided function class to be public.
Public because Flink requires the provided function class to be public.
Publishes every container's ports on the loopback address the Docker host resolves to, rather
than on the wildcard address, so a host-port collision with an unrelated local process cannot
form (ADR-0132).
What the Bigtable table sink does with a
MAP family's qualifiers a row does not carry.A materialised Bigtable row and the stable byte estimate measured while it was decoded.
Materialises SDK rows while measuring their decoded wire content without traversing them twice.
Shared boundary handling for connector metric values.
Fails a reader that holds unacknowledged messages but has never been asked for a checkpoint.
Checkpointed timer for a service partition absent from the coordinator ledger.
JSON values describing the changes to one watched table row.
The operation represented by one Spanner change-stream modification.
Accepts row mutations for one table and reports each one's outcome separately.
Creates one
MutationBatcher per table the writer resolves, and owns whatever those
batchers share.Appends row batches to one buffered write stream at explicit offsets.
The checks every connector's option builders repeat, in one place.
Assertions shared by the Table API option-set tests.
Applies Table API option values to DataStream builder setters, renaming a rejected value's
failure to the option key the SQL caller actually wrote (issue #1030, ADR-0133).
Whether the source preserves Pub/Sub ordering-key delivery order.
How
StagingFormat.PARQUET staging files are compressed.Reports successful end-of-query after all child-partitions records were forwarded.
Lifecycle of one Spanner Change Streams partition in the checkpointed coordinator ledger.
What one planning call answered with: a snapshot, and the partitions of the read at it.
Opens a batch read and asks the service to plan it into partitions.
Mints the
PartitionPlanner one enumerator plans through.Reader heartbeat progress used to keep the coordinator's recovery watermark current.
Reader progress used to keep the coordinator's assigned-partition ledger current.
Reports that a partition closed and names each successor range and its continuation token.
One target range from CloseStream paired with the token that arrived from this parent.
How much a paused split may buffer before the reader stops its subscriber.
Successor partition parked until its parent tokens cover the whole target range.
Immutable physical resource metadata under the
gcp dataset facet key.A split as the planner produced it, with the size the samples suggest it holds.
What the sink does with a write Firestore refuses because its
lastUpdateTime precondition
no longer holds.The Datastore API's
PropertyMask path grammar, shared by the lookup's mask and a merge's:
a path is property names joined by dots, and a name holding a dot or a backslash must be escaped
(datastore.proto).Adds fields to a protobuf row descriptor without changing its surrounding file structure.
ProtoMessageSerializationSchema<T extends com.google.protobuf.Message>
A
BigQueryProtoSerializationSchema for records that already are protobuf messages.One validated field consumed by the generic protobuf row-augmentation engine.
Serializable provider whose validation may report a row-level
IOException.Write-only fields accepted by BigQuery's documented Storage Write API contract.
Adds ordered physical or write-only fields to delegate protobuf rows.
Rewrites protobuf messages into
DynamicMessages conforming to a BigQuery-storage
compatible target descriptor (as produced by BQTableSchemaToProtoDescriptor from the
schema derived by ProtoToTableSchemaConverter).Options controlling how protobuf descriptors are mapped to BigQuery schemas.
Builder for
ProtoSchemaOptions.Converts serialized proto rows (the
BigQueryProtoSerializationSchema wire
form, whose descriptors follow the BQTableSchemaToProtoDescriptor conventions) into Avro
GenericRecords conforming to the schema produced by TableSchemaToAvroConverter.Derives a BigQuery
TableSchema from a protobuf Descriptors.Descriptor.Creates
TopicPublisher instances for the writer's per-topic publisher map.Builds
PubSubPublisherOptions from the table options.AckTracker keeping the acknowledgement handles in memory until their checkpoint
completes.The
WITH options of the pubsub table connector.Loads the credentials shared by the Pub/Sub source and sink client families.
A
DeadLetterQueue publishing every terminally failed element to a Pub/Sub topic, used
through FailureHandler.sendToDeadLetterQueue(DeadLetterQueue).Builder for
PubSubDeadLetterQueue.Deserializes Pub/Sub messages into source records.
A table sink publishing rows to one Pub/Sub topic.
A table source consuming rows from one or more Pub/Sub subscriptions.
The
pubsub table connector factory.The Pub/Sub emulator image shared by every harness that starts the emulator, so they cannot drift
apart.
Checkpointed state of the split enumerator: the subscriptions it resolved.
Serializer for
PubSubEnumeratorState.Every metric name this connector registers itself, in one place so that this file is the
connector's inventory: what it reports can be read here without opening a writer, a reader or the
enumerator.
Tuning options for the sink's Pub/Sub publishers and its writer: SDK batching and publish-retry
settings, message ordering, the writer's in-flight caps, the backoff budget of the topic
auto-creation recovery, and the writer's shutdown budget.
Builder for
PubSubPublisherOptions.At-least-once sink publishing through
google-cloud-pubsub Publisher instances
with dynamic per-record topic destinations.Deserializes received messages and emits them, using the Pub/Sub publish time as the event
timestamp.
Serializes sink records into Pub/Sub messages.
Extracts message attributes from a record.
Extracts the ordering key from a record.
What this connector's bounded teardowns leave behind, counted for the lifetime of the class
loader rather than of a task.
Entry point for building a Pub/Sub sink.
Builder for Pub/Sub sinks, obtained from
PubSubSink.builder().Immutable sink configuration assembled by
PubSubSinkBuilder.Entry point for building Pub/Sub sources.
Builder for Pub/Sub sources, obtained from
PubSubSource.builder().Immutable configuration of a Pub/Sub source, assembled by
PubSubSourceBuilder and shipped
in the job graph.Reader subtask consuming its assigned subscription splits.
The reader's Pub/Sub-specific metrics.
Verifies the subscriptions, then assigns subscription splits to reader subtasks.
Multiplexes the streaming-pull subscribers of one reader subtask's splits.
Fetch-loop helpers for driving a Pub/Sub
SplitReader directly in tests, typed against the
flink-connector-base interface so the connector's reader class never crosses the module
boundary.Where the source starts consuming a subscription.
Which starting point a
PubSubStartPosition names.At-least-once source consuming Pub/Sub subscriptions through
google-cloud-pubsub
Subscriber streaming pull.Tuning options for the source's Pub/Sub subscribers and its reader: SDK flow control, the
streaming-pull connection count and the acknowledgement-deadline extension settings, plus the
source's own drain size, subscriber shutdown budget and first-checkpoint watchdog.
Builder for
PubSubSubscriberOptions.Default
SubscriptionAdmin backed by the Pub/Sub SubscriptionAdminClient.The Pub/Sub admin, publish and pull machinery every integration-test harness needs, parameterised
over the transport: a plaintext channel with no credentials against the emulator, or
application-default credentials against the real service.
Default
TopicAdmin backed by the Pub/Sub TopicAdminClient.At-least-once writer publishing records to dynamic per-record Pub/Sub topic destinations.
PullAssignmentSplitEnumerator<SplitT extends org.apache.flink.api.connector.source.SourceSplit,StateT,PlanT>
A bounded source's split enumerator that plans once, asynchronously, and then hands the plan out
one split per request.
Bridges Pub/Sub's push-style streaming pull to the pull-style
SplitReader contract: the client
library delivers messages on its own threads into an in-memory buffer, and the fetcher thread
drains that buffer.Continues a query after a document, the one operation both the split reader's paging and a
checkpoint's resume point are made of.
The deterministic identity under which a query job may be reused across a re-plan.
Everything one terminal BigQuery query job needs, decoupled from the client for testability.
Reads one page of a query: one
RunQuery call.Reads one page of a query at a snapshot time.
What planning answers with: the snapshot time, the queries every split reads at it, and why the
query was not split when it was not.
What planning answers with: the snapshot time and the queries every split reads at it.
Plans a read: picks its snapshot time and cuts it into queries.
Plans a read: picks its snapshot time and cuts it into queries.
Mints the
QueryPlanner one enumerator plans with.Mints the
QueryPlanner one enumerator plans with.What running the query produced: the table the result landed in, and whether it came from a
previous attempt's job rather than a fresh submission.
Runs the source's query and says which table its result landed in.
The query job a
QueryRunner is asked to run.One query still to be read, at the job's snapshot time.
One query still to be read, at the job's snapshot time.
Serializer for
QuerySplit.Serializer for
QuerySplit.How far through one
QuerySplit the source has emitted.How far through one
QuerySplit the source has emitted.A fully-qualified Cloud Tasks queue reference: project, location and queue.
Opens
ReadRows calls through a BigQueryReadClient.Creates read sessions through a
BigQueryReadClient.Absolute number of additional partition splits one reader can accept.
Adapts a
SourceReaderContext to the context a DeserializationSchema expects when
a source opens it.Immutable row key and ordered rules, independent of the resolved destination.
Internal bridge from read-modify-write models to the shared request runtime.
Successful response containing the resolved destination and the final changed cells.
Serializes the destination's three UTF-8 components followed by the changed row.
Snapshot for the connector-owned field format.
Supplies the connector-owned serializer for
ReadModifyWriteResult.A
ReadModifyWriteRow: one or more append or increment rules over one row, applied
atomically in order.One rule of a
ReadModifyWriteRow: an append of bytes to a cell, or an increment of a
cell holding a 64-bit big-endian signed integer (an unset cell counts as empty or zero).One immutable append or signed increment, applied in its request's list order.
Checks the operand types before a read-modify-write sink reaches a TaskManager.
Builds a read-modify-write request.
A service database as a Flink catalog database: its description, if the service keeps one, is the
comment.
Creates the source's read session.
Mints the
ReadSessionCreator one enumerator plans through.One Storage Read API
ReadStream, plus how far into it the source has already consumed.Serializer for
ReadStreamSplit.The mutable reading progress of one
ReadStreamSplit.Runs a query at a read time: the one read both the planner's probe and the readers' pages make.
REST plumbing shared by the ITCases gated on real-GCP credentials: the
BQ_IT_* variables,
a REST client over application-default credentials, table creation/deletion and query read-back.Cloud Storage plumbing for the gated FILE_LOADS ITCases: the
BQ_IT_GCS_BUCKET variable, a
client over the same application-default credentials RealBigQuery uses, and
staging-prefix upload, listing and cleanup.The failure boundary of a single-row request: which failures concern the row alone, which leave
the service's state unknown, and what each is reported as.
The classes a failed request falls into.
Maps single-row runtime options, leaving absent values at their builder defaults.
A read-only
Counter view of one PubSubShutdownResidue adder, so a residue
registers as the counter it is rather than as a gauge over a monotonic total.A configured physical resource reported in a
PhysicalResourceFacet.The check a configured value earns by being a component of a resource path this project composes.
A
TableAdmin failure that repeating the same call can fix — a rate limit, a quota that
refills, a server-side error.Support for the connectors' hand-written retry loops.
A
TableAdmin that repeats a creation, or a temporary table's preparation, the service
answered with a RetriableTableAdminException, within a fixed budget.An immutable retry schedule: exponential backoff from an initial delay up to a cap, a bounded
number of attempts, and optional proportional jitter (to de-synchronize parallel subtasks
retrying against the same destination).
Appends batches of serialized rows to one destination's default write stream.
Creates
RowAppenders for destinations.How
RowTypeToTableSchemaConverter derives a BigQuery schema from a SQL table's columns —
the RowData counterpart of AvroSchemaOptions, carrying the same three knobs under
the same names.Builder for
RowDataSchemaOptions.Turns a table row into the entity write the
datastore table sink sends.Turns a table row into the document write the
firestore table sink sends.Serializes table-runtime rows into mutations for one configured table.
Derives the protobuf row descriptor a BigQuery table schema is written through.
How a Bigtable table option represents a row-key bound.
One row key the service offered as a section boundary, with the approximate number of bytes
stored before it.
Asks the service where a table's sections begin.
Mints the
RowKeySampler one enumerator plans against.The single-row request-response RPCs of Bigtable that this family issues, named as the service
names them.
The module's row-key range algebra: emptiness, containment, cutting, coalescing, intersection,
truncation and the unsigned key comparison they all rest on, each defined once here.
One contiguous row-key range still to be read.
Cuts the configured row-key ranges at the boundaries the service sampled.
Serializer for
RowRangeSplit.How far through one
RowRangeSplit the source has emitted.One single-row request, built from a record and started against a client once its destination is
known.
Turns a record into the single-row request the sink surface sends for it.
One open
ReadRows call: the response blocks of one stream, from one offset onwards.One open
ReadRows call, consumed a row at a time.Opens
ReadRows calls on a read session's streams.Opens a
ReadRows call over one row-key range.Which Bigtable read surface a table source uses.
The Spanner read path selected by a table DDL.
Computes the union of a destination table's live schema and the serializer's desired schema — the
schema the sink proposes when connector-driven schema updates are enabled.
A checked rejection of a schema union: the desired schema requires a change the rules (or the
configured options) do not permit.
The result of a union: the unified schema and whether it differs from the existing one.
Options gating connector-driven destination table schema updates.
Builder for
SchemaUpdateOptions.The physical primary key and value-format payload of a selected-cell table.
Creates synthetic service-account keys for credential wiring tests: structurally valid enough for
the Google auth libraries to parse — a real RSA key in a real PEM envelope — while naming an
example.invalid account no service would ever accept, so a test that leaks one at a real
endpoint fails loudly rather than authenticating.Loads the service-account key file a connector is configured with.
Locates the uber-jar a
flink-sql-connector-gcp-* module builds, for the tests that
inspect or run against it.The two request-response RPCs of a Bigtable data client, as the single-row runtime issues them.
Leases
SingleRowClients per table over instance-shared data clients, the single-row
family's counterpart of the batching sink's MutationBatcherFactory.Immutable configuration of a single-row request sink: what the per-operation sink builders
assemble and the runtime reads.
The single-row runtime's counters and gauges, registered on a sink writer's group or an async
function's operator group.
The
createWriter path the per-operation single-row sinks share: credentials, the client
factory, the serializer's and the handler's open, and the release of all three when the
writer cannot be built.At-least-once writer sending one single-row request per record to the Bigtable tables a
DestinationResolver names, and discarding the answers.What the split enumerator checkpoints: whether the read has been planned, and the splits nobody
is holding yet.
Serializer for
SpannerBatchReadEnumeratorState.The bounded batch source over Spanner's partitioned reads.
Opens one batch read, asks Spanner to plan it into partitions, and hands the partitions out one
at a time.
Creates the
spanner catalog from a CREATE CATALOG statement's options.The
WITH options of the spanner catalog, created by CREATE CATALOG ...
WITH ('type' = 'spanner', ...).Builds BigQuery CDC sequence numbers from typed Spanner change-stream coordinates.
Transcodes Spanner change-stream coordinates into BigQuery CDC hexadecimal sections.
Metadata operations the Spanner Change Streams coordinator needs during initialization.
Serializable job-graph seam that opens the coordinator's runtime metadata client.
Turns one Spanner data-change record into zero or more user records.
An unbounded Table API source for one table observed through a Spanner change stream.
Checkpointed Spanner Change Streams partition state.
Connector-owned checkpoint format for the Spanner Change Streams partition ledger.
Releases readers after the coordinator has validated Change Stream metadata and restore age.
Opens asynchronous partition queries for one source subtask.
Serializable factory for the TaskManager-local Change Streams query client.
Reads several Spanner Change Streams partition queries concurrently in one source subtask.
Applies connector-side table and column filters to decoded data-change records.
Result of applying table and column filters to one data-change record.
FLIP-27 source for Cloud Spanner Change Streams.
Builds a
SpannerChangeStreamSource.Immutable configuration assembled by
SpannerChangeStreamSourceBuilder.Coordinates the checkpointed parent-child lifecycle of Spanner Change Streams partitions.
Advances every reader subtask to the coordinator's unfinished-ledger watermark.
Calculates the non-early Flink watermark for the complete Spanner partition ledger.
Builds the Spanner service handle both directions of this connector open.
The
WITH options of the spanner table connector.Loads credentials shared by every direction of this connector: the sink writer, the bounded
source's planner and readers, the Change Streams coordinator and query clients, and the table
lookup path.
The narrow view of a Spanner database the writer needs: read the schema once, then apply batches.
Receives the outcome of one mutation group of a batch write.
Opens a
SpannerDatabaseAccess on the task manager.The
spanner table sink, backed by the DataStream SpannerSink.Collects values for the immutable sink.
A bounded table scan and primary-key point-lookup source for Spanner tables.
Creates the
spanner table source and sink from a SQL DDL.The Spanner emulator image shared by every harness that starts the emulator, so they cannot drift
apart.
Lookup options owned by the Spanner table source.
Every metric name this connector registers, in one place.
Turns a stream record into the Spanner
Mutation the sink writes.At-least-once sink applying one mutation per record through
batchWriteAtLeastOnce of
google-cloud-spanner.A table's schema and name as one Flink object name: the default schema's tables unqualified, a
named schema's as
schema.table, each part in the dialect's canonical quoting.What the batch source reads: either a query, or a table with its columns and key set, optionally
through an index.
Internal bridge between a table-layer deferred read and the batch partition planner.
Resolves a table-layer read after the batch transaction has selected its snapshot.
Deserializes each row and hands its output records on.
Asynchronous primary-key lookup producing the projected table row.
Synchronous primary-key lookup producing the projected table row.
The priority Spanner schedules this connector's requests at.
Entry point for building a Spanner sink.
Builder for the Spanner sink; created through
SpannerSink.builder().The immutable configuration
SpannerSinkBuilder builds and the sink carries to the task
managers.Reads rows from a Cloud Spanner database.
Builds a
SpannerSource.Everything the batch source was built with, assembled by the builder and carried into the job
graph.
Reads the partitions this subtask is assigned, asking the enumerator for the next one each time
it finishes one.
The batch reader's own counters, beside the ones Flink registers for every source.
Reads the partitions this subtask was assigned, a bounded batch at a time.
Turns a Spanner row into zero or more output records.
Immutable physical identities carried from the Table factory to its runtime providers.
A table name as rendered for Spanner APIs and compared with
INFORMATION_SCHEMA.The name of one table-local access path, such as an index.
The physical DDL schema and its corresponding native Spanner types.
One physical column.
Stock Spanner clients pointed at an emulator endpoint.
The Spanner sink's writer: buffers mutations, applies them with
batchWriteAtLeastOnce,
and decides per mutation what a refusal means.Tuning options for the sink's writer: how large a batch write request grows, how Spanner should
schedule it, and the recovery budget the writer spends on transient failures.
Builder for
SpannerWriterOptions.The split universe of a job and its subtask ownership, computed deterministically from the
subscription list, the ordering mode and the source parallelism.
Decides whether a query can be cut into key ranges, and builds the query a kind scan reads.
Maps checkpoint-owned delivery options without performing remote service validation.
Reads the metadata that authorizes a staged target send; injectable for deterministic failures.
The file format
FILE_LOADS stages rows in before loading them, set by
FileLoadsOptions.Builder#stagingFormat.The Cloud Storage staging area of the FILE_LOADS write method.
Where a change-stream source starts reading when no checkpointed state is restored.
Builds a
PubSubStartPosition from the table options.Resolves change-stream start positions against one startup instant and one retention window.
The retained-window evidence for one expired restored partition.
Discovers how long a connector's change stream retains records.
Extraction of gRPC status codes from the exceptions the Google Cloud clients surface.
Converts a Storage API
TableSchema (the schema form of the serializer SPI) into the REST
client Schema needed to create tables.Default
RowAppenderFactory backed by Storage Write API StreamWriters on the
destination's default write stream.One open read of a partition, consumed a row at a time.
Rejoins a batch read's snapshot and opens one of its partitions.
The parts of
WriterInitContext a sink reads when creating a writer; everything a sink has
no reason to touch is unsupported, so a new dependency on the context shows up as a failing test
rather than as a silent null.Reports that one source reader could not admit a delivery within its subscriber-buffer bound.
Creates the
google-cloud-pubsub Subscriber backing one split.Receives a message together with the handle that settles it.
Builds
PubSubSubscriberOptions from the table options.Subscription administration used by the enumerator's startup check, abstracting the Pub/Sub admin
client so the check can be unit-tested without one.
Settings the source applies when it creates a subscription that does not exist.
Builder for
SubscriptionCreateOptions.Builds per-subscription
SubscriptionCreateOptions from the table options.A fully-qualified Pub/Sub subscription reference: project and subscription.
The settings of an existing subscription that the source's preflight reads.
Builder for
SubscriptionInfo.A unit of work for one reader subtask: a streaming-pull connection to one subscription.
Serializer for
SubscriptionSplit.Runs one source deserialization call with a direct, non-null collector.
One deserialization call that emits through the supplied collector.
Administers destination tables: creation for
CreateDisposition.CREATE_IF_NEEDED and schema
reads/updates for schema evolution, and the FILE_LOADS temporary tables laid out for a
partitioned or clustered destination, which are created here rather than by the jobs that fill
them so they share the creation retry RetryingTableAdmin gives.Table administration operations used by the sink, abstracting the Bigtable admin client so writer
logic can be unit-tested without one.
What an
TableAdmin.ensureTable(io.github.flink.gcp.connector.bigtable.TableDestination, io.github.flink.gcp.connector.bigtable.sink.TableCreateOptions) call actually did, so the caller can report creation and
family-addition separately without this SPI knowing about metrics.A table-administration failure that retains whether the operation requested creation.
Options applied when the sink creates a destination table under
CreateDisposition.CREATE_IF_NEEDED: time partitioning, clustering, and the table's description
and labels.The settings for the table the sink creates under
CreateDisposition.CREATE_IF_NEEDED: its
column families, their value types and optional garbage-collection rules.Builder for
TableCreateOptions.Builder for
TableCreateOptions.Granularity of time-based partitioning.
Maps the
sink.table-create.* options onto TableCreateOptions.Builds
TableCreateOptions from the table options and the DDL's column families.Supplies per-destination
TableCreateOptions for tables created under CreateDisposition.CREATE_IF_NEEDED.A fully-qualified BigQuery table reference: project, dataset and table.
A fully-qualified Bigtable table reference: project, instance and table.
The parts of a table's storage layout a copy job requires its sources to share: time or integer
range partitioning, and clustering.
A Table sink's runtime
Sink, reporting Lineage.tableSink(String, String, List)
under the table's catalog name (ADR-0160).A Table source's runtime
Source, reporting Lineage.tableSource(String, String, Boundedness, List) under the table's catalog name (ADR-0160).A point-in-time read of a destination table's schema, as returned by
TableAdmin.getSchema(io.github.flink.gcp.connector.bigquery.sink.TableDestination).Converts a Storage API
TableSchema (the schema form of the serializer SPI) into the Avro
schema staged files are written with.Validated fixed target values used by the table sink.
Creates Cloud Tasks tasks, one RPC per task.
Creates the writer's
TaskCreator.Extracts the deduplication key of a record, opting the sink into named tasks.
Shared test stubs for
SinkWriter.Context.Random resource-name helpers for tests whose resources share a namespace — the gated real-GCP
suites above all, where concurrent runs must not collide on resource names and a crashed run must
leave behind names that identify it, but also an emulator harness giving each test a database of
its own.
A
SinkCommitterMetricGroup whose metrics can be read back by the names they registered
under — TestSinkCommitterMetricGroup is to a committer what TestSinkWriterMetricGroup is to a writer, and for the same reason:
UnregisteredMetricsGroup.createSinkCommitterMetricGroup() hands out a fresh
SimpleCounter per call, so a counter the committer registered is unreachable afterwards.A
SinkWriterMetricGroup whose metrics can be read back by the names they registered
under, so a sink writer's metrics are asserted the way a reporter would see them.Transcodes a TiCDC commit TSO into a BigQuery CDC sequence section.
Derives BigQuery CDC sequence numbers from TiCDC's Debezium-protocol source properties.
The writer's view of time: reading the clock and waiting out a retry backoff.
Topic administration operations used by the sink, abstracting the Pub/Sub admin client so writer
logic can be unit-tested without one.
Settings the sink applies when it creates a topic that does not exist.
Builder for
TopicCreateOptions.Builds
TopicCreateOptions from the table options.A fully-qualified Pub/Sub topic reference: project and topic.
Publishes messages to one Pub/Sub topic.
What a fixed-width decoder does with a cell or row key longer than the declared type's layout.
What
type-mismatch-policy does with a stored value whose Datastore type does not match
the declared column: a kind has no schema, so its entities can hold, under one property name,
values the table's type cannot represent.What
type-mismatch-policy does with a stored value whose Firestore type does not match
the declared column: a schemaless collection can hold, under one field name, values the table's
type cannot represent.An explicit, record-specific destination-resolution failure.
Parses the table layer's string carrier as a native Spanner UUID value.
The value-capture policy that was active when a data change was recorded.
Production
BufferedStreamService over one BigQueryWriteClient.Default
BufferedStreamServiceFactory creating WriteClientBufferedStreamServices.How data lands in a destination table that already contains data.
The mechanism used to write records to BigQuery.
Destination-side write operation selected by sink.write-mode.
The entity write
sink.write-mode selects for a table with a PRIMARY KEY.The document operation
sink.write-mode selects for a table with a PRIMARY KEY.Refuses explicit options that the selected write operation cannot use.
Builds
BigtableWriterOptions from the table options.Maps Table API options onto the shared delivery mode, writer and staging settings.
Builds
DatastoreWriterOptions from the table options.Builds
FirestoreWriterOptions from the table options.Maps table options onto
SpannerWriterOptions.