Serializer benchmarks for the 0.1.0 development milestone#

This report compares equivalent logical records through Flink serializers. The benchmark is development tooling and makes no production API or state-compatibility promise. The measurement contract is ADR-0004 .

Reproduce#

The ordinary build does not require benchmark tools. Run these commands from the v0.1.0 tag, whose benchmark pins differ from later revisions, with a compiler cache directory that has not built another Thrift version. For opt-in generation, install CMake, Flex, Bison 3.8 or newer, and a C++ compiler; build the pinned Thrift 0.23.0 compiler in an external cache:

mise x -- just benchmark-tools /tmp/flink-protobuf-benchmark-tools
export BENCHMARK_THRIFT_COMPILER=/tmp/flink-protobuf-benchmark-tools/build/compiler/cpp/bin/thrift
mise x -- just benchmark-smoke 2.3.0 3 /tmp/flink-protobuf-smoke-3
mise x -- just benchmark-run 2.3.0 3 /tmp/flink-protobuf-baseline-3
mise x -- just benchmark-run 2.3.0 4 /tmp/flink-protobuf-baseline-4

Each output directory must be new. The recipes clean between runtime/gencode combinations, generate all three schema representations, build the library, run the benchmark contract tests, check JAR isolation, and then execute independent validation and measurement JVMs. benchmark-check performs the full correctness inventory without timing; benchmark-smoke additionally measures every operation and supported lane on the collections workload with one 100 ms warmup and one 100 ms measurement. A full run measures all five workloads with five independent JVMs per cell, five one-second warmups and five one-second measurements per JVM. It uses one benchmark thread, -Xms1g -Xmx1g -XX:+UseG1GC, and JMH’s GC profiler. Two complete Protobuf profile runs take several hours; run them sequentially without other builds or heavy work. Keep the Mac connected to power and record interruptions or concurrent load.

On macOS, ensure the selected Bison is modern; /usr/bin/bison may be too old. On macOS the recipe selects the installed SDK and its matching C++ include path through xcrun. These toolchain settings do not change the Thrift source. The compiler tarball is checked against SHA-256 1859d932d2ae1f13d16c5a196931208c116310a5ff50f2bfd11d3db03be8f46f. Generation checks thrift --version before using it.

Corpus and representation mapping#

The seed is 30012026, with 64 distinct records per workload. Every representation carries the same nine fields: signed integer ID, long timestamp, boolean, name, optional note, bytes, a child record, a list of longs, and a string-to-long map. All workloads use the same record schema; omitted payload categories retain their defined empty/default values, so fixed-field formats still encode those fields. This measures populated shapes within one schema, not minimal separately optimized schemas for each format.

The optional note cycles through absent, present-empty, and present-nonempty. Absent means POJO/Kryo null, Avro union null, Protobuf unset optional, Thrift unset optional, and Tuple (false, ""). Tuple present values use (true, value). Empty strings, zero scalars, empty lists, empty maps, and a present default child remain valid values. Top-level nulls and null collection entries are excluded. The Protobuf schema uses proto3 and valid UTF-8, so it does not exercise the proto2 ambiguous-string counting path covered by the separate implementation performance check . Maps compare by key/value equality, not iteration order; their initial insertion order is deterministic. Byte arrays, ByteString, and ByteBuffer compare by payload bytes; Avro strings normalize to Java strings.

The Kryo lanes retain Flink’s reference tracking and registrationRequired=false policy. The comparison inventory records the actual adapter, root registration ID, loaded Kryo/adapter artifacts, and serializer graph. Avro’s type discovery may log a generic ByteBuffer field; the selected runtime serializer is independently asserted to be AvroSerializer. The POJO lane checks its actual nested serializers, rather than assuming the top-level type proves the entire route.

Timed operations and interpretation#

serialize, deserialize, objectCopy, and streamCopy each process one record per invocation, cycling over the corpus. deserializeReuse and objectCopyReuse pass the previous result as the reuse candidate. serializeSameInstance repeatedly writes corpus element zero. buildAndSerialize constructs a fresh representation from a prebuilt logical value inside the timed method, including representation allocation and serialization; it does not include pseudorandom corpus generation. String and scalar values are shared with the prebuilt logical corpus; each conversion creates fresh record/container objects and byte storage. The logical-to-representation conversion is common instrumentation, not a production ingestion recommendation.

Correctness subprocesses and JMH forks load the packaged production library JAR, whose hash is retained alongside the test instrumentation hashes. Each JMH fork asserts the library’s loaded location against the packaged JAR before timing. Trial setup constructs type information, creates serializers, validates round-trips, pre-encodes the input records, and warms copy/stream-copy storage. Repeated serialization therefore measures previously encoded instances; fresh construction creates new generated messages without reusing their memoized sizes. Native sizing traverses populated fields on every write; Chill uses toByteArray, which consults the generated runtime’s memoized sizes. The benchmark never resets private Protobuf caches or subtracts construction costs from measured results. Initialization, descriptor discovery, and fresh-classloader startup are outside the timed regions.

Each thread owns its serializer, input view, output buffer, and reuse candidate. The output buffer is preallocated to 128 KiB and cleared for each write; input views are reset to the exact encoded record. Buffer resets and corpus indexing are included in the measured operation. Outputs and returned objects escape through JMH to prevent dead-code elimination. Allocation results include work performed by the timed method, including any internal adapter buffers.

Native Protobuf object copy returns its immutable input reference. Chill’s Protobuf adapter serializes and parses a new value even though Protobuf messages are immutable. Other serializers’ mutation independence and reuse identities are recorded by correctness checks. Flink Kryo stream copy deserializes and serializes; native Protobuf stream copy transfers the framed bytes through reusable storage. These are the actual implementations of the Flink operations and are not interchangeable amounts of copying work. Do not interpret their ratios as general encoding-speed ratios.

Encoded size includes every byte emitted by the Flink TypeSerializer, including the native four-byte payload-length prefix and any Kryo/adapter framing. It excludes Flink network envelopes, checkpoints, compression, and transport overhead. The support inventory reports mean, minimum, and maximum encoded sizes over all 64 values. Throughput in measurements.csv is explicitly derived as 1e9 / ns_per_op; it is not a separate throughput-mode experiment. Native deterministic writing is a separately labeled serialize/collections supplement; it does not change the primary comparison configuration.

Correctness and performance matrices#

Correctness checks decode all records, exercise both copy overloads and both deserialize overloads, mutate copies of mutable representations, and ensure a deserialize or stream copy consumes exactly one of two adjacent records. JMH setup repeats value checks inside each measurement fork before timing. A missing prerequisite, incorrect serializer, wrong value, or missing allocation metric fails the run. Known adapter linkage or Java module-access failures are recorded as unsupported without substitution.

CI covers Flink 2.2.1 and 2.3.0 with JDK 17/21, and Flink 1.20.4 with JDK 17, each with Protobuf 3.25.9/protoc 3.25.9 and Protobuf 4.33.6/protoc 4.33.6. The representative long-run matrix is Flink 2.3.0 on this Mac’s Temurin 17 for both Protobuf profiles. JMH is 1.37, Avro runtime/generator is 1.11.4, Thrift runtime/compiler is 0.23.0, and both Chill adapters are 0.10.0. The report reader uses Jackson 2.14.3 through an explicit benchmark-only dependency. Exact runtime JARs and hashes are retained in each run’s classpath manifest. Generic comparison JVMs on Flink 1.20 additionally try --add-opens=java.base/java.util=ALL-UNNAMED after an unsupported plain-JVM result. Native tests and benchmarks never inherit that flag.

Statistical contract and retained evidence#

The practical goal is at most 10% additional native Protobuf time relative to Chill on the same workload, operation, and input policy. Five measurement blocks place the native and Chill forks adjacent, reversing their order in alternating blocks. The remaining lane groups use a fixed shuffled order within each workload/operation block. For each block, average the five timed iterations, then compute the native/Chill log ratio. Use the mean and Student-t interval of those five log ratios (four degrees of freedom, critical value 2.7764451051977987), and exponentiate the limits. This retains fork-to-fork uncertainty instead of treating 25 iterations as independent JVM runs.

An upper 95% limit at or below 1.10 is established; a lower limit above 1.10 is missed; all other intervals are inconclusive. The intervals are per comparison and assume approximately normal independent log ratios, not simultaneous confidence across every reported cell. Identity-copy timings near the harness overhead deserve particular caution. Noisy overlaps between independent JMH scores do not establish the tolerance. Allocation and encoded size are reported separately without inventing an additional acceptance threshold.

Each run retains JMH JSON with per-iteration raw data and GC metrics, per-fork logs, completed result inventory, support properties, machine/JVM/configuration manifest, and SHA-256 hashes of the classpath and packaged library. measurements.csv and parity.csv are reproducible summaries of that evidence. The source revision identifies the measurement source; report-only commits may follow without relabeling its provenance. Results apply to the recorded machine and runtime, do not rank serializers universally, and do not establish snapshot compatibility or whole-job throughput.

Baseline results#

The native serializer misses the predeclared time tolerance in several encoding and decoding cells on this host. Large-payload serialization and the copy operations favor the native implementation, while small-record writes expose substantial allocation and sizing costs. These measurements do not establish a universal fastest serializer or parity across the corpus. The retained evidence includes both complete runs, all per-cell results, archive hashes, and separate diagnostic event exports.

Host and run validity#

Both runs used the clean measurement source f1fdd36137b8baa59202652da83a7a8a94d98c3f, Flink 2.3.0, Temurin 17.0.20+8, an Apple M1 Pro with 10 logical CPUs and 32 GiB memory, and macOS 26.6.2 as reported by the JVM. The system manifest separately reports Darwin 25.6.0 on arm64. The JMH fork settings and comparison contract above were unchanged between profiles. The version in the packaged development artifact is 0.1.0-SNAPSHOT; 0.1.0 identifies a development milestone, not a published release.

Each profile contains 1,600 primary forks and five deterministic supplements. An independent audit checked all 3,210 results for the expected inventory, source/JAR provenance, fork/iteration settings, finite timing and allocation data, and agreement of all 80 paired intervals with the report reader.

Before each build and after each build, the start gate required at least five minutes of sampled CPU idle averaging 90% or more, with an 85% minimum, one-minute load below 10, no observed swap-out, and at most 1 MiB of swap-in per observation. The gate sampled two-second windows approximately every 30 seconds; it did not establish exclusive host access or continuous CPU isolation. Its initial stricter CPU/page-in thresholds were refined before any retained timing, as recorded in the environment notes. The profiles ran sequentially on different days, so ambient host conditions may differ.

An initial run under background load was stopped and excluded. The first overnight Protobuf 4 attempt was excluded in full after the system recorded clamshell sleep on battery power at 2026-09-13 01:43:46 JST and wake on AC at 10:35:01. No results from that attempt enter these tables. The accepted replacement used closed-display operation with USB-C AC power, power observations every 30 seconds, and an external guard that invalidated power-source changes or observation gaps exceeding 120 seconds. Its retained power-events.log contains no sleep events; the accepted Protobuf 3 interval also had no recorded sleep events. The Protobuf 3 archive includes a separate power-event-audit.txt; the Protobuf 4 archive retains the event log and power observations without that separate audit summary. These power checks address interruptions, not background CPU scheduling or thermal variation.

Native and Chill time tolerance#

All 15 copy cells per profile establish the tolerance, with the different copy semantics described above. They do not offset missed encoding/decoding cells. The following selected cells show the paired geometric time ratio and its 95% interval; lower ratios favor native Protobuf. The archives retain every cell, including reuse, same-instance, and copy results.

Protobuf 3 text buildAndSerialize has a broad interval, so its point estimate is insufficient for an acceptance claim. The narrower text serialize interval also crosses the 1.10 tolerance and is inconclusive. Across profiles, Chill text serialization falls from 1,176.8 to 519.4 ns/op with the same 1,814.5 B/op, while native text serialization changes from 1,303.2 to 1,343.0 ns/op. The changed comparator contributes to the different text verdicts; the profile totals do not establish a native-serializer regression between Protobuf versions. The profiles differ in runtime/gencode and ran on different days, so these observations do not isolate a Protobuf implementation effect from host/run variation. Five independent pairs give limited information about host drift and distribution shape. The fixed corpus and warm setup also limit transfer to fresh application objects and first-use costs. serializeSameInstance uses record zero while serialize cycles through all 64 values; their difference does not isolate memoization.

Cross-format encoding costs#

Each entry below is the arithmetic mean across five fork scores for ordinary corpus-cycling serialization. Time is ns/op and allocation is B/op from the GC profiler; encoded bytes are the corpus mean including serializer framing. The two values in each cell are Protobuf profile 3 / profile 4. Every lane was rerun in each profile, so cross-profile timing differences can include host and run variation as well as the changed Protobuf runtime/gencode. Use per-fork raw data for uncertainty; this table supplies descriptive means, not additional statistical acceptance tests.

The full 640-cell summaries include deserialize, reuse, object/stream copy, same-instance serialization, and fresh construction, with minimum and maximum fork scores. Derived operations per second remain available in measurements.csv.

Allocation and optimization candidates#

On Protobuf 4, native scalar serialization allocates about 6,089 B/op versus 175 B/op for Chill, and scalar deserialization allocates about 4,496 versus 414 B/op. Native large-payload serialization allocates about 6,145 B/op versus 65,716 B/op for Chill and takes about 0.366 times the paired Chill time. Large-payload deserialization still allocates about 132,056 versus 131,532 B/op and has an inconclusive time-tolerance interval. Avoiding an intermediate output payload array therefore does not make the whole serializer allocation-free or uniformly faster.

Separate short JFR diagnostics on the same Protobuf 4 source/configuration corroborate allocation sites in the stream encoder/decoder buffers and native sizing traversal. Worker-thread allocation samples include byte arrays allocated by CodedOutputStream.AbstractBufferedEncoder and CodedInputStream.StreamDecoder, and sizing Frame, Child, and collection objects. Execution samples also reach sizing traversal, identity maps, generated field access, and decoder refill paths. The retained allocation/execution event exports cover native/Chill scalar serialize/deserialize and native collection serialization; diagnostic timings are not included in the baseline.

The measured source explains these sites: serialization and deserialization create stream-based coded wrappers for each record, while safe sizing builds per-call traversal metadata and reads populated fields. Protobuf 4.33.6’s CodedOutputStream.newInstance(OutputStream) and CodedInputStream.newInstance(InputStream) use 4,096-byte default buffers; Chill’s adapter uses toByteArray for writing and parses a payload array when reading. The pinned Protobuf source locations are CodedOutputStream.DEFAULT_BUFFER_SIZE, CodedInputStream.StreamDecoder, and AbstractMessageLite.toByteArray; dependency identities are in the classpath manifest. JFR supports identifying exercised allocation sites, but sampling weights do not quantify each site’s contribution to the baseline time gap. No controlled ablation was performed.

The first optimization candidates are reducing fixed per-record buffers for small frames and reducing sizing traversal allocations while preserving overflow, depth, size, framing, classloading, and concurrency guarantees. Construction benchmarks and collection-heavy messages must remain separate acceptance cases when evaluating such changes. This issue establishes the baseline and investigation; it makes no production optimization or fastest-serializer claim. The buffer-allocation report measures the later small-frame buffer change separately, on a different machine. The sizing-allocation report measures the later sizing-traversal change the same way.

Benchmark remeasurement for 1.0.0#

Before releasing 1.0.0, repeat the long serializer benchmarks from a fixed release-candidate commit for both supported Protobuf runtime/gencode profiles. Record the candidate commit, packaged library SHA-256, exact dependency and toolchain versions, and host configuration with the raw results. Use the same logical corpus, operation definitions, warmup and measurement settings, independent paired blocks, and predeclared practical tolerance as the development baseline. Use temporary Google Compute Engine VMs for the next benchmark run and the 1.0.0 candidate measurements. Use matching machine types, CPU platforms, regions, operating-system images, and JDK settings, and record their exact identities in the evidence. Run one Protobuf profile per VM so that the profiles can execute concurrently without competing within a VM. Keep the native/Chill pair ordering and independent fork contract within each profile unchanged. Record VM interruptions and exclude interrupted runs before interpreting their results. Export raw results and provenance, verify that the retained evidence is complete, and then delete the temporary VMs and their disks. Recheck current machine availability and pricing when provisioning.

Compare the candidate with the retained 0.1.0 development baseline by workload and operation, including time, allocation, and encoded size. If the supported Flink, JDK, Protobuf, adapter, or host versions changed, list those differences explicitly and do not attribute the entire measured difference to library changes. Preserve the earlier baseline unchanged and publish a separate 1.0.0 report with raw evidence and uncertainty. Benchmark results do not replace release API, state-compatibility, or published-artifact validation. The 1.0.0 release-candidate report records this remeasurement.