Serializer benchmarks for the 1.0.0 release candidate#

This report is the release-candidate performance evidence for 1.0.0 that #45 requires. It repeats the long serializer benchmarks of the 0.1.0 report on temporary Google Compute Engine VMs, with the same corpus, lanes, operations, fork and iteration settings, paired blocks, and practical tolerance, all defined in ADR-0004 . It adds the two DynamicMessage lanes that ADR-0004 describes. The results describe the recorded VMs and runtimes; they do not rank serializers universally and do not establish whole-job throughput or saved-state compatibility.

Reproduce#

The plan measured commit 9e7d175341fc0e418a97f199168d63c1a0f8cb33 with this specification:

{"runId": "issue45-rc-1", "mode": "run", "source": "9e7d175341fc0e418a97f199168d63c1a0f8cb33",
 "profiles": [{"protobuf": 4}, {"protobuf": 3}]}

mise x -- just gce-benchmark plan rc.json /tmp/issue45-gce/plan
mise x -- just gce-benchmark execute /tmp/issue45-gce/plan --approve
mise x -- just gce-benchmark collect /tmp/issue45-gce/plan /tmp/issue45-gce/results

Temporary GCE execution describes the commands, the load gate, maintenance handling, and cleanup. Each VM ran just benchmark-build 2.3.0 <major> and just benchmark-measure run 2.3.0 <major> on a checkout of the source commit, as each archive’s invocation.txt records. A rerun needs a new run ID and new output directories, and its VMs are not comparable causally with these.

Candidate and hosts#

The source commit is the merge of #173 , which set the version to 1.0.0-SNAPSHOT. The commits that follow it on main up to ed4db42 change no production source, POM, or dependency, so they do not change the measured library. A later change to production code or a dependency before the v1.0.0 tag requires a rerun or an explicit invalidation of this evidence.

just benchmark-build packages the library for each Protobuf profile, with -Pprotobuf<major> at Flink 2.3.0. The measured jars have SHA-256 cdc10744cc934f9d8c427aa09de066fce26ec6dcbbd8b2eca4bb732a925ff80e (Protobuf 4) and 0b47620f7758467cee99e9662d92d487ff960e567b9ce184fb405bb2a6c5fc5a (Protobuf 3). The release builds each artifact line once with Protobuf 4, at Flink 2.2.1 for the 2.x line, and ships that jar for both runtimes. A local rebuild of the source commit with Temurin 17.0.20 in the release configuration and in both benchmark configurations produced byte-identical class files and resources, which differed only in zip entry timestamps. The measured classes are therefore those of the release configuration, as Temurin 17 compiles them; the archives do not retain the VM jars, so this comparison covers the local rebuild only. #177 tracks measuring the release-configuration jar directly.

Each profile holds 2,000 primary forks, ten lanes by five workloads by eight operations by five blocks, and the five deterministic supplements. Both VMs ran concurrently, one profile each, from the same image, machine type, and CPU platform. Their maintenance-event logs record only NONE, and both load gates passed before and after the build with a mean CPU idle of at least 99.95% and a minimum of at least 99.88%. Recomputing every native/Chill interval from measurements.csv reproduces parity.csv within floating-point rounding. The collection deleted both VMs and their disks, and a later cleanup check found no instance or disk labelled with the run.

Native and Chill time tolerance#

No cell misses the 10% tolerance in either profile. All 15 copy cells per profile establish it, with upper limits of at most 0.114; their copy semantics differ from Chill’s, as the 0.1.0 report explains. The table shows the paired geometric native/Chill time ratio with its 95% interval for the encoding cells, next to the 0.1.0 point estimate; lower ratios favor native Protobuf. The archives retain every cell.

The two inconclusive Protobuf 3 cells have point estimates below the tolerance. Text buildAndSerialize, at 0.627 (0.245, 1.604), belongs to the Protobuf 3 text write cells whose JVM forks take one of two times: native text serialize forks take 639 to 642 or 1,467 to 1,469 ns/op, and Chill buildAndSerialize forks 1,496 or 3,193 to 3,359 ns/op. The standard deviation of the five log ratios is 0.39 to 0.76 for text serialize, serializeSameInstance, and buildAndSerialize, against at most 0.16 for every other cell; the sizing-allocation , generated-size , and large-read reports recorded the same spread, and #178 investigates it. Collections serializeSameInstance, at 0.945 (0.794, 1.125), crosses the tolerance because one native fork and one Chill fork each took about 20% longer than the other four of its lane.

Changes since the 0.1.0 baseline#

ADR-0004 does not compare a VM result causally with the Mac baseline, so this section describes differences without attributing them. Between the two runs the following changed:

• Host: an Apple M1 Pro Mac with macOS 26.6.2 became a c3-standard-8 VM with Ubuntu 24.04.5.
• JDK: Temurin 17.0.20+8 became 17.0.21+9.
• Protobuf 4 profile: protobuf-java and protoc 4.33.6 became 4.36.2; the Protobuf 3 profile kept 3.25.9.
• Thrift: runtime and compiler 0.23.0 became 0.25.0. Flink 2.3.0, Avro, Chill, Kryo, JMH, and the corpus are unchanged.
• Library: the per-record buffer change of #42 , the sizing changes of #43 and #112 , the large-frame read of #113 , DynamicMessage support, and the 1.0.0 persisted-format restart.

The unchanged Chill comparator took 1.06 to 2.66 times as long on the VM as on the Mac for serialize and deserialize, so absolute times do not compare across the two runs. The native/Chill ratio divides out what the host change does equally to both lanes, but on Protobuf 4 it also contains the comparator’s runtime change. Of the 33 encoding cells that missed the tolerance in 0.1.0, 32 establish it now, and Protobuf 3 collections serializeSameInstance is inconclusive. The large-payload serialize and buildAndSerialize ratios stayed within 0.03 of their 0.1.0 values, while deserialize and deserializeReuse fell from 1.09–1.10 to 0.54–0.55; the large-read report measured that change of #113 against a baseline on the same VM.

Encoded sizes are unchanged for both Protobuf lanes in both profiles, for example 23.9 bytes for scalar, 498.1 for collections, and 65,563.9 for large native records. Native allocation, the mean of five forks, fell in every cell of this table:

The buffer-allocation and sizing-allocation reports measured the main reductions against baselines on the same VM. The remaining Protobuf 4 write allocation grows with the number of string fields: text serialize allocates 1,957.1 B/op on Protobuf 4 against 136.0 on Protobuf 3. Chill and DynamicMessage carry the same increase on Protobuf 4, and Chill’s own text serialize allocation rose from 1,814.5 B/op with 4.33.6 in 0.1.0 to 3,642.0 B/op with 4.36.2, so it comes from the protobuf-java 4.36.2 runtime rather than from this library.

Cross-format encoding costs#

Each entry is the arithmetic mean of five fork scores for ordinary corpus-cycling serialize; time is ns/op, allocation is B/op from the GC profiler, and encoded bytes are the corpus mean including serializer framing. The two values in each cell are profile 3 / profile 4, measured on different VMs of one plan. These are descriptive means; the paired intervals above and the per-fork data carry the uncertainty.

Native scalar serialization is the fastest of all lanes, at 54.5 to 63.8 ns/op. Collections have the largest native cost relative to the fixed-layout lanes: native serialize takes 777 to 861 ns/op, 0.88 to 0.95 times Chill’s time but 2.5 to 2.8 times Tuple’s and 2.0 to 2.2 times POJO’s. Protobuf writes each of the 16 map entries as a length-delimited message and each long as a varint, while Flink’s POJO and Tuple serializers write longs as fixed eight-byte values and no per-entry length; this report does not measure how much each difference contributes. Native large serialization takes 0.34 times Chill’s time, but 1.08 to 1.10 times Kryo’s, and allocates about 4.5 KB per record for the stream encoder buffer that #176 addresses. Generated-message serialize allocates 96 B/op more than DynamicMessage serialize of the same content in the scalar, text, and nested workloads of both profiles; #175 investigates that fixed allocation.

DynamicMessage lanes#

PROTOBUF_DYNAMIC values carry the serializer’s own descriptors, taken from a value it decoded, and PROTOBUF_DYNAMIC_FOREIGN values carry an equal-content descriptor from another instance, which the serializer accepts through its cache of accepted descriptor pairs. Both are measured in the same blocks as the generated and Chill lanes, and ADR-0004 gives their ratios as descriptive results without a tolerance or verdict. The table shows the encoding operations; dynamic.csv in each archive holds all 40 cells.

Decoding a DynamicMessage takes 5.3 to 6.3 times as long as decoding the generated message for the scalar, nested, and collections records, 1.6 times for text, and 1.1 times for large. The serializer decodes both through the parser of the value’s type, so the difference is the cost of protobuf-java’s DynamicMessage parsing; DynamicMessage collections decoding also allocates 9.5 to 12.6 KB per record against 3.7 to 3.8 KB. Writing takes 1.5 to 1.8 times as long for the scalar, nested, and collections records and 0.81 to 1.06 times as long for text and large, where the Protobuf 3 text interval, 0.465 to 1.415, is wide. buildAndSerialize also includes building the DynamicMessage value with its builder, which is the benchmark’s conversion, not serializer work.

The foreign/own point estimates lie between 0.91 and 1.05 across all 80 cells. Eight of their intervals exclude 1, by at most 3%, and four of those are copy or stream-copy cells, which run the same code for both lanes; such differences are therefore within the variation between adjacent lanes. Values with descriptors from another instance thus show no measurable cost from the write-acceptance cache beyond that variation.

Remaining gaps and release decision#

• Collections encoding remains the narrowest native margin over Chill, 0.88 to 0.95, and costs several times the fixed-layout lanes.
• Scalar and nested encoding are now faster than Chill in every operation, against 2.1 to 10.1 times Chill’s time in 0.1.0.
• On Protobuf 4.36.2, every Protobuf lane allocates an additional amount per string field when writing.
• Protobuf 3 text write cells stay bimodal across forks, which leaves one of them inconclusive (#178 ).
• DynamicMessage decoding costs 5.3 to 6.3 times generated decoding for small records, which applications choosing DynamicMessage pay to protobuf-java’s parser.
• #175 and #176 track the two library allocations above, and #177 the measured jar, all for 1.1.0.

No cell misses the tolerance, and the only encoding ratio above its 0.1.0 estimate, Protobuf 3 large buildAndSerialize, rose from 0.631 to 0.649 across the host and runtime changes listed above. The measurements therefore show no material regression and raise no performance objection to releasing this candidate as 1.0.0. None of the follow-ups requires an API, state-format, or default change.

Retained evidence#

The retained evidence holds both complete result directories with their GCE run records, checksums, and inventories.