Reduce Kafka Integer Payload Size with Bijou Serializers

If you use Kafka to send numbers (like counters, IDs, or sequence numbers), the default LongSerializer always sends 8 bytes—even for small numbers like 1.

That is fine until the topic is hot. Then those fixed-width longs show up in broker disk, replication, and egress.

I wrapped Ink & Switch’s bijou encodings as a Java library with Kafka SerDes. The idea is the same as their bijou64 write-up: variable length, one canonical encoding per value, swap the serializer and keep your Long (or Integer / BigInteger) app code.

Repo: github.com/Joel-hanson/bijou · Maven Central: io.github.joel-hanson:bijou-kafka-serializers:0.4.1

(Older bijou64-kafka-serializers stops at 0.3.0. Java package is still org.bijou64.)

How the encoding works

#

For bijou64, values 0..247 are a single byte. From 248 up, a tag (F8..FF) says how many payload bytes follow; the payload is big-endian (value - offset) so each size range is unique.

On decode, check the first byte. If it is under 248 (0xF8), that single byte is the number. If it is 248 or higher, it is a tag that says how many more bytes follow.

From 0.4.1 the library is a small family, not only Bijou64:

Widths are not interchangeable on the wire. Pick one format per topic (or field) and use matching SerDes on both ends.

What actually shrinks

#

For sequential integers from 1 to N, the average value shrinks from 8.0 bytes to about 3.0-3.7 bytes (about 54-62% smaller). Full TCP captures only drop about 20%, because keys and Kafka framing still take space.

Millisecond timestamps (around 1.7×10¹²) take about 7 bytes with Bijou64. Still under 8, just not the cute 1-byte demo. Counters and sequences get the big savings.

Plain Bijou64 treats Java long bits as unsigned. Negatives still round-trip, but they look huge on the wire and cost 9 bytes. Use Bijou64s (zigzag) when you need signed values that stay compact.

When it is worth it

#

Use it when values are mostly integers, volume is high, and you own producer and consumer.

Skip it for JSON / Avro / Protobuf payloads (turn on Kafka compression first), or if you cannot change both ends. Existing LongSerializer topics need a coordinated cutover or a new topic.

Drop it into Kafka

#

<dependency>
  <groupId>io.github.joel-hanson</groupId>
  <artifactId>bijou-kafka-serializers</artifactId>
  <version>0.4.1</version>
</dependency>

value.serializer=org.bijou64.kafka.serialization.Bijou64Serializer
value.deserializer=org.bijou64.kafka.serialization.Bijou64Deserializer

Same package for the siblings (Bijou32Serializer, Bijou64sSerializer, Bijou128Serializer, …).

There are two implementations under the hood: a pure Java encoder, and a native JNI path that calls the original bijou package. I added both so I could compare performance, and both stayed. Native is the default when the library loads; set bijou.useJava=true to force Java. If the native library is missing, Java is used anyway.

byte[] encoded = Bijou64.encode(300L); // F8 34
long value = Bijou64.decode(encoded);

Quick local demo:

cd examples/kafka-counter-app
mvn -B package && docker compose up -d
mvn exec:java -Dexec.mainClass=org.bijou64.examples.kafka.CounterProducer

Numbers from CI

#

From a recent CI producer run (sequential ints, no compression):

Same integer values. About 62% less payload for Bijou vs Long. Throughput is in the same ballpark; Bijou32 edged ahead on this runner.

Absolute rates depend on the machine and harness, so treat these as a rough shape, not a promise.

To reproduce: perf/kafka plus ./scripts/compare-benchmarks.sh. For encode-only timing: ./scripts/run-jmh.sh.

Bottom line

#

For mixed payloads, turn on Kafka compression first. For integer-heavy topics, you are often sending 8-byte values that never needed 8 bytes, then compressing them anyway. Bijou shrinks them at the serializer instead.

The wire formats come from Ink & Switch. This repo is the Java / Kafka layer. If it helps, star the repo.