How to parse variable sized integer from metadata log

This is for Stage #EA7.

I’ve been assuming that the any length is 1 byte, but then I saw that the value length of the record is 2 bytes. After further reading, I realized that the spec said its a signed variable size integer. How does one parse a variable sized integer i.e how does one know when to stop reading a variable sized thing?

Hi @pdtran3k6, here’s an example illustrating how to parse 0x8147 as decimal 199.


how does one know when to stop reading a variable sized thing

Stop when a byte has a leading 0.

0x8201 has two bytes:

  • 1000 0010: continue
  • 0000 0001: stop

what’s the software that you use here to see the hex in binary like that?

and also, if I read the other post correctly, in this case, the final number after concat would be:

000 0010 + 000 0001
= 10 000 0001
= 1 0000 0001 = 257

which is definitely not 65 like the link here
and even if I intepret it as a signed integer, that would be -255, also not 65

It’s the Calculator app on mac in programmer mode:


You’re absolutely right! It’s 257. I was a bit confused as well, because the length of Value (Partition Record) is apparently 65 (= 1 + 16 x 4):

Turns out that VARINT in Kafka involves two additional considerations:

  1. Small/Big-endian order
  2. Zig-zag encoding
000 0010 + 000 0001 reverses order due to endianness:

000 0001 + 000 0010

= 1 000 0010 = 130

130 becomes 130 / 2 = 65 due to zig-zag encoding.