Kafka DescribeTopicPartitions request payload lacks bytes

Build your own Kafka (Go edition)

I’m not stuck per se - I can work around the seemingly invalid request - but:

I don’t believe the request that is being sent for testing stage ‘List for an unknown topic’
(The Software Pro's Best Kept Secret.) is correct.

Several bytes are missing at the end, if it is supposed to follow the protocol
specification.

DescribeTopicPartitions Request (Version: 0) => [topics] response_partition_limit cursor TAG_BUFFER 
  topics => name TAG_BUFFER 
    name => COMPACT_STRING
  response_partition_limit => INT32
  cursor => topic_name partition_index TAG_BUFFER 
    topic_name => COMPACT_STRING
    partition_index => INT32

Following the hexdump of the request payload below

  • After the ‘unknown topic’ string, there’s an empty TAG_BUFFER (1 byte).
  • Then 4 bytes goes towards response_partition_limit (0x00000001). Then
    there’s only 2 bytes left (0xFF00).
  • 0xFF on its own is not valid as an UNSIGNED_VARINT (which is what COMPACT_STRING)
    starts with (looks like a signed int8? - meant to signal a nullable string?)

Either way, that’s not to mention the 4 bytes from partition_index and 2 tag_buffers
(one for cursor and for the request as a whole)

(Note: the dump does not including the leading 4 bytes for request length)

 0: 0000002d 004b0000 514df2b3 000c6b61  | ░░░-░K░░QM░░░░ka
16: 666b612d 74657374 65720002 0e756e6b  | fka-tester░░░unk
32: 6e6f776e 2d746f70 69630000 000001ff  | nown-topic░░░░░░
48: 00                   ^          ^    | ░
                         |          |     
                         |          + end of response_partition_limit
                         +-end of 'unknown topic' string

After response_partition_limit, there would have to be atleast 7 bytes:

  • 1 byte for empty compact string
  • 4 bytes for partition index
  • 1 byte for tag_buffer (cursor)
  • 1 byte for tag_buffer (request)

So it seems, after inspecting BinspecVisualizer I realize that next_cursor is a nullable field?

How would you get that from the spec?

@leflings Yep, Cursor is nullable. (0xFF represents -1, indicating that the cursor is null here.)

Unfortunately, there’s no way to get that from the official spec. The author of this challenge, @ryan-gang, spent countless hours reverse-engineering Kafka and digging through its source code, so you don’t have to!

The json files below contain the entire schema of a request / response, and the parser + serializer + documentation is auto generated from them:

2 Likes

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.