# Kafka DescribeTopicPartitions request payload lacks bytes

**URL:** <https://forum.codecrafters.io/t/kafka-describetopicpartitions-request-payload-lacks-bytes/2429>\
**Category:** Challenges\
**Tags:** challenge:kafka\
**Created:** [October 17, 2024, 11:29am UTC](https://forum.codecrafters.io/t/kafka-describetopicpartitions-request-payload-lacks-bytes/2429 "2024-10-17T11:29:33Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![leflings](https://yyz1.discourse-cdn.com/flex003/user_avatar/forum.codecrafters.io/leflings/32/2412_2.png) [@leflings](https://forum.codecrafters.io/u/leflings)\
**Post date:** [October 17, 2024, 11:29am UTC](https://forum.codecrafters.io/t/kafka-describetopicpartitions-request-payload-lacks-bytes/2429/1 "2024-10-17T11:29:33Z")

</div>

Build your own Kafka (Go edition)

I’m not stuck per se - I can work around the seemingly invalid request - but:

I don’t believe the request that is being sent for testing stage ‘List for an unknown topic’  
([The Software Pro's Best Kept Secret.](https://app.codecrafters.io/courses/kafka/stages/vt6)) is correct.

Several bytes are missing at the end, if it is supposed to follow the protocol  
specification.

```auto
DescribeTopicPartitions Request (Version: 0) => [topics] response_partition_limit cursor TAG_BUFFER 
  topics => name TAG_BUFFER 
    name => COMPACT_STRING
  response_partition_limit => INT32
  cursor => topic_name partition_index TAG_BUFFER 
    topic_name => COMPACT_STRING
    partition_index => INT32

```

Following the hexdump of the request payload below

- After the ‘unknown topic’ string, there’s an empty TAG\_BUFFER (1 byte).
- Then 4 bytes goes towards `response_partition_limit` (`0x00000001`). Then  
there’s only 2 bytes left (`0xFF00`).
- `0xFF` on its own is not valid as an UNSIGNED\_VARINT (which is what COMPACT\_STRING)  
starts with (looks like a signed int8? - meant to signal a nullable string?)

Either way, that’s not to mention the 4 bytes from `partition_index` and 2 tag\_buffers  
(one for cursor and for the request as a whole)

(Note: the dump does not including the leading 4 bytes for request length)

```hexdump
 0: 0000002d 004b0000 514df2b3 000c6b61 | ░░░-░K░░QM░░░░ka
16: 666b612d 74657374 65720002 0e756e6b | fka-tester░░░unk
32: 6e6f776e 2d746f70 69630000 000001ff | nown-topic░░░░░░
48: 00 ^ ^ | ░
                         | |     
                         | + end of response_partition_limit
                         +-end of 'unknown topic' string

```

After response\_partition\_limit, there would have to be atleast 7 bytes:

- 1 byte for empty compact string
- 4 bytes for partition index
- 1 byte for tag\_buffer (cursor)
- 1 byte for tag\_buffer (request)

---

<div class="post-metadata">

**Author:** ![leflings](https://yyz1.discourse-cdn.com/flex003/user_avatar/forum.codecrafters.io/leflings/32/2412_2.png) [@leflings](https://forum.codecrafters.io/u/leflings)\
**Post date:** [October 17, 2024, 1:13pm UTC](https://forum.codecrafters.io/t/kafka-describetopicpartitions-request-payload-lacks-bytes/2429/3 "2024-10-17T13:13:28Z")

</div>

So it seems, after inspecting [BinspecVisualizer](https://binspec.org/kafka-describe-topic-partitions-response-v0-unknown-topic?highlight=43-43) I realize that next\_cursor is a nullable field?

How would you get that from the [spec](https://kafka.apache.org/protocol.html#The_Messages_DescribeTopicPartitions)?

---

<div class="post-metadata">

**Author:** ![andy1li](https://yyz1.discourse-cdn.com/flex003/user_avatar/forum.codecrafters.io/andy1li/32/10_2.png) [@andy1li](https://forum.codecrafters.io/u/andy1li)\
**Post date:** [October 17, 2024, 2:11pm UTC](https://forum.codecrafters.io/t/kafka-describetopicpartitions-request-payload-lacks-bytes/2429/4 "2024-10-17T14:11:15Z")

</div>

@leflings Yep, `Cursor` is nullable. (`0xFF` represents -1, indicating that the cursor is null here.)

Unfortunately, there’s no way to get that from the official spec. The author of this challenge, @ryan-gang, spent countless hours reverse-engineering Kafka and digging through its source code, so you don’t have to!

The json files below contain the entire schema of a request / response, and the parser + serializer + documentation is auto generated from them:

> <https://github.com/apache/kafka/blob/05a68986103818a057d488a498698c7eb3185aa3/clients/src/main/resources/common/message/DescribeTopicPartitionsRequest.json#L33-L33>

---

<div class="post-metadata">

**Author:** ![system](https://canada1.discourse-cdn.com/flex003/uploads/codecrafters/original/3X/7/0/700657133935c15703e22c7c8870394f6e6dc27d.svg) [@system](https://forum.codecrafters.io/u/system)\
**Post date:** [October 22, 2024, 2:14pm UTC](https://forum.codecrafters.io/t/kafka-describetopicpartitions-request-payload-lacks-bytes/2429/6 "2024-10-22T14:14:15Z")

</div>

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.