How to decide which Response Header to use?

ganimtron-10 · March 6, 2025, 4:14pm

While solving test #VT6 the issue I faced was that the DescribeTopicPartition Response was being expected with Response header v1 ie correlationID and then TagBuffer, but in the subsequent test (#YK1) the Response was being expected with Response header v0 ie correlationId only no TagBuffer. (I figured it out by analysing the request response from the debug logs). For now I have added a flag to check if its APIVersionResponse then return Response Header v0. But what is ideal mechanism like how to identify which response header to send?

Also attaching Github link for any reference: GitHub - ganimtron-10/akfak: A Go-based implementation of the Kafka messaging system, built from scratch for understanding how things work under the hood.

Failing Logs

remote: ------------------------------------------------------------------------
remote: 
remote: 
remote:      ___            _          ___              __  _                   
remote:     / __\ ___    __| |  ___   / __\_ __  __ _  / _|| |_  ___  _ __  ___ 
remote:    / /   / _ \  / _` | / _ \ / /  | '__|/ _` || |_ | __|/ _ \| '__|/ __|
remote:   / /___| (_) || (_| ||  __// /___| |  | (_| ||  _|| |_|  __/| |   \__ 
remote:   \____/ \___/  \__,_| \___|\____/|_|   \__,_||_|   \__|\___||_|   |___/
remote: 
remote: 
remote:    Welcome to CodeCrafters! Your commit was received successfully.
remote: 
remote: ------------------------------------------------------------------------
remote: 
remote: ⚡ This is a turbo test run. https://codecrafters.io/turbo
remote: 
remote: Running tests on your code. Logs should appear shortly...
remote: 
remote: [compile] Moved ./.codecrafters/run.sh → ./your_program.sh
remote: [compile] Compilation successful.
remote: 
remote: Debug = true
remote: 
remote: [tester::#VT6] Running tests for Stage #VT6 (Listing Partitions - List for an unknown topic)
remote: [tester::#VT6] $ ./your_program.sh /tmp/server.properties
remote: [tester::#VT6] Connecting to broker at: localhost:9092
remote: [your_program] Starting Akfak on port 9092...
remote: [tester::#VT6] Connection to broker at localhost:9092 successful
remote: [tester::#VT6] Sending "DescribeTopicPartitions" (version: 0) request (Correlation id: 1919854771)
remote: [tester::#VT6] Hexdump of sent "DescribeTopicPartitions" request: 
remote: [tester::#VT6] Idx  | Hex                                             | ASCII
remote: [tester::#VT6] -----+-------------------------------------------------+-----------------
remote: [tester::#VT6] 0000 | 00 00 00 31 00 4b 00 00 72 6e a8 b3 00 0c 6b 61 | ...1.K..rn....ka
remote: [tester::#VT6] 0010 | 66 6b 61 2d 74 65 73 74 65 72 00 02 12 75 6e 6b | fka-tester...unk
remote: [tester::#VT6] 0020 | 6e 6f 77 6e 2d 74 6f 70 69 63 2d 73 61 7a 00 00 | nown-topic-saz..
remote: [tester::#VT6] 0030 | 00 00 01 ff 00                                  | .....
remote: [tester::#VT6] 
remote: [your_program] &{RequestHeader:{messageSize:49 apiKey:75 apiVersion:0 correlationId:1919854771 clientId:kafka-tester} names:[unknown-topic-saz] responsePartitionLimit:1 topicName: partitionIndex:0}
remote: [your_program] &{throttleTime:0 topics:[{errorCode:3 name:unknown-topic-saz topicId:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] isInternal:false partitions:[] topicAuthorizedOperations:0}] nextCursor:{topicName: partitionIndex:0}}
remote: [tester::#VT6] Hexdump of received "DescribeTopicPartitions" response: 
remote: [tester::#VT6] Idx  | Hex                                             | ASCII
remote: [tester::#VT6] -----+-------------------------------------------------+-----------------
remote: [tester::#VT6] 0000 | 00 00 00 37 72 6e a8 b3 00 00 00 00 00 02 00 03 | ...7rn..........
remote: [tester::#VT6] 0010 | 12 75 6e 6b 6e 6f 77 6e 2d 74 6f 70 69 63 2d 73 | .unknown-topic-s
remote: [tester::#VT6] 0020 | 61 7a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | az..............
remote: [tester::#VT6] 0030 | 00 00 00 01 00 00 00 00 00 ff 00                | ...........
remote: [tester::#VT6] 
remote: [tester::#VT6] [Decoder] - .ResponseHeader
remote: [tester::#VT6] [Decoder]   - .correlation_id (1919854771)
remote: [tester::#VT6] [Decoder]   - .TAG_BUFFER
remote: [tester::#VT6] [Decoder] - .ResponseBody
remote: [tester::#VT6] [Decoder]   - .throttle_time_ms (0)
remote: [tester::#VT6] [Decoder]   - .topic.length (1)
remote: [tester::#VT6] [Decoder]   - .Topics[0]
remote: [tester::#VT6] [Decoder]     - .error_code (3)
remote: [tester::#VT6] [Decoder]     - .name (unknown-topic-saz)
remote: [tester::#VT6] [Decoder]     - .topic_id (00000000-0000-0000-0000-000000000000)
remote: [tester::#VT6] [Decoder]     - .is_internal (false)
remote: [tester::#VT6] [Decoder]     - .num_partitions (0)
remote: [tester::#VT6] [Decoder]     - .topic_authorized_operations (0)
remote: [tester::#VT6] [Decoder]     - .TAG_BUFFER
remote: [tester::#VT6] [Decoder]   - .next_cursor (null)
remote: [tester::#VT6] [Decoder]   - .TAG_BUFFER
remote: [tester::#VT6] ✓ Correlation ID: 1919854771
remote: [tester::#VT6] ✓ Throttle Time: 0
remote: [tester::#VT6]   ✓ TopicResponse[0] Error code: 3
remote: [tester::#VT6]   ✓ TopicResponse[0] Topic Name: unknown-topic-saz
remote: [tester::#VT6]   ✓ TopicResponse[0] Topic UUID: 00000000-0000-0000-0000-000000000000
remote: [tester::#VT6] Test passed.
remote: [tester::#VT6] Terminating program
remote: [your_program] Closing Connection, Error reading request:  EOF
remote: [tester::#VT6] Program terminated successfully
remote: 
remote: [tester::#YK1] Running tests for Stage #YK1 (Listing Partitions - Include DescribeTopicPartitions in APIVersions)
remote: [tester::#YK1] $ ./your_program.sh /tmp/server.properties
remote: [tester::#YK1] Connecting to broker at: localhost:9092
remote: [your_program] Starting Akfak on port 9092...
remote: [tester::#YK1] Connection to broker at localhost:9092 successful
remote: [tester::#YK1] Sending "ApiVersions" (version: 4) request (Correlation id: 1579994263)
remote: [tester::#YK1] Hexdump of sent "ApiVersions" request: 
remote: [tester::#YK1] Idx  | Hex                                             | ASCII
remote: [tester::#YK1] -----+-------------------------------------------------+-----------------
remote: [tester::#YK1] 0000 | 00 00 00 23 00 12 00 04 5e 2c cc 97 00 09 6b 61 | ...#....^,....ka
remote: [tester::#YK1] 0010 | 66 6b 61 2d 63 6c 69 00 0a 6b 61 66 6b 61 2d 63 | fka-cli..kafka-c
remote: [tester::#YK1] 0020 | 6c 69 04 30 2e 31 00                            | li.0.1.
remote: [tester::#YK1] 
remote: [tester::#YK1] Hexdump of received "ApiVersions" response: 
remote: [tester::#YK1] Idx  | Hex                                             | ASCII
remote: [tester::#YK1] -----+-------------------------------------------------+-----------------
remote: [tester::#YK1] 0000 | 00 00 00 1b 5e 2c cc 97 00 00 00 03 00 12 00 00 | ....^,..........
remote: [tester::#YK1] 0010 | 00 04 00 00 4b 00 00 00 00 00 00 00 00 00 00    | ....K..........
remote: [tester::#YK1] 
remote: [tester::#YK1] [Decoder] - .ResponseHeader
remote: [tester::#YK1] [Decoder]   - .correlation_id (1579994263)
remote: [tester::#YK1] [Decoder] - .ResponseBody
remote: [tester::#YK1] [Decoder]   - .error_code (0)
remote: [tester::#YK1] [Decoder]   - .num_api_keys (0)
remote: [tester::#YK1] [Decoder]   - .throttle_time_ms (50336256)
remote: [tester::#YK1] [Decoder]   - .TAG_BUFFER
remote: [tester::#YK1] Received:
remote: [tester::#YK1] Hex (bytes 7-22)                                | ASCII
remote: [tester::#YK1] ------------------------------------------------+------------------
remote: [tester::#YK1] 03 00 12 00 00 00 04 00 00 4b 00 00 00 00 00 00 | .........K......
remote: [tester::#YK1]                 ^                                      ^
remote: [tester::#YK1] Error: unexpected 15 bytes remaining in decoder after decoding ApiVersionsResponse
remote: [tester::#YK1] Context:
remote: [tester::#YK1] - ApiVersions v3
remote: [tester::#YK1]   - Response Body
remote: [tester::#YK1] 
remote: [tester::#YK1] Test failed
remote: [tester::#YK1] Terminating program
remote: [your_program] &{RequestHeader:{messageSize:35 apiKey:18 apiVersion:4 correlationId:1579994263 clientId:kafka-cli} clientSoftwareName:kafka-cli clientSoftwareVersion:0.1}
remote: [your_program] &{errorCode:0 numOfApiKeys:3 apiKeys:[{key:18 minVersion:0 maxVersion:4} {key:75 minVersion:0 maxVersion:0}] throttleTime:0}
remote: [tester::#YK1] Program terminated successfully
remote: 
remote: Try our CLI to run tests faster without Git: https://codecrafters.io/cli
remote: 
remote: View our article on debugging test failures: https://codecrafters.io/debug
remote:

gerbil-329 · March 7, 2025, 10:47am

I don’t think there’s a better way to do it that will satisfy the tester.

I think the technically correct thing would be to use response header v1 for any API versions that support tagged fields - so any request/response body that has the TagBuffer type. But that would include api versions v4, which we were already required to implement with response header v0.

Source:

RequestHeader Version 1

Requests within a “flexible version” will have a new version of the request header. The new RequestHeader version will be version 1, superseding version 0. In this new version, the RequestHeader’s ClientId string will be a COMPACT_STRING rather than STRING. Additionally, the header will contain space for tagged fields at the end. Supporting tagged fields in the request header will give us a natural place to put additional information that is common to all requests.

ResponseHeader Version 1

Responses within a “flexible version” will have a new version of the response header. The new ResponseHeader version will be version 1, superseding version 0. In this new version, the header will contain space for tagged fields at the end. Supporting tagged fields in the response header will give us a natural place to put additional information that is common to all responses.

Where “flexible version” is how they refer to message versions that implement tagged fields (and compact types). Note that this doc may not be fully accurate to the final implementation, as it mentions the client id being made a compact string when it is actually still just a nullable string to handle some compatibility issues in edge cases.

Topic		Replies	Views
Inconsistent ResponseHeader required - v0 vs V1 - ApiVersions vs Fetch Challenges challenge:kafka	2	78	October 12, 2024
Question about Handle APIVersions requests stage Challenges challenge:kafka	9	873	October 4, 2024
Question about nc5: Parse API Version: response message fields order Challenges challenge:kafka	8	112	October 21, 2024
Cant seem to solve the problem Challenges challenge:kafka	3	49	March 2, 2025
Question about Kafka: Parse correlation Id Challenges challenge:kafka	3	95	November 2, 2024

How to decide which Response Header to use?

Related topics