Question about Handle APIVersions requests stage

Challenges
1

@Eghizio moving this here.

The instructions seems to be off with what the actually is being tested:

  • The first 4 bytes of your response (the “message length”) are valid.
  • The correlation ID in the response header matches the correlation ID in the request header.
  • The error code in the response body is 0 (No Error).
  • The response body contains at least one entry for the API key 18 (API_VERSIONS).
  • The MaxVersion for the ApiKey 18 is at least 4.

While the tester seems to check more than that (for example throttle_time_ms which is not even mentioned in any of the challenge steps):

remote: [tester::#PV1] [Decoder] - .ResponseHeader
remote: [tester::#PV1] [Decoder]   ✔️ .correlation_id (810919229)
remote: [tester::#PV1] [Decoder] - .ResponseBody
remote: [tester::#PV1] [Decoder]   ✔️ .error_code (0)
remote: [tester::#PV1] [Decoder]   ✔️ .num_api_keys (0)
remote: [tester::#PV1] [Decoder]   ✔️ .throttle_time_ms (256)
remote: [tester::#PV1] Received:
remote: [tester::#PV1] Hex (bytes 21-25)                               | ASCII
remote: [tester::#PV1] ------------------------------------------------+------------------
remote: [tester::#PV1] 00 00 00 00 00                                  | .....
remote: [tester::#PV1]                 ^                                      ^
remote: [tester::#PV1] Error: Unexpected end of data
remote: [tester::#PV1] Context:
remote: [tester::#PV1] - ApiVersions v3
remote: [tester::#PV1]   - Response Body
remote: [tester::#PV1]     - TAG_BUFFER
remote: [tester::#PV1]       - TAGGED_FIELD_ARRAY
remote: [tester::#PV1]         - UNSIGNED_VARINT
remote: [tester::#PV1]
remote: [tester::#PV1] Test failed

Also there is a mention of ApiVersions v3 where in the challenge it is written that the tester will use v4.

1 Like
2

We’ll update the docs soon to mention v4 - this was a last minute change to match a soon-to-be-released version of Kafka that introduces v4. Details on this here: kafka/clients/src/main/resources/common/message/ApiVersionsRequest.json at 7a321f29a2495d519165206e50b10595a0ceae9c · apache/kafka · GitHub

3

Re: throttle_time_ms, we’ll add this to that list! Screenshot from official docs:

Screenshot 2024-09-15 at 00.19.25
Screenshot 2024-09-15 at 00.19.252014×476 23.3 KB

4

Hello !
Since we don’t parse the Request content, I am curious why not only implement ApiVersions Request v2 ?
Best,
Romain

1 Like
5

Since I spent the last two hours trying to determine what was expected from the tests and it wasn’t fun because the instructions are unclear and nowhere can we find the right sequence that’s expected.

I’ll share them here since it doesn’t seem the format of the expected response is listed on Kafka doc, it’s not exactly the v3 response as @rohitpaulk shared but rather something a bit different:

ApiVersions Response (Version: CodeCrafters) => error_code num_of_api_keys [api_keys] throttle_time_ms TAG_BUFFER
error_code => INT16
num_of_api_keys => INT8
api_keys => api_key min_version max_version
api_key => INT16
min_version => INT16
max_version => INT16
_tagged_fields
throttle_time_ms => INT32
_tagged_fields

Further explanations

For the num_of_api_keys I had to input 2 despite having only one API key, not sure why but the decoder seems to decode my 2 as a 1, when I input 1 it errors in the decode.

The _tagged_field is just a single byte with a value of 0

15 Likes
6

For num_of_api_keys, I think this is just the way lengths are encoded (see COMPACT_ARRAY type definition) :

Represents a sequence of objects of a given type T. Type T can be either a primitive type (e.g. STRING) or a structure. First, the length N + 1 is given as an UNSIGNED_VARINT. Then N instances of type T follow. A null array is represented with a length of 0. In protocol documentation an array of T instances is referred to as [T].

I don’t understand why they do this instead of just using 0 for 0, 1 for 1, … but adding this extra + 1 for non empty arrays made it work

Regarding the TAG_BUFFER, they said we don’t use tag in the CodeCrafter challenge, so it os just one byte to encode a length of 0 for the tag array. At least, that’s the more or less logical explanation I came up with after being puzzled just like you are and deriving this 0 on one byte empirically TAG_BUFFER explanation · romainfd/codecrafters-kafka-python@8b3ecf9 · GitHub

So I agree the instructions are not super clear but I think this is also a problem of the Kafka doc not always being super clear…

I also had a LOT of trouble finding out when to use Response Header V0 vs V1…

2 Likes
8

Since we don’t parse the Request content, I am curious why not only implement ApiVersions Request v2 ?

@romainfd We implemented ApiVersionsRequest:v4 to stay on the bleeding edge of Kafka. Our users probably don’t want to implement older versions of requests.

I don’t understand why they do this instead of just using 0 for 0, 1 for 1, … but adding this extra + 1 for non empty arrays made it work

For COMPACT_ARRAY, the reason why Kafka uses N + 1 as the length is to differentiate a null array (represented as 0) and an empty array (actual length of 0, but represented as 1).

I also had a LOT of trouble finding out when to use Response Header V0 vs V1

If you don’t need to support older versions of Kafka, use Response Header v1, because of KIP-482:

In order to have flexible version support across all requests and responses, we will bump the version of all requests and responses.
…
The new ResponseHeader version will be version 1, superseding version 0.

The only exception is that “ApiVersionsResponse always includes a v0 header”.

1 Like
9

Reverse engineering master. Instructions should be improved A LOT

7 Likes
11

Closing this thread due to inactivity. If you still need assistance, feel free to reopen or start a new discussion!