# Question about pv1: Handle APIVersions requests: Tester receiving extra bytes

**URL:** <https://forum.codecrafters.io/t/question-about-pv1-handle-apiversions-requests-tester-receiving-extra-bytes/3178>\
**Category:** Challenges\
**Tags:** challenge:kafka\
**Created:** [November 28, 2024, 8:14am UTC](https://forum.codecrafters.io/t/question-about-pv1-handle-apiversions-requests-tester-receiving-extra-bytes/3178 "2024-11-28T08:14:05Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![silversword05](https://yyz1.discourse-cdn.com/flex003/user_avatar/forum.codecrafters.io/silversword05/32/3123_2.png) [@silversword05](https://forum.codecrafters.io/u/silversword05)\
**Post date:** [November 28, 2024, 8:14am UTC](https://forum.codecrafters.io/t/question-about-pv1-handle-apiversions-requests-tester-receiving-extra-bytes/3178/1 "2024-11-28T08:14:05Z")

</div>

When I try to replicate the test in my local environment, the code works but the tester fails.

Tester output:

```auto
......

[tester::#PV1] Running tests for Stage #PV1 (Handle APIVersions requests)
[tester::#PV1] $ ./your_program.sh /tmp/server.properties
[tester::#PV1] Connecting to broker at: localhost:9092
[your_program] Waiting for a client to connect...
[your_program] Logs from your program will appear here!
[tester::#PV1] Connection to broker at localhost:9092 successful
[tester::#PV1] Sending "ApiVersions" (version: 4) request (Correlation id: 526337192)
[tester::#PV1] Hexdump of sent "ApiVersions" request:
[tester::#PV1] Idx | Hex | ASCII
[tester::#PV1] -----+-------------------------------------------------+-----------------
[tester::#PV1] 0000 | 00 00 00 23 00 12 00 04 1f 5f 44 a8 00 09 6b 61 | ...#....._D...ka
[tester::#PV1] 0010 | 66 6b 61 2d 63 6c 69 00 0a 6b 61 66 6b 61 2d 63 | fka-cli..kafka-c
[tester::#PV1] 0020 | 6c 69 04 30 2e 31 00 | li.0.1.
[tester::#PV1]
[your_program] Client connected
[your_program] Received 39 bytes from client
[your_program] Received API Versions Request: ApiVersionsRequestMessage{RequestHeader{message_size=35, request_api_key=18, request_api_version=4, corellation_id=526337192, client_id=kafka-clo}}
[your_program] Supported version: 4
[your_program] Sending msg to client: ApiVersionsResponseMessage{message_size=23, corellation_id=526337192, error_code=0, api_key=18, api_keys_count=2, min_version=2, max_version=2, throttle_time=0, tagged_fields=TaggedFields{fieldCount=0}, tagged_fields2=TaggedFields{fieldCount=0}}
[your_program] Message sent to client: 23 bytes
[tester::#PV1] Hexdump of received "ApiVersions" response:
[tester::#PV1] Idx | Hex | ASCII
[tester::#PV1] -----+-------------------------------------------------+-----------------
[tester::#PV1] 0000 | 00 00 00 17 1f 5f 44 a8 00 00 02 00 12 00 02 00 | ....._D.........
[tester::#PV1] 0010 | 02 00 00 00 00 00 00 00 00 00 00 | ...........
[tester::#PV1]
[tester::#PV1] [Decoder] - .ResponseHeader
[tester::#PV1] [Decoder] - .correlation_id (526337192)
[tester::#PV1] [Decoder] - .ResponseBody
[tester::#PV1] [Decoder] - .error_code (0)
[tester::#PV1] [Decoder] - .num_api_keys (1)
[tester::#PV1] [Decoder] - .ApiKeys[0]
[tester::#PV1] [Decoder] - .api_key (18)
[tester::#PV1] [Decoder] - .min_version (2)
[tester::#PV1] [Decoder] - .max_version (2)
[tester::#PV1] [Decoder] - .TAG_BUFFER
[tester::#PV1] [Decoder] - .throttle_time_ms (0)
[tester::#PV1] [Decoder] - .TAG_BUFFER
[tester::#PV1] Received:
[tester::#PV1] Hex (bytes 14-22) | ASCII
[tester::#PV1] ------------------------------------------------+------------------
[tester::#PV1] 00 00 00 00 00 00 00 00 00 | .........
[tester::#PV1] ^ ^
[tester::#PV1] Error: unexpected 4 bytes remaining in decoder after decoding ApiVersionsResponse
[tester::#PV1] Context:
[tester::#PV1] - ApiVersions v3
[tester::#PV1] - Response Body
[tester::#PV1]
[tester::#PV1] Test failed
[tester::#PV1] Terminating program
[tester::#PV1] Program terminated successfully
[tester::#PV1] Running tests for Stage #PV1 (Handle APIVersions requests)
[tester::#PV1] $ ./your_program.sh /tmp/server.properties
[tester::#PV1] Connecting to broker at: localhost:9092
[your_program] Waiting for a client to connect...
[your_program] Logs from your program will appear here!
[tester::#PV1] Connection to broker at localhost:9092 successful
[tester::#PV1] Sending "ApiVersions" (version: 4) request (Correlation id: 526337192)
[tester::#PV1] Hexdump of sent "ApiVersions" request:
[tester::#PV1] Idx | Hex | ASCII
[tester::#PV1] -----+-------------------------------------------------+-----------------
[tester::#PV1] 0000 | 00 00 00 23 00 12 00 04 1f 5f 44 a8 00 09 6b 61 | ...#....._D...ka
[tester::#PV1] 0010 | 66 6b 61 2d 63 6c 69 00 0a 6b 61 66 6b 61 2d 63 | fka-cli..kafka-c
[tester::#PV1] 0020 | 6c 69 04 30 2e 31 00 | li.0.1.
[tester::#PV1]
[your_program] Client connected
[your_program] Received 39 bytes from client
[your_program] Received API Versions Request: ApiVersionsRequestMessage{RequestHeader{message_size=35, request_api_key=18, request_api_version=4, corellation_id=526337192, client_id=kafka-clo}}
[your_program] Supported version: 4
[your_program] Sending msg to client: ApiVersionsResponseMessage{message_size=23, corellation_id=526337192, error_code=0, api_key=18, api_keys_count=2, min_version=2, max_version=2, throttle_time=0, tagged_fields=TaggedFields{fieldCount=0}, tagged_fields2=TaggedFields{fieldCount=0}}
[your_program] Message sent to client: 23 bytes
[tester::#PV1] Hexdump of received "ApiVersions" response:
[tester::#PV1] Idx | Hex | ASCII
[tester::#PV1] -----+-------------------------------------------------+-----------------
[tester::#PV1] 0000 | 00 00 00 17 1f 5f 44 a8 00 00 02 00 12 00 02 00 | ....._D.........
[tester::#PV1] 0010 | 02 00 00 00 00 00 00 00 00 00 00 | ...........
[tester::#PV1]
[tester::#PV1] [Decoder] - .ResponseHeader
[tester::#PV1] [Decoder] - .correlation_id (526337192)
[tester::#PV1] [Decoder] - .ResponseBody
[tester::#PV1] [Decoder] - .error_code (0)
[tester::#PV1] [Decoder] - .num_api_keys (1)
[tester::#PV1] [Decoder] - .ApiKeys[0]
[tester::#PV1] [Decoder] - .api_key (18)
[tester::#PV1] [Decoder] - .min_version (2)
[tester::#PV1] [Decoder] - .max_version (2)
[tester::#PV1] [Decoder] - .TAG_BUFFER
[tester::#PV1] [Decoder] - .throttle_time_ms (0)
[tester::#PV1] [Decoder] - .TAG_BUFFER
[tester::#PV1] Received:
[tester::#PV1] Hex (bytes 14-22) | ASCII
[tester::#PV1] ------------------------------------------------+------------------
[tester::#PV1] 00 00 00 00 00 00 00 00 00 | .........
[tester::#PV1] ^ ^
[tester::#PV1] Error: unexpected 4 bytes remaining in decoder after decoding ApiVersionsResponse
[tester::#PV1] Context:
[tester::#PV1] - ApiVersions v3
[tester::#PV1] - Response Body
[tester::#PV1]
[tester::#PV1] Test failed
[tester::#PV1] Terminating program
[tester::#PV1] Program terminated successfully

```

But trying to replicate the same in my local environment

```auto
./your_program.sh # in terminal 1
-- Running vcpkg install
All requested packages are currently installed.
Total install time: 1.46 us
-- Running vcpkg install - done
-- Configuring done (0.5s)
-- Generating done (0.2s)
-- Build files have been written to: /mnt/d/CodeCrafters/codecrafters-kafka-cpp/build
[100%] Built target kafka
Waiting for a client to connect...
Logs from your program will appear here!
Client connected
Received 39 bytes from client
Received API Versions Request: ApiVersionsRequestMessage{RequestHeader{message_size=35, request_api_key=18, request_api_version=4, corellation_id=1113080347, client_id=@j�@��}}
Supported version: 4
Sending msg to client: ApiVersionsResponseMessage{message_size=23, corellation_id=1113080347, error_code=0, api_key=18, api_keys_count=2, min_version=2, max_version=2, throttle_time=0, tagged_fields=TaggedFields{fieldCount=0}, tagged_fields2=TaggedFields{fieldCount=0}}
Message sent to client: 23 bytes
Closing file descriptor 4
Closing file descriptor 3

$ echo -n "00000023001200044258421b00096b61666b612d636c69000a6b61666b612d636c6904302e3100" | xxd -r -p | nc localhost 9092 | hexdump -C # in terminal 2
00000000 00 00 00 17 42 58 42 1b 00 00 02 00 12 00 02 00 |....BXB.........|
00000010 02 00 00 00 00 00 00 |.......|
00000017

$ echo -n "00000023001200044258421b00096b61666b612d636c69000a6b61666b612d636c6904302e3100" | xxd -r -p | nc localhost 9092 | wc --bytes # in terminal 2
23

```

I am not sure why the tester is receiving more than 23 bytes while my localhost receives exactly 23 bytes.

---

<div class="post-metadata">

**Author:** ![silversword05](https://yyz1.discourse-cdn.com/flex003/user_avatar/forum.codecrafters.io/silversword05/32/3123_2.png) [@silversword05](https://forum.codecrafters.io/u/silversword05)\
**Post date:** [November 28, 2024, 8:17am UTC](https://forum.codecrafters.io/t/question-about-pv1-handle-apiversions-requests-tester-receiving-extra-bytes/3178/2 "2024-11-28T08:17:39Z")

</div>

My write function for reference.

```cpp
// Uses C++20 concepts. Templated function on a 23 byte struct
void TCPManager::writeBufferOnClientFd(const Fd &client_fd,
                                       const auto &response_message) const {

    std::cout << "Sending msg to client: " << response_message.toString()
              << "\n";

    std::string buffer = response_message.toBuffer();

    // Write message Length
    if (write(client_fd, buffer.data(), buffer.size()) != buffer.size()) {
        perror("send failed: ");
        throw std::runtime_error("Failed to send msgLen to client: ");
    }

    std::cout << "Message sent to client: " << buffer.size() << " bytes\n";

    // Flush and close write side
    // int optval = 1;
    // setsockopt(client_fd, IPPROTO_TCP, TCP_NODELAY, &optval, sizeof(optval));
    // shutdown(client_fd, SHUT_WR);

    // Hack to keep the program running for a while so that netcat can read the
    // buffer
    using namespace std::chrono_literals;
    std::this_thread::sleep_for(1000ms);
}

```

I have also pushed my code to master for reference.

---

<div class="post-metadata">

**Author:** ![andy1li](https://yyz1.discourse-cdn.com/flex003/user_avatar/forum.codecrafters.io/andy1li/32/10_2.png) [@andy1li](https://forum.codecrafters.io/u/andy1li)\
**Post date:** [November 28, 2024, 8:21am UTC](https://forum.codecrafters.io/t/question-about-pv1-handle-apiversions-requests-tester-receiving-extra-bytes/3178/3 "2024-11-28T08:21:02Z")

</div>

Hi @silversword05, could you elaborate a bit on “the tester is receiving more than 23 bytes”?

---

<div class="post-metadata">

**Author:** ![silversword05](https://yyz1.discourse-cdn.com/flex003/user_avatar/forum.codecrafters.io/silversword05/32/3123_2.png) [@silversword05](https://forum.codecrafters.io/u/silversword05)\
**Post date:** [November 28, 2024, 2:25pm UTC](https://forum.codecrafters.io/t/question-about-pv1-handle-apiversions-requests-tester-receiving-extra-bytes/3178/4 "2024-11-28T14:25:48Z")

</div>

Hi @andy1li , thanks for looking into this. I was trying multiple things and I pasted the wrong output. I have updated the write functions and output slightly. You can now see that the tester receives 4 extra bytes that I don’t see when I try to repeat this locally.

Can you provide me the exact command the tester is running so that I can try it locally?

---

<div class="post-metadata">

**Author:** ![andy1li](https://yyz1.discourse-cdn.com/flex003/user_avatar/forum.codecrafters.io/andy1li/32/10_2.png) [@andy1li](https://forum.codecrafters.io/u/andy1li)\
**Post date:** [November 28, 2024, 3:12pm UTC](https://forum.codecrafters.io/t/question-about-pv1-handle-apiversions-requests-tester-receiving-extra-bytes/3178/5 "2024-11-28T15:12:30Z")

</div>

Absolutely, here’s the source code of our tester for this stage (#PV1):

> <https://github.com/codecrafters-io/kafka-tester/blob/main/internal/stage5.go>

---

<div class="post-metadata">

**Author:** ![silversword05](https://yyz1.discourse-cdn.com/flex003/user_avatar/forum.codecrafters.io/silversword05/32/3123_2.png) [@silversword05](https://forum.codecrafters.io/u/silversword05)\
**Post date:** [November 28, 2024, 3:19pm UTC](https://forum.codecrafters.io/t/question-about-pv1-handle-apiversions-requests-tester-receiving-extra-bytes/3178/6 "2024-11-28T15:19:54Z")

</div>

Okay I could resolve the issue. Basically, I was writing message size wrong (message size should exclude the first 4 bytes of itself). That’s why the tester was seeing the extra 4 bytes.

TBH I think the description should include message formats and details like this. I used “strace” with the code example (thanks to you for providing one working example) to trace the network calls and discovered this issue.

---

<div class="post-metadata">

**Author:** ![andy1li](https://yyz1.discourse-cdn.com/flex003/user_avatar/forum.codecrafters.io/andy1li/32/10_2.png) [@andy1li](https://forum.codecrafters.io/u/andy1li)\
**Post date:** [November 29, 2024, 12:58am UTC](https://forum.codecrafters.io/t/question-about-pv1-handle-apiversions-requests-tester-receiving-extra-bytes/3178/7 "2024-11-29T00:58:07Z")

</div>

We’re still working on releasing the instructions for stage #PV1. In the meantime, feel free to review the structure of API Versions Response (v4) at [BinspecVisualizer](https://binspec.org/kafka-api-versions-Response-v4).

---

<div class="post-metadata">

**Author:** ![system](https://canada1.discourse-cdn.com/flex003/uploads/codecrafters/original/3X/7/0/700657133935c15703e22c7c8870394f6e6dc27d.svg) [@system](https://forum.codecrafters.io/u/system)\
**Post date:** [December 4, 2024, 12:58am UTC](https://forum.codecrafters.io/t/question-about-pv1-handle-apiversions-requests-tester-receiving-extra-bytes/3178/8 "2024-12-04T00:58:23Z")

</div>

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.