Question about pv1: Handle APIVersions requests: Tester receiving extra bytes

silversword05 · November 28, 2024, 8:14am

When I try to replicate the test in my local environment, the code works but the tester fails.

Tester output:

......

[tester::#PV1] Running tests for Stage #PV1 (Handle APIVersions requests)
[tester::#PV1] $ ./your_program.sh /tmp/server.properties
[tester::#PV1] Connecting to broker at: localhost:9092
[your_program] Waiting for a client to connect...
[your_program] Logs from your program will appear here!
[tester::#PV1] Connection to broker at localhost:9092 successful
[tester::#PV1] Sending "ApiVersions" (version: 4) request (Correlation id: 526337192)
[tester::#PV1] Hexdump of sent "ApiVersions" request:
[tester::#PV1] Idx  | Hex                                             | ASCII
[tester::#PV1] -----+-------------------------------------------------+-----------------
[tester::#PV1] 0000 | 00 00 00 23 00 12 00 04 1f 5f 44 a8 00 09 6b 61 | ...#....._D...ka
[tester::#PV1] 0010 | 66 6b 61 2d 63 6c 69 00 0a 6b 61 66 6b 61 2d 63 | fka-cli..kafka-c
[tester::#PV1] 0020 | 6c 69 04 30 2e 31 00                            | li.0.1.
[tester::#PV1]
[your_program] Client connected
[your_program] Received 39 bytes from client
[your_program] Received API Versions Request: ApiVersionsRequestMessage{RequestHeader{message_size=35, request_api_key=18, request_api_version=4, corellation_id=526337192, client_id=kafka-clo}}
[your_program] Supported version: 4
[your_program] Sending msg to client: ApiVersionsResponseMessage{message_size=23, corellation_id=526337192, error_code=0, api_key=18, api_keys_count=2, min_version=2, max_version=2, throttle_time=0, tagged_fields=TaggedFields{fieldCount=0}, tagged_fields2=TaggedFields{fieldCount=0}}
[your_program] Message sent to client: 23 bytes
[tester::#PV1] Hexdump of received "ApiVersions" response:
[tester::#PV1] Idx  | Hex                                             | ASCII
[tester::#PV1] -----+-------------------------------------------------+-----------------
[tester::#PV1] 0000 | 00 00 00 17 1f 5f 44 a8 00 00 02 00 12 00 02 00 | ....._D.........
[tester::#PV1] 0010 | 02 00 00 00 00 00 00 00 00 00 00                | ...........
[tester::#PV1]
[tester::#PV1] [Decoder] - .ResponseHeader
[tester::#PV1] [Decoder]   - .correlation_id (526337192)
[tester::#PV1] [Decoder] - .ResponseBody
[tester::#PV1] [Decoder]   - .error_code (0)
[tester::#PV1] [Decoder]   - .num_api_keys (1)
[tester::#PV1] [Decoder]   - .ApiKeys[0]
[tester::#PV1] [Decoder]     - .api_key (18)
[tester::#PV1] [Decoder]     - .min_version (2)
[tester::#PV1] [Decoder]     - .max_version (2)
[tester::#PV1] [Decoder]     - .TAG_BUFFER
[tester::#PV1] [Decoder]   - .throttle_time_ms (0)
[tester::#PV1] [Decoder]   - .TAG_BUFFER
[tester::#PV1] Received:
[tester::#PV1] Hex (bytes 14-22)                               | ASCII
[tester::#PV1] ------------------------------------------------+------------------
[tester::#PV1] 00 00 00 00 00 00 00 00 00                      | .........
[tester::#PV1]                 ^                                      ^
[tester::#PV1] Error: unexpected 4 bytes remaining in decoder after decoding ApiVersionsResponse
[tester::#PV1] Context:
[tester::#PV1] - ApiVersions v3
[tester::#PV1]   - Response Body
[tester::#PV1]
[tester::#PV1] Test failed
[tester::#PV1] Terminating program
[tester::#PV1] Program terminated successfully
[tester::#PV1] Running tests for Stage #PV1 (Handle APIVersions requests)
[tester::#PV1] $ ./your_program.sh /tmp/server.properties
[tester::#PV1] Connecting to broker at: localhost:9092
[your_program] Waiting for a client to connect...
[your_program] Logs from your program will appear here!
[tester::#PV1] Connection to broker at localhost:9092 successful
[tester::#PV1] Sending "ApiVersions" (version: 4) request (Correlation id: 526337192)
[tester::#PV1] Hexdump of sent "ApiVersions" request:
[tester::#PV1] Idx  | Hex                                             | ASCII
[tester::#PV1] -----+-------------------------------------------------+-----------------
[tester::#PV1] 0000 | 00 00 00 23 00 12 00 04 1f 5f 44 a8 00 09 6b 61 | ...#....._D...ka
[tester::#PV1] 0010 | 66 6b 61 2d 63 6c 69 00 0a 6b 61 66 6b 61 2d 63 | fka-cli..kafka-c
[tester::#PV1] 0020 | 6c 69 04 30 2e 31 00                            | li.0.1.
[tester::#PV1]
[your_program] Client connected
[your_program] Received 39 bytes from client
[your_program] Received API Versions Request: ApiVersionsRequestMessage{RequestHeader{message_size=35, request_api_key=18, request_api_version=4, corellation_id=526337192, client_id=kafka-clo}}
[your_program] Supported version: 4
[your_program] Sending msg to client: ApiVersionsResponseMessage{message_size=23, corellation_id=526337192, error_code=0, api_key=18, api_keys_count=2, min_version=2, max_version=2, throttle_time=0, tagged_fields=TaggedFields{fieldCount=0}, tagged_fields2=TaggedFields{fieldCount=0}}
[your_program] Message sent to client: 23 bytes
[tester::#PV1] Hexdump of received "ApiVersions" response:
[tester::#PV1] Idx  | Hex                                             | ASCII
[tester::#PV1] -----+-------------------------------------------------+-----------------
[tester::#PV1] 0000 | 00 00 00 17 1f 5f 44 a8 00 00 02 00 12 00 02 00 | ....._D.........
[tester::#PV1] 0010 | 02 00 00 00 00 00 00 00 00 00 00                | ...........
[tester::#PV1]
[tester::#PV1] [Decoder] - .ResponseHeader
[tester::#PV1] [Decoder]   - .correlation_id (526337192)
[tester::#PV1] [Decoder] - .ResponseBody
[tester::#PV1] [Decoder]   - .error_code (0)
[tester::#PV1] [Decoder]   - .num_api_keys (1)
[tester::#PV1] [Decoder]   - .ApiKeys[0]
[tester::#PV1] [Decoder]     - .api_key (18)
[tester::#PV1] [Decoder]     - .min_version (2)
[tester::#PV1] [Decoder]     - .max_version (2)
[tester::#PV1] [Decoder]     - .TAG_BUFFER
[tester::#PV1] [Decoder]   - .throttle_time_ms (0)
[tester::#PV1] [Decoder]   - .TAG_BUFFER
[tester::#PV1] Received:
[tester::#PV1] Hex (bytes 14-22)                               | ASCII
[tester::#PV1] ------------------------------------------------+------------------
[tester::#PV1] 00 00 00 00 00 00 00 00 00                      | .........
[tester::#PV1]                 ^                                      ^
[tester::#PV1] Error: unexpected 4 bytes remaining in decoder after decoding ApiVersionsResponse
[tester::#PV1] Context:
[tester::#PV1] - ApiVersions v3
[tester::#PV1]   - Response Body
[tester::#PV1]
[tester::#PV1] Test failed
[tester::#PV1] Terminating program
[tester::#PV1] Program terminated successfully

But trying to replicate the same in my local environment

./your_program.sh # in terminal 1
-- Running vcpkg install
All requested packages are currently installed.
Total install time: 1.46 us
-- Running vcpkg install - done
-- Configuring done (0.5s)
-- Generating done (0.2s)
-- Build files have been written to: /mnt/d/CodeCrafters/codecrafters-kafka-cpp/build
[100%] Built target kafka
Waiting for a client to connect...
Logs from your program will appear here!
Client connected
Received 39 bytes from client
Received API Versions Request: ApiVersionsRequestMessage{RequestHeader{message_size=35, request_api_key=18, request_api_version=4, corellation_id=1113080347, client_id=@j�@��}}
Supported version: 4
Sending msg to client: ApiVersionsResponseMessage{message_size=23, corellation_id=1113080347, error_code=0, api_key=18, api_keys_count=2, min_version=2, max_version=2, throttle_time=0, tagged_fields=TaggedFields{fieldCount=0}, tagged_fields2=TaggedFields{fieldCount=0}}
Message sent to client: 23 bytes
Closing file descriptor 4
Closing file descriptor 3

$ echo -n "00000023001200044258421b00096b61666b612d636c69000a6b61666b612d636c6904302e3100" | xxd -r -p | nc localhost 9092 | hexdump -C # in terminal 2
00000000  00 00 00 17 42 58 42 1b  00 00 02 00 12 00 02 00  |....BXB.........|
00000010  02 00 00 00 00 00 00                              |.......|
00000017

$  echo -n "00000023001200044258421b00096b61666b612d636c69000a6b61666b612d636c6904302e3100" | xxd -r -p | nc localhost 9092 | wc --bytes # in terminal 2
23

I am not sure why the tester is receiving more than 23 bytes while my localhost receives exactly 23 bytes.

silversword05 · November 28, 2024, 8:17am

My write function for reference.

// Uses C++20 concepts. Templated function on a 23 byte struct
void TCPManager::writeBufferOnClientFd(const Fd &client_fd,
                                       const auto &response_message) const {

    std::cout << "Sending msg to client: " << response_message.toString()
              << "\n";

    std::string buffer = response_message.toBuffer();

    // Write message Length
    if (write(client_fd, buffer.data(), buffer.size()) != buffer.size()) {
        perror("send failed: ");
        throw std::runtime_error("Failed to send msgLen to client: ");
    }

    std::cout << "Message sent to client: " << buffer.size() << " bytes\n";

    // Flush and close write side
    // int optval = 1;
    // setsockopt(client_fd, IPPROTO_TCP, TCP_NODELAY, &optval, sizeof(optval));
    // shutdown(client_fd, SHUT_WR);

    // Hack to keep the program running for a while so that netcat can read the
    //  buffer
    using namespace std::chrono_literals;
    std::this_thread::sleep_for(1000ms);
}

I have also pushed my code to master for reference.

andy1li · November 28, 2024, 8:21am

Hi @silversword05, could you elaborate a bit on “the tester is receiving more than 23 bytes”?

silversword05 · November 28, 2024, 2:25pm

Hi @andy1li , thanks for looking into this. I was trying multiple things and I pasted the wrong output. I have updated the write functions and output slightly. You can now see that the tester receives 4 extra bytes that I don’t see when I try to repeat this locally.

Can you provide me the exact command the tester is running so that I can try it locally?

andy1li · November 28, 2024, 3:12pm

Absolutely, here’s the source code of our tester for this stage (#PV1):

github.com

codecrafters-io/kafka-tester/blob/main/internal/stage5.go

package internal

import (
	"fmt"

	"github.com/codecrafters-io/kafka-tester/internal/kafka_executable"
	"github.com/codecrafters-io/kafka-tester/protocol"
	kafkaapi "github.com/codecrafters-io/kafka-tester/protocol/api"
	"github.com/codecrafters-io/kafka-tester/protocol/serializer"
	"github.com/codecrafters-io/tester-utils/logger"
	"github.com/codecrafters-io/tester-utils/test_case_harness"
)

func testAPIVersion(stageHarness *test_case_harness.TestCaseHarness) error {
	b := kafka_executable.NewKafkaExecutable(stageHarness)
	if err := b.Run(); err != nil {
		return err
	}

	quietLogger := logger.GetQuietLogger("")

This file has been truncated. show original

silversword05 · November 28, 2024, 3:19pm

Okay I could resolve the issue. Basically, I was writing message size wrong (message size should exclude the first 4 bytes of itself). That’s why the tester was seeing the extra 4 bytes.

TBH I think the description should include message formats and details like this. I used “strace” with the code example (thanks to you for providing one working example) to trace the network calls and discovered this issue.

andy1li · November 29, 2024, 12:58am

We’re still working on releasing the instructions for stage #PV1. In the meantime, feel free to review the structure of API Versions Response (v4) at BinspecVisualizer.

Topic		Replies	Views
Clarifying Questions about pv1: Handle APIVersions requests - What's going wrong? Challenges challenge:kafka	7	178	May 11, 2025
Bytes left into the decode buffer Challenges challenge:kafka	9	139	June 7, 2025
Question about pv1: Handle APIVersions requests Challenges challenge:kafka	3	284	October 31, 2024
Question about pv1: Handle APIVersions requests in rust Challenges challenge:kafka	10	291	December 15, 2024
[tester::#PV1] error reading from connection: unexpected EOF Challenges challenge:kafka	6	299	November 4, 2024

Question about pv1: Handle APIVersions requests: Tester receiving extra bytes

Related topics