Clarifying Questions about pv1: Handle APIVersions requests - What's going wrong?

I honestly just don’t understand what I’m supposed to be doing for this one. The documentation provided is confusing and requirements aren’t particularly descriptive. I get an error when I push my code. This is what my code looks like:

package main

import (
	"encoding/binary"
	"fmt"
	"net"
	"os"
)

func main() {
	// First create a server that can connect on port 9092
	fmt.Println("Creating listener . . . ")
	l, err := net.Listen("tcp", "0.0.0.0:9092")
	if err != nil {
		fmt.Println("Failed to bind to port 9092")
		os.Exit(1)
	}
	defer l.Close()

	fmt.Println("Listening on port 9092 . . .")

	// Accept the incoming connection
	for {
		fmt.Println("Accepting an oncoming connection . . .")
		netConnection, err := l.Accept()
		if err != nil {
			fmt.Println("Error accepting connection: ", err.Error())
			continue
		}
		go handleConnection(netConnection)
	}
}

// This function takes in a net.Conn struct and obtains the header value from it. Following this, it will return a response
func handleConnection(conn net.Conn) {
	// 3 parts to our response
	var message_size int32
	var correlation_id int32 // This is part of the response header
	var request_api_version int16
	var request_api_key int16

	/*
		Steps:
		1.) The client generates a correlation_id
		2.) The client sends a request that includes the correlation_id
		3.) The server sends a response that includes the same correlation_id
		4.) The client receives the response and matches the received correlation_id to the original request
	*/

	// Getting the correlation ID:

	//Since we know the message size is 4 bytes long (i.e. 32 bits . . .) and the
	// correlationID is part of the header, where it is the 32nd through 64th bits, then we know that bits 64-86 are the necessary bits
	buffer := make([]byte, 12) // We need to take in 12 bytes
	_, err := conn.Read(buffer)
	if err != nil {
		fmt.Println("Failed to read message: ", err)
		os.Exit(1)
	}

	// message_size is first 4 bytes
	message_size = int32(binary.BigEndian.Uint32(buffer[0:4])) // A number value representing the number of bits/bytes including the 32 bits of the message_size portion!
	// Now we need to parse so that of the 12 bytes we have, we take the 8th through 11th bytes
	correlation_id = int32(binary.BigEndian.Uint32(buffer[8:12]))     // 4 bytes
	request_api_version = int16(binary.BigEndian.Uint16(buffer[6:8])) // 2 bytes
	request_api_key = int16(binary.BigEndian.Uint16(buffer[4:6]))     // 2 bytes

	//Now we want to send the binary values
	binary.Write(conn, binary.BigEndian, message_size)   // 4 bytes in length. The value we write may be subject to change
	binary.Write(conn, binary.BigEndian, correlation_id) // 4 bytes
	api_error_code := valid_version(request_api_version)
	binary.Write(conn, binary.BigEndian, api_error_code) // 2 bytes

	binary.Write(conn, binary.BigEndian, request_api_key) // 2 bytes
	binary.Write(conn, binary.BigEndian, int16(3))        //min version - 2 bytes
	binary.Write(conn, binary.BigEndian, int16(4))        //max version - 2 bytes
	binary.Write(conn, binary.BigEndian, int32(0))        // throttle time - 4 bytes

}

// Check that we're dealing with API version 4 or above!
func valid_version(api_version int16) int16 {
	if api_version > 4 {
		return 35
	}
	return 0
}

This is the error I get:

remote: ⏳ Turbo test runners busy. You are in queue.
remote: 
remote: Upgrade to skip the wait: https://codecrafters.io/turbo
remote: 
remote: Running tests on your code. Logs should appear shortly...
remote: 
remote: [compile] Moved ./.codecrafters/run.sh → ./your_program.sh
remote: [compile] Compilation successful.
remote: 
remote: Debug = true
remote: 
remote: [tester::#PV1] Running tests for Stage #PV1 (Handle APIVersions requests)
remote: [tester::#PV1] $ ./your_program.sh /tmp/server.properties
remote: [tester::#PV1] Connecting to broker at: localhost:9092
remote: [your_program] Creating listener . . . 
remote: [your_program] Listening on port 9092 . . .
remote: [your_program] Accepting an oncoming connection . . .
remote: [your_program] Accepting an oncoming connection . . .
remote: [tester::#PV1] Connection to broker at localhost:9092 successful
remote: [tester::#PV1] Sending "ApiVersions" (version: 4) request (Correlation id: 1578789074)
remote: [tester::#PV1] Hexdump of sent "ApiVersions" request: 
remote: [tester::#PV1] Idx  | Hex                                             | ASCII
remote: [tester::#PV1] -----+-------------------------------------------------+-----------------
remote: [tester::#PV1] 0000 | 00 00 00 23 00 12 00 04 5e 1a 68 d2 00 09 6b 61 | ...#....^.h...ka
remote: [tester::#PV1] 0010 | 66 6b 61 2d 63 6c 69 00 0a 6b 61 66 6b 61 2d 63 | fka-cli..kafka-c
remote: [tester::#PV1] 0020 | 6c 69 04 30 2e 31 00                            | li.0.1.
remote: [tester::#PV1] 
remote: [tester::#PV1] Hexdump of received "ApiVersions" response: 
remote: [tester::#PV1] Idx  | Hex                                             | ASCII
remote: [tester::#PV1] -----+-------------------------------------------------+-----------------
remote: [tester::#PV1] 0000 | 00 00 00 23 5e 1a 68 d2 00 00 00 12 00 03 00 04 | ...#^.h.........
remote: [tester::#PV1] 0010 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
remote: [tester::#PV1] 0020 | 00 00 00 00 00 00 00                            | .......
remote: [tester::#PV1] 
remote: [tester::#PV1] [Decoder] - .ResponseHeader
remote: [tester::#PV1] [Decoder]   - .correlation_id (1578789074)
remote: [tester::#PV1] [Decoder] - .ResponseBody
remote: [tester::#PV1] [Decoder]   - .error_code (0)
remote: [tester::#PV1] [Decoder]   - .num_api_keys (0)
remote: [tester::#PV1] [Decoder]   - .throttle_time_ms (301990656)
remote: [tester::#PV1] [Decoder]   - .TAG_BUFFER
remote: [tester::#PV1] Received:
remote: [tester::#PV1] Hex (bytes 15-30)                               | ASCII
remote: [tester::#PV1] ------------------------------------------------+------------------
remote: [tester::#PV1] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
remote: [tester::#PV1]                 ^                                      ^
remote: [tester::#PV1] Error: unexpected 15 bytes remaining in decoder after decoding ApiVersionsResponse
remote: [tester::#PV1] Context:
remote: [tester::#PV1] - ApiVersions v3
remote: [tester::#PV1]   - Response Body
remote: [tester::#PV1] 
remote: [tester::#PV1] Test failed
remote: [tester::#PV1] Terminating program
remote: [tester::#PV1] Program terminated successfully

Hey @omara569, could you upload your code to GitHub and share the link? It will be much easier to debug if I can run it directly.

Hi @andy1li , here you go: GitHub - omara569/Kafka-from-Scratch

Thanks!

@omara569 There might be a few issues at play. The first I noticed is that the API Versions Array is not encoded in the expected format:

I’d recommend checking BinSpec to review the expected format for reference.

Let me know if you’d like further clarification!

2 Likes

Thank you for the binspec visualizer, it’s incredibly helpful.

I guess for my followup question, for the format of the API Versions I’m not clear on what I’m supposed to be setting the values as for the tag buffer and throttle time.

Also, according to the BinSpec, I’m supposed to support 3 API Versions? I’m not clear on what those values are supposed to be set to - are they supposed to be obtained from the orignal message received?

Edit: If I change my code as shown in the screenshot below:

Pushing my code, it complains there’s an extra 2 bytes, which is an improvement over what I had previously. I followed the format shown in the BinSpec you shared (though I don’t know if I’m inputting valid values). I’ve also committed my code to the same github link as before if you’d like to run it yourself.

Also, according to the BinSpec, I’m supposed to support 3 API Versions?

BinSpec shows the expected format (the structure of the message), but not the exact content or values you should send.

For this stage, you only need to satisfy these two requirements:

In a real-world scenario, a broker would return the full list of supported API versions. For now, keeping it minimal to pass the tester is totally fine.


@omara569 The message_size doesn’t look correct. It should be calculated based on the response payload, not copied directly from the request:

That all seems to have done the trick! Thanks!

1 Like