I’m stuck on Stage #FD8.
I’ve tried debugging to capture my response locally (using codecrafters test) and passing it to my lib, try to debug it hoping to find error, but it pass, I mean the response can be parse…
Here are my logs:
vscode ➜ /workspaces/codecrafters-kafka-rust (master) $ codecrafters test
Initiating test run...
⏳ Turbo test runners busy. You are in queue.
Upgrade to skip the wait: https://codecrafters.io/turbo
Running tests. Logs should appear shortly...
[compile] Compiling codecrafters-kafka v0.1.0 (/app)
[compile] Finished `release` profile [optimized] target(s) in 3.42s
[compile] Moved ./.codecrafters/run.sh → ./your_program.sh
[compile] Compilation successful.
Debug = true
[tester::#FD8] Running tests for Stage #FD8 (Consuming Messages - Fetch multiple messages from disk)
[tester::#FD8] [Serializer] Writing log files to: /tmp/kraft-combined-logs
[tester::#FD8] [Serializer] - Wrote file to: /tmp/server.properties
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/meta.properties
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/.kafka_cleanshutdown
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/baz-0/partition.metadata
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/pax-0/partition.metadata
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/paz-0/partition.metadata
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/paz-1/partition.metadata
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/__cluster_metadata-0/partition.metadata
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/baz-0/00000000000000000000.log
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/pax-0/00000000000000000000.log
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/paz-0/00000000000000000000.log
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/paz-1/00000000000000000000.log
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log
[tester::#FD8] [Serializer] Finished writing log files to: /tmp/kraft-combined-logs
[tester::#FD8] $ ./your_program.sh /tmp/server.properties
[tester::#FD8] Connecting to broker at: localhost:9092
[your_program] Logs from your program will appear here!
[tester::#FD8] Connection to broker at localhost:9092 successful
[tester::#FD8] Sending "Fetch" (version: 16) request (Correlation id: 1983587668)
[tester::#FD8] Hexdump of sent "Fetch" request:
[tester::#FD8] Idx | Hex | ASCII
[tester::#FD8] -----+-------------------------------------------------+-----------------
[tester::#FD8] 0000 | 00 00 00 60 00 01 00 10 76 3b 25 54 00 09 6b 61 | ...`....v;%T..ka
[tester::#FD8] 0010 | 66 6b 61 2d 63 6c 69 00 00 00 01 f4 00 00 00 01 | fka-cli.........
[tester::#FD8] 0020 | 03 20 00 00 00 00 00 00 00 00 00 00 00 02 00 00 | . ..............
[tester::#FD8] 0030 | 00 00 00 00 40 00 80 00 00 00 00 00 00 30 02 00 | ....@........0..
[tester::#FD8] 0040 | 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00 ff | ................
[tester::#FD8] 0050 | ff ff ff ff ff ff ff ff ff ff ff 00 10 00 00 00 | ................
[tester::#FD8] 0060 | 00 01 01 00 | ....
[tester::#FD8]
[your_program] accepted new connection
[your_program] read 167 bytes from paz-0/00000000000000000000.log
[your_program] read 0 bytes from paz-1/00000000000000000000.log
[tester::#FD8] Hexdump of received "Fetch" response:
[tester::#FD8] Idx | Hex | ASCII
[tester::#FD8] -----+-------------------------------------------------+-----------------
[tester::#FD8] 0000 | 00 00 01 15 76 3b 25 54 00 00 00 00 00 00 00 00 | ....v;%T........
[tester::#FD8] 0010 | 00 00 00 02 00 00 00 00 00 00 40 00 80 00 00 00 | ..........@.....
[tester::#FD8] 0020 | 00 00 00 30 03 00 00 00 00 00 00 00 00 00 00 00 | ...0............
[tester::#FD8] 0030 | 00 00 00 ff ff ff ff ff ff ff ff ff ff ff ff ff | ................
[tester::#FD8] 0040 | ff ff ff 00 ff ff ff ff a8 01 00 00 00 00 00 00 | ................
[tester::#FD8] 0050 | 00 00 00 00 00 44 00 00 00 00 02 ab fd 04 91 00 | .....D..........
[tester::#FD8] 0060 | 00 00 00 00 00 00 00 01 91 e0 5b 6d 8b 00 00 01 | ..........[m....
[tester::#FD8] 0070 | 91 e0 5b 6d 8b 00 00 00 00 00 00 00 00 00 00 00 | ..[m............
[tester::#FD8] 0080 | 00 00 00 00 00 00 01 24 00 00 00 01 18 48 65 6c | .......$.....Hel
[tester::#FD8] 0090 | 6c 6f 20 4b 61 66 6b 61 21 00 00 00 00 00 00 00 | lo Kafka!.......
[tester::#FD8] 00a0 | 00 01 00 00 00 4b 00 00 00 00 02 55 60 53 93 00 | .....K.....U`S..
[tester::#FD8] 00b0 | 00 00 00 00 00 00 00 01 91 e0 5b 6d 8b 00 00 01 | ..........[m....
[tester::#FD8] 00c0 | 91 e0 5b 6d 8b 00 00 00 00 00 00 00 00 00 00 00 | ..[m............
[tester::#FD8] 00d0 | 00 00 00 00 00 00 01 32 00 00 00 01 26 48 65 6c | .......2....&Hel
[tester::#FD8] 00e0 | 6c 6f 20 43 6f 64 65 43 72 61 66 74 65 72 73 21 | lo CodeCrafters!
[tester::#FD8] 00f0 | 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 | ................
[tester::#FD8] 0100 | ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff | ................
[tester::#FD8] 0110 | 00 ff ff ff ff 00 00 00 00 | .........
[tester::#FD8]
[tester::#FD8] [Decoder] - .ResponseHeader
[tester::#FD8] [Decoder] - .correlation_id (1983587668)
[tester::#FD8] [Decoder] - .TAG_BUFFER
[tester::#FD8] [Decoder] - .ResponseBody
[tester::#FD8] [Decoder] - .throttle_time_ms (0)
[tester::#FD8] [Decoder] - .error_code (0)
[tester::#FD8] [Decoder] - .session_id (0)
[tester::#FD8] [Decoder] - .num_responses (1)
[tester::#FD8] [Decoder] - .TopicResponse[0]
[tester::#FD8] [Decoder] - .topic_id (00000000-0000-4000-8000-000000000030)
[tester::#FD8] [Decoder] - .num_partitions (2)
[tester::#FD8] [Decoder] - .PartitionResponse[0]
[tester::#FD8] [Decoder] - .partition_index (0)
[tester::#FD8] [Decoder] - .error_code (0)
[tester::#FD8] [Decoder] - .high_watermark (0)
[tester::#FD8] [Decoder] - .last_stable_offset (-1)
[tester::#FD8] [Decoder] - .log_start_offset (-1)
[tester::#FD8] [Decoder] - .num_aborted_transactions (0)
[tester::#FD8] [Decoder] - .preferred_read_replica (-1)
[tester::#FD8] [Decoder] - .compact_records_length (167)
[tester::#FD8] [Decoder] - .RecordBatch[0]
[tester::#FD8] [Decoder] - .base_offset (0)
[tester::#FD8] [Decoder] - .batch_length (68)
[tester::#FD8] [Decoder] - .partition_leader_epoch (0)
[tester::#FD8] [Decoder] - .magic_byte (2)
[tester::#FD8] [Decoder] - .crc (-1409481583)
[tester::#FD8] [Decoder] - .record_attributes (0)
[tester::#FD8] [Decoder] - .last_offset_delta (0)
[tester::#FD8] [Decoder] - .base_timestamp (1726045973899)
[tester::#FD8] [Decoder] - .max_timestamp (1726045973899)
[tester::#FD8] [Decoder] - .producer_id (0)
[tester::#FD8] [Decoder] - .producer_epoch (0)
[tester::#FD8] [Decoder] - .base_sequence (0)
[tester::#FD8] [Decoder] - .num_records (1)
[tester::#FD8] [Decoder] - .Record[0]
[tester::#FD8] [Decoder] - .length (18)
[tester::#FD8] [Decoder] - .attributes (0)
[tester::#FD8] [Decoder] - .timestamp_delta (0)
[tester::#FD8] [Decoder] - .offset_delta (0)
[tester::#FD8] [Decoder] - .key_length (-1)
[tester::#FD8] [Decoder] - .key ("")
[tester::#FD8] [Decoder] - .value_length (12)
[tester::#FD8] [Decoder] - .value ("Hello Kafka!")
[tester::#FD8] [Decoder] - .num_headers (0)
[tester::#FD8] [Decoder] - .RecordBatch[1]
[tester::#FD8] [Decoder] - .base_offset (1)
[tester::#FD8] [Decoder] - .batch_length (75)
[tester::#FD8] [Decoder] - .partition_leader_epoch (0)
[tester::#FD8] [Decoder] - .magic_byte (2)
[tester::#FD8] [Decoder] - .crc (1432376211)
[tester::#FD8] [Decoder] - .record_attributes (0)
[tester::#FD8] [Decoder] - .last_offset_delta (0)
[tester::#FD8] [Decoder] - .base_timestamp (1726045973899)
[tester::#FD8] [Decoder] - .max_timestamp (1726045973899)
[tester::#FD8] [Decoder] - .producer_id (0)
[tester::#FD8] [Decoder] - .producer_epoch (0)
[tester::#FD8] [Decoder] - .base_sequence (0)
[tester::#FD8] [Decoder] - .num_records (1)
[tester::#FD8] [Decoder] - .Record[0]
[tester::#FD8] [Decoder] - .length (25)
[tester::#FD8] [Decoder] - .attributes (0)
[tester::#FD8] [Decoder] - .timestamp_delta (0)
[tester::#FD8] [Decoder] - .offset_delta (0)
[tester::#FD8] [Decoder] - .key_length (-1)
[tester::#FD8] [Decoder] - .key ("")
[tester::#FD8] [Decoder] - .value_length (19)
[tester::#FD8] [Decoder] - .value ("Hello CodeCrafters!")
[tester::#FD8] [Decoder] - .num_headers (0)
[tester::#FD8] [Decoder] - .RecordBatch[2]
[tester::#FD8] [Decoder] - .base_offset (16777216)
[tester::#FD8] [Decoder] - .batch_length (0)
[tester::#FD8] [Decoder] - .partition_leader_epoch (255)
[tester::#FD8] [Decoder] - .magic_byte (-1)
[tester::#FD8] [Decoder] - .crc (-1)
[tester::#FD8] Received:
[tester::#FD8] Hex (bytes 253-268) | ASCII
[tester::#FD8] ------------------------------------------------+------------------
[tester::#FD8] ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 00 | ................
[tester::#FD8] ^ ^
[tester::#FD8] Error: CRC mismatch: calculated 00000000, expected ffffffff
[tester::#FD8] Context:
[tester::#FD8] - Fetch Response v16
[tester::#FD8] - Response Body
[tester::#FD8] - TopicResponse[0]
[tester::#FD8] - PartitionResponse[0]
[tester::#FD8] - RecordBatch[2]
[tester::#FD8] - crc
[tester::#FD8]
[tester::#FD8] Test failed
[tester::#FD8] Terminating program
[tester::#FD8] Program terminated successfully
What I find interesting here, tester seem adamant that there’s third RecordBatch within first PartitionResponse when actually it only consist of 2 RecordBatch then follow by another PartitionResponse with empty RecordBatch.
I have write an unit test which only parse the same response here. You can find my code at abastian/codecrafters-kafka-rust
