Weird bug in stage #FD8: Fetch multiple messages from disk

fortwoone · September 20, 2025, 5:55pm

Hello,

So I’ve finally reached the last stage, but I can’t seem to understand this error I’m encountering. For some reason, the tester considers the CRC checksum incorrect, but after repeated attempts, I can’t figure out what is wrong for the life of me. My code should already be available on GitHub, if anyone wants to test this.

Here are the tester logs I get:

remote: [tester::#FD8] Hexdump of sent "Fetch" request: 
remote: [tester::#FD8] Idx  | Hex                                             | ASCII
remote: [tester::#FD8] -----+-------------------------------------------------+-----------------
remote: [tester::#FD8] 0000 | 00 00 00 60 00 01 00 10 6b bf 88 fa 00 09 6b 61 | ...`....k.....ka
remote: [tester::#FD8] 0010 | 66 6b 61 2d 63 6c 69 00 00 00 01 f4 00 00 00 01 | fka-cli.........
remote: [tester::#FD8] 0020 | 03 20 00 00 00 00 00 00 00 00 00 00 00 02 00 00 | . ..............
remote: [tester::#FD8] 0030 | 00 00 00 00 40 00 80 00 00 00 00 00 00 50 02 00 | ....@........P..
remote: [tester::#FD8] 0040 | 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00 ff | ................
remote: [tester::#FD8] 0050 | ff ff ff ff ff ff ff ff ff ff ff 00 10 00 00 00 | ................
remote: [tester::#FD8] 0060 | 00 01 01 00                                     | ....
remote: [tester::#FD8] 
remote: [your_program] Client connected at file descriptor: 4        
remote: [your_program] Request API Key: 1        
remote: [your_program] Request API Version: 16        
remote: [your_program] Request Correlation ID: 1807714554        
remote: [tester::#FD8] Hexdump of received "Fetch" response: 
remote: [tester::#FD8] Idx  | Hex                                             | ASCII
remote: [tester::#FD8] -----+-------------------------------------------------+-----------------
remote: [tester::#FD8] 0000 | 00 00 00 f7 6b bf 88 fa 00 00 00 00 00 00 00 00 | ....k...........
remote: [tester::#FD8] 0010 | 00 00 00 02 00 00 00 00 00 00 40 00 80 00 00 00 | ..........@.....
remote: [tester::#FD8] 0020 | 00 00 00 50 02 00 00 00 00 00 00 00 00 00 00 00 | ...P............
remote: [tester::#FD8] 0030 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
remote: [tester::#FD8] 0040 | 00 00 00 01 00 00 00 00 03 00 00 00 00 00 00 00 | ................
remote: [tester::#FD8] 0050 | 00 00 00 00 4b 00 00 00 00 02 55 60 53 93 00 00 | ....K.....U`S...
remote: [tester::#FD8] 0060 | 00 00 00 00 00 00 01 91 e0 5b 6d 8b 00 00 01 91 | .........[m.....
remote: [tester::#FD8] 0070 | e0 5b 6d 8b 00 00 00 00 00 00 00 00 00 00 00 00 | .[m.............
remote: [tester::#FD8] 0080 | 00 00 00 00 00 01 19 00 00 00 ff ff ff ff 0f 13 | ................
remote: [tester::#FD8] 0090 | 48 65 6c 6c 6f 20 43 6f 64 65 43 72 61 66 74 65 | Hello CodeCrafte
remote: [tester::#FD8] 00a0 | 72 73 21 00 00 00 00 00 00 00 00 01 00 00 00 44 | rs!............D
remote: [tester::#FD8] 00b0 | 00 00 00 00 02 64 61 7c 4a 00 00 00 00 00 00 00 | .....da|J.......
remote: [tester::#FD8] 00c0 | 00 01 91 e0 5b 6d 8b 00 00 01 91 e0 5b 6d 8b 00 | ....[m......[m..
remote: [tester::#FD8] 00d0 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
remote: [tester::#FD8] 00e0 | 01 12 00 00 00 ff ff ff ff 0f 0c 48 65 6c 6c 6f | ...........Hello
remote: [tester::#FD8] 00f0 | 20 45 61 72 74 68 21 00 00 00 00                |  Earth!....
remote: [tester::#FD8] 
remote: [tester::#FD8] [Decoder] - .ResponseHeader
remote: [tester::#FD8] [Decoder]   - .correlation_id (1807714554)
remote: [tester::#FD8] [Decoder]   - .TAG_BUFFER
remote: [tester::#FD8] [Decoder] - .ResponseBody
remote: [tester::#FD8] [Decoder]   - .throttle_time_ms (0)
remote: [tester::#FD8] [Decoder]   - .error_code (0)
remote: [tester::#FD8] [Decoder]   - .session_id (0)
remote: [tester::#FD8] [Decoder]   - .num_responses (1)
remote: [tester::#FD8] [Decoder]   - .TopicResponse[0]
remote: [tester::#FD8] [Decoder]     - .topic_id (00000000-0000-4000-8000-000000000050)
remote: [tester::#FD8] [Decoder]     - .num_partitions (1)
remote: [tester::#FD8] [Decoder]     - .PartitionResponse[0]
remote: [tester::#FD8] [Decoder]       - .partition_index (0)
remote: [tester::#FD8] [Decoder]       - .error_code (0)
remote: [tester::#FD8] [Decoder]       - .high_watermark (0)
remote: [tester::#FD8] [Decoder]       - .last_stable_offset (0)
remote: [tester::#FD8] [Decoder]       - .log_start_offset (0)
remote: [tester::#FD8] [Decoder]       - .num_aborted_transactions (0)
remote: [tester::#FD8] [Decoder]       - .preferred_read_replica (0)
remote: [tester::#FD8] [Decoder]       - .compact_records_length (2)
remote: [tester::#FD8] [Decoder]       - .RecordBatch[0]
remote: [tester::#FD8] [Decoder]         - .base_offset (0)
remote: [tester::#FD8] [Decoder]         - .batch_length (75)
remote: [tester::#FD8] [Decoder]         - .partition_leader_epoch (0)
remote: [tester::#FD8] [Decoder]         - .magic_byte (2)
remote: [tester::#FD8] [Decoder]         - .crc (1432376211)
remote: [tester::#FD8] Received:
remote: [tester::#FD8] Hex (bytes 85-100)                              | ASCII
remote: [tester::#FD8] ------------------------------------------------+------------------
remote: [tester::#FD8] 02 55 60 53 93 00 00 00 00 00 00 00 00 01 91 e0 | .U`S............
remote: [tester::#FD8]                 ^                                      ^
remote: [tester::#FD8] Error: CRC mismatch: calculated 4f2b7365, expected 55605393
remote: [tester::#FD8] Context:
remote: [tester::#FD8] - Fetch Response v16
remote: [tester::#FD8]   - Response Body
remote: [tester::#FD8]     - TopicResponse[0]
remote: [tester::#FD8]       - PartitionResponse[0]
remote: [tester::#FD8]         - RecordBatch[0]
remote: [tester::#FD8]           - crc
remote: [tester::#FD8] 
remote: [tester::#FD8] Test failed

@andy1li do you have an idea why I failed this, perchance?

fortwoone · September 20, 2025, 8:10pm

For convenience, here is a link to my repo: GitHub - fortwoone/cpp_kafka: CPP light implementation of Kafka for Codecrafters' "Build your own Kafka" challenge

fortwoone · September 21, 2025, 9:38am

Any help would be greatly appreciated here

fortwoone · September 22, 2025, 11:07am

The reason I get confused is that the tester says another checksum was computed, but then the CRC checksum sent is literally the same, so I don’t understand what is expected here, because then the tester simply ignores the rest of the record batch entirely. I can only guess some misplacement happens, but since the tester does not complain beforehand, I am unable to actually debug this further

fortwoone · September 22, 2025, 6:59pm

@andy1li sorry for the bother, but I really can’t find a solution for this…

andy1li · September 22, 2025, 11:33pm

Hey @fortwoone, looks like there’re two issues:

compact_records_length is incorrect:
The RecordBatch is read and reconstructed manually, which is unnecessary.

To clarify, the records field in FetchResponse is of type COMPACT_RECORDS (COMPACT_NULLABLE_BYTES):

That means the compact_records_length should represent the number of raw bytes in the RecordBatch.

You don’t need to rebuild the RecordBatch. Just read it from disk as-is and prepend the correct length prefix.

fortwoone · September 23, 2025, 6:07am

Okay, I actually didn’t understand the length was supposed to be referring to the number of bytes, I thought it was just the batch count. Thanks for giving me that pointer, I failed to understand this from the docs themselves. Will try to implement them

fortwoone · September 23, 2025, 6:15am

As for the batch structure, I’m only reading them this way to prepare for future stages in this challenge

fortwoone · September 23, 2025, 7:48am

Okay, uh, it seems the tester is stuck right now @andy1li, the git push output shows repeated compilations that all are successful, and then it stops right after attempting the connection. Every time this happens it just rebuilds the executable only to wait for some reason
The code seems correct though

fortwoone · September 23, 2025, 7:59am

Logs:

remote: ------------------------------------------------------------------------        
remote: 
remote: 
remote:      ___            _          ___              __  _                   
remote:     / __\ ___    __| |  ___   / __\_ __  __ _  / _|| |_  ___  _ __  ___ 
remote:    / /   / _ \  / _` | / _ \ / /  | '__|/ _` || |_ | __|/ _ \| '__|/ __|
remote:   / /___| (_) || (_| ||  __// /___| |  | (_| ||  _|| |_|  __/| |   \__ 
remote:   \____/ \___/  \__,_| \___|\____/|_|   \__,_||_|   \__|\___||_|   |___/
remote: 
remote: 
remote:    Welcome to CodeCrafters! Your commit was received successfully.        
remote: 
remote: ------------------------------------------------------------------------        
remote: 
remote: We're warming up test runners for your repository, please bear with us.
remote: Subsequent builds will be snappy ⚡
remote: 
remote: Streaming build logs...        
remote: 
remote: [build]Starting build...
remote: [build]If you don't see logs for 60s+, please contact us at hello@codecrafters.io
remote: [build] Step 1 complete.        
remote: [build] Step 2 complete.        
remote: [build] Step 3 complete.        
remote: [build] Step 4 complete.        
remote: [build] Step 5 complete.        
remote: [build] Step 6 complete.        
remote: [build] Step 7 complete.        
remote: [build] Step 8 complete.        
remote: [build] Step 9 complete.        
remote: [build] Step 10 complete.        
remote: [build] Step 11 complete.        
remote: [build] Step 10 complete.        
remote: [build] Step 12 complete.        
remote: [build] Step 13 complete.        
remote: [build] Step 14 complete.        
remote: [build] Step 15 complete.        
remote: [build] Step 16 complete.        
remote: [build] Step 17 complete.        
remote: [build] > All requested packages are currently installed.        
remote: [build] > All requested installations completed successfully in: 32.4 us        
remote: [build] Step 18 complete.        
remote: [build] Step 19 complete.        
remote: [build] Step 20 complete.        
remote: [build] Step 21 complete.        
remote: [build] Step 22 complete.        
remote: [build] Step 23 complete.        
remote: [build] Step 24 complete.        
remote: [build] Step 25 complete.        
remote: [build] Step 27 complete.        
remote: [build] Step 26 complete.        
remote: [build]Build successful.
remote: 
remote: 
remote: Running tests on your code. Logs should appear shortly...        
remote: 
remote: [build]Starting build...
remote: [build]If you don't see logs for 60s+, please contact us at hello@codecrafters.io
remote: [build] Step 1 complete.        
remote: [build] Step 2 complete.        
remote: [build] Step 3 complete.        
remote: [build] Step 4 complete.        
remote: [build] Step 5 complete.        
remote: [build] Step 6 complete.        
remote: [build] Step 7 complete.        
remote: [build] Step 8 complete.        
remote: [build] Step 9 complete.        
remote: [build] Step 10 complete.        
remote: [build] Step 11 complete.        
remote: [build] Step 10 complete.        
remote: [build] Step 12 complete.        
remote: [build] Step 13 complete.        
remote: [build] Step 14 complete.        
remote: [build] Step 15 complete.        
remote: [build] Step 16 complete.        
remote: [build] Step 17 complete.        
remote: [build] > All requested packages are currently installed.        
remote: [build] > All requested installations completed successfully in: 32.4 us        
remote: [build] Step 18 complete.        
remote: [build] Step 19 complete.        
remote: [build] Step 20 complete.        
remote: [build] Step 21 complete.        
remote: [build] Step 22 complete.        
remote: [build] Step 23 complete.        
remote: [build] Step 24 complete.        
remote: [build] Step 25 complete.        
remote: [build] Step 27 complete.        
remote: [build] Step 26 complete.        
remote: [build]Build successful.
remote: 
remote: [compile] -- Running vcpkg install        
remote: [compile] All requested packages are currently installed.        
remote: [compile] All requested installations completed successfully in: 16.8 us        
remote: [compile] -- Running vcpkg install - done        
remote: [compile] -- The C compiler identification is GNU 14.2.0        
remote: [compile] -- The CXX compiler identification is GNU 14.2.0        
remote: [compile] -- Detecting C compiler ABI info        
remote: [compile] -- Detecting C compiler ABI info - done        
remote: [compile] -- Check for working C compiler: /usr/bin/cc - skipped        
remote: [compile] -- Detecting C compile features        
remote: [compile] -- Detecting C compile features - done        
remote: [compile] -- Detecting CXX compiler ABI info        
remote: [compile] -- Detecting CXX compiler ABI info - done        
remote: [compile] -- Check for working CXX compiler: /usr/local/bin/c++ - skipped        
remote: [compile] -- Detecting CXX compile features        
remote: [compile] -- Detecting CXX compile features - done        
remote: [compile] -- Configuring done (0.3s)        
remote: [compile] -- Generating done (0.0s)        
remote: [compile] -- Build files have been written to: /app/build        
remote: [compile] [ 20%] Building CXX object CMakeFiles/kafka.dir/src/cluster_metadata.cpp.o        
remote: [compile] [ 40%] Building CXX object CMakeFiles/kafka.dir/src/main.cpp.o        
remote: [compile] [ 60%] Building CXX object CMakeFiles/kafka.dir/src/requests.cpp.o        
remote: [compile] [ 80%] Building CXX object CMakeFiles/kafka.dir/src/varint_type.cpp.o        
remote: [compile] [100%] Linking CXX executable kafka        
remote: [compile] [100%] Built target kafka        
remote: [compile] Moved ./.codecrafters/run.sh → ./your_program.sh        
remote: [compile]Compilation successful.
remote: 
remote: Debug = true        
remote: 
remote: [tester::#FD8] Running tests for Stage #FD8 (Consuming Messages - Fetch multiple messages from disk)
remote: [tester::#FD8] $ ./your_program.sh /tmp/server.properties
remote: [tester::#FD8] Connecting to broker at: localhost:9092
remote: [compile] -- Running vcpkg install        
remote: [compile] All requested packages are currently installed.        
remote: [compile] All requested installations completed successfully in: 17.8 us        
remote: [compile] -- Running vcpkg install - done        
remote: [compile] -- The C compiler identification is GNU 14.2.0        
remote: [compile] -- The CXX compiler identification is GNU 14.2.0        
remote: [compile] -- Detecting C compiler ABI info        
remote: [compile] -- Detecting C compiler ABI info - done        
remote: [compile] -- Check for working C compiler: /usr/bin/cc - skipped        
remote: [compile] -- Detecting C compile features        
remote: [compile] -- Detecting C compile features - done        
remote: [compile] -- Detecting CXX compiler ABI info        
remote: [compile] -- Detecting CXX compiler ABI info - done        
remote: [compile] -- Check for working CXX compiler: /usr/local/bin/c++ - skipped        
remote: [compile] -- Detecting CXX compile features        
remote: [compile] -- Detecting CXX compile features - done        
remote: [compile] -- Configuring done (0.3s)        
remote: [compile] -- Generating done (0.0s)        
remote: [compile] -- Build files have been written to: /app/build        
remote: [compile] [ 20%] Building CXX object CMakeFiles/kafka.dir/src/cluster_metadata.cpp.o        
remote: [compile] [ 40%] Building CXX object CMakeFiles/kafka.dir/src/main.cpp.o        
remote: [compile] [ 60%] Building CXX object CMakeFiles/kafka.dir/src/requests.cpp.o        
remote: [compile] [ 80%] Building CXX object CMakeFiles/kafka.dir/src/varint_type.cpp.o        
remote: [compile] [100%] Linking CXX executable kafka        
remote: [compile] [100%] Built target kafka        
remote: [compile] Moved ./.codecrafters/run.sh → ./your_program.sh        
remote: [compile]Compilation successful.
remote: 
remote: Debug = true        
remote: 
remote: [tester::#FD8] Running tests for Stage #FD8 (Consuming Messages - Fetch multiple messages from disk)
remote: [tester::#FD8] $ ./your_program.sh /tmp/server.properties
remote: [tester::#FD8] Connecting to broker at: localhost:9092
remote: [compile] -- Running vcpkg install        
remote: [compile] All requested packages are currently installed.        
remote: [compile] All requested installations completed successfully in: 23.5 us        
remote: [compile] -- Running vcpkg install - done        
remote: [compile] -- The C compiler identification is GNU 14.2.0        
remote: [compile] -- The CXX compiler identification is GNU 14.2.0        
remote: [compile] -- Detecting C compiler ABI info        
remote: [compile] -- Detecting C compiler ABI info - done        
remote: [compile] -- Check for working C compiler: /usr/bin/cc - skipped        
remote: [compile] -- Detecting C compile features        
remote: [compile] -- Detecting C compile features - done        
remote: [compile] -- Detecting CXX compiler ABI info        
remote: [compile] -- Detecting CXX compiler ABI info - done        
remote: [compile] -- Check for working CXX compiler: /usr/local/bin/c++ - skipped        
remote: [compile] -- Detecting CXX compile features        
remote: [compile] -- Detecting CXX compile features - done        
remote: [compile] -- Configuring done (0.3s)        
remote: [compile] -- Generating done (0.0s)        
remote: [compile] -- Build files have been written to: /app/build        
remote: [compile] [ 20%] Building CXX object CMakeFiles/kafka.dir/src/cluster_metadata.cpp.o        
remote: [compile] [ 40%] Building CXX object CMakeFiles/kafka.dir/src/main.cpp.o        
remote: [compile] [ 60%] Building CXX object CMakeFiles/kafka.dir/src/requests.cpp.o        
remote: [compile] [ 80%] Building CXX object CMakeFiles/kafka.dir/src/varint_type.cpp.o        
remote: [compile] [100%] Linking CXX executable kafka        
remote: [compile] [100%] Built target kafka        
remote: [compile] Moved ./.codecrafters/run.sh → ./your_program.sh        
remote: [compile]Compilation successful.
remote: 
remote: Debug = true        
remote: 
remote: [tester::#FD8] Running tests for Stage #FD8 (Consuming Messages - Fetch multiple messages from disk)
remote: [tester::#FD8] $ ./your_program.sh /tmp/server.properties
remote: [tester::#FD8] Connecting to broker at: localhost:9092
remote: [compile] -- Running vcpkg install        
remote: [compile] All requested packages are currently installed.        
remote: [compile] All requested installations completed successfully in: 29.9 us        
remote: [compile] -- Running vcpkg install - done        
remote: [compile] -- The C compiler identification is GNU 14.2.0        
remote: [compile] -- The CXX compiler identification is GNU 14.2.0        
remote: [compile] -- Detecting C compiler ABI info        
remote: [compile] -- Detecting C compiler ABI info - done        
remote: [compile] -- Check for working C compiler: /usr/bin/cc - skipped        
remote: [compile] -- Detecting C compile features        
remote: [compile] -- Detecting C compile features - done        
remote: [compile] -- Detecting CXX compiler ABI info        
remote: [compile] -- Detecting CXX compiler ABI info - done        
remote: [compile] -- Check for working CXX compiler: /usr/local/bin/c++ - skipped        
remote: [compile] -- Detecting CXX compile features        
remote: [compile] -- Detecting CXX compile features - done        
remote: [compile] -- Configuring done (0.4s)        
remote: [compile] -- Generating done (0.0s)        
remote: [compile] -- Build files have been written to: /app/build        
remote: [compile] [ 20%] Building CXX object CMakeFiles/kafka.dir/src/cluster_metadata.cpp.o        
remote: [compile] [ 40%] Building CXX object CMakeFiles/kafka.dir/src/main.cpp.o        
remote: [compile] [ 60%] Building CXX object CMakeFiles/kafka.dir/src/requests.cpp.o        
remote: [compile] [ 80%] Building CXX object CMakeFiles/kafka.dir/src/varint_type.cpp.o        
remote: [compile] [100%] Linking CXX executable kafka        
remote: [compile] [100%] Built target kafka        
remote: [compile] Moved ./.codecrafters/run.sh → ./your_program.sh        
remote: [compile]Compilation successful.
remote: 
remote: Debug = true        
remote: 
remote: [tester::#FD8] Running tests for Stage #FD8 (Consuming Messages - Fetch multiple messages from disk)
remote: [tester::#FD8] $ ./your_program.sh /tmp/server.properties
remote: [tester::#FD8] Connecting to broker at: localhost:9092
remote: [compile] -- Running vcpkg install        
remote: [compile] All requested packages are currently installed.        
remote: [compile] All requested installations completed successfully in: 21.8 us        
remote: [compile] -- Running vcpkg install - done        
remote: [compile] -- The C compiler identification is GNU 14.2.0        
remote: [compile] -- The CXX compiler identification is GNU 14.2.0        
remote: [compile] -- Detecting C compiler ABI info        
remote: [compile] -- Detecting C compiler ABI info - done        
remote: [compile] -- Check for working C compiler: /usr/bin/cc - skipped        
remote: [compile] -- Detecting C compile features        
remote: [compile] -- Detecting C compile features - done        
remote: [compile] -- Detecting CXX compiler ABI info        
remote: [compile] -- Detecting CXX compiler ABI info - done        
remote: [compile] -- Check for working CXX compiler: /usr/local/bin/c++ - skipped        
remote: [compile] -- Detecting CXX compile features        
remote: [compile] -- Detecting CXX compile features - done        
remote: [compile] -- Configuring done (0.3s)        
remote: [compile] -- Generating done (0.0s)        
remote: [compile] -- Build files have been written to: /app/build        
remote: [compile] [ 20%] Building CXX object CMakeFiles/kafka.dir/src/cluster_metadata.cpp.o        
remote: [compile] [ 40%] Building CXX object CMakeFiles/kafka.dir/src/main.cpp.o        
remote: [compile] [ 60%] Building CXX object CMakeFiles/kafka.dir/src/requests.cpp.o        
remote: [compile] [ 80%] Building CXX object CMakeFiles/kafka.dir/src/varint_type.cpp.o        
remote: [compile] [100%] Linking CXX executable kafka        
remote: [compile] [100%] Built target kafka        
remote: [compile] Moved ./.codecrafters/run.sh → ./your_program.sh        
remote: [compile]Compilation successful.
remote: 
remote: Debug = true        
remote: 
remote: [tester::#FD8] Running tests for Stage #FD8 (Consuming Messages - Fetch multiple messages from disk)
remote: [tester::#FD8] $ ./your_program.sh /tmp/server.properties
remote: [tester::#FD8] Connecting to broker at: localhost:9092
remote: Looks like we failed to execute tests on time.        
remote: 
remote: Please try again? Let us know at hello@codecrafters.io if this keeps happening.        
remote: 
remote: 
remote: We couldn't fetch the results of your submission. Please try again?        
remote: Let us know at hello@codecrafters.io if this error persists.        
remote:

(EDIT: it was one of my changes, my fault)

fortwoone · September 23, 2025, 9:00am

Actually, it seems a bugfix I tried causes this for some reason

fortwoone · September 23, 2025, 9:36am

Oh, no, I fixed it

system · September 28, 2025, 9:36am

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Stuck on stage #FD8, tester failing to test valid reponse Bug Reports challenge:kafka	4	67	April 4, 2025
Crc Reading Kafka Metadata log Challenges challenge:kafka	9	170	April 6, 2025
Question about fd8 Fetch multiple messages from disk Challenges challenge:kafka	3	79	December 21, 2024
Question about eg2: Fetch single message from disk (Records Array) Challenges challenge:kafka	4	110	December 22, 2024
Question about fd8: Fetch multiple messages from disk. Data isn't being prepended to response from log file correctly Challenges challenge:kafka	3	40	January 11, 2026

Weird bug in stage #FD8: Fetch multiple messages from disk

Related topics