Question about fd8: Fetch multiple messages from disk. Data isn't being prepended to response from log file correctly

whorst · December 25, 2025, 10:29pm

Seems like I’m having trouble parsing the log partition file incorrectly. I’m receiving the following from the command line:

remote: [tester::#FD8] [client] Successfully connected to broker at localhost:9092
remote: [tester::#FD8] [Encoder] - FetchRequest
remote: [tester::#FD8] [Encoder]   - Header
remote: [tester::#FD8] [Encoder]     - APIKey (1)
remote: [tester::#FD8] [Encoder]     - APIVersion (16)
remote: [tester::#FD8] [Encoder]     - CorrelationID (330266902)
remote: [tester::#FD8] [Encoder]     - ClientID (kafka-tester)
remote: [tester::#FD8] [Encoder]   - Body
remote: [tester::#FD8] [Encoder]     - MaxWaitMS (500)
remote: [tester::#FD8] [Encoder]     - MinBytes (1)
remote: [tester::#FD8] [Encoder]     - MaxBytes (2147483647)
remote: [tester::#FD8] [Encoder]     - IsolationLevel (0)
remote: [tester::#FD8] [Encoder]     - SessionID (0)
remote: [tester::#FD8] [Encoder]     - SessionEpoch (0)
remote: [tester::#FD8] [Encoder]     - Topics
remote: [tester::#FD8] [Encoder]       - Length (2 (Array length(1) + 1))
remote: [tester::#FD8] [Encoder]       - Topics[0]
remote: [tester::#FD8] [Encoder]         - Topic
remote: [tester::#FD8] [Encoder]           - UUID (71a59a51-8968-4f8b-937e-1b233c64eacb)
remote: [tester::#FD8] [Encoder]           - Partitions
remote: [tester::#FD8] [Encoder]             - Length (2 (Array length(1) + 1))
remote: [tester::#FD8] [Encoder]             - Partitions[0]
remote: [tester::#FD8] [Encoder]               - Partition
remote: [tester::#FD8] [Encoder]                 - ID (0)
remote: [tester::#FD8] [Encoder]                 - CurrentLeaderEpoch (-1)
remote: [tester::#FD8] [Encoder]                 - FetchOffset (0)
remote: [tester::#FD8] [Encoder]                 - LastFetchedOffset (-1)
remote: [tester::#FD8] [Encoder]                 - LogStartOffset (-1)
remote: [tester::#FD8] [Encoder]                 - PartitionMaxBytes (2147483647)
remote: [tester::#FD8] [Encoder]     - ForgottenTopics
remote: [tester::#FD8] [Encoder]       - Length (1 (Array length(0) + 1))
remote: [tester::#FD8] [Encoder]     - RackID ("")
remote: [tester::#FD8] [client] Sending "Fetch" request
remote: [tester::#FD8] [client] Hexdump of sent "Fetch" request: 
remote: [tester::#FD8] [client] Idx  | Hex                                             | ASCII
remote: [tester::#FD8] [client] -----+-------------------------------------------------+-----------------
remote: [tester::#FD8] [client] 0000 | 00 00 00 63 00 01 00 10 13 af 79 16 00 0c 6b 61 | ...c......y...ka
remote: [tester::#FD8] [client] 0010 | 66 6b 61 2d 74 65 73 74 65 72 00 00 00 01 f4 00 | fka-tester......
remote: [tester::#FD8] [client] 0020 | 00 00 01 7f ff ff ff 00 00 00 00 00 00 00 00 00 | ................
remote: [tester::#FD8] [client] 0030 | 02 71 a5 9a 51 89 68 4f 8b 93 7e 1b 23 3c 64 ea | .q..Q.hO..~.#<d.
remote: [tester::#FD8] [client] 0040 | cb 02 00 00 00 00 ff ff ff ff 00 00 00 00 00 00 | ................
remote: [tester::#FD8] [client] 0050 | 00 00 ff ff ff ff ff ff ff ff ff ff ff ff 7f ff | ................
remote: [tester::#FD8] [client] 0060 | ff ff 00 00 01 01 00                            | .......
remote: [tester::#FD8] [client] 
remote: [tester::#FD8] [client] Hexdump of received "Fetch" response: 
remote: [tester::#FD8] [client] Idx  | Hex                                             | ASCII
remote: [tester::#FD8] [client] -----+-------------------------------------------------+-----------------
remote: [tester::#FD8] [client] 0000 | 00 00 00 93 13 af 79 16 00 00 00 00 00 00 00 00 | ......y.........
remote: [tester::#FD8] [client] 0010 | 00 00 00 02 71 a5 9a 51 89 68 4f 8b 93 7e 1b 23 | ....q..Q.hO..~.#
remote: [tester::#FD8] [client] 0020 | 3c 64 ea cb 02 00 00 00 00 00 00 00 00 00 00 00 | <d..............
remote: [tester::#FD8] [client] 0030 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
remote: [tester::#FD8] [client] 0040 | 00 00 00 01 00 00 00 00 4b 00 00 00 00 00 00 00 | ........K.......
remote: [tester::#FD8] [client] 0050 | 00 00 00 00 3e 00 00 00 00 02 07 c6 ce e2 00 00 | ....>...........
remote: [tester::#FD8] [client] 0060 | 00 00 00 00 00 00 01 91 e0 5b 6d 8b 00 00 01 91 | .........[m.....
remote: [tester::#FD8] [client] 0070 | e0 5b 6d 8b 00 00 00 00 00 00 00 00 00 00 00 00 | .[m.............
remote: [tester::#FD8] [client] 0080 | 00 00 00 00 00 01 18 00 00 00 01 0c 6f 72 61 6e | ............oran
remote: [tester::#FD8] [client] 0090 | 67 65 01 00 00 00 00                            | ge.....
remote: [tester::#FD8] [client] 
remote: [tester::#FD8] [Decoder] - FetchResponse
remote: [tester::#FD8] [Decoder]   - Header
remote: [tester::#FD8] [Decoder]     - CorrelationID (330266902)
remote: [tester::#FD8] [Decoder]   - Body
remote: [tester::#FD8] [Decoder]     - ThrottleTimeMS (0)
remote: [tester::#FD8] [Decoder]     - ErrorCode (0)
remote: [tester::#FD8] [Decoder]     - SessionID (0)
remote: [tester::#FD8] [Decoder]     - Topics
remote: [tester::#FD8] [Decoder]       - Length (2 (Array length(1) + 1))
remote: [tester::#FD8] [Decoder]       - Topics[0]
remote: [tester::#FD8] [Decoder]         - UUID (71a59a51-8968-4f8b-937e-1b233c64eacb)
remote: [tester::#FD8] [Decoder]         - Partitions
remote: [tester::#FD8] [Decoder]           - Length (2 (Array length(1) + 1))
remote: [tester::#FD8] [Decoder]           - Partitions[0]
remote: [tester::#FD8] [Decoder]             - Id (0)
remote: [tester::#FD8] [Decoder]             - ErrorCode (0)
remote: [tester::#FD8] [Decoder]             - HighWaterMark (0)
remote: [tester::#FD8] [Decoder]             - LastStableOffset (0)
remote: [tester::#FD8] [Decoder]             - LogStartOffset (0)
remote: [tester::#FD8] [Decoder]             - AbortedTransactions
remote: [tester::#FD8] [Decoder]               - Length (1 (Array length(0) + 1))
remote: [tester::#FD8] [Decoder]             - PreferredReadReplica (0)
remote: [tester::#FD8] [Decoder]             - RecordBatches
remote: [tester::#FD8] [Decoder]               - Size (75 (Record size(74) + 1))
remote: [tester::#FD8] [Decoder]               - RecordBatches[0]
remote: [tester::#FD8] [Decoder]                 - Offset (0)
remote: [tester::#FD8] [Decoder]                 - Length (62)
remote: [tester::#FD8] [Decoder]                 - PartitionLeaderEpoch (0)
remote: [tester::#FD8] [Decoder]                 - MagicByte (2)
remote: [tester::#FD8] [Decoder]                 - CRC (130469602)
remote: [tester::#FD8] [Decoder]                 - Attributes (0)
remote: [tester::#FD8] [Decoder]                 - LastOffsetDelta (0)
remote: [tester::#FD8] [Decoder]                 - FirstTimeStamp (1726045973899)
remote: [tester::#FD8] [Decoder]                 - MaxTimeStamp (1726045973899)
remote: [tester::#FD8] [Decoder]                 - ProducerID (0)
remote: [tester::#FD8] [Decoder]                 - ProducerEpoch (0)
remote: [tester::#FD8] [Decoder]                 - BaseSequence (0)
remote: [tester::#FD8] [Decoder]                 - Records
remote: [tester::#FD8] [Decoder]                   - Length (1)
remote: [tester::#FD8] [Decoder]                   - Records[0]
remote: [tester::#FD8] [Decoder]                     - Record
remote: [tester::#FD8] [Decoder]                       - Length (12)
remote: [tester::#FD8] [Decoder]                       - Attributes (0)
remote: [tester::#FD8] [Decoder]                       - TimestampDelta (0)
remote: [tester::#FD8] [Decoder]                       - OffsetDelta (0)
remote: [tester::#FD8] [Decoder]                       - KeyLength (-1)
remote: [tester::#FD8] [Decoder]                       - ValueLength (6)
remote: [tester::#FD8] [Decoder]                       - Value ([111 114 97 110 103 101] -> UTF-8: (orange))
remote: [tester::#FD8] [Decoder]                       - HeadersLength (-1)
remote: [tester::#FD8] ✓ TopicResponses Length: 1
remote: [tester::#FD8] ✓ TopicResponse[0] TopicUUID: 71a59a51-8968-4f8b-937e-1b233c64eacb
remote: [tester::#FD8] ✓ PartitionResponses Length: 1
remote: [tester::#FD8] ✓ PartitionResponse[0] ErrorCode: 0 (NO_ERROR)
remote: [tester::#FD8] ✓ PartitionResponse[0] PartitionId: 0
remote: [tester::#FD8] 
remote: [tester::#FD8] Expected (bytes 24-124), hexadecimal:                       | ASCII:
remote: [tester::#FD8] 00 00 00 00 00 01 91 e0 5b 6d 8b 00 00 01 91 e0 5b 6d 8b 00 | ........[m......[m..
remote: [tester::#FD8] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 18 00 00 | ....................
remote: [tester::#FD8] 00 01 0c 6f 72 61 6e 67 65 01                               | ...orange.
remote: [tester::#FD8] 
remote: [tester::#FD8] Actual (bytes 24-124), hexadecimal:                         | ASCII:
remote: [tester::#FD8] 00 00 00 00 00 01 91 e0 5b 6d 8b 00 00 01 91 e0 5b 6d 8b 00 | ........[m......[m..
remote: [tester::#FD8] 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 18 00 00 | ....................
remote: [tester::#FD8] 00 01 0c 6f 72 61 6e 67 65 01 00 00 00 00 00 00 00 01 00 00 | ...orange...........
remote: [tester::#FD8] 00 3d 00 00 00 00 02 80 d7 c9 b3 00 00 00 00 00 00 00 00 01 | .=..................
remote: [tester::#FD8] 91 e0 5b 6d 8b 00 00 01 91 e0 5b 6d 8b 00 00 00 00 00 00 00 | ..[m......[m........
remote: [tester::#FD8] remote: [tester::#FD8] RecordBatch bytes do not match with the contents on disk
remote: [tester::#FD8] Test failed

I think simply appending the entire data file as I am here (not much mutation) is correct. And here is where I add all the data together. What could be happening? Adding screenshots for readability

andy1li · December 25, 2025, 10:29pm

Hi, thanks for your post!

I’m currently out of the office for the holidays and will be back on January 5. I’ll get back to you as soon as possible once I return.

whorst · December 25, 2025, 11:32pm

One other thing, before my program gives it’s output after an invocation, I’m seeing in the logs it’s sending back

What the program says is expected
00 00 01 91 e0 5b 6d 8b 00 00 01 91 e0 5b 6d 8b 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 1e 00 00 00 01 12 72 61 73 70 62 65 72 72 79 01

What the program says I’m returning ( in the log Actual (bytes 27-127), hexadecimal)
00 00 01 91 e0 5b 6d 8b 00 00 01 91 e0 5b 6d 8b 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 1e 00 00 00 01 12 72 61 73 70 62 65 72 72 79 01 00 00 00 00 00 00 00 01 00 00 00 41 00 00 00 00 02 cf 35 bf 62 00 00 00 00 00 00 00 00 01 91 e0 5b 6d 8b 00 00 01 91 e0 5b 6d 8b 00 00 00 00 00 00 00

Actual Return Value (My Logs)
00 00 00 96 0f ca b3 0e 00 00 00 00 00 00 00 00 00 00 00 02 71 a5 9a 51 89 68 4f 8b 93 7e c2 88 2a b8 39 a4 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 4e 00 00 00 00 00 00 00 00 00 00 00 41 00 00 00 00 02 b2 66 93 fc 00 00 00 00 00 00 00 00 01 91 e0 5b 6d 8b 00 00 01 91 e0 5b 6d 8b 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 1e 00 00 00 01 12 72 61 73 70 62 65 72 72 79 01 00 00 00 00

Topic		Replies	Views
Question about fd8: Fetch multiple messages from disk problem reading second batch Challenges challenge:kafka	3	31	August 23, 2025
Question about fd8: Fetch multiple messages from dis Challenges challenge:kafka	3	48	August 28, 2025
Question about fd8: Fetch multiple messages from disk - Expected partitions.length to be 1, got 2 Challenges challenge:kafka	8	122	May 9, 2025
Question about fd8 Fetch multiple messages from disk Challenges challenge:kafka	3	79	December 21, 2024
Question about eg2: Fetch single message from disk (Records Array) Challenges challenge:kafka	4	107	December 22, 2024

Question about fd8: Fetch multiple messages from disk. Data isn't being prepended to response from log file correctly

Related topics