I’m stuck on Stage #eg2
I’ve tried simplifying my reading from file abstractions following the protocol to the T I’m reading a unsigned 32-bit integer from the file but it all comes back weird all my other values are correct, so I am just confused
Here are my logs:
[tester::#EG2] Hexdump of received "Fetch" response:
[tester::#EG2] Idx | Hex | ASCII
[tester::#EG2] -----+-------------------------------------------------+-----------------
[tester::#EG2] 0000 | 00 00 00 97 5d 94 28 cd 00 00 00 00 00 00 00 00 | ....].(.........
[tester::#EG2] 0010 | 00 00 00 02 00 00 00 00 00 00 40 00 80 00 00 00 | ..........@.....
[tester::#EG2] 0020 | 00 00 00 96 02 00 00 00 00 00 00 00 00 00 00 00 | ................
[tester::#EG2] 0030 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
[tester::#EG2] 0040 | 00 00 00 00 00 00 01 01 02 00 00 00 00 00 00 00 | ................
[tester::#EG2] 0050 | 00 00 00 00 45 00 00 00 00 02 64 61 7c 4a 00 00 | ....E.....da|J..
[tester::#EG2] 0060 | 00 00 00 00 00 00 01 91 e0 5b 6d 8b 00 00 01 91 | .........[m.....
[tester::#EG2] 0070 | e0 5b 6d 8b 00 00 00 00 00 00 00 00 00 00 00 00 | .[m.............
[tester::#EG2] 0080 | 00 00 00 00 00 01 24 00 00 00 00 0c 48 65 6c 6c | ......$.....Hell
[tester::#EG2] 0090 | 6f 20 45 61 72 74 68 21 00 00 00 | o Earth!...
[tester::#EG2]
[tester::#EG2] [Decoder] - .ResponseHeader
[tester::#EG2] [Decoder] - .correlation_id (1569990861)
[tester::#EG2] [Decoder] - .TAG_BUFFER
[tester::#EG2] [Decoder] - .ResponseBody
[tester::#EG2] [Decoder] - .throttle_time_ms (0)
[tester::#EG2] [Decoder] - .error_code (0)
[tester::#EG2] [Decoder] - .session_id (0)
[tester::#EG2] [Decoder] - .num_responses (1)
[tester::#EG2] [Decoder] - .TopicResponse[0]
[tester::#EG2] [Decoder] - .topic_id (00000000-0000-4000-8000-000000000096)
[tester::#EG2] [Decoder] - .num_partitions (1)
[tester::#EG2] [Decoder] - .PartitionResponse[0]
[tester::#EG2] [Decoder] - .partition_index (0)
[tester::#EG2] [Decoder] - .error_code (0)
[tester::#EG2] [Decoder] - .high_watermark (0)
[tester::#EG2] [Decoder] - .last_stable_offset (0)
[tester::#EG2] [Decoder] - .log_start_offset (0)
[tester::#EG2] [Decoder] - .num_aborted_transactions (0)
[tester::#EG2] [Decoder] - .preferred_read_replica (257)
[tester::#EG2] [Decoder] - .compact_records_length (1)
[tester::#EG2] [Decoder] - .RecordBatch[0]
[tester::#EG2] [Decoder] - .base_offset (0)
[tester::#EG2] [Decoder] - .batch_length (69)
[tester::#EG2] [Decoder] - .partition_leader_epoch (0)
[tester::#EG2] [Decoder] - .magic_byte (2)
[tester::#EG2] [Decoder] - .crc (1684110410)
[tester::#EG2] Received:
[tester::#EG2] Hex (bytes 85-100) | ASCII
[tester::#EG2] ------------------------------------------------+------------------
[tester::#EG2] 02 64 61 7c 4a 00 00 00 00 00 00 00 00 01 91 e0 | .da|J...........
[tester::#EG2] ^ ^
[tester::#EG2] Error: CRC mismatch: calculated 75dd6e94, expected 64617c4a
[tester::#EG2] Context:
[tester::#EG2] - Fetch Response v16
[tester::#EG2] - Response Body
[tester::#EG2] - TopicResponse[0]
[tester::#EG2] - PartitionResponse[0]
[tester::#EG2] - RecordBatch[0]
[tester::#EG2] - crc
[tester::#EG2]
[tester::#EG2] Test failed
[tester::#EG2] Terminating program
[tester::#EG2] Program terminated successfully
And here’s a snippet of my code:
This is for the reading
RecordBatchReader &readHeader() {
readInt64(batch.base_offset)
.readInt32(batch.batch_length)
.readInt32(batch.partition_leader_epoch)
.readRaw(batch.magic_byte);
readUint32(batch.crc);
std::cout << "CRC Batch: " << std::hex << batch.crc << std::dec
<< std::endl;
return *this;
}
Then for writing
FetchResponse &FetchResponse::writeRecordBatchHeader(
const RecordBatchReader::RecordBatch &batch) {
writeInt64(batch.base_offset)
.writeInt32(batch.batch_length + 1)
.writeInt32(batch.partition_leader_epoch)
.writeInt8(batch.magic_byte)
.writeUInt32(batch.crc)
.writeInt16(batch.attributes)
.writeInt32(batch.last_offset_delta)
.writeInt64(batch.base_timestamp)
.writeInt64(batch.max_timestamp)
.writeInt64(batch.producer_id)
.writeInt16(batch.producer_epoch)
.writeInt32(batch.base_sequence)
.writeInt32(batch.records_count);
return *this;
}
I’ll take a look shortly.
Hi @botirk38 , the compact_records_length
highlighted in red is incorrect. It should represent the number of bytes in the raw RecordBatch
, not the length of the RecordBatch
array.
Specifically, there is no need to parse the RecordBatch
. Instead, you should read it from disk and send it as-is.
Thanks @andy1li your help has resulted in the issue being resolved.
1 Like
[tester::#FD8] Running tests for Stage #FD8 (Consuming Messages - Fetch multiple messages from disk)
[tester::#FD8] [Serializer] Writing log files to: /tmp/kraft-combined-logs
[tester::#FD8] [Serializer] - Wrote file to: /tmp/server.properties
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/meta.properties
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/.kafka_cleanshutdown
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/baz-0/partition.metadata
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/foo-0/partition.metadata
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/quz-0/partition.metadata
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/quz-1/partition.metadata
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/__cluster_metadata-0/partition.metadata
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/baz-0/00000000000000000000.log
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/foo-0/00000000000000000000.log
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/quz-0/00000000000000000000.log
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/quz-1/00000000000000000000.log
[tester::#FD8] [Serializer] - Wrote file to: /tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log
[tester::#FD8] [Serializer] Finished writing log files to: /tmp/kraft-combined-logs
[tester::#FD8] $ ./your_program.sh /tmp/server.properties
[tester::#FD8] Connecting to broker at: localhost:9092
[your_program] Server listening on port 9092
[tester::#FD8] Connection to broker at localhost:9092 successful
[tester::#FD8] Sending "Fetch" (version: 16) request (Correlation id: 276740288)
[tester::#FD8] Hexdump of sent "Fetch" request:
[tester::#FD8] Idx | Hex | ASCII
[tester::#FD8] -----+-------------------------------------------------+-----------------
[tester::#FD8] 0000 | 00 00 00 60 00 01 00 10 10 7e b8 c0 00 09 6b 61 | ...`.....~....ka
[tester::#FD8] 0010 | 66 6b 61 2d 63 6c 69 00 00 00 01 f4 00 00 00 01 | fka-cli.........
[tester::#FD8] 0020 | 03 20 00 00 00 00 00 00 00 00 00 00 00 02 00 00 | . ..............
[tester::#FD8] 0030 | 00 00 00 00 40 00 80 00 00 00 00 00 00 66 02 00 | ....@........f..
[tester::#FD8] 0040 | 00 00 00 ff ff ff ff 00 00 00 00 00 00 00 00 ff | ................
[tester::#FD8] 0050 | ff ff ff ff ff ff ff ff ff ff ff 00 10 00 00 00 | ................
[tester::#FD8] 0060 | 00 01 01 00 | ....
[tester::#FD8]
[your_program] Processing fetch request with 1 topics
[your_program] Topic Count: Loading cluster metadata from /tmp/kraft-combined-logs
[your_program] Loading cluster metadata from: /tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log
[your_program] Reading metadata record batches
[your_program] Found 4 record batches
[your_program] Processing batch with 1 records
[your_program] Processing batch with 2 records
[your_program] Found topic record: baz (ID: 0x40008000000000000076)
[your_program] Found partition record for topic baz, partition 0
[your_program] Loading partition log for baz-0
[your_program] Reading partition log from: /tmp/kraft-combined-logs/baz-0/00000000000000000000.log
[your_program] Successfully opened partition log file
[your_program] Read 1 record batches from partition log
[your_program] Loaded 1 record batches for partition
[your_program] Added to map
[your_program] Processing batch with 2 records
[your_program] Found topic record: foo (ID: 0x40008000000000000031)
[your_program] Found partition record for topic foo, partition 0
[your_program] Loading partition log for foo-0
[your_program] Reading partition log from: /tmp/kraft-combined-logs/foo-0/00000000000000000000.log
[your_program] Successfully opened partition log file
[your_program] Read 0 record batches from partition log
[your_program] Loaded 0 record batches for partition
[your_program] Added to map
[your_program] Processing batch with 3 records
[your_program] Found topic record: quz (ID: 0x40008000000000000066)
[your_program] Found partition record for topic quz, partition 0
[your_program] Loading partition log for quz-0
[your_program] Reading partition log from: /tmp/kraft-combined-logs/quz-0/00000000000000000000.log
[your_program] Successfully opened partition log file
[your_program] Read 2 record batches from partition log
[your_program] Loaded 2 record batches for partition
[your_program] Added to map
[your_program] Found partition record for topic quz, partition 1
[your_program] Loading partition log for quz-1
[your_program] Reading partition log from: /tmp/kraft-combined-logs/quz-1/00000000000000000000.log
[your_program] Successfully opened partition log file
[your_program] Read 0 record batches from partition log
[your_program] Loaded 0 record batches for partition
[your_program] Added to map
[your_program] Completed loading metadata for 3 topics
[your_program] Loaded cluster metadata
[your_program] Processing topic ID: 0x40008000000000000066 with 1 partitions
[tester::#FD8] Hexdump of received "Fetch" response:
[tester::#FD8] Idx | Hex | ASCII
[tester::#FD8] -----+-------------------------------------------------+-----------------
[tester::#FD8] 0000 | 00 00 00 72 10 7e b8 c0 00 00 00 00 00 00 00 00 | ...r.~..........
[tester::#FD8] 0010 | 00 00 00 02 00 00 00 00 00 00 40 00 80 00 00 00 | ..........@.....
[tester::#FD8] 0020 | 00 00 00 66 02 00 00 00 00 00 00 00 00 00 00 00 | ...f............
[tester::#FD8] 0030 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
[tester::#FD8] 0040 | 00 00 00 00 00 00 01 01 2a 00 00 00 00 00 00 00 | ........*.......
[tester::#FD8] 0050 | 00 00 00 00 47 00 00 00 00 02 b8 7c 9c ff 00 00 | ....G......|....
[tester::#FD8] 0060 | 00 00 00 00 00 01 00 00 00 4b 00 00 00 00 02 55 | .........K.....U
[tester::#FD8] 0070 | 60 53 93 00 00 00 | `S....
[tester::#FD8]
[tester::#FD8] [Decoder] - .ResponseHeader
[tester::#FD8] [Decoder] - .correlation_id (276740288)
[tester::#FD8] [Decoder] - .TAG_BUFFER
[tester::#FD8] [Decoder] - .ResponseBody
[tester::#FD8] [Decoder] - .throttle_time_ms (0)
[tester::#FD8] [Decoder] - .error_code (0)
[tester::#FD8] [Decoder] - .session_id (0)
[tester::#FD8] [Decoder] - .num_responses (1)
[tester::#FD8] [Decoder] - .TopicResponse[0]
[tester::#FD8] [Decoder] - .topic_id (00000000-0000-4000-8000-000000000066)
[tester::#FD8] [Decoder] - .num_partitions (1)
[tester::#FD8] [Decoder] - .PartitionResponse[0]
[tester::#FD8] [Decoder] - .partition_index (0)
[tester::#FD8] [Decoder] - .error_code (0)
[tester::#FD8] [Decoder] - .high_watermark (0)
[tester::#FD8] [Decoder] - .last_stable_offset (0)
[tester::#FD8] [Decoder] - .log_start_offset (0)
[tester::#FD8] [Decoder] - .num_aborted_transactions (0)
[tester::#FD8] [Decoder] - .preferred_read_replica (257)
[tester::#FD8] [Decoder] - .compact_records_length (41)
[tester::#FD8] [Decoder] - .RecordBatch[0]
[tester::#FD8] [Decoder] - .base_offset (0)
[tester::#FD8] [Decoder] - .batch_length (71)
[tester::#FD8] [Decoder] - .partition_leader_epoch (0)
[tester::#FD8] [Decoder] - .magic_byte (2)
[tester::#FD8] [Decoder] - .crc (-1199792897)
[tester::#FD8] Received:
[tester::#FD8] Hex (bytes 85-100) | ASCII
[tester::#FD8] ------------------------------------------------+------------------
[tester::#FD8] 02 b8 7c 9c ff 00 00 00 00 00 00 00 01 00 00 00 | ..|.............
[tester::#FD8] ^ ^
[tester::#FD8] Error: CRC mismatch: calculated 00000000, expected b87c9cff
[tester::#FD8] Context:
[tester::#FD8] - Fetch Response v16
[tester::#FD8] - Response Body
[tester::#FD8] - TopicResponse[0]
[tester::#FD8] - PartitionResponse[0]
[tester::#FD8] - RecordBatch[0]
[tester::#FD8] - crc
[tester::#FD8]
[tester::#FD8] Test failed
[tester::#FD8] Terminating program
[your_program] Processing partition 0
[your_program] Fetch response completed with offset: 76
[tester::#FD8] Program terminated successfully
Just when I thought I’m done, I get this I cant make out the issue because I haven’t changed the recordbatch reading from the last stage
It seems that the correct data is right there, just off by a few bytes.
I am not sure what would change since this is just the first batch it should be the same as the last challenge unless there is some other data when we have multiple batches.
system
Closed
February 2, 2025, 11:57am
11
This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.