Question about fd8: Fetch multiple messages from disk - Expected partitions.length to be 1, got 2

[asdfman]

The test has 2 partitions for the topic. One partition has 2 record (batches) and the other has nothing.

It’s easily fixed by not including partitions with no records in the response but that will cause an earlier test to fail:
Stage #CM4 (Consuming Messages - Fetch with an empty topic)

It’s quite unclear what the desired behavior is and the 2 stages seem contradicting.

What am I missing here?

Relevant code:

fn topic_handler(topic_record: &TopicRecord, metadata: &MetadataFile) -> FetchTopicResponse {
    let mut partitions = vec![];
    let partition_iter = metadata.get_topic_partitions(&topic_record.uuid);
    for partition in partition_iter {
        let mut records = vec![];
        if let Ok(partition_metadata) =
            read_partition_metadata(topic_record.topic_name.data.clone(), partition.partition_id)
        {
            for record in partition_metadata.record_batches {
                records.push(record);
            }
        }
        partitions.push(FetchTopicPartition {
            partition_index: partition.partition_id,
            error_code: ErrorCode::NoError,
            high_watermark: 0,
            last_stable_offset: 0,
            log_start_offset: 0,
            aborted_transactions: CompactArray::new(vec![]),
            preferred_read_replica: 0,
            record_batches: records,
        });
    }
    FetchTopicResponse {
        topic_id: topic_record.uuid,
        partitions: CompactArray::new(partitions),
    }
}

[andy1li]

Hey @asdfman, could you upload your code to GitHub and share the link? It will be much easier to debug if I can run it directly.

[asdfman]

Sure thing

GitHub - asdfman/codecrafters-kafka-rust

[andy1li]

@asdfman The empty partition should be excluded (unless certain conditions apply). What specific errors are you seeing for stage #CM4 “Fetch with an empty topic”?

Here’s the relevant Kafka code for reference:

core/src/main/scala/kafka/server/FetchSession.scala

[asdfman]

#CM4 expects one partition in the response

Changed code (passes last stage now)

CM4 instructions seem pretty clear about expecting a partition in the response even though it has no data:

Unless I’m missing something it seems like CM4 contradicts the Kafka code you posted above..?

[andy1li]

@asdfman You’re absolutely right! My previous reply was incorrect. The filtering only applies to incremental fetches.

The real reason why #fd8 doesn’t expect the empty partition isn’t because it’s empty, but because the request doesn’t ask for it. The request only asks for Partition 0:

Note that the empty partition is Partition 1, which the request doesn’t ask for:

[asdfman]

Ah, that explains it. I knew I had to be missing something. I was not reading the fetch topic request beyond topic name so I never even realized it specifies partition as well.

Thanks for your help.

As a future improvement I would suggest changing the instructions in these stages:

• The partitions array has 1 element, and in that element:
• The partition_index field is 0.

Should rather mention that partition index should match what was requested, that would make it a lot clearer. An instruction change would go a long way even if the tester always looks for index 0, if test case parameters cannot be randomized.

[andy1li]

Quick update: we’ve improved the instructions a bit in this PR.

Thanks again for highlighting the issue! @asdfman