Kafka #CM4 missing details

General
1

I’m stuck on Stage #CM4.

It’s clear we need to distinguish between the previous stage and this stage, but I’m not clear how that is. done. I read there is a partition record metadata file involved, but where does that come from? I don’t see that in the request.

4

Hey @samer-hamood Yes, you’ll need to read the metadata file (/tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log) to determine whether the requested topic exists.

For stage #cm4 “Fetch with an empty topic”, you can assume that if the requested topic exists, it contains no records.

For stage #VT6 “List for an unknown topic”, the topic does not exist at all. Your broker should respond accordingly.

Let me know if you’d like any further clarification!

6

And where does that file come from? The expectation was that all the required input to complete the exercise was either coming from the request or the instructions.

Is there a dependency I’m missing, or am I not fully parsing the input, as that file is also being sent over the wire?

Thanks for your help.

7

The expectation was that all the required input to complete the exercise was either coming from the request or the instructions.

@samer-hamood Would you mind sharing where that expectation came from? We might need to tweak the instructions to avoid this expectation.

as that file is also being sent over the wire?

Nope, the file exists locally on the server and isn’t transmitted over the wire.

Just a quick reminder: since you’re implementing a Kafka broker (i.e. server), it’s expected to manage messages and maintain metadata locally (on the server).

8

Would you mind sharing where that expectation came from? We might need to tweak the instructions to avoid this expectation.

Sure, it might be just me but:

  1. based on previous stages being relatively independent and having instructions only on what to return
  2. the title of the stage being similar to the others, all suggesting a separate functionality, irrespective of previous requests, all of which didn’t require maintaining any record of them
  3. In this stage, you'll implement the Fetch response for a topic with no messages.In this stage, you'll implement the Fetch response for a topic with no messages. Response to me suggests a stateless operation, not taking into account previous requests
  4. No mention of storing state in any file or that very specific metadata file, in this stage or previous ones

Yes, you’ll need to read the metadata file (/tmp/kraft-combined-logs/__cluster_metadata-0/00000000000000000000.log ) to determine whether the requested topic exists.

So this is a specific file the broker creates to store the metadata, correct? I don’t think a topic has been created so far, so this is the first time we need it, as there was no test indicating we did previously, and those for this one fails on regression for me. Maybe this is a file we’re supposed to create for this stage and is needed from now on.

Just a quick reminder: since you’re implementing a Kafka broker (i.e. server), it’s expected to manage messages and maintain metadata locally (on the server).

Fair enough; it’s not this that I’m unaware of, but how and when to implement each required part of the server in the way that it actually should be. We’re not implement the server all at once, right?

10

@samer-hamood, thanks the clarification!

You might want to try tackling the Listing Partitions extension first. It introduces the idea of reading from the metadata file in a more guided way.

image
image1326×998 68.6 KB

Let me know if anything is still unclear!