The desired record value is [72 101 108 108 111 32 87 111 114 108 100 33], however, i do not see any record value equal to it in provided records.(I print the record value out to see whether the desired record value exists?) Do i misunderstand the requirements, or is it a bug?
Hey @page0egap, could you upload your code to GitHub and share the link? It will be much easier to debug if I can run it directly.
I do not know how to share the repository. I upload whole code to the public github repository GitHub - page0egap/page0egap-codecrafters-kafka. I hope it help.
@page0egap I tried running your code against the previous stages, but it’s actually no longer passing the previous stage #CM4 (Consuming Messages - Fetch with an empty topic).
Suggestions:
- Use our CLI to test against previous stages by running:
codecrafters test --previous
- Focus on fixing the early stages first, as later stages depend on them.
@andy1li I update my code. But the stage CM4 still do not pass since i found that the request topic is not empty. The topic does contain one partion record. Here is an example. I’m confused of the meaning of “empty topic”. Could you explain it?
I’m confused of the meaning of “empty topic”. Could you explain it?
@page0egap An “empty” topic is one with no messages stored in it.
Looks like you placed partition records in the spot where message records should be:
The actual message records should come from the log on disk, rather than from metadata structures like partition records.
Let me know if you’d like further clarification!
@Andy1li What’s the meaning of message records and partition records? Are they all from disk log? I know the record batch has many records. Each record has their types like partition record or topic record. I assume the record batch contains the first topic record then following partition records with the same topic id. I send the rebuild the record batch by removing the topic record and combine the remaining partition records together. Then the record batch is put in the response parition “records” field. What’s wrong with it?
And my question is why the request topic id do has parition records in the disk log, the expected result from tester is “Expected recordBatches.length to be 0, got 1”. I think if the request topic id do has records in the disk log, the recordBatches(records) field should not be empty. Could you explain it?
@page0egap There are two types of log files:
-
Data logs where Kafka stores actual messages for each partition.
Example:/tmp/kraft-combined-logs/foo-0/00000000000000000000.log -
Metadata logs where Kafka stores internal cluster metadata (topics and partitions).
Example:/tmp/kraft-combined-logs/__cluster_metadata-0/
Up until #hn6 Fetch with an unknown topic, you’ve been working with metadata logs. These contain records about topics and partitions, but not actual messages.
Starting from #cm4 Fetch with an empty topic and #eg2 Fetch single message from disk, you’ll need to also read from data logs, which store the actual messages.
And my question is why the request topic id do has parition records in the disk log, the expected result from tester is “Expected recordBatches.length to be 0, got 1”. I think if the request topic id do has records in the disk log, the recordBatches(records) field should not be empty. Could you explain it?
The key distinction here is between the existence of partitions and the presence of messages.
A topic can have one or more partitions (as defined in metadata logs), but that doesn’t necessarily mean any messages have been written to it yet (as stored in data logs).
Let me know if you need further clarification!
@andy1li So does it mean that we just need to check whether the topic’s partition exists in metadata log and then fill the raw file of corresponding message log to records field to build the repsonse.
For me, i do not understand how kafka log file works and its structure. I know the learning project is under development. But i still hope that the task gives all needed information to help us finish it at least. Thank you all for valuable works.
1 Like
This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.