I’m currently performing the golang Kafka implmentation and seem to run into a pesky issue regarding an “End of Data” when pushing my code.
I’ve repeatedly checked the message size upon debugging and it looks right to me. I’m not sure what’s causing this issue with the end of data if it’s not message sizing.
My code can be found here:
Thank you!
The error is in the encoding of the Topic Name. The protocol expects the name to be encoded as a CompactString, which is encoded by prefixing length(string) + 1 as a varint to the actual string bytes, but you are encoding length(string).
Changing line 162 to binary.Write(bytesBuffer, binary.BigEndian, uint8(len(topic) + 1)) //Array length should fix the same.
After making that change, the tester would successfully parse the request, but would fail due to incorrect name:
The issue is that the exact bytes of the topic name are not correct (they have an extra 0 byte at their end, see the following message for better logs). You would need to address that as well.
@andy1li Can we change the tester so as to
Also print the expected number of bytes in the string in brackets, as we are doing with arrays? This would help to debug the issues with one-by-off bytes as in the case of CompactString encoding! Something like this:
In case if any parsed string that we are comparing has any non printable characters, we should explicitly log a message to the user about the same. This would help to avoid seemingly bogus message likes the one posted in the above screenshot. Some like this:
I had made these changes to the tester itself while trying to help debug this implementation, so if it feels helpful I can open a PR to the tester, and we can see if there would be other places as well where these would help (I personally feel 2 is a just have for all string compassion assertions that we are making. I just suffered too a great deal myself due to such extra bytes while implementing the binary parsing myself!)
Hi @EshaanAgg,
Thanks for your help! It seems that the suggestion you made might have fixed the issue I was running into. I’m not entirely sure I understood what the problem was, but another relevant detail was that, besides changing the part you mentioned, I also changed lines 106 and 107 to the following:
topic_names[i] = string(buffer[start_idx : start_idx+name_length-1])
start_idx += name_length
I did this because when reading the strings, during debugging, my “topic_names” variable contained the following string: “foo\x00”. It seems to be a golang quirk, though I’m not sure.
In any case, I hope that rounds things down a bit. Thanks for your help!
That isn’t a GoLang quirk, but rather again the encoding scheme. In the cluster metadata logfiles, the names of the topics and the partitions are both stored as CompactStrings, so the same note of n + 1 encoding applies: this time you have to read one character less than the length that was parsed!
Ah I see, that makes sense - thanks!