Has anyone built their own Kafka? (Or know others who have?)

General
1

We’re working on build your own Kafka as our next challenge (:shushing_face: super secret, of course) - this is the consistently one of the highest voted challenges, but I haven’t been able to find a lot of resources online related to this.

The closest I could find is minikafka — write your own kafka from scratch | by Henry Fuheng Wu | Medium, but that’s pretty barebones.

Has anyone here built their own Kafka / know someone else who has? Would love to show you a preview of our challenge outline and see if there’s a better way to approach things.

2

Hey,

3

Hey,

4

In case you need me to do any POC in Golang or Java I will be happy to do it.

5

@VishalMCF nice finds, thanks! These don’t actually mimic Kafka though, they only build something similar. What I was initially looking for was something that tries to actually implement the Kafka protocol in a byte-compatible way (and similarly to actually read Kafka’s on-disk storage format).

If anyone knows of resources along these lines, please do share!

6

Paul by on disk storage format do you mean append only log?

7

@VishalMCF yep! Specifically, these files: Deep Dive Into Apache Kafka | Storage Internals – Rohith Sankepally

8

Marking this as closed for now - pasting all the links that @ryan-gang gathered as part of internal research, in case it helps:

Client libraries:

  1. Sarama (docs)/ kafka-python / kafka-python-ng (docs)
  2. Sarama usage examples
  3. Sarama initial broker implementation 11yr ago
  4. Barebones sarama

Kafka Docs

  1. Quickstart
  2. Protocol doc: Apache Kafka
  3. Log Implementation
  4. Confluent docs: Intro to Apache Kafka: How Kafka Works
  5. Javadoc

Articles

  1. Deep Dive Into Apache Kafka | Storage Internals – Rohith Sankepally
  2. Apache Kafka and the File System | Confluent Documentation
  3. Apache Kafka: What Is and How It Works | by João Guilherme Berti Sczip | The Startup | Medium
  4. A Practical Introduction to Kafka Storage Internals · FreBlogg
  5. Kafka 101