Data Engineering

4 readers

1 users here now

News and discussion on Data Engineering topics

founded 1 year ago

MODERATORS

ephemeral404@lemmy.ml

Visualising Kafka Internals (lemmy.ml)

submitted 1 year ago by bahmanm@lemmy.ml to c/dataengineering@lemmy.ml

2 comments fedilink hide all child comments

I plan to run a few tests to determine if Kafka is suitable for a certain usecase I have in mind.

My idea is to run a local cluster of Kafka servers (either VMs or containers), produce/consume a series of messages, observe a bunch of metrics (Prometheus & Grafana) and custom business logic outcomes.

What are some good tools to record and visualise the internals of Kafka cluster?

I'm looking for things like consumer lag, topic replication, possibly tracing messages, ...

Originally posted on https://mastodon.social/@bahmanm/110662538718523380

you are viewing a single comment's thread
view the rest of the comments

[–] bahmanm@lemmy.ml 1 points 1 year ago

Thanks 🙏 That's what I've gathered so far too. I will keep this post, well, posted in case I stumble upon something else.