15. Appendices A & B
15.6 B.3 Monitoring and data exploration
"A critical part to running Kafka is to ensure that your cluster AND your clients are healthy. Like many applications, Kafka exposes numerous metrics and other telemetry, BUT MAKING SENSE OF IT CAN BE CHALLENGING. Many of the larger monitoring platforms (such as Prometheus) can easily fetch metrics from Kafka brokers and clients."
| Tool | Purpose |
|---|---|
| Xinfra Monitor (formerly Kafka Monitor) — LinkedIn | ⭐ "monitor AVAILABILITY of Kafka clusters and brokers. It does this by using a set of topics to generate SYNTHETIC DATA through the cluster and measuring latency, availability, and completeness. It's a valuable tool for measuring your Kafka deployment's health WITHOUT REQUIRING DIRECT INTERACTION WITH YOUR CLIENTS." (Ch. 13 §9) |
| Burrow — LinkedIn | ⭐ "holistic monitoring of CONSUMER LAG... provides a view into the health of the consumers WITHOUT NEEDING TO DIRECTLY INTERACT WITH THEM. Burrow is actively supported by the community and has its OWN ECOSYSTEM OF TOOLS to connect it with other components." (Ch. 13 §8) |
| Kafka Dashboard — DataDog | "an excellent Kafka Dashboard to give you a head start on integrating Kafka clusters into your monitoring stack. Designed to provide a SINGLE-PANE view, simplifying the view of many metrics." |
| Streams Explorer — bakdata | "visualizing the flow of data through applications and connectors in a KUBERNETES deployment. ⚠️ It heavily relies on structuring your deployments using either Kafka Streams or Faust through bakdata's tools, but it can then provide an easily comprehensible view of those applications and their metrics." |
| kcat (formerly kafkacat) | 💡 "a much-loved alternative to the console producer and consumer... It is SMALL, FAST, and WRITTEN IN C, SO IT DOES NOT HAVE JVM OVERHEAD. It also supports limited views into cluster status by showing METADATA OUTPUT for the cluster." |
(Xinfra Monitor and Burrow are the two external monitors Ch. 13 says you cannot do without — because the broker fundamentally cannot report whether clients can use it, or how far behind consumers are.)