micrometer-kafka

This project contains several advanced Micrometer metrics for Kafka. The metrics provided by Micrometer out of the box are useful for publishing the metrics collected by the Kafka producer and consumer clients themselves, and for metrics provided via JMX. However this library calculates metrics without using JMX so is suitable for use with Amazon MSK for example.

How to use

Import this project from Maven central.

For each metric you want to publish, create an instance of that metric, providing a function to create the Kafka Admin Client, along with a cadence of how often to refresh the metrics from Kafka.

Note that if you have a large number of topics, or a large number of consumer groups you may wish to only fetch the metrics every few minutes.

Then for each metric instance, bind to your registry.

For example:

val registry = LoggingMeterRegistry()

val metric = TopicTimestampMetric(
   setOf("topic1", "topic2"),
   Duration.ofSeconds(30)
) {
   val props = Properties()
   props[CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG] = "localhost:9092"
   AdminClient.create(props)
}

metric.bind(registry)

Available metrics

Metric Name	Description
ConsumerGroupMemberCountMetric	Provides a gauge for the member counts of each of the specified consumer groups.
ConsumerGroupStatesMetric	Provides a gauge counting how many of the specified consumer groups are in each of the consumer group states.
NonPreferredLeadersMetric	Provides a gauge for each of the specified topics showing the count of non preferred leaders for that topic.
OffsetLagMetric	Provides a gauge for each of the specified topics showing the combined offset lag for a specified consumer across all partitions of that topic.
OutOfSyncReplicasMetric	Provides a gauge for each of the specified topics counting how many replicas are currently ouf of sync for that topic.
TopicProduceCountMetric	Provides a counter for the increase in records on the specified topics. Unlike the producer counts generated out of the box, this metric will count the increase on the topic as a whole and not individual producer rates.
TopicRetentionMetric	Provides a gauge for each of the specified topics showing retention times in milliseconds.
TopicRecordCountMetric	Provides a gauge for each of the specified topics showing an estimated count of records across all partitions of that topic.
TopicTimestampMetric	Provides a gauge for each of the specified topics showing the max timestamp of all records currently on the topic.

GitHub

View Github