Posted On: Nov 23, 2020
HAQM Managed Streaming for Apache Kafka (MSK) now offers consumer lag metrics for new HAQM MSK clusters by default, making it easier for you to track whether your applications are consuming the latest data available in your Apache Kafka cluster. Consumer lag metrics quantify the difference between the latest data written to Apache Kafka topics in HAQM MSK and the data read by your applications. Monitoring consumer lag metrics allows application developers to identify slow or stuck consumers that are not keeping up with the latest data available in an Apache Kafka topic so they can take remedial actions such as scaling or rebooting those consumers.
Consumer lag metrics can be monitored using HAQM CloudWatch or Open Monitoring for Prometheus. Additionally, HAQM MSK has moved ten broker level and two topic level metrics into its default monitoring tier for all clusters, which allows you to monitor key cluster metrics such as bytes-in and bytes-out for free.
Consumer lag metrics are available for all new HAQM MSK clusters in all AWS regions where the service is currently available. For existing clusters, reach out to AWS Support to ask for enablement of consumer lag metrics. To learn more about monitoring your HAQM MSK clusters, see our documentation.