HAQM OpenSearch Ingestion now supports ingesting streaming data from HAQM MSK Serverless
HAQM OpenSearch Ingestion now allows you to ingest streaming data from HAQM Managed Streaming for Apache Kafka (MSK) Serverless, enabling you to seamlessly index the data from HAQM MSK Serverless clusters in HAQM OpenSearch Service managed clusters or Serverless collections without the need for any third-party data connectors. With this integration, you can now use HAQM OpenSearch Ingestion to perform near- real-time aggregations, sampling and anomaly detection on data ingested from HAQM MSK Serverless, helping you to build efficient data pipelines to power your complex observability and analytics use cases.
HAQM OpenSearch Ingestion pipelines can consume data from one or more topics in an HAQM MSK Serverless cluster and transform the data before writing it to HAQM OpenSearch Service or HAQM S3. While reading data from HAQM MSK Serverless via HAQM OpenSearch Ingestion, you can configure the number of consumers per topic and tune different fetch parameters for high and low priority data. Furthermore, you can also optionally use AWS Glue Schema Registry to specify your data schema to dynamically read custom data schema at ingest time.
This feature is available in all the 13 AWS commercial regions where HAQM OpenSearch Ingestion and HAQM MSK Serverless are currently available: US East (Ohio), US East (N. Virginia), US West (Oregon), Europe (Ireland), Europe (London), Europe (Frankfurt), Asia Pacific (Tokyo), Asia Pacific (Sydney), Asia Pacific (Singapore), Asia Pacific (Mumbai), Asia Pacific (Seoul), Canada (Central), and Europe (Stockholm).
To learn more, see the HAQM OpenSearch Ingestion webpage and the HAQM OpenSearch Service Developer Guide.