Posted On: Nov 29, 2022
HAQM Redshift now supports real-time streaming ingestion for HAQM Kinesis Data Streams (KDS) and HAQM Managed Streaming for Apache Kafka (MSK). HAQM Redshift streaming ingestion eliminates the need to stage streaming data in HAQM S3 before ingesting it into HAQM Redshift, enabling customers to achieve low latency, measured in seconds, while ingesting hundreds of megabytes of streaming data per second into their data warehouse.
Data engineers, data analysts, and big data developers are evolving their analytics from batch to real-time, adopting streaming engines like HAQM KDS and HAQM MSK, to implement near real-time responsive logic and analytics on streaming application data. Currently, customers who want to ingest real-time data from services like HAQM KDS and HAQM MSK, into HAQM Redshift must first stage the data in HAQM S3 and use the COPY command, which achieves latency in the minutes. With the new streaming ingestion capability in HAQM Redshift, you can use SQL (Structured Query Language) within Redshift to provide the ability to connect to and directly ingest data from multiple HAQM KDS streams or multiple HAQM MSK topics simultaneously. HAQM Redshift streaming ingestion simplifies data pipelines by letting you create materialized views on top of streams directly. The materialized views can also include SQL transforms as part of your ELT (Extract Load Transform) pipeline.
Once the materialized views are defined, streaming data is automatically and continuously ingested from the KDS stream or MSK topic into the HAQM Redshift streaming materialized view when the Auto Refresh feature is enabled. You can also choose to manually refresh the streaming materialized view when direct control over ingest scheduling is desired. This approach allows you to perform downstream processing and transformations of streaming data using existing HAQM Redshift tools and SQL that you are familiar with, at no additional cost.
HAQM Redshift streaming ingestion is now generally available in all HAQM Web Services (AWS) regions where HAQM Redshift is available. To get started with HAQM Redshift streaming ingestion, provision an HAQM Redshift cluster in the CURRENT track. For more information on how to get started with HAQM Redshift streaming ingestion, see the HAQM Redshift documentation page.