Posted On: Feb 10, 2022
HAQM Redshift launches Streaming Ingestion support for Kinesis Data Streams (KDS). HAQM Redshift’s Streaming Ingestion eliminates the need to stage data in HAQM S3 before ingesting it into HAQM Redshift, enabling customers to achieve low latency in the seconds while ingesting hundreds of megabytes of streaming data per second into their data warehouse.
Data engineers, data analysts, and big data developers are evolving their analytics from batch to real-time, adopting streaming engines like HAQM Kinesis, to implement near real-time responsive logic and analytics on streaming application data. Currently, customers who want to ingest real-time data, from services like HAQM Kinesis, into HAQM Redshift must first stage the data in HAQM S3 and use the COPY command, which achieves latency in the minutes. With the new streaming ingestion capability in HAQM Redshift, you can use SQL (Structured Query Language) within Redshift to provide the ability to connect to and directly ingest data from multiple Kinesis data streams simultaneously. HAQM Redshift streaming ingestion simplifies data pipelines by letting you create materialized views on top of streams directly. The materialized views can also include SQL transforms as part of your ELT (Extract Load Transform) pipeline. Once the materialized views are defined, you can manually refresh them to allow you to query the most recent stream data. This approach allows you to perform downstream processing and transformations of streaming data using existing tools and SQL that you are familiar with, at no additional cost.
HAQM Redshift Streaming Ingestion preview is now available in all HAQM Web Services (AWS) regions where HAQM Redshift is available. To get started with HAQM Redshift Streaming Ingestion, provision an HAQM Redshift cluster on the CURRENT Track and verify your cluster is running version 1.0.35480 or newer. Please note, preview features are provided primarily for evaluation and testing purposes. They should not be used in production systems or with production data. For preview terms and conditions, see Beta Service Participation in AWS Service Terms. For more information on how to get started with HAQM Redshift Streaming Ingestion, see the HAQM Redshift documentation page. Please do not hesitate to contact us with questions or feedback at: redshiftstreamingingestion@haqm.com.