Process HAQM Kinesis Aggregated Data with AWS Lambda

Ian Meyers is a Solutions Architecture Senior Manager with AWS

Last year, we introduced the HAQM Kinesis Producer Library (KPL) to simplify the development of applications that need to send data to HAQM Kinesis Streams. Many customers use aggregation, which allows you to send multiple records to a single HAQM Kinesis Streams record.  Although the KPL made it easy to aggregate events, it left you with the problem of unpacking this information when you consumed the record from the stream.

We are excited to announce that you can now process this data easily using HAQM Kinesis de-aggregation modules. These modules support Java, Python, and Node.js and allow you to extract user records from a KPL-aggregated stream to AWS Lambda or your multi-lang KCL application.

We’ve also launched version 1.2.0 of the Lambda-based HAQM Kinesis Streams to HAQM Kinesis Firehose Forwarder, which automatically extracts KPL user records from your stream before forwarding for archiving in HAQM S3 or loading to HAQM Redshift.

If you have questions or suggestions, please leave a comment below.

——————————

Related

Implementing Efficient and Reliable Producers with the HAQM Kinesis Producer Library

Looking to learn more about Big Data or Streaming Data? Check out our Big Data and Streaming data educational pages.