AWS Big Data Blog
Category: Kinesis Data Firehose
Uplevel your data architecture with real- time streaming using HAQM Data Firehose and Snowflake
Today’s fast-paced world demands timely insights and decisions, which is driving the importance of streaming data. Streaming data refers to data that is continuously generated from a variety of sources. The sources of this data, such as clickstream events, change data capture (CDC), application and service logs, and Internet of Things (IoT) data streams are […]
Deliver decompressed HAQM CloudWatch Logs to HAQM S3 and Splunk using HAQM Data Firehose
You can use HAQM Data Firehose to aggregate and deliver log events from your applications and services captured in HAQM CloudWatch Logs to your HAQM Simple Storage Service (HAQM S3) bucket and Splunk destinations, for use cases such as data analytics, security analysis, application troubleshooting etc. By default, CloudWatch Logs are delivered as gzip-compressed objects. […]
Exploring real-time streaming for generative AI Applications
Foundation models (FMs) are large machine learning (ML) models trained on a broad spectrum of unlabeled and generalized datasets. FMs, as the name suggests, provide the foundation to build more specialized downstream applications, and are unique in their adaptability. They can perform a wide range of different tasks, such as natural language processing, classifying images, […]
Build an end-to-end serverless streaming pipeline with Apache Kafka on HAQM MSK using Python
The volume of data generated globally continues to surge, from gaming, retail, and finance, to manufacturing, healthcare, and travel. Organizations are looking for more ways to quickly use the constant inflow of data to innovate for their businesses and customers. They have to reliably capture, process, analyze, and load the data into a myriad of […]
Gain insights from historical location data using HAQM Location Service and AWS analytics services
Many organizations around the world rely on the use of physical assets, such as vehicles, to deliver a service to their end-customers. By tracking these assets in real time and storing the results, asset owners can derive valuable insights on how their assets are being used to continuously deliver business improvements and plan for future […]
Run Kinesis Agent on HAQM ECS
February 9, 2024: HAQM Kinesis Data Firehose has been renamed to HAQM Data Firehose. Read the AWS What’s New post to learn more. Kinesis Agent is a standalone Java software application that offers a straightforward way to collect and send data to HAQM Kinesis Data Streams and HAQM Kinesis Data Firehose. The agent continuously monitors […]
Perform HAQM Kinesis load testing with Locust
February 9, 2024: HAQM Kinesis Data Firehose has been renamed to HAQM Data Firehose. Read the AWS What’s New post to learn more. Building a streaming data solution requires thorough testing at the scale it will operate in a production environment. Streaming applications operating at scale often handle large volumes of up to GBs per […]
Migrate from HAQM Kinesis Data Analytics for SQL Applications to HAQM Managed Service for Apache Flink Studio
February 9, 2024: HAQM Kinesis Data Firehose has been renamed to HAQM Data Firehose. Read the AWS What’s New post to learn more. August 30, 2023: HAQM Kinesis Data Analytics has been renamed to HAQM Managed Service for Apache Flink. Read the announcement in the AWS News Blog and learn more. In this post, we […]
Stream VPC Flow Logs to Datadog via HAQM Kinesis Data Firehose
February 9, 2024: HAQM Kinesis Data Firehose has been renamed to HAQM Data Firehose. Read the AWS What’s New post to learn more. It’s common to store the logs generated by customer’s applications and services in various tools. These logs are important for compliance, audits, troubleshooting, security incident responses, meeting security policies, and many other […]
Accelerate data insights with Elastic and HAQM Kinesis Data Firehose
February 9, 2024: HAQM Kinesis Data Firehose has been renamed to HAQM Data Firehose. Read the AWS What’s New post to learn more. This is a guest post co-written with Udayasimha Theepireddy from Elastic. Processing and analyzing log and Internet of Things (IoT) data can be challenging, especially when dealing with large volumes of real-time […]