AWS Big Data Blog
Category: HAQM VPC
Introducing shared VPC support on HAQM MWAA
In this post, we demonstrate automating deployment of HAQM Managed Workflows for Apache Airflow (HAQM MWAA) using customer-managed endpoints in a VPC, providing compatibility with shared, or otherwise restricted, VPCs. Data scientists and engineers have made Apache Airflow a leading open source tool to create data pipelines due to its active open source community, familiar […]
Stream VPC flow logs to HAQM OpenSearch Service via HAQM Kinesis Data Firehose
February 9, 2024: HAQM Kinesis Data Firehose has been renamed to HAQM Data Firehose. Read the AWS What’s New post to learn more. HAQM Virtual Private Cloud (HAQM VPC) flow logs enable you to track the IP traffic going to and from the network interfaces in your VPC for your workloads. Analyzing VPC logs helps […]
Enrich VPC Flow Logs with resource tags and deliver data to HAQM S3 using HAQM Kinesis Data Firehose
February 9, 2024: HAQM Kinesis Data Firehose has been renamed to HAQM Data Firehose. Read the AWS What’s New post to learn more. VPC Flow Logs is an AWS feature that captures information about the network traffic flows going to and from network interfaces in HAQM Virtual Private Cloud (HAQM VPC). Visibility to the network […]
Optimize performance and reduce costs for network analytics with VPC Flow Logs in Apache Parquet format
VPC Flow Logs help you understand network traffic patterns, identify security issues, audit usage, and diagnose network connectivity on AWS. Customers often route their VPC flow logs directly to HAQM Simple Storage Service (HAQM S3) for long-term retention. You can then use a custom format conversion application to convert these text files into an Apache […]
HAQM QuickSight deployment models for cross-account and cross-Region access to HAQM Redshift and HAQM RDS
Many AWS customers use multiple AWS accounts and Regions across different departments and applications within the same company. However, you might deploy services like HAQM QuickSight using a single-account approach to centralize users, data source access, and dashboard management. This post explores how you can use different HAQM Virtual Private Cloud (HAQM VPC) private connectivity features to connect QuickSight […]
Connect to and run ETL jobs across multiple VPCs using a dedicated AWS Glue VPC
In this blog post, we’ll go through the steps needed to build an ETL pipeline that consumes from one source in one VPC and outputs it to another source in a different VPC. We’ll set up in multiple VPCs to reproduce a situation where your database instances are in multiple VPCs for isolation related to security, audit, or other purposes.