Data integration with AWS

Connect to and act on all your data, no matter where it lives

Get started with data integration on AWS

Overview

The most impactful data-driven insights come from getting a full picture of your business and customers. You can only achieve this when you connect data from all your data sources across multiple departments, services, on-premises tools, and third-party applications.

Data integration with AWS makes it easy to connect to and act on all your data, no matter where it lives. With AWS data integration capabilities, you can bring together data from multiple sources; you can transform, operationalize, and manage data to deliver high quality data across your data lakes and data warehouses.

Watch the re:Invent 2023 session on new releases

AWS Announces Four Zero-ETL Integrations to Make Data Access and Analysis Faster and Easier Across Data Stores

Data integration graphic

Benefits of Data integration with AWS

Your data comes from many sources in a variety of formats—third-party hosted applications, on-premises data stores, and operational data stores. AWS services connect to hundreds of data sources including third-party Software-as-a-Service (SaaS), on premises, and other clouds. Once you’ve connected or moved data into data lakes, warehouses and databases, you can make it securely available across your entire organization.
AWS Glue, one of the many AWS data integration services, consolidates major data integration capabilities into one place including data discovery, extract, transform, and load (ETL), cleansing, transforming, and centralized cataloging. It is serverless and can automatically provision and manage workers as needed.
AWS provides tools that meet the needs of data engineers, ETL developers, and business analysts, helping users at all technical levels to interactively explore and work with their data. You can visually transform data with a drag-and-drop interface in AWS Glue Studio, clean and normalize data with data preparation tool AWS Glue DataBrew, and test data using your preferred integrated development environment (IDE) or notebook.
Oftentimes organizations must support a variety of data processing frameworks, like ETL, reverse ETL, and extract, load, and transform (ELT), as well as, different workloads like batch, micro-batch, and streaming. AWS provides flexible support for all frameworks and workloads, and enables portability by leveraging open-source standards.

Address data integration challenges with AWS

AWS helps you access and integrate data from anywhere you store it, letting your data integration teams focus on high-value activities that maximize the value of your data.

AWS Glue makes it easy to discover, prepare and integrate all your data at any scale. AWS Database Migration Service helps move database and analytics workloads to AWS quickly, securely, and with minimal downtime and zero data loss. HAQM Managed Workflows for Apache Airflow (MWAA) provides secure and highly available managed workflow orchestration for Apache Airflow. AWS Data Exchange connects with third-party data from 300+ data providers and 3,500+ data products.

For complex use cases where you need to transform data, normalize data, check data quality before ingesting data from a raw data store, or reduce table columns, rows, or data size, AWS Glue makes it easier for you to prepare and integrate data.

For common use cases where ETL is regularly repeated, you can use AWS zero-ETL capabilities, which eliminate the need to build and manage ETL pipelines. With AWS zero-ETL capabilities, you can directly ingest streaming data from HAQM Managed Streaming for Apache Kafka to HAQM Redshift, analyze HAQM Aurora data with HAQM Redshift in near real time, and so much more.

Users across the business have varying technical abilities to interact with the data, and may not be able to do so without the right tools.

AWS provides skill-specific interfaces and job-authoring tools for all user types, from developers to business users. AWS Glue Studio automatically generates ETL code, and lets ETL developers and business analysts transform data with a no-code interface. AWS Glue also lets developers and engineers use their preferred IDE, notebook, and processing engines. HAQM Managed Workflows for Apache Airflow enables scientists and engineers to orchestrate end-to-end data pipelines.

AWS provides support for various workloads with no lock-in.

AWS Glue Studio helps you author highly scalable ETL jobs without becoming an Apache Spark expert, and load structured and unstructured data to data warehouses and data lakes. HAQM Managed Streaming for Apache Kafka (MSK) and HAQM Kinesis make it easy to ingest and process streaming data in real-time. Other common workloads include batch data transformation, database replication, data ingestion from SaaS, data sharing among teams, and subscription to third-party data.

AWS Glue – discover, prepare, and integrate all your data at any scale

AWS Q for Data Integration - generative AI-powered AWS Glue capability, enables you to build data integration jobs using natural language.

HAQM Managed Workflows for Apache Airflow – secure and highly available managed workflow orchestration for Apache Airflow

HAQM AppFlow – automate data flows between software as a service (SaaS) and AWS services

HAQM Aurora zero-ETL Integrations with HAQM Redshift – perform near real-time analytics and ML on petabytes of transactional data in Aurora

HAQM Aurora PostgreSQL zero-ETL integration with HAQM

HAQM DynamoDB zero-ETL integration with HAQM Redshift

HAQM RDS for MySQL zero-ETL integration with HAQM Redshift

HAQM DynamoDB zero-ETL integration with HAQM OpenSearch Service

AWS Database Migration Service – move your database and analytics workflows to AWS quickly, securely, and with minimal downtime and zero data loss

HAQM Athena – analyze petabyte-scale data where it lives with ease and flexibility

HAQM Redshift – best price-performance for cloud data warehousing

AWS Lake Formation – build, manage, and secure data lakes in days

AWS Data Exchange – easily find, subscribe to, and use third-party data in the cloud

AWS Glue Catalog – store, annotate and share metadata in the AWS cloud

HAQM DataZone – unlock data across organizational boundaries with built-in governance

What's new

  • Date (Newest to Oldest)
No results found
1 …
…