HAQM Aurora MySQL zero-ETL integration with HAQM Redshift is now generally available

“Data is at the center of every application, process, and business decision,” wrote Swami Sivasubramanian, VP of Database, Analytics, and Machine Learning at AWS, and I couldn’t agree more. A common pattern customers use today is to build data pipelines to move data from HAQM Aurora to HAQM Redshift. These solutions help them gain insights to grow sales, reduce costs, and optimize their businesses.

To help you focus on creating value from data instead of preparing data for analysis, we announced HAQM Aurora zero-ETL integration with HAQM Redshift at AWS re:Invent 2022 and in public preview for HAQM Aurora MySQL-Compatible Edition in June 2023.

Now generally available: HAQM Aurora MySQL zero-ETL integration with HAQM Redshift
Today, we announced the general availability of HAQM Aurora MySQL zero-ETL integration with HAQM Redshift. With this fully managed solution, you no longer need to build and maintain complex data pipelines in order to derive time-sensitive insights from your transactional data to inform critical business decisions.

This zero-ETL integration between HAQM Aurora and HAQM Redshift unlocks opportunities for you to run near real-time analytics and machine learning (ML) on petabytes of transactional data in HAQM Redshift. As this data gets written into Aurora, it will be available in HAQM Redshift within seconds.

It also enables you to run consolidated analytics from multiple Aurora MySQL database clusters in HAQM Redshift to derive holistic insights across many applications or partitions. HAQM Aurora MySQL zero-ETL integration with HAQM Redshift processes over 1 million transactions per minute (an equivalent of 17.5 million insert/update/delete row operations per minute) from multiple Aurora databases and makes them available in HAQM Redshift in less than 15 seconds (p50 latency lag).

Furthermore, you can take advantage of the analytics and built-in ML capabilities of HAQM Redshift, such as materialized views, cross-Region data sharing, and federated access to multiple data stores and data lakes.

Let’s get started
In this article, I’ll highlight some steps along with information on how you can get started easily. I will use my existing HAQM Aurora MySQL serverless database and HAQM Redshift data warehouse.

To get started, I need to navigate to HAQM RDS and select Create zero-ETL integration on the Zero-ETL integrations page.

On the Create zero-ETL integration page, I need to follow a few steps to configure the integration for my HAQM Aurora database cluster and my HAQM Redshift data warehouse.

First, I define an identifier for my integration and select Next.

On the next page, I need to select the source database by selecting Browse RDS databases.

Here, I can select my existing database as the source.

The next step asks me the target HAQM Redshift data warehouse. Here, I have the flexibility to choose the HAQM Redshift Serverless or RA3 data warehouse in my account or in different account. I select Browse Redshift data warehouses.

Then, I choose the target data warehouse.

Because HAQM Aurora needs to replicate into the data warehouse, we need to add an additional resource policy and add the Aurora database as an authorized integration source in the HAQM Redshift data warehouse.

I can solve this by manually updating in the HAQM Redshift console or let HAQM RDS fix it for me. I tick the checkbox.

On the next page, it shows me the changes that HAQM RDS will perform for us. I select Continue.

On the next page, I can configure the tags and also the encryption. By default, zero-ETL integration encrypts your data using AWS Key Management Service (AWS KMS), and I have the option to use my own key.

Then, I need to review all the configurations and select Create zero-ETL integration to create the integration.

After a few minutes, my zero-ETL integration is sucessfully created. Then, I switch to HAQM Redshift, and on the Zero-ETL integrations page, I can see that I have my recently created zero-ETL integration.

Since the integration does not yet have a target database inside HAQM Redshift, I need to create one.

Now the integration configuration is complete. On this page, I can see the integration status is active, and there is one table that has been replicated.

For testing, I create a new table in my HAQM Aurora database and insert a record into this table.

Then I switched to the Redshift query editor v2 inside HAQM Redshift. Here I can make a connection to the database that I formed as part of the integration. By running a simple query, I can see that my data is already available inside HAQM Redshift.

I found this zero-ETL integration very convenient for two reasons. First, I could unify all data from multiple database clusters together and analyze it in aggregate. Second, within seconds of the transactional data being written into HAQM Aurora MySQL, this zero-ETL integration seamlessly made the data available in HAQM Redshift.

Things to know

Availability – HAQM Aurora zero-ETL integration with HAQM Redshift is available in US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Europe (Frankfurt), Europe (Ireland), and Europe (Stockholm).

Supported Database Engines – HAQM Aurora zero-ETL Integration with HAQM Redshift currently supports MySQL-compatible editions of HAQM Aurora. Support for HAQM Aurora PostgreSQL-Compatible Edition is a work in progress.

Pricing –  HAQM Aurora zero-ETL integration with HAQM Redshift is provided at no additional cost. You pay for existing HAQM Aurora and HAQM Redshift resources used to create and process the change data created as part of a zero-ETL integration.

We’re one step closer to helping you focus more on creating value from data instead of preparing it for analysis. To learn more on how to get started, please visit the HAQM Aurora MySQL zero-ETL integration with HAQM Redshift page.

Happy integrating!
— Donnie