Configure AWS VPC Traffic Mirroring to troubleshoot replication errors in AWS MGN and AWS DRS

Organizations using replication-based solutions for cloud migration or disaster recovery sometimes face challenges when troubleshooting replication errors. These challenges can impact migration timelines, increase operational overhead, and affect business continuity plans. Without proper visibility into network communication, identifying and resolving replication problems becomes a complex and time-consuming task.

AWS Application Migration Service simplifies and expedites the migration of your source servers from physical, virtual, or cloud platforms to run natively on AWS, serving as AWS’s recommended service for lift-and-shift migrations. AWS Elastic Disaster Recovery enables fast, reliable recovery of your on-premises and cloud-based applications to AWS, reducing recovery point objectives (RPO) to seconds and recovery time objectives (RTO) to minutes. Both services use a service-managed replication server in AWS to copy data from your source environment. The replication server stores this data temporarily in a staging area before copying it over to the target instances for migration cutover or disaster recovery failover. Since the replication servers are service-managed, standard network troubleshooting methods aren’t available when replication errors occur. This is where AWS VPC Traffic Mirroring can help, by providing a secure way to capture and analyze network traffic between the source and replication server without disrupting the replication process.

In this post, we demonstrate how you can configure VPC Traffic Mirroring to capture network traffic from replication servers in an Application Migration Service environment. While we don’t delve into resolving specific replication errors, this guide equips you with the tools to gather network data needed for further analysis. The process followed in this post can also be applied to an Elastic Disaster Recovery environment. Implementing these techniques helps you collect valuable information to pinpoint when and where traffic is blocked or impacted, helping you resolve problems and ensure a successful replication.

Understanding Traffic Mirroring

Traffic Mirroring is a feature of HAQM Virtual Private Cloud (HAQM VPC) that allows you to capture and inspect network traffic. Mirroring traffic from the network interfaces of your instances allows you to send this data to a target such as another instance or a network monitoring appliance for analysis.

The main components of Traffic Mirroring are:

• Mirror source: A traffic mirror source is the network interface from which you want to capture traffic. In the context of Application Migration Service, this is the network interface of the replication server running in the staging area.
• Mirror target: A traffic mirror target is the destination for mirrored traffic. Traffic mirroring allows you to copy network traffic from an elastic network interface and to a traffic mirror target for monitoring and analysis.
• Mirror filter: A traffic mirror filter is a set of inbound and outbound rules that determine which traffic is copied from the traffic mirror source and sent to the traffic mirror target.
• Mirror session: A traffic mirror session establishes a relationship between a traffic mirror source and a traffic mirror target.

Traffic mirroring can help identify several common replication errors, including:

• SSL interception (also known as SSL/TLS inspection), a process where firewalls decrypt and inspect encrypted traffic between the source and replication server. This security measure, while intended to protect networks, can interfere with the required secure communication between the two servers.
• Maximum Transmission Unit (MTU) and Maximum Segment Size (MSS) mismatches between the source and replication server, or between intermediary firewalls and the replication server. These mismatches can lead to packet fragmentation or dropped packets, resulting in slower replication speeds or failed data transfers.
• Firewalls adding flags to the traffic, leading the replication server to discard the incoming traffic from the source server.
• OS-level security software rejecting or blocking the communication between the source and replication server.

Solution Overview

The process of setting up VPC Traffic Mirroring to troubleshoot replication errors involves the following steps:

1. Identify replication server: Identify the replication server and its network interface that is used as the source for traffic mirroring.
2. Set up traffic mirror target instance: Create and configure an HAQM Elastic Compute Cloud (HAQM EC2) instance that serves as the destination for the mirrored traffic.
3. Configure traffic mirror target: Configure the traffic mirror target using the network interface of the traffic mirror target instance.
4. Configure traffic mirror filter: Set up the traffic mirror filter with appropriate inbound and outbound rules to determine which network traffic should be captured and forwarded.
5. Set up traffic mirror session: Create a traffic mirror session to establish the connection between the replication server and the traffic mirror target instance. This session defines how traffic will be mirrored.
6. Monitor and analyze traffic: Implement traffic capture and analysis using tcpdump on the target instance to inspect the mirrored traffic communication.

Figure 1: Application Migration Service server traffic mirroring architecture

Prerequisites

You need the following to configure traffic mirroring:

1. Access to an AWS account with necessary permissions to configure traffic mirroring in the staging area subnet.
2. An active migration using Application Migration Service with a source server that has AWS Replication Agent installed and is replicating data to a replication server in the staging area subnet.

Identify replication server

Identify the replication server’s network interface, which serves as the source for traffic mirroring.

Identify replication server’s IP address

You can identify the replication server’s IP address by connecting to your source server and running netstat.

• Linux

$ sudo netstat -nap | grep ":1500"

• Windows

C:\Users\Administrator> netstat -an | find "1500"

You should see five connections from the source server to the replication server.

Figure 2: netstat output showing replication server connections on port 1500

If you don’t see five connections, it indicates that there might be connectivity errors between the source server and replication server. In this case, check the agent log file to find the last attempted connection.

• Linux

$ sudo cat /var/lib/aws-replication-agent/agent.log.0 | grep :1500 | tail -n 1

• Windows (64-bit)

PS C:\Users\Administrator> Get-Content -Path "C:\Program Files (x86)\AWS Replication Agent\agent.log.0" | Select-String ":1500" | Select-Object -Last 1

• Windows (32-bit)

PS C:\Users\Administrator> Get-Content -Path "C:\Program Files\AWS Replication Agent\agent.log.0" | Select-String ":1500" | Select-Object -Last 1

Figure 3: Agent log file inspection for latest replication server’s IP address

Identify replication server’s network interface

In the HAQM EC2 console, under Network interfaces, locate the network interface with the replication server’s IP address. This network interface serves as the source for traffic mirroring.

Note: Record the private IP of the replication server for use during the packet capture step.

Figure 4: Replication server network Interface as traffic mirror source

The same can also be done using AWS Command Line Interface (AWS CLI). Replace <Replication-Server-Private-IP> and <Replication-Server-Public-IP> with the replication server’s IP address.

• For private network configuration:

aws ec2 describe-network-interfaces \
    --filters "Name=addresses.private-ip-address,Values=<Replication-Server-Private-IP>" \
    --query 'NetworkInterfaces[*].[NetworkInterfaceId,Description]' \
    --output table

• For public network configuration:

aws ec2 describe-network-interfaces \
    --filters "Name=association.public-ip,Values=<Replication-Server-Public-IP>" \
    --query 'NetworkInterfaces[*].[NetworkInterfaceId,Description]' \
    --output table

Set up traffic mirror target instance

Create a Linux-based HAQM EC2 instance in the same subnet as the replication server. This can also be placed in a different subnet with appropriate networking setup. Configure its security group to allow inbound traffic from the replication server’s private IP address on port 4789/UDP.

Figure 5: Traffic mirror target instance’s network interface

Record the network interface ID of the traffic mirror target instance for use during the creation of traffic mirror target.

Configure traffic mirror target

The traffic mirror target acts as the destination for your mirrored network traffic from the replication server. Follow these steps to create and configure a traffic mirror target.

1. Navigate to VPC Dashboard in the AWS Region where your staging area subnet exists
2. Under Traffic mirroring, navigate to Mirror targets
3. Click Create traffic mirror target
4. Configure settings:
   • Select Network Interface as Target type
   • Choose the replication server’s network interface as Target
5. Click Create

Figure 6: Create traffic mirror target

Figure 7: Traffic mirror target ID

Configure traffic mirror filter

A traffic mirror filter defines which network packets to capture through inbound and outbound rules. For replication troubleshooting, we’ll create a filter that mirrors all inbound and outbound traffic from the replication server to the traffic mirror target instance. Follow these steps to configure the filter.

1. Under Traffic mirroring, navigate to Mirror filters
2. Click Create traffic mirror filter
3. Configure inbound rule
   

   Source CIDR block	Destination CIDR block
   0.0.0.0/0	0.0.0.0/0

4. Configure outbound rule
   

   Source CIDR block	Destination CIDR block
   0.0.0.0/0	0.0.0.0/0

5. Click Create

Figure 8: Create traffic mirror filter

Figure 9: Traffic mirror filter ID

Set up traffic mirror session

A traffic mirror session connects your traffic mirror source (the replication server’s network interface) to your mirror target using the filter rules. The session establishes the traffic mirroring configuration. Follow these steps to set up the session.

1. Under Traffic mirroring, navigate to Mirror sessions
2. Click Create traffic mirror session
3. Configure the following:
   • Mirror source: Select the replication server’s network interface.
   • Mirror target: Select the traffic mirror target (from Create Traffic Mirror Target step).
   • Session number: Enter 1.
   • Filter: Select the traffic mirror filter (from Configure traffic mirror filter step).
4. Click Create

Figure 10: Create traffic mirror session

Monitor and analyze traffic

To monitor the mirrored traffic, connect to the traffic mirror target instance that you configured to receive the traffic mirroring. Execute a tcpdump command using the host filter. You can use this data to analyze the network traffic coming to the replication server and detect any issues impacting the replication.

Note: Use the replication server’s private IP address when setting up the host filter. This is because the internal network communication uses private addressing.

$ tcpdump -nn -tt -vv host <Replication-Server-Private-IP>

Figure 11: Monitoring mirrored traffic with tcpdump

To analyze the network traffic in more detail, capture and store the packets in a pcap (packet capture) file format. You can then utilize Wireshark, a network protocol analyzer, to examine the captured data in depth. This approach provides a comprehensive view of the network communication and allows for thorough investigation of any potential errors or patterns in the traffic flow.

$ tcpdump -nn -tt -vv host <Replication-Server-Private-IP> -w tcpdump-output.pcap

You can also compare the traffic capture from the source and the replication server to identify any factors impacting the network traffic between these two entities. For a comprehensive analysis, perform packet captures on other network components (such as firewalls) in your environment that are involved in the communication between the source and the replication server. This additional step precisely pinpoints where an error first emerges.

Cleanup

Clean up the resources used for traffic mirroring to avoid unnecessary charges after you complete your traffic mirroring analysis:

• Delete the traffic mirror session
• Delete the traffic mirror target
• Delete the traffic mirror filter
• Terminate the Linux instance used as target for the traffic mirroring

Conclusion

In this post, we demonstrated how to configure VPC Traffic Mirroring to capture network traffic for troubleshooting replication errors in Application Migration Service and Elastic Disaster Recovery Service environments. This approach provides you deeper visibility into network communication between source servers and replication servers, while maintaining operational stability.

Using VPC Traffic Mirroring as shown provides detailed visibility into traffic patterns, enables faster problem resolution and provides the ability to capture and analyze network communications for both immediate troubleshooting and future reference.

Learn more about VPC Traffic Mirroring capabilities in AWS VPC Traffic Mirroring documentation. Stay up-to-date on Application Migration Service and Elastic Disaster Recovery services with our posts in AWS Migration and Modernization Blog and AWS Storage Blog channels.

About the Author