Cross-region connectivity is typically required when hosting applications in active-active mode across multiple regions for high availability and resilience to regional failures.  A couple of use-cases which I wanted to address are:

• Database replication traffic (e.g. Azure SQLMI replication for failover groups)
• Application traffic from one region (webapp) to primary database in another region.

Given below are some options (using the hub-spoke deployment) for enabling cross-region connectivity between 2 Azure Regions for the above use-cases (workloads hosted in spoke VNets). In these options, I've used VMs as source and destination Azure resources in the spoke VNets, as well as VMs (IP forwarding enabled) to simulate routers/NVAs in the hub VNets. In the real world, these hub VMs would be replaced by either Azure Firewall instances (expensive!) or 3rd-party NVAs (PAN-OS, Fortinet, etc.).

NOTE: For completeness, I've included route tables with User-Defined Routes (UDRs) for all options with the assumption that any traffic from on-premises to spoke VNets and vice versa must be steered through the hub routers/NVAs.

OPTION 1 : ExpressRoute Gateway

Pre-requisites:

• ExpressRoute is used for connectivity between on-premises and Azure
• The Hub VNets in both Azure regions are connected to the same ExpressRoute circuit.

Implementation:

If you meet the pre-requisites, then there's nothing further to do. Since both hub VNets are connected to the same ExpressRoute circuit, they're part of the same routing domain and so the spoke VMs can communicate with each other via their locally-peered Hubs and ExpressRoute gateways as shown in the image below, unless you're blocking traffic in the hub NVAs.

OPTION 2 : Hub-to-Hub Global VNet peering

Pre-requisites:

• Global VNet peering between the hub VNets

Implementation:

With Global VNet peering the hub VNets, you can steer traffic between your spoke VNets via the Hub-to-Hub peering with UDRs as shown in the image below.

OPTION 3 : Spoke-to-Spoke Global VNet peering

Pre-requisites:

• Global VNet peering between the spoke VNets

Implementation:

With Global VNet peering the spoke VNets, traffic will be automatically steered between your spoke VNets via the Spoke-to-Spoke peering with the default routes (no UDRs) as shown in the image below.

Network Performance Testing

I used qperf and tcpping to test throughput and RTT (round-trip time) for connectivity between VMs in both spokes (10.10.1.5 to 10.20.1.5) for all 3 options. Given below are configuration specifications that have a bearing on the test results (throughput limitations):

An additional connectivity test for throughput was performed for both VMs in the same spoke subnet only to verify the network bandwidth allocation for the VM SKU DS3_v2 (verified!) and the results are depicted in the image below.

The throughput for the ExpressRoute path in the above image is limited by the bandwidth allocation for the ExpressRoute circuit (500 Mb/s). Increasing the bandwidth of the ExpressRoute circuit as well as changing the SKU of the ExpressRoute gateway to cater to higher throughput will likely be more expensive than any of the Global VNet peering options.

Irrespective of the bandwidth allocated to the ExpressRoute circuit or SKU selected for the ExpressRoute gateway, traffic via ExpressRoute (OPTION 1) will have the highest latency and so this option will not be appropriate for latency-sensitive traffic.

Based on the above observations, I would address the use-cases as below:

• Use spoke-to-spoke Global VNet peering (OPTION 3) for Azure SQLMI replication traffic. This is also a recommendation from Microsoft. A recommended way to do this is to allocate a separate VNet (spoke) for SQLMI in each region. Further, there is no benefit in inspecting this traffic.
• Use hub-to-hub Global VNet peering (Option 2) for application-to-database traffic so that you may inspect/police the traffic as well as have favorable latencies and throughput.

You may provide feedback on this article by clicking "Show comments" below and voting or adding comments.

• Redis is a popular in-memory key-value datastore.
• Azure Cache for Redis is a managed service that offers both Redis open-source (OSS Redis) and the commercial product from Redis Inc. (Redis Enterprise).
• For the latest Azure Cache for Redis Service Tiers, Features and Pricing, refer:
  • https://docs.microsoft.com/en-us/azure/azure-cache-for-redis/
  • https://azure.microsoft.com/en-us/pricing/details/cache/

Decision Tree

Here's a basic decision tree that may help you choose a service tier for the Azure Cache for Redis service:

Application Gateway WAF Triage workbook

The Application Gateway WAF Triage workbook is a handy workbook that will help users drill down to WAF policy violations that affect their web applications. This workbook is easily to deploy and use. Given below are some guidelines:

• Enable diagnostic settings for the Application Gateway and send logs to a Log Analytics workspace.
• When deploying the Application Gateway WAF Triage workbook, link it to the Log Analytics workspace by entering the resource id of the Log Analytics workspace as the Workbook Source Id during deployment.

• For users who need to access this workbook for troubleshooting, assign the following roles:
  • Log Analytics Reader to the Log Analytics workspace
  • Reader to the workbook

NOTE: With the above roles, users will have access to read WAF policy violations for all web applications behind the Application Gateway.

When troubleshooting connectivity between on-premises and your Azure private resources, you'd typically look at network routes and firewall policies, among other things. Given below are two ways to check if the relevant on-premises networks are visible on Azure (advertised via Azure Private peering) in its ExpressRoute circuit route table.

Azure Portal

Check the route table on the relevant ExpressRoute circuit circuit as shown in the screenshot below. You may also download the route table (as a csv) for both primary and secondary ExpressRoute circuit connections.

Azure CLI

Use the following command structure to view the route table JSON for both primary and secondary ExpressRoute circuit connections.

az network express-route list-route-tables -g <resource-group-name> -n <expressroute-circuit-name> --path <primary or secondary> --peering-name AzurePrivatePeering

For example, to view the route table JSON for a primary connection on ExpressRoute circuit cg-erc in resource group cg-rg, use the following command:

az network express-route list-route-tables -g cg-rg -n cg-erc --path primary --peering-name AzurePrivatePeering

In order to just obtain a convenient list of networks in the route table, you may add the additional --query switch as shown below:

az network express-route list-route-tables -g cg-rg -n cg-erc --path primary --peering-name AzurePrivatePeering --query 'value[].[network]' -o tsv

• Azure DNS offers both Public DNS and Private DNS services.
• Azure Private DNS zones provide a secure DNS service only for your Azure virtual networks. The records in Azure private DNS zones cannot be resolved via DNS lookups from outside Azure virtual networks.
• To resolve the records of an Azure private DNS zone from your virtual network, you must link the virtual network with the zone. Such a link is called a Virtual Network Link
• Azure's default DNS service for Virtual Networks is provided via 168.63.129.16. All DNS lookups to this IP address must be recursive.

Why Custom DNS?

• If your infrastructure is entirely on Azure, you may use the Azure Private DNS zones as your DNS service for private workloads deployed in your Azure virtual networks and avoid the deployment of any custom DNS solution.
• If your infrastructure is deployed across both on-premises and Azure, then you may leverage your on-premises custom DNS to resolve records for both on-premises private domains and Azure private DNS zones.

UPDATE (17th MAY 2022): Microsoft announced the preview of a new service called Azure DNS Private Resolver which removes the need of the custom DNS solution explained in the remainder of this article. This service is currently in public preview only in certain regions, but once it becomes GA globally, it will help customers eliminate the cost and administration of custom DNS solutions.

Primary use-case for Azure private DNS zone

If you use private endpoints for Azure PaaS services, then your PaaS service private endpoints will use private DNS zones for the DNS A records associated with the private endpoints. For example, if you you deploy an Azure SQL PaaS service with endpoint azsql.database.windows.net and create a private endpoint for this service, Azure will create a DNS A record for azsql.privatelink.database.windows.net (associated with a private IP from the relevant virtual network) in the privatelink.database.windows.net private DNS zone.

Using custom DNS with Azure private DNS zones

Even if you extend your distributed on-premises DNS solution (e.g. AD DNS, EfficientIP, Infoblox, Bluecat, etc.) to Azure by deploying instances within your Azure virtual networks, lookups from your on-premises clients for records in Azure private DNS zones will still be unresolvable as the DNS queries will originate from outside your Azure virtual networks. For such lookups to work, you must do the following:

• Use conditional forwarders on your custom DNS for Azure PaaS public DNS zones (e.g. database.windows.net) to send traffic to a DNS forwarding instance within your Azure virtual network. Azure creates a CNAME in the public DNS zone (e.g. database.windows.net) that redirects resolution to the private DNS zone (e.g. privatelink.database.windows.net).
• Deploy a solution to forward DNS lookups from your custom DNS to Azure's DNS service (168.63.129.16). This solution may be another DNS server (not part of your distributed custom DNS platform), DNS proxy on Azure Firewall or a Custom Firewall NAT policy that forwards traffic destined to a specific IP, to the Azure DNS service (168.63.129.16). If you already have firewalls in Azure, then using NAT policies with the firewall is a good solution. For example, you may choose a specific IP (not attached to anything) in your virtual network (say 10.10.0.100) for the conditional forwarder for database.windows.net on your custom DNS to forward lookups and configure a NAT policy on your firewalls in Azure to send this traffic to Azure's DNS. Of course, the DNS traffic from your conditional forwarder must pass through the firewall to reach 10.10.0.100 for this to work.

Putting all the above together, the image below depicts how 2 on-premises custom DNS solutions (1 PROD and 1 DEV) may be used along with 2 Azure tenants (1 PROD and 1 DEV) with Azure Private DNS zones in the PROD Azure tenant. With this deployment, both Azure and on-premises workloads can resolve records in both Azure and on-premises private DNS zones, thereby facilitating communication between workloads on Azure and on-premises and providing a seamless DNS service.

References

1. Azure Private DNS

2. Private Endpoint DNS Integration Scenarios

Microsoft offers the Azure ExpressRoute service for private connections between your on-premises datacenters and Azure regions.

ExpressRoute connectivity is enabled via ExpressRoute circuits, which are logical constructs representing connections between your on-premises and Microsoft edge locations via one or more communications provider.

Each ExpressRoute circuit offers peerings for Public services and Private Services, with each peering offering primary and secondary connections:

• Azure Private peering: Primary and Secondary connections for Azure private services in your VNets.
• Microsoft peering: Primary and Secondary connections for Azure public services (e.g. Storage, Dynamics 365)

Azure Private Peering

If you have multiple on-premises datacenters and use multiple Azure regions with multiple ExpressRoute circuits, then you may design for Disaster Recovery for your on-premises-to-Azure private traffic as per the topology shown below. This topology is listed as Scenario 1 in Microsoft's documentation. This scenario is appropriate when your on-premises datacenters much less apart than the Azure paired regions which you use. For example, in the topology below, the on-premises datacenters are less than 50 km part (not great for a regional disaster!) whereas the Azure regions are at least 300 miles apart.

Note that there are 2 levels of redundancy:

• Within an ExpressRoute circuit (primary and secondary connections)
• Between ExpressRoute circuits (traffic may be steered via the running ExpressRoute circuit should the other ExpressRoute circuit fail)

Multiple Traffic Paths for Azure Private Services

Each ExpressRoute circuit may be connected to your Azure VNets via a virtual network gateway connection between the ExpressRoute circuit and the Virtual Network Gateway in your VNet (within the GatewaySubnet).

So, as per the topology diagram below, configuring virtual network gateway connections from the virtual network gateways in both Azure regions to both ExpressRoute circuits provides mesh connectivity between your on-premises sites and the Azure regions.

For traffic from Azure to on-premises, a routing weight must be associated with each virtual network gateway connection to set up a connection preference. Connections with higher routing weights are preferred over those with lower routing weights.

For traffic from on-premises to Azure, AS_Path prepend may be used along with Azure private network prefix lists to prefer one ExpressRoute circuit over the other.

Microsoft Peering

If you have multiple on-premises datacenters and use multiple Azure regions with multiple ExpressRoute circuits, then you may design for Disaster Recovery for your on-premises-to-Azure publice service traffic as per the topology shown below. This scenario is appropriate when your on-premises datacenters much less apart than the Azure paired regions which you use. For example, in the topology below, the on-premises datacenters are less than 50 km part (not great for a regional disaster!) whereas the Azure regions are at least 300 miles apart.

Note that there are 2 levels of redundancy:

• Within an ExpressRoute circuit (primary and secondary connections)
• Between ExpressRoute circuits (traffic may be steered via the running ExpressRoute circuit should the other ExpressRoute circuit fail)

Multiple Traffic Paths for Azure Public Services

Each Azure Region will have a Route Filter that allows the BGP communities of specific Azure Public services to be advertised over associated ExpressRoute circuits. By configuring each route filter in each region to allow the BGP communities for Azure public services from both regions to be advertised over the associated expressroute circuits, we can ensure that the Azure public services in both regions may be accessed from on-premises via each expressroute circuit. For example, the image below shows that the Azure Storage public service for both Canada Central and Canada East regions has its BGP communities allowed and this route filter is a associated with a specific expressroute circuit.

Configuring route filters as explained above provides mesh connectivity between your on-premises sites and the Azure regions for accessing Azure public services over ExpressRoute. The image below shows the topology.

For traffic from on-premises to Azure, AS_Path prepend may be used along with Azure public service BGP community lists to prefer one ExpressRoute circuit over the other.

References

1. BCDR - Azure Paired Regions

2. Designing for disaster recovery with ExpressRoute private peering