Pushing VPN Performance Limits in Azure virtualWan

May 15, 2025

Structure of this article

Introduction
Concept: Link aggregation in Azure Virtual WAN

Introduction

Virtual WAN is a networking service that brings a lot of functionalities like routing and security together. It offers full managed mesh connectivity and various connectivity options.

Anyone planning site-to-site VPN connections in Azure VWAN is probably aware of the ~1 GBit/s bandwidth limit per tunnel.

https://github.com/MicrosoftDocs/azure-docs/blob/main/includes/virtual-wan-limits.md

This article shows a way to push the vpn perfomance by aggregating multiple tunnels (up to 8) to one interface in a Virtual WAN setup.

Concept: Link aggregation in Azure Virtual WAN

Azure Virtual WAN site-to-site VPN can aggregate up to 4 links to transfer Data to a single site. One Link consists of 2 VPN Tunnels. (Primary/Backup run in active/active mode) When taken to the maximum you can have ~ >9 Gbit/s site-to-site VPN Gateway throughput to a single site. A site can be anything from an on-premise site , or another azure virtual WAN.

Links	Tunnels	achievable Bandwidth
1	2	~ >2 Gbit/s
2	4	~ >4.5 GBit/s
3	6	~ >7 Gbit/s
4	8	~ >9 Gbit/s

This article is accompanied by a lab setup. The lab consists of two Virtual WANs in two different Azure Regions which are interconnected via a 2 Link (4 Tunnel) site-to-site VPN:

In the lab setup the vpn tunnels are terminated on another Virtual WAN Hub.

It will work with any VPN termination device that is capable of aggergating multiple tunnels into one logic interface.

I successfully tested this setup with a Fortigate Firewall with 8 VPN Tunnels in one aggregation interface and a thoughput >> 9GBPs.

Structure of the bicep deployment

The diagram above shows the Azure resource elements of which the solution consists.

The virtualWan forms the outermost bracket around the other network components in a region. The virtualWan contains the regional virtualHub in which we create a vpnGateway. The vpnGateway by default has 2 Instances (0 and 1) with a publicIp each.

Also inside the virtualWan we create a vpnSite with 2 Links. The vpnSite describes the VPN remote station to which we want to connect.

Each Link manages and establishes 2 Tunnels with the remote site. The values for Vendor and Bandwidth are just for informative reasons (at least I didnt notice an effect).

The vpnSite is connected to the vpnGateway through a vpnConnection resource.

The vpnGateway is deployed through a module. so we can get the Public IPs of the two vpnGateway instances back from the module.

In addition to that there is a conditional deployment of the vnets-and-vms.bicep file which contains the VMs and VNETs to run the bandwidth tests. Set ‘deployVms’ to true if that shall be included.

Code & Configuration

Virtual Wan

The bicep code is available in this repo here:

https://github.com/juggah/vwan-s2s-bw-max/tree/main/src

main.bicep

// deploy the vwans in both regions and the corresponding vhubs 

param region1 string = 'northeurope'
param region2 string = 'germanywestcentral'
param prefixRegion1 string = '10.10.0.0/16'
param prefixRegion2 string = '10.20.0.0/16'
param prefix2Region1 string = '192.168.1.0/24'
param prefix2Region2 string = '192.168.2.0/24'
param vmSize string = Standard_D8_v5 // Size of VM should be appropriate for the bandwidth test
param deployVms boolean = false //deploy the VMs for the bandwidth test ? set to true if desired

resource vWanRegion1 'Microsoft.Network/virtualWans@2024-05-01' = {
  name: 'vwan-${region1}'
  location: region1
  properties: {
    allowBranchToBranchTraffic: true
    type: 'Standard'
  }
}
resource vhubRegion1 'Microsoft.Network/virtualHubs@2024-05-01' = {
    name: 'vhub-${region1}'
    location: region1
    properties: {
      addressPrefix: prefixRegion1
      routeTable: {
        routes: []
      }
      virtualRouterAutoScaleConfiguration: {
        minCapacity: 2
      }
      virtualWan: {
        id: vWanRegion1.id
      }
      hubRoutingPreference: 'ASPath'
    }
  }

resource vWanRegion2 'Microsoft.Network/virtualWans@2024-05-01' = {
  name: 'vwan-${region2}'
  location: region2
  properties: {
    allowBranchToBranchTraffic: true
    type: 'Standard'
  }
}

 

resource vhubRegion2 'Microsoft.Network/virtualHubs@2024-05-01' = {
  name: 'vhub-${region2}'
  location: region2
  properties: {
    addressPrefix: prefixRegion2
    routeTable: {
      routes: []
    }
    virtualRouterAutoScaleConfiguration: {
      minCapacity: 2
    }
    virtualWan: {
      id: vWanRegion2.id
    }
    hubRoutingPreference: 'ASPath'
  }
}

// the vpn gateways inside the hubs are called via module so we get the public IPs back
// the public ips  are needed to feed them into the vpnsite definitions

module vpnRegion1 'vpngw.bicep' = {
  name: 'vpngw-${region1}-${uniqueString(resourceGroup().id)}'
  params : {
  vpnGwName: 'vpngw-${region1}'
  vpnGwLoc: region1
   vHubId: vhubRegion1.id
   }
 }
module vpnRegion2 'vpngw.bicep' = {
 name: 'vpngw-${region2}-${uniqueString(resourceGroup().id)}'
 params : {
  vpnGwName: 'vpngw-${region2}'
  vpnGwLoc: region2
  vHubId: vhubRegion2.id
  }
}

// the vpnSite needs the ip address output of the vpnGateway module. Sometimes there seems to be a delay between the 
// provisioning of the vpnGw and the pip which lets the whole bicep script runs endlessly. 
// this is why there is the conditional.

resource vpnSiteRegion1 'Microsoft.Network/vpnSites@2023-05-01' = {
  name: 'vpnsite-${region1}'
  location: region1
  properties: {
    deviceProperties: {
      deviceVendor: 'Generic'
    }
    virtualWan: {
      id: vWanRegion2.id
    }
    addressSpace: {
      addressPrefixes: [
     prefixRegion1
     prefix2Region1
      ]
    }
    vpnSiteLinks: [
      {
        name: 'link-1'
        properties: {
          ipAddress: vpnRegion1.outputs.vpnGwPublicIps[0] == null ? '1.1.1.1' : vpnRegion1.outputs.vpnGwPublicIps[0]
        }
      }
      {
        name: 'link-2'
        properties: {
          ipAddress: vpnRegion1.outputs.vpnGwPublicIps[1] == null ? '2.2.2.2' : vpnRegion1.outputs.vpnGwPublicIps[1]
        }
      }
    ]
  }
}
resource vpnSiteRegion2 'Microsoft.Network/vpnSites@2023-05-01' = {
  name: 'vpnsite-${region2}'
  location: region2
  properties: {
    deviceProperties: {
      deviceVendor: 'Generic'
    }
    virtualWan: {
      id: vWanRegion1.id
    }
    addressSpace: {
      addressPrefixes: [
     prefixRegion2
     prefix2Region2
      ]
    }
    vpnSiteLinks: [
      {
        name: 'link-1'
        properties: {
          ipAddress: vpnRegion2.outputs.vpnGwPublicIps[0] == null ? '3.3.3.3' : vpnRegion2.outputs.vpnGwPublicIps[0]
        }
      }
      {
        name: 'link-2'
        properties: {
          ipAddress: vpnRegion2.outputs.vpnGwPublicIps[1] == null ? '4.4.4.4' :  vpnRegion2.outputs.vpnGwPublicIps[1]
        }
      }
    ]
  }
}

 

resource hubVpnConnectionRegion1 'Microsoft.Network/vpnGateways/vpnConnections@2020-05-01' = {
  name: 'vpngw-${region1}/HubToOnPremConnection'
  properties: {
    //enableBgp: false
    remoteVpnSite: {
      id: vpnSiteRegion2.id
    }
    vpnLinkConnections: [
      {
        name: 'link-1'
        properties: {
          vpnSiteLink: {
            id: resourceId('Microsoft.Network/vpnSites/vpnSiteLinks', 'vpnsite-${region2}', 'link-1')
          }
          //enableBgp: false
          sharedKey: 'test'
        }
      }
      {
        name: 'link-2'
        properties: {
          vpnSiteLink: {
            id: resourceId('Microsoft.Network/vpnSites/vpnSiteLinks', 'vpnsite-${region2}', 'link-2')
          }
          //enableBgp: false
          sharedKey: 'test'
        }
      }
    ]
  }
}

resource hubVpnConnectionRegion2 'Microsoft.Network/vpnGateways/vpnConnections@2020-05-01' = {
  name: 'vpngw-${region2}/HubToOnPremConnection'
  properties: {
    //enableBgp: false
    remoteVpnSite: {
      id: vpnSiteRegion1.id
    }
    vpnLinkConnections: [
      {
        name: 'link-1'
        properties: {
          vpnSiteLink: {
            id: resourceId('Microsoft.Network/vpnSites/vpnSiteLinks', 'vpnsite-${region1}', 'link-1')
          }
          //enableBgp: false
          sharedKey: 'test'
        }
      }
      {
        name: 'link-2'
        properties: {
          vpnSiteLink: {
            id: resourceId('Microsoft.Network/vpnSites/vpnSiteLinks', 'vpnsite-${region1}', 'link-2')
          }
          //enableBgp: false
          sharedKey: 'test'
        }
      }
    ]
  }
}

//this part will include the deployment of the vms needed fo the Bandwidth test.
module vms 'vnets-and-vms.bicep' if (deployVms) = {
  name: 'vmsvnets-${region1}-${region2}-${uniqueString(resourceGroup().id)}'
  region1: region1
  region2: region2
  prefix2Region1: prefix2Region1
  prefix2Region2: prefix2Region2
}

Module for the VPN Gateway

vpngw.bicep

//VPN Gateway Module
param vpnGwName string
param vpnGwLoc string
param vHubId string
resource vpngw 'Microsoft.Network/vpnGateways@2024-05-01' = {
  name: vpnGwName
  location: vpnGwLoc

  properties: {
    connections: []
    virtualHub: {
      id: vHubId
    }
    bgpSettings: {
      asn: 65515
      peerWeight: 0

    }
    vpnGatewayScaleUnit: 1
    
    natRules: []
    enableBgpRouteTranslationForNat: false
    isRoutingPreferenceInternet: false
  }
}
output vpnGwPublicIps array = [ 
  vpngw.properties.ipConfigurations[0].publicIpAddress
  vpngw.properties.ipConfigurations[1].publicIpAddress
]

Module for the Bandwidth Test VMs and Peering

vnets-and-vms.bicep

param region1 string = 'northeurope'
param region2 string = 'germanywestcentral'
param prefix2Region1 string = '192.168.1.0/24'
param prefix2Region2 string = '192.168.2.0/24'
param vmSize string = 'Standard_D16s_v5'
param adminUserName string = 'azureuser'
@secure()
param adminPassword string = 'Password1234!'


var imageReference = {
  'Ubuntu-2204': {
    publisher: 'Canonical'
    offer: '0001-com-ubuntu-server-jammy'
    sku: '22_04-lts-gen2'
    version: 'latest'
  }
}
//Import vHubs
resource vHubRegion1 'Microsoft.Network/virtualHubs@2024-05-01' existing = {
  name: 'vhub-${region1}'
}
resource vHubRegion2 'Microsoft.Network/virtualHubs@2024-05-01' existing = {
  name: 'vhub-${region2}'
}
// Deploy VNETs and vHub Connections

resource vNetRegion1 'Microsoft.Network/virtualNetworks@2024-05-01' = {
  name: 'vnet-${region1}'
  location: region1
  properties: {
    addressSpace: {
      addressPrefixes: [
        prefix2Region1
      ]
    }
     subnets: [
      {
        name: 'snet-${region1}'
        properties: {
          addressPrefix: prefix2Region1
        }
      }
    ]
  }
}

resource vNetRegion2 'Microsoft.Network/virtualNetworks@2024-05-01' = {
  name: 'vnet-${region2}'
  location: region2
  properties: {
    addressSpace: {
      addressPrefixes: [
        prefix2Region2
      ]
    }
     subnets: [
      {
        name: 'snet-${region2}'
        properties: {
          addressPrefix: prefix2Region2
        }
      }
    ]
  }
}

resource nicRegion1 'Microsoft.Network/networkInterfaces@2020-08-01' = {
  name: 'nic-vm-${region1}'
  location: region1
  properties:{
    enableIPForwarding: true
    ipConfigurations: [
      {
        name: 'ipv4config'
        properties:{
          primary: true
          privateIPAllocationMethod: 'Static'
          privateIPAddressVersion: 'IPv4'
          subnet: {
            id: vNetRegion1.id
          }
          privateIPAddress: '192.168.1.4'         
        }
      }
    ]
  }
}

resource vmRegion1 'Microsoft.Compute/virtualMachines@2020-12-01' = {
  name: 'vm-${region1}'
  location: region1
  zones: [ '1' ]
  properties: {
    hardwareProfile:{
      vmSize: vmSize
    }
    storageProfile: {
      osDisk: {
        createOption: 'FromImage'
        managedDisk: {
          storageAccountType: 'Standard_LRS'
        }
      }
      imageReference: imageReference['Ubuntu-2204']
    }
      osProfile:{
        computerName: 'vm-${region1}'
        adminUsername: adminUserName
        adminPassword: adminPassword
        linuxConfiguration: {
          patchSettings: {
            patchMode: 'ImageDefault'
          }
        }
      }
      diagnosticsProfile: {
        bootDiagnostics: {
          enabled: true
        }
     }
      networkProfile: {
        networkInterfaces: [
        {
          id: nicRegion1.id
        }
      ]
    }
  }
  }

resource nicRegion2 'Microsoft.Network/networkInterfaces@2020-08-01' = {
  name: 'vm-${region2}'
  location: region2
  properties:{
    enableIPForwarding: true
    ipConfigurations: [
      {
        name: 'ipv4config'
        properties:{
          primary: true
          privateIPAllocationMethod: 'Static'
          privateIPAddressVersion: 'IPv4'
          subnet: {
            id: vNetRegion2.id
          }
          
          privateIPAddress: '192.168.2.4'         
        }
      }
    ]
  }
}
resource vmRegion2 'Microsoft.Compute/virtualMachines@2020-12-01' = {
  name: 'vm-${region2}'
  location: region2
  zones: [ '1' ]
  properties: {
    hardwareProfile:{
      vmSize: vmSize
    }
    storageProfile: {
      osDisk: {
        createOption: 'FromImage'
        managedDisk: {
          storageAccountType: 'Standard_LRS'
        }
      }
      imageReference: imageReference['Ubuntu-2204']
    }
      osProfile:{
        computerName: 'vm-${region2}'
        adminUsername: adminUserName
        adminPassword: adminPassword
        linuxConfiguration: {
          patchSettings: {
            patchMode: 'ImageDefault'
          }
        }
      }
      diagnosticsProfile: {
        bootDiagnostics: {
          enabled: true
   //       storageUri: bootstUri
        }
     }
      networkProfile: {
        networkInterfaces: [
        {
          id: nicRegion2.id
        }
      ]
    }
  }
}

resource vWanHubToVnet 'Microsoft.Network/virtualHubs/hubVirtualNetworkConnections@2020-05-01' = {
  parent: vHubRegion1
  name: 'VMvnet-region1-to-hub-comm'
  properties: {
    routingConfiguration: {
      associatedRouteTable: {
        id: resourceId('Microsoft.Network/virtualHubs/hubRouteTables', vHubRegion1.name, 'defaultRouteTable')
      }
      propagatedRouteTables: {
        labels: [
          'default'
        ]
        ids: [
          {
            id: resourceId('Microsoft.Network/virtualHubs/hubRouteTables', vHubRegion1.name, 'defaultRouteTable')
          }
        ]
      }
    }
    remoteVirtualNetwork: {
      id: vNetRegion1.id
  }
}
}

resource vWanHubToVnet2 'Microsoft.Network/virtualHubs/hubVirtualNetworkConnections@2020-05-01' = {
  parent: vHubRegion2
  name: 'VMvnet-region2-to-hub-comm'
  properties: {
    routingConfiguration: {
      associatedRouteTable: {
        id: resourceId('Microsoft.Network/virtualHubs/hubRouteTables', vHubRegion2.name, 'defaultRouteTable')
      }
      propagatedRouteTables: {
        labels: [
          'default'
        ]
        ids: [
          {
            id: resourceId('Microsoft.Network/virtualHubs/hubRouteTables', vHubRegion2.name, 'defaultRouteTable')
          }
        ]
      }
    }
    remoteVirtualNetwork: {
      id: vNetRegion2.id
  }
}
}

Note: From Azure side, as long as same address ranges are advertised from both of the tunnels, the traffic distribution uses ECMP and each TCP or UDP flow will follow the same tunnel or path.

Important! You must also configure ECMP (L3/4 Loadsharing) for your on premises devices. This is important otherwise the connection is disturbed and will show strange phenomena.

Bandwidth Test in the Lab

I test the throughput with the help of one Virtual Machine in each Region. As a testing tool I use the free NTTTCP by Microsoft. iPerf3 is as well suitable for these bandwidth tests.

https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-bandwidth-testing?tabs=windows

To test throughput one VM is configured as the reciever (vm-northeurope) and the other one as the sender (vm-germanywestcentral).

For demonstration purposes I limit the duration to 60 seconds to save costs (Traffic!). The VM Type shall be an Standard_D16s_v5 in order to be able to handle the load. Login to the vms with the credentials azureuser/Password1234! by using the ‘serial connection’

Note: It is important that the VM size is sufficient to support the network traffic and load tool. Otherwise the bandwidth tests will not show the maximum throughput possible.

Install NTTTCP on both VMs:

# For Ubuntu, install build-essential and git.

sudo apt-get update
sudo apt-get -y install build-essential
sudo apt-get -y install git

# Make and install NTTTCP-for-Linux.

git clone https://github.com/Microsoft/ntttcp-for-linux
cd ntttcp-for-linux/src
sudo make && sudo make install

Finally set one machine up as the sender and the other as the reciever

VM-Northeurope
ntttcp -r -m 20,*,192.168.1.4 -t 60

VM-Germanywestcentral
ntttcp -s -m 20,*,192.168.1.4 -t 60

The Screenshot of the results shows that there is around 4.7 GBps of bandwidth with parallel 20 streams.

Northeurope

Germanywestcentral

Conclusion

By combining multiple vWan vpnSite Links you can dynamically expand site-to-site bandwidth up to ~ >>9 GBps.

Take into Consideration that the load sharing is set to Equal Cost Multi Pathing on your on-prem device.

Important! Take into consideration that one TCP Stream will not be able to exceed ~ 1,25 GBps. Multiple senders will be able to utilise the full bandwidth potential of your aggregated interface.

Note Take into consideration that you will need an apropriate amount of public IP adresses on your remote site matching the amount of links you want to configure.

Also if you need that much bandwidth it might be feasible to opt for an Express Route. The provisioning of multiple VPN Links and scaling the VPN Gateways to meet the bandwidth requirements can quickly reach high costs. Also the remote site needs to be provisioned hardware- & and admin expertise wise.

As a temporary solution for mid bandwidth requirements this might be an apropriate solution though.

Stability wise it doesnt have any downsides compared to a solution with a single ipsec tunnel from azure to onprem (According to my expierience with vWan and Fortigate).