Tag Archives: Networking

Azure Networking, VPN, Express Route, DNS Services, Traffic Manager & Application Gateway.

URL Routing with Azure Application Gateway

I have a scenario perfect for a Layer-7 Load Balancer / Reverse Proxy:

  • Multiple web server clusters to be routed under one URL hierarchy (one domain name)
  • Redirect HTTP traffic to the same URL on HTTPS
  • Have reverse proxy performing SSL termination (or SSL offloading), i.e. accepting HTTPS but routing to underlying servers using HTTP

On paper, Azure Application Gateway can do all of those.  Let’s fine out in practice.

Azure Application Gateway Concepts

From the documentation:

Application Gateway is a layer-7 load balancer.  It provides failover, performance-routing HTTP requests between different servers, whether they are on the cloud or on-premises. Application Gateway provides many Application Delivery Controller (ADC) features including HTTP load balancing, cookie-based session affinity, Secure Sockets Layer (SSL) offload, custom health probes, support for multi-site, and many others.

Before we get into the meat of it, there are a bunch of concepts Application Gateway uses and we need to understand:

  • Back-end server pool: The list of IP addresses of the back-end servers. The IP addresses listed should either belong to the virtual network subnet or should be a public IP/VIP.
  • Back-end server pool settings: Every pool has settings like port, protocol, and cookie-based affinity. These settings are tied to a pool and are applied to all servers within the pool.
  • Front-end port: This port is the public port that is opened on the application gateway. Traffic hits this port, and then gets redirected to one of the back-end servers.
  • Listener: The listener has a front-end port, a protocol (Http or Https, these values are case-sensitive), and the SSL certificate name (if configuring SSL offload).
  • Rule: The rule binds the listener, the back-end server pool and defines which back-end server pool the traffic should be directed to when it hits a particular listener.

On top of those, we should probably add probes that are associated to a back-end pool to determine its health.

Proof of Concept

As a proof of concept, we’re going to implement the following:

image

We use Windows Virtual Machine Scale Sets (VMSS) for back-end servers.

In a production setup, we would go for exposing the port 443 on the web, but for a POC, this should be sufficient.

As of this writing, there are no feature to allow automatic redirection from port 80 to port 443.  Usually, for public web site, we want to redirect users to HTTPS.  This could be achieve by having one of the VM scale set implementing the redirection and routing HTTP traffic to it.

ARM Template

We’ve published the ARM template on GitHub.

First, let’s look at the visualization.

image

The template is split within 4 files:

  • azuredeploy.json, the master ARM template.  It simply references the others and passes parameters around.
  • network.json, responsible for the virtual network and Network Security Groups
  • app-gateway.json, responsible for the Azure Application Gateway and its public IP
  • vmss.json, responsible for VM scale set, a public IP and a public load balancer ; this template is invoked 3 times with 3 different set of parameters to create the 3 VM scale sets

We’ve configured the VMSS to have public IPs.  It is quite typical to want to connect directly to a back-end servers while testing.  We also optionally open the VMSS to RDP traffic ; this is controlled by the ARM template’s parameter RDP Rule (Allow, Deny).

Template parameters

Here are the following ARM template parameters.

Parameter Description
Public DNS Prefix The DNS suffix for each VMSS public IP.
They are then suffixed by ‘a’, ‘b’ & ‘c’.
RDP Rule Switch allowing or not allowing RDP network traffic to reach VMSS from public IPs.
Cookie Based Affinity Switch enabling / disabling cookie based affinity on the Application Gateway.
VNET Name Name of the Virtual Network (default to VNet).
VNET IP Prefix Prefix of the IP range for the VNET (default to 10.0.0).
VM Admin Name Local user account for administrator on all the VMs in all VMSS (default to vmssadmin).
VM Admin Password Password for the VM Admin (same for all VMs of all VMSS).
Instance Count Number of VMs in each VMSS.
VM Size SKU of the VMs for the VMSS (default to Standard DS2-v2).

Routing

An important characteristic of URL-based routing is that requests are routed to back-end servers without alteration.

This is important.  It means that /a/ on the Application Gateway is mapped to /a/ on the Web Server.  It isn’t mapped to /, which seems more intuitive as that would seem like the root of the ‘a’ web servers.  This is because URL-base routing can be more general than just defining suffix.

Summary

This proof of concept gives a fully functional example of Azure Application Gateway using URL-based routing.

This is a great showcase for Application Gateway as it can then reverse proxy all traffic while keeping user affinity using cookies.

Joining an ARM Linux VM to AAD Domain Services

Active Directory is one of the most popular domain controller / LDAP server around.

In Azure we have Azure Active Directory (AAD).  Despite the name, AAD isn’t just a multi-tenant AD.  It is built for the cloud.

Sometimes though, it is useful to have a traditional domain controller…  in the cloud.  Typically this is with legacy workloads built to work with Active Directory.  But also, a very common scenario is to join an Azure VM to a domain so that users authenticate on it with the same accounts they authenticate to use the Azure Portal.

The underlying directory could even be synced with a corporate network, in which case users could log into the VMs using their corporate account.  I won’t cover this here but you can read about it in a previous article for AD Connect part.

The straightforward option is to build an Active Directory cluster on Azure VMs.  This will work but requires the maintenance of those 2 VMs.

mont-saint-michel-france-normandy-europe[1]

An easier option is AAD Domain Services (AADDS).  AADDS exposes an AAD tenant as a managed domain service.  It does it by provisioning some variant of Active Directory managed cluster, i.e. we do not see or care about the underlying VMs.

The cluster is synchronized one-way (from AAD to AADDS).  For this reason, AAD is read only through its LDAP interface, e.g. we can’t reset a password using LDAP.

The Azure documentation walks us through such an integration with classic (ASM) VMs.  Since ARM has been around for more than a year, I recommend to always go with ARM VMs.  This article aims at showing how to do this.

I’ll heavily leveraged the existing documentation and detail only what differs from the official documentation.

Also, keep in mind this article is written in January 2017.  Azure AD will transition to ARM in the future and will likely render this article obsolete.

Dual Networks

The main challenge we face is that AAD is an ASM service and AAD Domain Service are exposed within an ASM Virtual Network (VNET), which is incompatible with our ARM VM.

Thankfully, we now have Virtual Network peering allowing us to peer an ASM and an ARM VNET together so they can act as if they were one.

image

Peering

As with all VNET peering, the two VNETs must be of mutually exclusive IP addresses space.

I created two VNETs in the portal (https://portal.azure.com).  I recommend creating them in the new portal explicitly, this way even the classic one will be part of the desired resource group.

The classic one has 10.0.0.0/24 address range while the ARM one has 10.1.0.0/24.

The peering can be done from the portal too.  In the Virtual Network pane (say the ARM one), select Peerings:

image

We should see an empty list here, so let’s click Add.

We need to give the peering a name.  Let’s type PeeringToDomainServices.

We then select Classic in the Peer details since we want to peer with a classic VNET.

Finally, we’ll click on Choose a Virtual Network.

image

From there we should see the classic VNET we created.

Configuring AADDS

The online documentation is quite good for this.

Just make sure you select the classic VNET we created.

You can give a domain name that is different that the AAD domain name (i.e. *.onmicrosoft.com).

Enabling AADDS takes up to 30 minutes.  Don’t hold your breath!

Joining a VM

We can create a Linux VM, put it in the ARM VNET we created, and join it to the AADDS domain now.

Again, the online documentation does a great job of walking us through the process.  The documentation is written for Red Hat.

When I tried it, I used a CentOS VM and I ended up using different commands, namely the realmd command (ignoring the SAMBA part of the article).

Conclusion

It is fairly straightforward to enable Domain Services in AAD and well documented.

A challenge we currently have currently is to join or simply communicate from an ARM VM to AADDS.  For this we need two networks, a classic (ASM) one and an ARM one, and we need to peer them together.

Troubleshooting NSGs using Diagnostic Logs

I’ve wrote about how to use Network Security Group (NSG) before.

Chances are, once you get a complicated enough set of rules in a NSG, you’ll find yourself with NSGs that do not do what you think they should do.

Troubleshooting NSGs isn’t trivial.

I’ll try to give some guidance here but to this date (January 2017), there is no tool where you can just say “please follow packet X and tell me against which wall it bumps”.  It’s more indirect than that.

 

Connectivity

First thing, make sure you can connect to your VNET.

If you are connecting to a VM via a public IP, make sure you have access to that IP (i.e. you’re not sitting behind an on premise firewall blocking the outgoing port you are trying to use), that the IP is connected to the VM either directly or via a Load Balancer.

If you are connecting to a VM via a private IP through a VPN Gateway of some sort, make sure you can connect and that your packets are routed to the gateway and from there they get routed to the proper subnet.

An easy way to make sure of that is to remove all NSGs and replace them by a “let everything go in”.  Of course, that’s also opening your workloads to hackers, so I recommend you do that with a test VM that you destroy afterwards.

Diagnose

Then I would recommend to go through the official Azure guidelines to troubleshoot NSGs.  This walks you through the different diagnosis tools.

Diagnostic Logs

If you reached this section and haven’t achieve greatness yet, well…  You need something else.

What we’re going to do here is use NSG Diagnostic Logs to understand a bit more what is going on.

By no means is this magic and especially in an environment already in use where a lot of traffic is occurring, it might be difficult to make sense of what the logs are going to give us.

Nevertheless, the logs give us a picture of what really is happening.  They are aggregated though, so we won’t see your PC IP address for instance.  The aggregation is probably what limit the logs effectiveness the most.

Sample configuration

I provide here a sample configuration I’m going to use to walk through the troubleshooting process.

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "VM Admin User Name": {
      "defaultValue": "myadmin",
      "type": "string"
    },
    "VM Admin Password": {
      "defaultValue": null,
      "type": "securestring"
    },
    "Disk Storage Account Name": {
      "defaultValue": "<your prefix>vmpremium",
      "type": "string"
    },
    "Log Storage Account Name": {
      "defaultValue": "<your prefix>logstandard",
      "type": "string"
    },
    "VM Size": {
      "defaultValue": "Standard_DS2",
      "type": "string",
      "allowedValues": [
        "Standard_DS1",
        "Standard_DS2",
        "Standard_DS3"
      ],
      "metadata": {
        "description": "SKU of the VM."
      }
    },
    "Public Domain Label": {
      "type": "string"
    }
  },
  "variables": {
    "Vhds Container Name": "vhds",
    "VNet Name": "MyVNet",
    "Ip Range": "10.0.1.0/24",
    "Public IP Name": "MyPublicIP",
    "Public LB Name": "PublicLB",
    "Address Pool Name": "addressPool",
    "Subnet NSG Name": "subnetNSG",
    "VM NSG Name": "vmNSG",
    "RDP NAT Rule Name": "RDP",
    "NIC Name": "MyNic",
    "VM Name": "MyVM"
  },
  "resources": [
    {
      "type": "Microsoft.Network/publicIPAddresses",
      "name": "[variables('Public IP Name')]",
      "apiVersion": "2015-06-15",
      "location": "[resourceGroup().location]",
      "tags": {
        "displayName": "Public IP"
      },
      "properties": {
        "publicIPAllocationMethod": "Dynamic",
        "idleTimeoutInMinutes": 4,
        "dnsSettings": {
          "domainNameLabel": "[parameters('Public Domain Label')]"
        }
      }
    },
    {
      "type": "Microsoft.Network/loadBalancers",
      "name": "[variables('Public LB Name')]",
      "apiVersion": "2015-06-15",
      "location": "[resourceGroup().location]",
      "tags": {
        "displayName": "Public Load Balancer"
      },
      "properties": {
        "frontendIPConfigurations": [
          {
            "name": "LoadBalancerFrontEnd",
            "comments": "Front end of LB:  the IP address",
            "properties": {
              "publicIPAddress": {
                "id": "[resourceId('Microsoft.Network/publicIPAddresses/', variables('Public IP Name'))]"
              }
            }
          }
        ],
        "backendAddressPools": [
          {
            "name": "[variables('Address Pool Name')]"
          }
        ],
        "loadBalancingRules": [
          {
            "name": "Http",
            "properties": {
              "frontendIPConfiguration": {
                "id": "[concat(resourceId('Microsoft.Network/loadBalancers', variables('Public LB Name')), '/frontendIPConfigurations/LoadBalancerFrontEnd')]"
              },
              "frontendPort": 80,
              "backendPort": 80,
              "enableFloatingIP": false,
              "idleTimeoutInMinutes": 4,
              "protocol": "Tcp",
              "loadDistribution": "Default",
              "backendAddressPool": {
                "id": "[concat(resourceId('Microsoft.Network/loadBalancers', variables('Public LB Name')), '/backendAddressPools/', variables('Address Pool Name'))]"
              },
              "probe": {
                "id": "[concat(resourceId('Microsoft.Network/loadBalancers', variables('Public LB Name')), '/probes/TCP-Probe')]"
              }
            }
          }
        ],
        "probes": [
          {
            "name": "TCP-Probe",
            "properties": {
              "protocol": "Tcp",
              "port": 80,
              "intervalInSeconds": 5,
              "numberOfProbes": 2
            }
          }
        ],
        "inboundNatRules": [
          {
            "name": "[variables('RDP NAT Rule Name')]",
            "properties": {
              "frontendIPConfiguration": {
                "id": "[concat(resourceId('Microsoft.Network/loadBalancers', variables('Public LB Name')), '/frontendIPConfigurations/LoadBalancerFrontEnd')]"
              },
              "frontendPort": 3389,
              "backendPort": 3389,
              "protocol": "Tcp"
            }
          }
        ],
        "outboundNatRules": [],
        "inboundNatPools": []
      },
      "dependsOn": [
        "[resourceId('Microsoft.Network/publicIPAddresses', variables('Public IP Name'))]"
      ]
    },
    {
      "type": "Microsoft.Network/virtualNetworks",
      "name": "[variables('VNet Name')]",
      "apiVersion": "2016-03-30",
      "location": "[resourceGroup().location]",
      "properties": {
        "addressSpace": {
          "addressPrefixes": [
            "10.0.0.0/16"
          ]
        },
        "subnets": [
          {
            "name": "default",
            "properties": {
              "addressPrefix": "[variables('Ip Range')]",
              "networkSecurityGroup": {
                "id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('Subnet NSG Name'))]"
              }
            }
          }
        ]
      },
      "resources": [],
      "dependsOn": [
        "[resourceId('Microsoft.Network/networkSecurityGroups', variables('Subnet NSG Name'))]"
      ]
    },
    {
      "apiVersion": "2015-06-15",
      "name": "[variables('Subnet NSG Name')]",
      "type": "Microsoft.Network/networkSecurityGroups",
      "location": "[resourceGroup().location]",
      "tags": {},
      "properties": {
        "securityRules": [
          {
            "name": "Allow-HTTP-From-Internet",
            "properties": {
              "protocol": "Tcp",
              "sourcePortRange": "*",
              "destinationPortRange": "80",
              "sourceAddressPrefix": "Internet",
              "destinationAddressPrefix": "*",
              "access": "Allow",
              "priority": 100,
              "direction": "Inbound"
            }
          },
          {
            "name": "Allow-RDP-From-Everywhere",
            "properties": {
              "protocol": "Tcp",
              "sourcePortRange": "*",
              "destinationPortRange": "3389",
              "sourceAddressPrefix": "*",
              "destinationAddressPrefix": "*",
              "access": "Allow",
              "priority": 150,
              "direction": "Inbound"
            }
          },
          {
            "name": "Allow-Health-Monitoring",
            "properties": {
              "protocol": "*",
              "sourcePortRange": "*",
              "destinationPortRange": "*",
              "sourceAddressPrefix": "AzureLoadBalancer",
              "destinationAddressPrefix": "*",
              "access": "Allow",
              "priority": 200,
              "direction": "Inbound"
            }
          },
          {
            "name": "Disallow-everything-else-Inbound",
            "properties": {
              "protocol": "*",
              "sourcePortRange": "*",
              "destinationPortRange": "*",
              "sourceAddressPrefix": "*",
              "destinationAddressPrefix": "*",
              "access": "Deny",
              "priority": 300,
              "direction": "Inbound"
            }
          },
          {
            "name": "Allow-to-VNet",
            "properties": {
              "protocol": "*",
              "sourcePortRange": "*",
              "destinationPortRange": "*",
              "sourceAddressPrefix": "*",
              "destinationAddressPrefix": "VirtualNetwork",
              "access": "Allow",
              "priority": 100,
              "direction": "Outbound"
            }
          },
          {
            "name": "Disallow-everything-else-Outbound",
            "properties": {
              "protocol": "*",
              "sourcePortRange": "*",
              "destinationPortRange": "*",
              "sourceAddressPrefix": "*",
              "destinationAddressPrefix": "*",
              "access": "Deny",
              "priority": 200,
              "direction": "Outbound"
            }
          }
        ],
        "subnets": []
      }
    },
    {
      "apiVersion": "2015-06-15",
      "name": "[variables('VM NSG Name')]",
      "type": "Microsoft.Network/networkSecurityGroups",
      "location": "[resourceGroup().location]",
      "tags": {},
      "properties": {
        "securityRules": [
          {
            "name": "Allow-HTTP-From-Internet",
            "properties": {
              "protocol": "Tcp",
              "sourcePortRange": "*",
              "destinationPortRange": "80",
              "sourceAddressPrefix": "Internet",
              "destinationAddressPrefix": "*",
              "access": "Allow",
              "priority": 100,
              "direction": "Inbound"
            }
          },
          {
            "name": "Allow-Health-Monitoring",
            "properties": {
              "protocol": "*",
              "sourcePortRange": "*",
              "destinationPortRange": "*",
              "sourceAddressPrefix": "AzureLoadBalancer",
              "destinationAddressPrefix": "*",
              "access": "Allow",
              "priority": 200,
              "direction": "Inbound"
            }
          },
          {
            "name": "Disallow-everything-else-Inbound",
            "properties": {
              "protocol": "*",
              "sourcePortRange": "*",
              "destinationPortRange": "*",
              "sourceAddressPrefix": "*",
              "destinationAddressPrefix": "*",
              "access": "Deny",
              "priority": 300,
              "direction": "Inbound"
            }
          },
          {
            "name": "Allow-to-VNet",
            "properties": {
              "protocol": "*",
              "sourcePortRange": "*",
              "destinationPortRange": "*",
              "sourceAddressPrefix": "*",
              "destinationAddressPrefix": "VirtualNetwork",
              "access": "Allow",
              "priority": 100,
              "direction": "Outbound"
            }
          },
          {
            "name": "Disallow-everything-else-Outbound",
            "properties": {
              "protocol": "*",
              "sourcePortRange": "*",
              "destinationPortRange": "*",
              "sourceAddressPrefix": "*",
              "destinationAddressPrefix": "*",
              "access": "Deny",
              "priority": 200,
              "direction": "Outbound"
            }
          }
        ],
        "subnets": []
      }
    },
    {
      "type": "Microsoft.Network/networkInterfaces",
      "name": "[variables('NIC Name')]",
      "apiVersion": "2016-03-30",
      "location": "[resourceGroup().location]",
      "properties": {
        "ipConfigurations": [
          {
            "name": "ipconfig",
            "properties": {
              "privateIPAllocationMethod": "Dynamic",
              "subnet": {
                "id": "[concat(resourceId('Microsoft.Network/virtualNetworks', variables('VNet Name')), '/subnets/default')]"
              },
              "loadBalancerBackendAddressPools": [
                {
                  "id": "[concat(resourceId('Microsoft.Network/loadBalancers', variables('Public LB Name')), '/backendAddressPools/', variables('Address Pool Name'))]"
                }
              ],
              "loadBalancerInboundNatRules": [
                {
                  "id": "[concat(resourceId('Microsoft.Network/loadBalancers', variables('Public LB Name')), '/inboundNatRules/', variables('RDP NAT Rule Name'))]"
                }
              ]
            }
          }
        ],
        "dnsSettings": {
          "dnsServers": []
        },
        "enableIPForwarding": false,
        "networkSecurityGroup": {
          "id": "[resourceId('Microsoft.Network/networkSecurityGroups', variables('VM NSG Name'))]"
        }
      },
      "resources": [],
      "dependsOn": [
        "[resourceId('Microsoft.Network/virtualNetworks', variables('VNet Name'))]",
        "[resourceId('Microsoft.Network/loadBalancers', variables('Public LB Name'))]"
      ]
    },
    {
      "type": "Microsoft.Compute/virtualMachines",
      "name": "[variables('VM Name')]",
      "apiVersion": "2015-06-15",
      "location": "[resourceGroup().location]",
      "properties": {
        "hardwareProfile": {
          "vmSize": "[parameters('VM Size')]"
        },
        "storageProfile": {
          "imageReference": {
            "publisher": "MicrosoftWindowsServer",
            "offer": "WindowsServer",
            "sku": "2012-R2-Datacenter",
            "version": "latest"
          },
          "osDisk": {
            "name": "[variables('VM Name')]",
            "createOption": "FromImage",
            "vhd": {
              "uri": "[concat('https', '://', parameters('Disk Storage Account Name'), '.blob.core.windows.net', concat('/', variables('Vhds Container Name'),'/', variables('VM Name'), '-os-disk.vhd'))]"
            },
            "caching": "ReadWrite"
          },
          "dataDisks": []
        },
        "osProfile": {
          "computerName": "[variables('VM Name')]",
          "adminUsername": "[parameters('VM Admin User Name')]",
          "windowsConfiguration": {
            "provisionVMAgent": true,
            "enableAutomaticUpdates": true
          },
          "secrets": [],
          "adminPassword": "[parameters('VM Admin Password')]"
        },
        "networkProfile": {
          "networkInterfaces": [
            {
              "id": "[resourceId('Microsoft.Network/networkInterfaces', concat(variables('NIC Name')))]"
            }
          ]
        }
      },
      "resources": [],
      "dependsOn": [
        "[resourceId('Microsoft.Storage/storageAccounts', parameters('Disk Storage Account Name'))]",
        "[resourceId('Microsoft.Network/networkInterfaces', variables('NIC Name'))]"
      ]
    },
    {
      "type": "Microsoft.Storage/storageAccounts",
      "name": "[parameters('Disk Storage Account Name')]",
      "sku": {
        "name": "Premium_LRS",
        "tier": "Premium"
      },
      "kind": "Storage",
      "apiVersion": "2016-01-01",
      "location": "[resourceGroup().location]",
      "properties": {},
      "resources": [],
      "dependsOn": []
    },
    {
      "type": "Microsoft.Storage/storageAccounts",
      "name": "[parameters('Log Storage Account Name')]",
      "sku": {
        "name": "Standard_LRS",
        "tier": "standard"
      },
      "kind": "Storage",
      "apiVersion": "2016-01-01",
      "location": "[resourceGroup().location]",
      "properties": {},
      "resources": [],
      "dependsOn": []
    }
  ]
}

The sample has one VM sitting in a subnet protected by a NSG.  The VM’s NIC is also protected by NSG, to make our life complicated (as we do too often).  The VM is exposed on a Load Balanced Public IP and RDP is enabled via NAT rules on the Load Balancer.

The VM is running on a Premium Storage account but the sample also creates a standard storage account to store the logs.

The Problem

The problem we are going to try to find using Diagnostic Logs is that the subnet’s NSG let RDP in via “Allow-RDP-From-Everywhere” rule while the NIC’s doesn’t and that type of traffic will get blocked, as everything else, by the “Disallow-everything-else-Inbound” rule.

In practice, you’ll likely have something more complicated going on, maybe some IP filtering, etc.  .   But the principles remain.

Enabling Diagnostic Logs

I couldn’t enable the Diagnostic Logs via the ARM template as it isn’t possible to do so yet.  We can do that via the Portal or PowerShell.

I’ll illustrate the Portal here, since it’s for troubleshooting, chances are you won’t automate it.

I’ve covered Azure Monitor in a previous article.  We’ve seen that different providers expose different schemas.

NSGs expose two categories of Diagnostic LogsEvent and Rule Counter.  We’re going to use Rule Counter only.

Rule Counter will give us a count of how many times a given rule was triggered for a given target (MAC address / IP).  Again, if we have lots of traffic flying around, that won’t be super useful.  This is why I recommend to isolate the network (or recreate an isolated one) in order to troubleshoot.

We’ll start by the subnet NSG.

image

Scrolling all the way down on the NSG’s pane left menu, we select Diagnostics Logs.

image

The pane should look as follow since no diagnostics are enabled.  Let’s click on Turn on diagnostics.

image

We then turn it on.

image

For simplicity here, we’re going to use the Archive to a storage account.

image

We will configure the storage account to send the logs to.

image

For that, we’re selecting the standard account created by the template or whichever storage account you fancy.  Log Diagnostics will go and create a blob container for each category in the selected account.  The names a predefined (you can’t choose).

We select the NetworkSecurityGroupRuleCounter category.

image

And finally we hit the save button on the pane.

We’ll do the same thing with the VM NSG.

image

Creating logs

No we are going to try to get through our VM.  We are going to describe how to that with the sample I gave but if you are troubleshooting something, just try the faulty connection.

We’re going to try to RDP to the public IP.  First we need the public IP domain name.  So in the resource group:

image

At the top of the pane we’ll find the DNS name that we can copy.

image

We can then paste it in an RDP window.

image

Trying to connect should fail and it should leave traces in the logs for us to analyse.

Analysis

We’ll have to wait 5-10 minutes for the logs to get in the storage account as this is done asynchronously.

Actually, a way to make sure to get clean logs is to delete the blob container and then try the RDP connection.  The blob container should reappear after 5-10 minutes.

To get the logs in the storage account we need some tool.  I use Microsoft Azure Storage Explorer.

image

The blob container is called insights-logs-networksecuritygrouprulecounter.

The logs are hidden inside a complicated hierarchy allowing us to send all our diagnostic logs from all our NSGs over time there.

Basically, resourceId%3D / SUBSCRIPTIONS / <Your subscription ID> / RESOURCEGROUPS / NSG / PROVIDERS / MICROSOFT.NETWORK / NETWORKSECURITYGROUPS / we’ll see two folders:  SUBNETNSG & VMNSG.  Those are our two NSGs.

If we dig under those two folders, we should find one file (or more if you’ve waited for a while).

Let’s copy those file with appropriate naming somewhere to analyse them.

Preferably, use a viewer / editor that understands JSON (I use Visual Studio).  If you use notepad…  you’re going to have fun.

If we look at the subnet NSG logs first and search for “RDP”, we’ll find this entry:

    {
      "time": "2017-01-09T11:46:44.9090000Z",
      "systemId": "...",
      "category": "NetworkSecurityGroupRuleCounter",
      "resourceId": ".../RESOURCEGROUPS/NSG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/SUBNETNSG",
      "operationName": "NetworkSecurityGroupCounters",
      "properties": {
        "vnetResourceGuid": "{50C7B76A-4B8F-481A-8029-73569E5C7D87}",
        "subnetPrefix": "10.0.1.0/24",
        "macAddress": "00-0D-3A-00-B6-B5",
        "primaryIPv4Address": "10.0.1.4",
        "ruleName": "UserRule_Allow-RDP-From-Everywhere",
        "direction": "In",
        "type": "allow",
        "matchedConnections": 0
      }
    },

The most interesting part is the matchedConnections property, which is zero because we didn’t achieve connections.

If we look in the VM logs, we’ll find this:

    {
      "time": "2017-01-09T11:46:44.9110000Z",
      "systemId": "...",
      "category": "NetworkSecurityGroupRuleCounter",
      "resourceId": ".../RESOURCEGROUPS/NSG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/VMNSG",
      "operationName": "NetworkSecurityGroupCounters",
      "properties": {
        "vnetResourceGuid": "{50C7B76A-4B8F-481A-8029-73569E5C7D87}",
        "subnetPrefix": "10.0.1.0/24",
        "macAddress": "00-0D-3A-00-B6-B5",
        "primaryIPv4Address": "10.0.1.4",
        "ruleName": "UserRule_Disallow-everything-else-Inbound",
        "direction": "In",
        "type": "block",
        "matchedConnections": 2
      }
    },

Where matchedConnections is 2 (because I tried twice).

So the logs tell us where the traffic when.

From here we could wonder why it hit that rule and look for a rule with a higher priority that allow RDP in, find none and conclude that’s our problem.

Trial & Error

If the logs are not helping you, the last resort is to modify the NSG until you understand what is going on.

A way to do this is to create a rule “allow everything in from anywhere”, give it maximum priority.

If traffic still doesn’t go in, you have another problem than NSG, so go back to previous steps.

If traffic goes in, good.  Move that allow-everything rule down until you find which rule is blocking you.  You may have a lot of rules, in which case I would recommend a dichotomic search algorithm:  put your allow-everything rule in the middle of your “real rules”, if traffic passes, move the rule to the middle of the bottom half, otherwise, the middle of the top half, and so on.  This way, you’ll only need log(N) steps where N is your number of rules.

Summary

Troubleshooting NSGs can be difficult but here I highlighted a basic methodology to find your way around.

Diagnostic Logs help to give us insight about what is really going on although it can be tricky to work with.

In general, as with every debugging experience just apply the principle of Sherlock Holmes:

Eliminate the impossible.  Whatever remains, however improbable, must be the truth.

In terms of debugging, that means remove all the noise, all the fat and then some meat, until what remains is so simply that the truth will hit you.

Virtual Machine with 2 NICs

Colorful Ethernet CableIn Azure Resource Manager (ARM), Network Interface Cards (NICs) are a first class resource.  You can defined them without a Virtual Machine.

UPDATE:  As a reader kingly point out, NIC means Network Interface Controller, not Network Interface Card as I initially wrote.  Don’t be fooled by the Azure logo 😉 

Let’s take a step back and look at how the different Azure Lego blocks snap together to get you a VM exposed on the web.  ARM did decouple a lot of infrastructure components, so each of those are much simpler (compare to the old ASM Cloud Service), but there are many of them.

Related Resources

Here’s a diagram that can help:

image

Let’s look at the different components:

  • Availability Set:  contains a set of (or only one) VMs ; see Azure basics: Availability sets for details
  • Storage Account:  VM hard drives are page blobs located in one or many storage accounts
  • NIC:  A VM has one or many NICs
  • Virtual Network:  a NIC is part of a subnet, where it gets its private IP address
  • Load Balancer:  a load balancer exposed the port of a NIC (or a pool of NICs) through a public IP address

The important point for us here:  the NIC is the one being part of a subnet, not the VM.  That means a VM can have multiple NICs in different subnets.

Also, something not shown on the diagram above, a Network Security Group (NSG) can be associated with each NIC of a VM.

One VM, many NICs

Not all VMs can have multiple NICs.  For instance, in the standard A series, the following SKUs can have only one NIC:  A0, A1, A2 & A5.

You can take a look at https://azure.microsoft.com/en-us/documentation/articles/virtual-machines-windows-sizes/ to see how many NICs a given SKU support.

Why would you want to have multiple NICs?

Typically, this is a requirement for Network Appliances and for VMs passing traffic from one subnet to another.

Having multiple NICs enables more control, such as better traffic isolation.

Another requirement I’ve seen, typically with customer with high security requirements, is to isolate management traffic and transactional traffic.

For instance, let’s say you have a SQL VM with its port 1443 open to another VM (web server).  That VM needs to open its RDP port for maintenance (i.e. sys admin people to log in and do maintenance).  But if both port are opened on the same NIC, then a sys admin having RDP access could also have access to the port 1443.  For some customer, that’s unacceptable.

So the way around that is to have 2 NICs.  One NIC will be used for port 1443 (SQL) and the other for RDP (maintenance).  Then you can put each NIC in different subnet.  The SQL-NIC will be in a subnet with NSG allowing the web server to access it while the RDP-NIC will be in a subnet accessible only from the VPN Gateway, by maintenance people.

Example

You will find here an ARM template (embedded in a Word document due to limitation of the Blog platform I’m using) deploying 2 VMs, each having 2 NICs, a Web NIC & a maintenance NIC.  The Web NICs are in the web subnet and are publically load balanced through a public IP while the maintenance NICs are in a maintenance subnet and accessible only via private IPs.  The maintenance subnet let RDP get in, via its NSG.

The template will take a little while to deploy, thanks to the fact it contains a VM.  You can see most of the resources deployed quite fast though.

If you’ve done VMs with ARM before, it is pretty much the same thing, except with two NICs references in the VM.  The only thing to be watchful for is that you have to specify which NIC is primary.  You do this with the primary property:


"networkProfile": {
  "networkInterfaces": [
    {
      "id": "[resourceId('Microsoft.Network/networkInterfaces', concat(variables('Web NIC Prefix'), '-', copyIndex()))]",
      "properties": {
        "primary": true
      }
    },
    {
      "id": "[resourceId('Microsoft.Network/networkInterfaces', concat(variables('Maintenance NIC Prefix'), '-', copyIndex()))]",
      "properties": {
        "primary": false
      }
    }
  ]
}

If you want to push the example and test it with a VPN gateway, consult https://azure.microsoft.com/en-us/documentation/articles/vpn-gateway-howto-point-to-site-rm-ps/ to do a point-to-site connection with your PC.

Conclusion

Somewhat a special case for VMs, a VM with 2 NICs allow you to understand a lot of design choices in ARM.  For instance, why the NICs are stand-alone resource, why they are the one to be part of a subnet and why NSG are associated to them (not the VM).

To learn more, see https://azure.microsoft.com/en-us/documentation/articles/virtual-networks-multiple-nics/

Load Balancing VMs in Azure Resource Manager

Here I want to show, in details, how you would go about to expose load balanced web server VMs using Azure Resource Manager (ARM) resources.

It sounds trivial but funnily enough I didn’t find an ARM template fully doing it without bugs.

I want to explain how it works and all the moving pieces (and there are a few).  I’ll walk you through the portal in order to create the artefacts and I’ll give the ARM template at the end.

Here’s a diagram of the final state:

image

Basically, we’ll have 2 VMs (which of course you could extend to n VMs) in a Virtual Network with their VHD in a blob storage, those 2 VMs will have HTTP requests load balanced to them from a public IP address and will have RDP NAT to individual VMs (i.e. port 3389 will go to VM-0, port 3390 will go to VM-1).

The RDP NAT isn’t mandatory ; it could have been done differently, for instance using a point-to-site VPN to enter the Virtual Network.  But since we covered the Load Balancer, I thought I might show the NAT rules while we’re there.

As you can see, there are quite a few puzzle pieces (NICs aren’t even represented!).  If you compare that with Cloud Service, it might seem complicated.  The thing is Cloud Service aggregated a bunch of features and at it end it was hard or impossible to cover other scenarios.  The new model is more modular with more (simpler) components.

Let’s start!

Resource Group

We’re in ARM and the first thing we’ll do is to create a resource group.  The resource group will contain all the Azure artefacts.

One of the nice things of resource group is that when you do some exploration or POCs, once you’re done you can simply delete the resource group and all the artefacts underneath will disappear.

Give the resource group the name you want and create it in the region you prefer.  It is absolutely possible to create a resource group in region X and artefacts underneath in region Y, but I find that confusing unless you’re addressing a DR scenario within one resource group.

Public IP Address

The first piece of the puzzle, the Public IP address.  Type “Public IP Address” in the marketplace and choose the one published by Microsoft.

image

For the name, call it Public-IP  but it really doesn’t matter:  this is the resource name and nobody else than you will see that.

Dynamic or Static?  In this post, we’ll go with dynamic because it’s the cheapest option and I’m stingy, am I?

Basically, a static IP address gives you a public IP address that won’t change.  It can be useful in some scenarios but is unnecessary in most and you pay for it.

A dynamic IP is an IP address that will change on Azure whims.  But you do not pay for it.  With dynamic IP addresses, you do not bind on the IP address, you bind on the DNS name label.  Azure makes sure the fully qualified domain name (FQDN) is in sync with the IP address whenever it changes.

Idle timeout is usually ok at the default value.  This configuration allows you to decide when to kill an idle TCP connection.

DNS label must be unique.  I often use my initials, VPL, but it isn’t the most professional approach.  You then select the resource group you just created and make sure it’s in the same region.

DNS Configuration

We have a public IP.  This is good but it’s on an Azure domain name.  It will be:

<The lable you chose>.<Name of the region>.cloudapp.azure.com

It’s ok for demo, might even be alright for dev & QA.  For production, it’s a bit like using your hotmail account for doing business:  it doesn’t blink “we are serious” very brightly.

So what you’ll wanna do is to map your real domain name, let’s say www.fantasticweb.com, to the cloudapp one.

This is done by entering a Canonical Name (CNAME) record either in your registrar or the upcoming Azure DNS Service.  What the hell is that?  A CNAME gives DNS levels of indirection to resolve DNS.  For instance we could have:

DNS Source Target Record Type
Your Registrar www.fantastic.com <something>.cloudapp.azure.com CNAME
Azure <something>.cloudapp.azure.com 143.32.45.12 A

So when somebody browses for www.fantastic.com, unbeknown to them, there is a bit of a dance happening to find the actual target IP address.

You can learn more about it here.

Virtual Network

Let’s create a Virtual Network to put our VMs in.

This is one of the major differences with ARM (vs Cloud Services):  VMs always live in a Virtual Network.  You can leave the door open if you like the air the get in, but you need a vnet.

So in the Azure Marketplace, type “Virtual Network”, take the one published by Microsoft.  Make sure you choose the Resource Manager before hitting Create

image

You can name the vnet…  “vnet”.  Make sure you put it in the right resource group and then hit create.

This creates a vnet with a default subnet.  In this post, I won’t secure the network.  If you want to do that, I suggest you read this post.

Storage

We’ll need storage to store the virtual hard drives of the VMs.

Let’s create a storage account.  Again, make sure you select “Resource Manager” as deployment model before hitting create.  Make sure you put it in the same resource group & region.

Availability Set

Next, we’ll create an availability set.  An availability set is a declarative construct that says to Azure:  please distribute the VMs that are in it across different failure & update domains.  For more details on availability sets, please read this post.

In the Azure Marketplace, search for “Availability Set”, select the one from Microsoft.

You must give it a name.  I suggest “AvailSet”.  Make sure you put it in the right resource group and Azure region.

You could configure the number of update domains & failure domains here.  This is why an availability set is “a thing” in ARM while it used to be simply a name in ASM:  in order to attach more configuration to it.

Virtual Machines

Here we’ll create 2 virtual machines.  I’m going to create Windows Server 2012 VMs but you could go ahead and create any Linux VMs.  The reminder of the post would be identical.

In Azure Marketplace, search for “windows 2012” and select “Windows Server 2012 R2 Datacenter” published by Microsoft.

image

VMs require a fair bit of configuration, so let’s go with it.

In the (1) Basics configuration, give your vm the name “VM-0”.  Give a login name / password ; this will be the local admin account of the VM.  Make sure it is in the right resource group and Azure region.

For the size (2), since it’s a demo, go with something small, such as a A2.

For the configuration (3), this is where the meat is.  Select the storage account we’ve created.  Select the virtual network we created.  For public IP address, select None.  For Network Security Group, select None.  At the bottom, for Availability Set, select “AvailSet” that we’ve just created.

You can now create the VM.  This is quite slow, so go ahead and create the second one.  Configurations will be identical but for its name, “VM-1”.

Load Balancer

Ok, now let’s stick everything together!

In the Azure Marketplace, type “Load Balancer” and select the one published by Microsoft.

Let’s call it “LB”, keep the scheme to “public”, choose the public IP as the one we’ve created, i.e. “Public-IP”.  As usual, let’s put it in the same region & resource group.

image

Let’s create it.

Then let’s configure the backend pools.

image

We’ll add one, then name it “Web” and “Add a virtual Machine”.

For the availability set, we choose the one we’ve created, i.e. “AvailSet” and for the virtual machines, we select both VM-0 & VM-1.

This defines the targets for the load balancer.

Now, let’s define a probe.

image

Let’s add one, call it “TCP-Probe”, give it the protocol TCP and leave the port to 80, interval to 5 seconds and unhealthy threshold to 2.

image

That probe is basically going to define when a VM is healthy or not.  When the VM can’t let the probe connect, it’s going to be removed from the load balancer roaster.

Load Balancing Rules

Now for the load balancing rules.

image

Let’s create one, call it “Http”, with protocol TCP, port 80, backend port 80, the back-end pool “Web” (that we just created), Probe “TCP-Probe” (that we also just created), Session Persistence “None”, Idle timeout 4 mins, Floating IP “Disabled”.

Basically, we say that TCP requests coming on the public IP will be forwarded to the availability set on their port 80.

NAT Rules

Let’s setup a NAT Rule to forward RDP requests as follow.

image

So, we’ll create 2 NAT rules.

image

For the first one, let’s call it “RDP-to-VM-0”, let’s select the RDP Service, protocol TCP, Port 3389, target “VM-0”, port mapping “default”.

For the second one, let’s call it “RDP-to-VM-1”, let’s select the RDP Service, protocol TCP, Port 3390 (override it), target “VM-1”, port mapping “custom” & target port 3389.

Testing

That’s all the configuration we need.

From here, you can connect to your VMs.

image

What we’ll want to do for testing is to configure a simple web site on each VM giving a different page each.  Let’s say that VM-0 prints out “VM-0 Site” and VM-1 prints out “VM-1 Site”.

To do that is a bit outside the scope of this post.  Basically, you need to add the “web server” role to your VMs then localize where IIS root folder is (typically in C:\inetpub\wwwroot) and drop a file “index.html” there with the desired output.

Now if you hit the FQDN of your public IP with a browser, you should see one of the two outputs.  If you hit F5, you’ll see…  the same?  Why is that?

That’s because your browser keeps the connection alive and we have 4 minutes before the load balancer times it out.

If you want to have a more spectacular test, go in Visual Studio and create a Console app with the following code:

using System;
using System.IO;
using System.Net;

namespace TestLbConsole
{
    class Program
    {
        static void Main(string[] args)
        {
            for (int i = 0; i != 10; ++i)
            {
                var request =
                    WebRequest.Create("http://vpl-ip.southcentralus.cloudapp.azure.com/") as HttpWebRequest;

                request.KeepAlive = false;

                using (var response = request.GetResponse())
                using (var stream = response.GetResponseStream())
                using (var reader = new StreamReader(stream))
                {
                    var payload = reader.ReadToEnd();

                    Console.WriteLine(payload);
                }
            }
        }
    }
}

Since we disable the KeepAlive, we now hit a different VM each time!

Conclusion

So, that was a bit complicated but hopefully, you followed all along.

There are lots of bit and pieces, but they each are simple.

Here is the ARM template allowing you to reproduce what we did in this article.

Please note I wasn’t able to incorporate the NAT rules into the ARM template.  This is due to a current limitation of the schema:  we can’t have loops within NAT rules.  So you’ll need to enter those rules manually in order to RDP into the VMs.


{
  "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "vm-admin-name": {
      "type": "string",
      "defaultValue": "Macgyver"
    },
    "vm-admin-password": {
      "type": "securestring"
    },
    "vm-prefix": {
      "type": "string",
      "defaultValue": "VM-"
    },
    "storage-account-name": {
      "type": "string"
    },
    "number-of-instance": {
      "type": "int",
      "defaultValue": 2
    },
    "vm-size": {
      "type": "string",
      "minLength": 1,
      "defaultValue": "Standard_A1",
      "allowedValues": [
        "Basic_A0",
        "Basic_A1",
        "Basic_A2",
        "Basic_A3",
        "Basic_A4",
        "Standard_A0",
        "Standard_A1",
        "Standard_A2",
        "Standard_A3",
        "Standard_A4",
        "Standard_A5",
        "Standard_A6",
        "Standard_A7",
        "Standard_D1",
        "Standard_D2",
        "Standard_D3",
        "Standard_D4",
        "Standard_D11",
        "Standard_D12",
        "Standard_D13",
        "Standard_D14"
      ]
    }
  },
  "variables": {
    "AvailSet-name": "AvailSet",
    "LoadBalancer-name": "LB",
    "backend-address-pool-name": "Web",
    "PublicIP-name": "Public-IP",
    "VNet-name": "VNet",
    "vm-subnet-name": "vm-subnet",
    "NIC-prefix": "NIC-",
    "VM-prefix": "VM-"
  },
  "resources": [
    {
      "type": "Microsoft.Compute/availabilitySets",
      "name": "[variables('AvailSet-name')]",
      "apiVersion": "2015-06-15",
      "location": "[resourceGroup().location]",
      "tags": {
        "displayName": "Availability Set"
      },
      "properties": {
        "platformUpdateDomainCount": 5,
        "platformFaultDomainCount": 3
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Network/networkInterfaces",
      "name": "[concat(variables('NIC-prefix'), copyindex())]",
      "location": "[resourceGroup().location]",
      "tags": {
        "displayName": "Network Interface"
      },
      "copy": {
        "name": "nic-loop",
        "count": "[parameters('number-of-instance')]"
      },
      "dependsOn": [
        "[concat('Microsoft.Network/virtualNetworks/', variables('VNet-Name'))]",
        "[concat('Microsoft.Network/loadBalancers/', variables('LoadBalancer-name'))]"
      ],
      "properties": {
        "ipConfigurations": [
          {
            "name": "ipconfig1",
            "properties": {
              "privateIPAllocationMethod": "Dynamic",
              "subnet": {
                "id": "[concat(resourceId('Microsoft.Network/virtualNetworks', variables('VNet-Name')), '/subnets/', variables('vm-subnet-name'))]"
              },
              "loadBalancerBackendAddressPools": [
                {
                  "id": "[concat(resourceId('Microsoft.Network/loadBalancers', variables('LoadBalancer-name')), '/backendAddressPools/', variables('backend-address-pool-name'))]"
                }
              ]
            }
          }
        ]
      }
    },
    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Compute/virtualMachines",
      "name": "[concat(variables('VM-prefix'), copyindex())]",
      "copy": {
        "name": "vm-loop",
        "count": "[parameters('number-of-instance')]"
      },
      "location": "[resourceGroup().location]",
      "tags": {
        "displayName": "Virtual Machines"
      },
      "dependsOn": [
        "[concat('Microsoft.Storage/storageAccounts/', parameters('storage-account-name'))]",
        "nic-loop",
        "[concat('Microsoft.Compute/availabilitySets/', variables('AvailSet-name'))]"
      ],
      "properties": {
        "availabilitySet": {
          "id": "[resourceId('Microsoft.Compute/availabilitySets', variables('AvailSet-name'))]"
        },
        "hardwareProfile": {
          "vmSize": "[parameters('vm-size')]"
        },
        "osProfile": {
          "computerName": "[concat(variables('VM-prefix'), copyIndex())]",
          "adminUsername": "[parameters('vm-admin-name')]",
          "adminPassword": "[parameters('vm-admin-password')]"
        },
        "storageProfile": {
          "imageReference": {
            "publisher": "MicrosoftWindowsServer",
            "offer": "WindowsServer",
            "sku": "2012-Datacenter",
            "version": "latest"
          },
          "osDisk": {
            "name": "osdisk",
            "vhd": {
              "uri": "[concat('http://', parameters('storage-account-name'), '.blob.core.windows.net/vhds/', 'osdisk', copyindex(), '.vhd')]"
            },
            "caching": "ReadWrite",
            "createOption": "FromImage"
          }
        },
        "networkProfile": {
          "networkInterfaces": [
            {
              "id": "[resourceId('Microsoft.Network/networkInterfaces', concat(variables('NIC-prefix'), copyindex()))]"
            }
          ]
        }
      }
    },
    {
      "type": "Microsoft.Network/loadBalancers",
      "name": "[variables('LoadBalancer-name')]",
      "apiVersion": "2015-06-15",
      "location": "[resourceGroup().location]",
      "tags": {
        "displayName": "Load Balancer"
      },
      "properties": {
        "frontendIPConfigurations": [
          {
            "name": "LoadBalancerFrontEnd",
            "properties": {
              "privateIPAllocationMethod": "Dynamic",
              "publicIPAddress": {
                "id": "[resourceId('Microsoft.Network/publicIPAddresses', variables('PublicIP-name'))]"
              }
            }
          }
        ],
        "backendAddressPools": [
          {
            "name": "[variables('backend-address-pool-name')]"
          }
        ],
        "loadBalancingRules": [
          {
            "name": "Http",
            "properties": {
              "frontendIPConfiguration": {
                "id": "[concat(resourceId('Microsoft.Network/loadBalancers', variables('LoadBalancer-name')), '/frontendIPConfigurations/LoadBalancerFrontEnd')]"
              },
              "frontendPort": 80,
              "backendPort": 80,
              "enableFloatingIP": false,
              "idleTimeoutInMinutes": 4,
              "protocol": "Tcp",
              "loadDistribution": "Default",
              "backendAddressPool": {
                "id": "[concat(resourceId('Microsoft.Network/loadBalancers', variables('LoadBalancer-name')), '/backendAddressPools/', variables('backend-address-pool-name'))]"
              },
              "probe": {
                "id": "[concat(resourceId('Microsoft.Network/loadBalancers', variables('LoadBalancer-name')), '/probes/TCP-Probe')]"
              }
            }
          }
        ],
        "probes": [
          {
            "name": "TCP-Probe",
            "properties": {
              "protocol": "Tcp",
              "port": 80,
              "intervalInSeconds": 5,
              "numberOfProbes": 2
            }
          }
        ],
        "inboundNatRules": [ ],
        "outboundNatRules": [ ],
        "inboundNatPools": [ ]
      },
      "dependsOn": [
        "[resourceId('Microsoft.Network/publicIPAddresses', variables('PublicIP-name'))]"
      ]
    },
    {
      "type": "Microsoft.Network/publicIPAddresses",
      "name": "[variables('PublicIP-name')]",
      "apiVersion": "2015-06-15",
      "location": "[resourceGroup().location]",
      "tags": {
        "displayName": "Public IP"
      },
      "properties": {
        "publicIPAllocationMethod": "Dynamic",
        "idleTimeoutInMinutes": 4,
        "dnsSettings": {
          "domainNameLabel": "vpl-ip"
        }
      }
    },
    {
      "type": "Microsoft.Network/virtualNetworks",
      "name": "[variables('VNet-name')]",
      "apiVersion": "2015-06-15",
      "location": "[resourceGroup().location]",
      "tags": {
        "displayName": "Virtual Network"
      },
      "properties": {
        "addressSpace": {
          "addressPrefixes": [
            "10.1.0.0/16"
          ]
        },
        "subnets": [
          {
            "name": "[variables('vm-subnet-name')]",
            "properties": {
              "addressPrefix": "10.1.0.0/24"
            }
          }
        ]
      }
    },
    {
      "type": "Microsoft.Storage/storageAccounts",
      "name": "[parameters('storage-account-name')]",
      "apiVersion": "2015-06-15",
      "location": "[resourceGroup().location]",
      "tags": {
        "displayName": "Storage Account"
      },
      "properties": {
        "accountType": "Standard_LRS"
      }
    }
  ]
}

Here are the template’s parameters:

Name Type Description
vm-admin-name string Name of the local admin user on the vms.
vm-admin-password secured string Password of the local admin user.
storage-account-name string Name of the storage account where the vhds are stored.

This needs to be unique within all storage accounts (not only yours).

number-of-instance int The number of vms to scale out.
vm-size string The size of vm (e.g. A2).

Using Network Security Groups (NSG) to secure network access to an environment

Quite a few demos (including mines) ommit security for the sake of simplicity.  One area where you can secure your applications in Azure is in terms of Networking.

Network Security Groups act as a firewall in the cloud.

In this post, I’ll show you how to create a virtual network with 3 subnets:  front-end, middle & back-end.  We’ll then secure network access to those subnets with the following rules:

  1. Front-end can only be accessed on port 80 by anything from the internet
  2. Front-end can only access the virtual network (not the internet)
  3. Middle can only be accessed by the front-end on port 80
  4. Middle can only access the virtual network
  5. Back-end can only be accessed by the middle on port 1433 (SQL default port)
  6. Back-end can’t access anything
  7. Azure Health Monitoring can access everyone (if we don’t allow that, every VMs within the subnet will be marked as unhealthy and be taken down)

This is a typical firewall configuration for a 3 tier application:

image

We’ll create it in the portal to get the feel of each feature.  At the end, I give you the Azure Resource Manager (ARM) template to generate it at the end.

So all of this is using Vnet v2 with ARM.

Resource Group

First, we’ll create a new resource group which will contain every artifact.

In the new portal, i.e. http://portal.azure.com/, you can select Resource groups on the left menu.  This will open a blade with all your resource groups listed.  Click the Add  button at the top.

For a resource group name, type whatever you want.  I suggest NSG.

Put it in a the region closest to your home.  E.g. East-US.

Hit create, it should take under a minute to create.

Having a resource group is mandatory for creating artefacts in Azure.  Having one for a little POC like this is very useful as you’ll be able to delete the resource group and every artefacts associated to it will be deleted at the same time.

Resource groups are also useful to associate RBAC rules, e.g. given access to a co-worker to a resource group.

Virtual Network

Let’s go ahead and create a Virtual Network inside the resource group we just created.

Open the resource group you just created, hit the Add  button then, in the filter text box, type network and hit enter.

Select Virtual Network (Microsoft as Publisher).

At the bottom of the blade, select “Resource Manager” as the deployment model, then hit create.

image

For the name, type “Poc-Net”.

In address space, type “10.0.0.0/24”.  This is using the Classless Inter-Domain Routing (CIDR, pronounced cider).  This means your virtual network starts at internal address 10.0.0.0 and spans 32-24 (i.e. 8) bits of address space, i.e. 256 addresses (28 = 256).

Leave the subnet name as “default”.  This is utterly useless and the first thing we’ll do is delete it.  Same thing for subnet address range.

Leave your subscription there.

Select the resource group we have created.

Ensure the location is the same as resource group and hit create.

Subnets

Create the virtual net (vnet) takes a little more time than a resource group.

Once it’s created, open it.  Go to the subnets settings.

image

As mention, first thing first:  delete the default subnet.  Select it then delete it.

Let’s create our three subnets:

Name Address Range
Front-end 10.0.0.0/28
Middle 10.0.0.16/28
Back-end 10.0.0.32/28

Those three subnets each span a range of 32-28 (4) bits addresses, i.e. 24 (16) addresses.

I’m a bit cheap on addresses…  If you need to put more than 16 VMs in each subnet, feel free to grow the address space.

You will have noted that there was the possibility to add Network Security Groups, but since we haven’t created them yet, let’s leave it to None for each subnet.

Network Security Groups

Finally, our NSGs!

An NSG in Azure is a stand alone artefact that you associate to subnets or VMs.  You can reuse them on multiple subnets or VMs, although we won’t do that here.

Let’s create our three NSGs, one for each subnet.

In the portal main menu, to the left, click the New button.  Type “Network Security Group” in the filter box, hit enter.  Select Network Security Group (published by Microsoft).  Read and understand every word of the legal statement.  Come on, do it!  No, just kidding.  Click Create.

Give it the name frontEndSecurityGroup.  Make sure you put it in the same resource group.  Hit Create.

Select Inbound security rules.

Inbound rules are the rules to apply to the traffic coming in a subnet or VM.  For the front end we want to allow 2 things:  Http-80 and Azure Health Monitoring.

Let’s add an inbound rule:

  • Name:  Allow-HTTP
  • Priority:  100
  • Source:  Tag
  • Source Tag:  Internet
  • Protocol:  TCP
  • Source port range:  *
  • Destination:  Any
  • Destination port range:  80
  • Action:  Allow

This is our first rule and it allows all traffic coming from the internet through port 80 using the TCP protocol.

Let’s create another one:

  • Name:  Allow-Health-Monitoring
  • Priority:  200
  • Source:  Tag
  • Source Tag:  AzureLoadBalancer
  • Protocol:  Any
  • Source port range:  *
  • Destination:  Any
  • Destination port range:  *
  • Action:  Allow

As mentionned, this is to allow Azure Health monitoring to monitor your VMs.

What is the priority?  Rules are apply by priority order until one actually makes sense:  is coming from the defined source & going to the defined destination using the defined protocol.  First rules that makes sense stops the evaluation process.  So if rule 1 makes sense, rule 2 won’t be applied.

Now, for the front end, we allowed everything we wanted to allow.  Let’s disallow everything else as our third rule:

  • Name:  Disallow-everything-else
  • Priority:  300
  • Source:  Any
  • Protocol:  Any
  • Source port range:  *
  • Destination:  Any
  • Destination port range:  *
  • Action:  Deny

This rules will make sure, for instance, that traffic coming from the virtual network can’t get in the front-end subnet (even on port 80).  That is because the first rule only allows traffic coming from the internet.

You should have the following inbound rules:

Priority Name Source Destination Service Action
100 Allow-HTTP Internet Any TCP/80 Allow
200 Allow-Health-Monitoring AzureLoadBalancer Any Any/Any Allow
300 Disallow-everything-else Any Any Any/Any Deny

For the outbound rule of the front-end, we only allow the front-end to speak to the middle.  We could state it explicitely like this but we’ll enforce that in the middle only.  For the front, we’ll let communication go “anywhere” in the vnet only.  So we’ll have the following outbound rules:

Priority Name Source Destination Service Action
100 Allow-to-VNet Any VirtualNetwork Any/Any Allow
200 Deny-All-Traffic Any Any Any/Any Deny

We will then create a new Network Security Group named middleSecurityGroup.  We’ll define the following inbound rules:

Priority Name Source Destination Service Action
100 Allow-Front 10.0.0.0/28 Any TCP/80 Allow
200 Allow-Health-Monitoring AzureLoadBalancer Any Any/Any Allow
300 Deny-All-Traffic Any Any Any/Any Deny

In the first rule we allow traffic coming from a CIDR block corresponding to the front-end subnet, hence only the front end can use this rule.

We then define the following outbound rules:

Priority Name Source Destination Service Action
100 Allow-to-VNet Any VirtualNetwork Any/Any Allow
200 Deny-All-Traffic Any Any Any/Any Deny

Finally, we’ll create a new Network Security Group named backSecurityGroup.  We’ll define the following inbound rules:

Priority Name Source Destination Service Action
100 Allow-Middle 10.0.0.16/28 Any TCP/1433 Allow
200 Allow-Health-Monitoring AzureLoadBalancer Any Any/Any Allow
300 Deny-All-Traffic Any Any Any/Any Deny

and the following outbound rules:

Priority Name Source Destination Service Action
100 Deny-All-Traffic Any Any Any/Any Deny

Attaching NSG to Subnets

Now that we have our virtual network, subnets & NSGs, we need to associate an NSG to each subnet.

Open the virtual network, select its subnets.  Select front-end subnet and then select Network Security Group (should be at None).  Select frontSecurityGroup and save.

Do the same thing with the other two subnets.

As mentionned, NSG can be associated with more than one subnet / VM and that is why they exist on their own.

There we go.  We have all our rules for a standard three tier architecture network.

If you want to test it, simply put VMs in each subnet and test the connectivity.

…  if you want to this, actually, you’ll need to add rules for allowing inbound TCP:3389 connections for letting your RDP coming in.

ARM Template

As promise, here is the ARM template to recreate that.

My best resources for ARM template are:

The last item is a feature of the new portal.  To access it, go on the main left menu & Browse.  Type Resource in the filter and select Resource Explorer.  You’ll be able to browser through your resources and see their ARM template definition.  This saves heaps of time.  Be mindful though:  resource explorer will show a couple of attributes you shouldn’t include in your template such as status, id, guids, etc.  .

I’ve used few tricks in this template:

  • I used variables to define the CIDR of the subnets ; this avoids duplication of the information
  • I used the resource group location to dictate the location of all the other resources ; this avoids defining an ultimately redundant parameter
  • I used dependsOn & resourceId to attach the Network Security Groups to the subnets.
{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
  },
  "variables": {
    "cidrNet": "10.0.0.0/24",
    "frontNet": "10.0.0.0/28",
    "middleNet": "10.0.0.16/28",
    "backNet": "10.0.0.32/28"
  },
  "resources": [
    {
      "apiVersion": "2015-06-15",
      "name": "Poc-Net",
      "type": "Microsoft.Network/virtualNetworks",
      "location": "[resourceGroup().location]",
      "dependsOn": [
        "Microsoft.Network/networkSecurityGroups/frontSecurityGroup",
        "Microsoft.Network/networkSecurityGroups/middleSecurityGroup",
        "Microsoft.Network/networkSecurityGroups/backSecurityGroup"
      ],
      "tags": { },
      "properties": {
        "addressSpace": {
          "addressPrefixes": [
            "[variables('cidrNet')]"
          ]
        },
        "subnets": [
          {
            "name": "front-end",
            "properties": {
              "addressPrefix": "[variables('frontNet')]",
              "networkSecurityGroup": {
                "id": "[resourceId('Microsoft.Network/networkSecurityGroups','frontSecurityGroup')]"
              }
            }
          },
          {
            "name": "middle",
            "properties": {
              "addressPrefix": "[variables('middleNet')]",
              "networkSecurityGroup": {
                "id": "[resourceId('Microsoft.Network/networkSecurityGroups','middleSecurityGroup')]"
              }
            }
          },
          {
            "name": "back-end",
            "properties": {
              "addressPrefix": "[variables('backNet')]",
              "networkSecurityGroup": {
                "id": "[resourceId('Microsoft.Network/networkSecurityGroups','backSecurityGroup')]"
              }
            }
          }
        ]
      }
    },
    {
      "apiVersion": "2015-06-15",
      "name": "frontSecurityGroup",
      "type": "Microsoft.Network/networkSecurityGroups",
      "location": "[resourceGroup().location]",
      "tags": { },
      "properties": {
        "securityRules": [
          {
            "name": "Allow-HTTP",
            "properties": {
              "protocol": "Tcp",
              "sourcePortRange": "*",
              "destinationPortRange": "80",
              "sourceAddressPrefix": "Internet",
              "destinationAddressPrefix": "*",
              "access": "Allow",
              "priority": 100,
              "direction": "Inbound"
            }
          },
          //{
          //  "name": "Allow-RDP",
          //  "properties": {
          //    "protocol": "Tcp",
          //    "sourcePortRange": "*",
          //    "destinationPortRange": "3389",
          //    "sourceAddressPrefix": "*",
          //    "destinationAddressPrefix": "*",
          //    "access": "Allow",
          //    "priority": 150,
          //    "direction": "Inbound"
          //  }
          //},
          {
            "name": "Allow-Health-Monitoring",
            "properties": {
              "protocol": "*",
              "sourcePortRange": "*",
              "destinationPortRange": "*",
              "sourceAddressPrefix": "AzureLoadBalancer",
              "destinationAddressPrefix": "*",
              "access": "Allow",
              "priority": 200,
              "direction": "Inbound"
            }
          },
          {
            "name": "Disallow-everything-else",
            "properties": {
              "protocol": "*",
              "sourcePortRange": "*",
              "destinationPortRange": "*",
              "sourceAddressPrefix": "*",
              "destinationAddressPrefix": "*",
              "access": "Deny",
              "priority": 300,
              "direction": "Inbound"
            }
          },
          {
            "name": "Allow-to-VNet",
            "properties": {
              "protocol": "*",
              "sourcePortRange": "*",
              "destinationPortRange": "*",
              "sourceAddressPrefix": "*",
              "destinationAddressPrefix": "VirtualNetwork",
              "access": "Allow",
              "priority": 100,
              "direction": "Outbound"
            }
          },
          {
            "name": "Deny-All-Traffic",
            "properties": {
              "protocol": "*",
              "sourcePortRange": "*",
              "destinationPortRange": "*",
              "sourceAddressPrefix": "*",
              "destinationAddressPrefix": "*",
              "access": "Deny",
              "priority": 200,
              "direction": "Outbound"
            }
          }
        ],
        "subnets": [ ]
      }
    },
    {
      "apiVersion": "2015-06-15",
      "name": "middleSecurityGroup",
      "type": "Microsoft.Network/networkSecurityGroups",
      "location": "[resourceGroup().location]",
      "tags": { },
      "properties": {
        "provisioningState": "Succeeded",
        "securityRules": [
          {
            "name": "Allow-Front",
            "properties": {
              "provisioningState": "Succeeded",
              "protocol": "Tcp",
              "sourcePortRange": "*",
              "destinationPortRange": "80",
              "sourceAddressPrefix": "[variables('frontNet')]",
              "destinationAddressPrefix": "*",
              "access": "Allow",
              "priority": 100,
              "direction": "Inbound"
            }
          },
          //{
          //  "name": "Allow-RDP",
          //  "properties": {
          //    "protocol": "Tcp",
          //    "sourcePortRange": "*",
          //    "destinationPortRange": "3389",
          //    "sourceAddressPrefix": "*",
          //    "destinationAddressPrefix": "*",
          //    "access": "Allow",
          //    "priority": 150,
          //    "direction": "Inbound"
          //  }
          //},
          {
            "name": "Allow-Health-Monitoring",
            "properties": {
              "provisioningState": "Succeeded",
              "protocol": "*",
              "sourcePortRange": "*",
              "destinationPortRange": "80",
              "sourceAddressPrefix": "AzureLoadBalancer",
              "destinationAddressPrefix": "*",
              "access": "Allow",
              "priority": 200,
              "direction": "Inbound"
            }
          },
          {
            "name": "Deny-Everything-Else",
            "properties": {
              "provisioningState": "Succeeded",
              "protocol": "*",
              "sourcePortRange": "*",
              "destinationPortRange": "80",
              "sourceAddressPrefix": "*",
              "destinationAddressPrefix": "*",
              "access": "Deny",
              "priority": 300,
              "direction": "Inbound"
            }
          },
          {
            "name": "Allow-to-VNet",
            "properties": {
              "provisioningState": "Succeeded",
              "protocol": "*",
              "sourcePortRange": "*",
              "destinationPortRange": "80",
              "sourceAddressPrefix": "*",
              "destinationAddressPrefix": "VirtualNetwork",
              "access": "Allow",
              "priority": 100,
              "direction": "Outbound"
            }
          },
          {
            "name": "Deny-All-Traffic",
            "properties": {
              "provisioningState": "Succeeded",
              "protocol": "*",
              "sourcePortRange": "*",
              "destinationPortRange": "80",
              "sourceAddressPrefix": "*",
              "destinationAddressPrefix": "*",
              "access": "Deny",
              "priority": 200,
              "direction": "Outbound"
            }
          }
        ],
        "subnets": [ ]
      }
    },
    {
      "apiVersion": "2015-06-15",
      "name": "backSecurityGroup",
      "type": "Microsoft.Network/networkSecurityGroups",
      "location": "[resourceGroup().location]",
      "tags": { },
      "properties": {
        "securityRules": [
          {
            "name": "Allow-Middle",
            "properties": {
              "provisioningState": "Succeeded",
              "protocol": "Tcp",
              "sourcePortRange": "*",
              "destinationPortRange": "1433",
              "sourceAddressPrefix": "[variables('middleNet')]",
              "destinationAddressPrefix": "*",
              "access": "Allow",
              "priority": 100,
              "direction": "Inbound"
            }
          },
          //{
          //  "name": "Allow-RDP",
          //  "properties": {
          //    "protocol": "Tcp",
          //    "sourcePortRange": "*",
          //    "destinationPortRange": "3389",
          //    "sourceAddressPrefix": "*",
          //    "destinationAddressPrefix": "*",
          //    "access": "Allow",
          //    "priority": 150,
          //    "direction": "Inbound"
          //  }
          //},
          {
            "name": "Allow-Health-Monitoring",
            "properties": {
              "provisioningState": "Succeeded",
              "protocol": "*",
              "sourcePortRange": "*",
              "destinationPortRange": "80",
              "sourceAddressPrefix": "AzureLoadBalancer",
              "destinationAddressPrefix": "*",
              "access": "Allow",
              "priority": 200,
              "direction": "Inbound"
            }
          },
          {
            "name": "Deny-Everything-Else",
            "properties": {
              "provisioningState": "Succeeded",
              "protocol": "*",
              "sourcePortRange": "*",
              "destinationPortRange": "80",
              "sourceAddressPrefix": "*",
              "destinationAddressPrefix": "*",
              "access": "Deny",
              "priority": 300,
              "direction": "Inbound"
            }
          },
          {
            "name": "Deny-All-Traffic",
            "properties": {
              "provisioningState": "Succeeded",
              "protocol": "*",
              "sourcePortRange": "*",
              "destinationPortRange": "80",
              "sourceAddressPrefix": "*",
              "destinationAddressPrefix": "*",
              "access": "Deny",
              "priority": 100,
              "direction": "Outbound"
            }
          }
        ],
        "subnets": [ ]
      }
    }
  ],
  "outputs": { }
}

Conclusion

You can see that you can “bring your own network” to Azure and define rules that mimic the rules of your on-premise firewall.

This adds a level of security and reliability.  Even if you use authentication for all your services on your middle tier, for instance, it is more secure to simply disallow traffic from the internet to be routed there.  Not only that, but if some attacker tries to breach in, he simply won’t get to your machine and his / her attacks won’t affect the performance of your VMs.  Hence your service will be more reliable.

Lots of variations are possible.  A popular one is to lock down the front end to an IP range to let only people from certain offices access them.

This was a quite standard network.  I chose that configuration since it is quite common but I’m sure you can see how you can start from that and customize it to fit your own requirements.