Azure Private Endpoints, Service Endpoints etc

There are quite a few things you need/should know when you start considering putting you Web App “into” your VNet. Or any Azure service for that matter… In my last post, Securing Azure Web Apps using Application Gateways and vNets, I explained how to set up a Web App behind an Application Gateway using Private Endpoints. However, there were so many things that I felt didn’t quite fit into that post. So I decided to write a more generic post on the topic.

The first thing I think you need when you start working on integrating your Web Apps in your VNet is a basic understanding of what a Private Endpoint and a Service Endpoint is, what VNet integration means, as well as what Subnet delegation means.

Note: I am by no means an expert on this, but I have spent quite some time working on it know, and will give you my view on it. It is not entirely simple to understand from the documentation in my opinion…

Private Endpoints / Private Link and Service Endpoints

Ok, so these are kind of 2 ways to perform the same thing. They allow you to communicate with an Azure service using a private connection. But they do it in 2 different ways, with some differences…

Let’s start with the Service Endpoint…

Service Endpoints

Adding a Service Endpoint for an Azure service to your subnet, allows traffic from your VNet to travel to the designated service over the Azure backbone instead of over the internet. This means 2 major things. First of all, the traffic stays securely inside the Azure backbone and never exits out to the internet. And secondly, this means that the service is able to see the origin of the traffic as being that subnet, which allows it to filter access based on this.

The service is still reached through its public IP address, and your applications call the service as usual. But under the hood, Azure networking is handling the request differently.

Imagine that you are building an application that needs to use an Azure SQL database. But you want to make sure that access to the database is only allowed from your VNet. To get this functionality, you need to configure a Service Endpoint and firewall rule as follows.

First, a VNet and a SQL database is needed. This can be set up by running the following commands

az network vnet create \
    -g MyRg \
    -n MyVNet \
    --address-prefix 10.0.0.0/16 \
    --subnet-name Default \
    --subnet-prefix 10.0.0.0/24

az sql server create \
    -g MyRg \
    -n MySqlServer \
    --admin-user zerokoll \
    --admin-password MyVeryS3cr3tP@ssword!

az sql db create \
    -g MyRg \
    -n MyDatabase \
    --server MySqlServer

Next you add a Service Endpoint for Microsoft.Sql to the Default subnet to allow you to talk to Azure SQL privately. This can be done by running a command like this

az network vnet subnet update \
  -g MyRg \
  -n Default \
  --vnet-name MyVNet \
  --service-endpoints Microsoft.Sql

As you notice from the command, Service Endpoints are configured per service. In this case, the service is Microsoft.Sql, but it could just as well have been another service type or even several services. If you want to see what services offer the ability to be added as Service Endpoints, have a look the Microsoft Docs.

However, if you tried to communicate with the database from your VNet at this point, you would be met with an error like this

Cannot open server ‘mysecretserver’ requested by the login. Client with IP address ‘XXX.XXX.XXX.XXX’ is not allowed to access the server. To enable access, use the Windows Azure Management Portal or run sp_set_firewall_rule on the master database to create a firewall rule for this IP address or address range. It may take up to five minutes for this change to take effect.

So far the Service Endpoint has been set up, and any traffic going to the SQL Server instance will use the Azure backbone to reach the service. However, it is still up to the SQL Server firewall to grant the incoming traffic access. The cool thing now though, is that because that we have a Service Endpoint set up for Microsoft.Sql, the firewall is able to detect traffic from our specific subnet, and grant it access based on that. All we need to do, is to add a subnet firewall rule by running

az sql server vnet-rule create \
    -g MyRg \
    -n AllowDefaultSubnet \
    --server MySqlServer \
    --subnet Default \
    --vnet-name MyVNet

This will give any traffic from the Default subnet access to the server. Awesome!

Note: Different services have different ways to configure access. Azure SQL Server uses a firewall that you can configure. Web Apps use “Access Restrictions”. And so on… So you need to figure out how to set it up for your specific service. But the general idea is the same. The subnet “integrates” with the service, and the service filters the traffic based on the origin.

Caveat: This works fine for things like VMs located inside the correct subnet, as the traffic originates from that network. Other Azure services that you might work with, like for example a Web App, will not be able to access the SQL Server database based on this, as the traffic doesn’t originate from the subnet. At least not by default… If we start looking at another feature called VNet integration, the story changes. I’ll cover that in a couple of minutes.

Important to note here is that the SQL Server is still “exposed” to the internet. However, the firewall will block any access unless you reconfigure it to explicitly allow access. So it should be very safe.

Ok, so that’s Service Endpoints… But what about Private Endpoints?

Private Endpoints / Private Link

Well, Private Endpoints are similar in the way that they allow you to talk to a Azure service in a way that allows you to configure access in a better way. But the implementation is very different.

Note: I write “Private Endpoints / Private Link” because both terms are often used to talk about this feature. Private Endpoints are enabled by a service called Private Link. This is a service that allows you to get a private link from your VNet to an external service. I’m not going to cover it in this post, but the Private Link service can be used to expose services privately to other Azure customers. Not just Azure’s own services, but also services created and managed by other Azure customers.

A Private Endpoint creates a virtual network interface (NIC) inside your VNet and connects that NIC to the Azure service of choice. This means that you can access the service inside your VNet using a private IP address from your own address range, and Azure makes sure that any communication to that NIC is sent to the right place…ish…

Let’s look at the same example before, but using a Private Endpoint for the SQL Server instead.

The first step is still to create a VNet and a SQL Server instance

az network vnet create \
    -g MyRg \
    -n MyVNet \
    --address-prefix 10.0.0.0/16 \
    --subnet-name Default \
    --subnet-prefix 10.0.0.0/24

az sql server create \
    -g MyRg \
    -n MySqlServer \
    --admin-user zerokoll \
    --admin-password MyVeryS3cr3tP@ssword!

az sql db create \
    -g MyRg \
    -n MyDatabase \
    --server MySqlServer

Next, we can start looking at adding the Private Endpoint. However, before we can do that, have to turn off private endpoint network policies on the subnet we want to add the Private Endpoint to. This is done by executing the following command

az network vnet subnet update \
    -g MyRg \
    -n Default \
    --vnet-name MyVNet \
    --disable-private-endpoint-network-policies true

Once private endpoint network policies have been disabled, the Private Endpoint can be added by running

az network private-endpoint create \
    -g MyRg \
    -n MySqlPrivateEndpoint \
    --vnet-name MyVNet \
    --subnet Default \
    --private-connection-resource-id \<RESOURCE ID OF SQL SERVER> \
    --group-id sqlServer \
    --connection-name MySqlPrivateEndpointConnection  

As you might be able to see from the command, we are creating a Private Endpoint called MySqlPrivateEndpoint. It will be added to the Default subnet on the VNet called MyVNet. The endpoint will point to a Azure SQL Server resource, as defined by the group-id parameter, and the specific Azure SQL Server instance we are connecting it to is defined by the private-connection-resource-id. The last parameter, connection-name, just defines the name of the private link connection that connects the Private Endpoint to the service.

This will create a new NIC called MySqlPrivateEndpoint.nic.XXXX inside the Default subnet. The NIC gets a private IP address that you can use to talk to the SQL Server instance. Well…ish… You can’t actually talk to the server using that IP address. Instead, you actually need to configure a Private DNS Zone that resolves the SQL Server host name to the internal IP address. Something that can be done by running a few commands like this

az network private-dns zone create \
    -g MyRg \
    -n "privatelink.database.windows.net"

az network private-dns link vnet create \
    -g MyRg \
    -n MyDNSLink \
    --zone-name "privatelink.database.windows.net" \
    --virtual-network MyVNet \
    --registration-enabled false

az network private-endpoint dns-zone-group create \
   -g MyRg \
   -n MyZoneGroup \
   --endpoint-name MySqlPrivateEndpoint \
   --private-dns-zone "privatelink.database.windows.net" \
   --zone-name sql

This creates a Private DNS Zone called privatelink.database.windows.net and links it to the VNet. Once the Private DNS Zone is up and running, we add the records needed for the SQL Server instance, pointing to the private IP address instead of the public one that will be returned by the public DNS. So any application inside the VNet will now be able to connect to the SQL Server privately.

Note: I will talk more about this DNS stuff in just a minute. So if it didn’t make much sense, just relax. It will hopefully make much more sense soon. And then you can come back here and have a look at it again… But I thought I would put it here as well just for the sake of completeness.

There is still one caveat though. The server is still exposed to the internet. Sure, the firewall will block any incoming requests, but we can do better. By running

az sql server update \
    -g MyRg \
    -n MySqlServer \
    --enable-public-network false

we can turn off public access to the SQL Server instance completely. This means that no traffic that originates from anywhere outside the VNet will be allowed to reach the server. Pretty much making the server behave as though it was completely inside your VNet. And since we have the Private Endpoint, all the communication from the VNet will access the server without going through the firewall.

Conclusion

So both services offer the same functionality, the ability to communicate with Azure services from your VNet without needing to pass over the internet. This allows for communication with the services in a much more secure way.

Service Endpoints allows resources on your subnet to talk Azure services as usual. However, the traffic is routed through the Azure backbone instead of the internet, and allows the Azure services to filter access based on the defined subnet.

Private Endpoints on the other hand give you a NIC inside your network, with a private IP address that “magically” connects to the service. This allows you to communicate securely with an Azure service inside your own network. This also allows you to completely shut off public access to the Azure service (if supported by the service).

Caveat: Unfortunately, Network Security Groups do not affect traffic to and from a Private Endpoint. So if you need to limit traffic to the Private Endpoint from the resources on your VNet, you will have to limit the outbound traffic at the source instead of trying to limit the inbound traffic to the Private Endpoint.

Both of these features are about communicating from your network to Azure services. If you want to go the other way around, and have the Azure service talk to resources inside you network, you need to look at VNet Integration.

VNet Integration

A VNet integration is kind of the inverse of the previous two services. If you for example want to let your Web App talk a service hosted on a VM inside your VNet, this is what you need.

It’s not overly complicated, and there is only one version of it, as opposed to Service Endpoints and Private Endpoints. On top of that, all you have to do, is to configure your service to have a VNet integration, and it gets “magically” connected to your network, allowing it to call your private resources inside the network.

There are a few caveats…

First of all, each VNet integration needs an empty subnet to integrate with. Sort of… For App Services for example, you need an empty subnet per App Service Plan, not per Web App. On top of that, you need to make sure there are enough IP addresses inside the subnet. Each service instance needs its own IP address. So for App Services for example, you need one IP address per App Service Plan instance. And don’t forget that Azure steals the first 4 addresses (0 to 3) and the last. So for a single instance, you need at least 5 addresses.

Very important: The VNet integration will use double the IP addresses if you scale up or down instead of in our out, as it temporarily will have duplicate instances running during the scaling.

Also, DNS look ups in the connected service won’t use the internal DNS from the VNet by default. This means that you can’t use the DNS to find the targets resources on the network. Instead, you have to either rely on IP addresses, or manually configure the service to use the internal DNS from the network. In the case of a Web App, this is done by setting an app setting called WEBSITE_DNS_SERVER to 168.63.129.16.

You also need to configure subnet delegation. This is basically you giving the Azure service access to modify your subnet. This is needed since the service is the only thing that knows how to route the traffic to your network. So by adding subnet delegation to that service, you are saying that you are OK with that service changing your subnet.

Subnet delegation is done per service. So if you are using App Services, you give “Microsoft.Web/serverFarms” access to play with your subnet. And if you are using another service, you give that specific service access… And you can only add subnet delegation to one service per subnet, which is fine, as every integration needs an empty subnet to work with anyway.

Ok, now that we understand Private Endpoints, Service Endpoints and VNet integration, let’s look at some more specific things…

Effects of adding a Private Endpoint to a Web App

I work a lot with App Services and Web Apps. Because of this, I have used these features mostly in that space. So I decided to give a little bit of extra information about that specific service.

The first thing I want to cover is what happens when you add a Private Endpoint to a Web App. I know some of it has already been covered, but I wanted to cover it a bit more in depth for this particular service.

Note: To be able to add a Private Endpoint, the App Service Plan has to be a Premium or Isolated SKU

Before you can even consider adding a Private Endpoint to your application, you need to make sure to turn off private endpoint network policies by running a command like this

az network vnet subnet update \
            -g <RESOURCE GROUP NAME> \
            -n <SUBNET NAME> \
            --vnet-name <VNET NAME> \
            --disable-private-endpoint-network-policies true

If you forget this step, the CLI will be nice enough to tell you but presenting you with a PrivateEndpointCannotBeCreatedInSubnetThatHasNetworkPoliciesEnabled error.

Once the private endpoint network policies have been turned off, you can add the Private Endpoint for a Web App like this

az network private-endpoint create \
            -g <RESOURCE GROUP NAME> \
            -n <ENDPOINT NAME>  \
            --vnet-name <VNET NAME> \
            --subnet <SUBNET NAME> \
            --private-connection-resource-id <WEB APP ID> \
            --connection-name <CONNECTION NAME> \
            --group-id sites

This will cause some network Voodoo to be performed performed (probably including chickens being killed, and incense being burned in an Azure Data Center), and you end up with a new NIC inside the subnet of your choice. And that NIC is connected to the Web App. This is very similar to what I already covered when explaining Private Endpoints using SQL Server. However, there is a bit more we should really know…

It really is private

As soon as you add a Private Endpoint to a Web App, it becomes an internal resource. All access from the public is turned off, and if you try to browse to the application using the azurewebsites.net address, you will be faced with a page like this

This also means that the Kudu site becomes private as well. This causes a few interesting things. It makes it impossible to get to the console if you for example need that to debug your app. And you can’t manage the apps site extensions etc. So, as soon as you add a Private Endpoint to your web app, you need to make sure that you have a way to access the site from inside the network.

Remember that this also means that if you are using a DevOps pipeline to do your deployments, you need to have a build agent inside the VNet to be able to do deployments. Funnily enough, right-click deploy from your dev machine still seems to works…

However, even if you might be able to right-click deploy, which I don’t recommend for real projects, you are likely going to need to access the app in some way anyway. Either to use it, manage it, or debug it.

There are a couple of solutions to get access to the app inside your network

• You can create a VM with a inside the network, add a public IP to it, and then RDP to that box. From there, you are free to browse to the xxx.azurewebsites.net or xxx.scm.azurewebsites.net address, as well as the Azure Portal. However, i wouldn’t recommend having a VM with a public endpoint running permanently in Azure due to the security concerns…

• You can create a VM without a public IP address, and then use a Azure Bastion service to connect to the VM securely using your browser. This is basically secure RDP inside your browser. The biggest downside to this solution is probably the price, which at the time of writing is $0.19/h.

• You can set up a point-to-site VPN, which would allow you to access the app and the Kudu site from your local machine. This is a lot cheaper than Bastion at $0,04/h. But on the other hand, there is a bit of work involved in setting it up.

Comment: If you need to expose the application to the internet, that’s a different story. And I covered that in my last post.

On top of the app becoming private, there are some interesting DNS changes being made…

Changes to the applications DNS records

When you set up a Web App, 2 DNS records are set up for you. One for xxx.azurewebsites.net and one for xxx.scm.azurewebsites.net (Kudu). And if you run nslookup before adding a Private Endpoint to the app, the result would be something like this

> nslookup foo.azurewebsites.net

foo.azurewebsites.net  canonical name = waws-prod-am2-365.sip.azurewebsites.windows.net.
waws-prod-am2-365.sip.azurewebsites.windows.net canonical name = waws-prod-am2-365-8740.westeurope.cloudapp.azure.com.
Name:   waws-prod-am2-365-8740.westeurope.cloudapp.azure.com
Address: 20.50.2.19

Ok, so the apps domain name is CNAME:d to an xxx.azurewebsites.windows.net address. This in turn is CNAME:d to an xxx.westeurope.cloudapp.azure.com address. Depending on the location the App Service is deployed in. And that domain name finally has the IP address of the App Service Plan (ish…Azure App Services are multi tenant, so the IP address is actually shared among a bunch of plans).

However, if a Private Endpoint is added, the result changes, and turns into this

> nslookup foo.azurewebsites.net

foo.azurewebsites.net  canonical name = foo.privatelink.azurewebsites.net.
foo.privatelink.azurewebsites.net  canonical name = waws-prod-am2-365.sip.azurewebsites.windows.net.
waws-prod-am2-365.sip.azurewebsites.windows.net canonical name = waws-prod-am2-365-8740.westeurope.cloudapp.azure.com.
Name:   waws-prod-am2-365-8740.westeurope.cloudapp.azure.com
Address: 20.50.2.19

Ok, it’s similar, but an extra CNAME has been added in there. The apps domain is now CNAME:d to foo.privatelink.azurewebsites.net, which in turn is then CNAME:d just like before. Basically they have added an extra step in the DNS lookup. This might not seem very important, but it is actually quite useful!

And why is it useful? Well, it allows us to add a DNS Zone inside of our VNet for privatelink.azurewebsites.net, and then use an A record inside that zone to redirect look ups to the internal IP address of the Private Endpoint NIC. So, from the outside, the DNS lookup returns exactly the same as before. But inside the VNet, the Private DNS Zone for privatelink.azurewebsites.net, causes it to resolve to the internal IP address. Very simple, but also very smart…

Note: This is exactly what we did before in the SQL example. However, for SQL the DNS Zone is privatelink.database.windows.net instead of privatelink.azurewebsites.net.

Comment: Why not just add a Private DNS Zone for azurewebsites.net? I assume that is because that would interfere with all azurewebsites.net look ups. If the Private DNS Zone didn’t have the specific host name you were looking for configured, it would return a “not found” result. Using a step in between, all other look ups will work unchanged, but we are allowed to alter the specific one we need.

With that DNS “indirection” in place, we can add and link a private DNS Zone for privatelink.azurewebsites.net like this

az network private-dns zone create \
    -g MyRg \
    -n privatelink.azurewebsites.net

az network private-dns link vnet create \
    -g MyRG \
    -n "MyDnsLink" \
    --virtual-network MyVNet \
    --zone-name privatelink.azurewebsites.net \
    --registration-enabled false

And then add we can create the required A records for the Private Endpoint like this

az network private-endpoint dns-zone-group create \
    -g MyRg \
    -n foo \
    --endpoint-name FooPrivateEndpoint \
    --private-dns-zone privatelink.azurewebsites.net \
    --zone-name privatelink.azurewebsites.net

This will add an A record for mywebapp.privatelink.azurewebsites.net, and one for mywebapp.scm.privatelink.azurewebsites.net.

This addition means that running nslookup inside the VNet will generate a result that looks like this

> nslookup foo.azurewebsites.net

foo.azurewebsites.net  canonical name = foo.privatelink.azurewebsites.net.
Name:   foo.privatelink.azurewebsites.net
Address: 10.0.1.4

That’s awesome in all of its simplicity!

There is really just one more thing that I want to cover in this post…

Routing traffic through the VNet

Setting up a VNet integration allows the Azure service to access resources on the private network. However, any outbound traffic, that is not targeting a private address range, will still be routed to the internet from the service itself. This might not be what you want in some cases…

Why would that be a problem? Well, the most obvious one is probably that we have no control over the network access being used. We can’t monitor what external resources are being used, and we can’t block unwanted outbound traffic.

Another reason is that the outbound traffic will originate from the Web App itself, meaning that the Web Apps public IP address will used as the origin. In some cases this is not the best solution. For example if you are the in the receiving end of the call wants to add firewall rules to limit incoming traffic to based on origin.

Remember, App Services is multi-tenant, so Azure is nice enough to provide a range of addresses that could become the origin address. But it is still a range of addresses, and on top that it is a range that is shared by many App Services…

So, is there anything we can do about these things?

Yes there is! We can add a NAT Gateway with a public IP address to our network, and route all traffic from the Web App through the VNet. This will not only make it possible to monitor and control the traffic, it also makes sure that all traffic originates from a single defined address.

The first step to get this solution up and running, is to create a Public IP address and a NAT Gateway like this

az network public-ip create \
    -g MyRg \
    -n MyNatGatewayIp \
    --sku standard \
    --allocation static

az network nat gateway create \
    -g MyRg \
    -n MyNatGateway \
    --public-ip-addresses MyNatGatewayIp \
    --idle-timeout 10

This will set up the Public IP address and the Gateway.

The next step is to integrate it with the VNet. This needs to be done on a per subnet. Any subnet that has a NAT Gateway assigned will use this when sending traffic out of the network.

To assign a the NAT Gateway to the Default subnet of the MyVNet VNet, we can run the following command

az network vnet subnet update \
    -g MyRg \
    -n Integration \
    --vnet-name MyVNet \
    --nat-gateway MyNatGateway

This is all there is to it!

Unfortunately it won’t actually work configured like this… The reason is that the Web App won’t send the traffic through the VNet integration by default. Only traffic targeted at private network addresses are sent into the integrated VNet. Luckily, it’s easy to fix! All you need to do, is to add a Web App configuration setting called WEBSITE_VNET_ROUTE_ALL to 1 like this

az webapp config appsettings set \
    -g MyRg \
    -n Foo \
    --settings WEBSITE_VNET_ROUTE_ALL=1

Now, all outbound traffic from the Web App will go through the VNet, and in turn its NAT Gateway, giving you a single origin IP address. And a way to monitor and limit the traffic.

That’s it! I covered what I wanted to cover, and as usual it turned into a HUGE post… Some day I will write a short post. I promise! But I would rather write what I want to write, than make sure that it is short at the expense of the content.

Conclusion

This is an interesting area to look into. Things used to be a lot simpler. You deployed your application to an Azure PaaS services, and you were good to go. We didn’t really have to consider the networking part of it to much…it just worked. But I can definitely see how this increases security and control over the solutions we deploy to the cloud. Even if it does add overhead and complexity to the set up, and costs a bit more. Security is never simple or cheap!

If you have any questions, I’m available at @ZeroKoll as always! And I will do my best to answer!