In my last post, I talked about Kubernetes Custom Resource Definitions, or CRDs. It was a fairly high-level intro with some code attached, which I guess could be useful. But to show something that shows that CRDs can be really useful, I decided to build a POC of how to use Kubernetes CRDs together with some .NET Core code and Azure to enable management of Azure DNS records using K8s resources.

Azure DNS

Have you never used Azure DNS? Well, it is actually quite simple…

Azure can manage our DNS records for through Azure DNS. To do so, you set up something called a DNS zone in Azure. This gives you a bunch of name server addresses that you can then configure your domain to use.

Once you have configured your domain to use the provided nameservers, you can assign DNS records to your zone by using something called record sets. Each record set has a name, a type, a TTL and so on.

In this sample, I want to show you how you can automate this process and make sure your DNS records are managed in the same way as you manage your other resources when working with Kubernetes.

This post is not about Azure DNS as such, so I won’t go into more detail.

Setting up Azure Credentials

To be allowed to modify a DNS zone, we need to have an account that has the right permissions. For that, we can create a new Service Principal in Azure, and assign it permission to modify the DNS zone.

To create a new Service Principal (SP), you can either go to the portal and create an application under the Azure Active Directory section, or you can log into the Azure CLI and run

az ad sp create-for-rbac --name http://DnsAdmin

Setting the name to whatever name you want it to use.

Note: I prefixed the name with http://. This is because a name should actually be a URL. If you don’t, Azure will automatically do it for you.

Running this command, will result in an output that looks something like this

{
  "appId": "b249b9e8-7abd-4554-ba1c-9c8609048be7",
  "displayName": "DnsAdmin",
  "name": "http://DnsAdmin",
  "password": "4a7cb9a0-c1b1-4ad6-9c88-6d809faa611e",
  "tenant": "d2238023-20ba-4e0b-a05a-b7de6d38a648"
}

You want to make sure that you write down the tenant, appId and password values, as they will be needed them later on. The password is especially important as this is the only time you will ever see it. Once you clear the screen, it is gone.

Tip: It is possible can generate a new password/secret if you loose it…

Note: I know I left my values here in plain text, which is a REALLY bad idea. I know that! But don’t worry, I have removed that principal already!

The newly created SP has Contributor access to your Azure account by default, which is a bit excessive, so I suggest removing that by running

az role assignment delete --assignee <APP ID> --role Contributor

and then add just the DNS Zone Contributor role for the DNS zone in question instead. This is done by running a command that looks like this

az role assignment create \
    --assignee <APP ID> \
    --role "DNS Zone Contributor" \
    --scope "/subscriptions/<SUBSCRIPTION ID>/resourceGroups/<RESOURCE GROUP NAME>/providers/Microsoft.Network/DnsZones/<ZONE NAME>/"

Ok, that should give us a service principal that we can use when modifying our DNS records!

The next part is to create the required CRDs

Creating the CRD

I’m going to need 2 different CRDs for this solution. The first one is used to set up the credentials to use when modifying the DNS records, and the second is used to define what DNS records that should be created.

The first one CRD is called AzureDnsCredential, and is defined using a yaml spec that looks like this

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: azurednscredentials.demos.fearofoblivion.com
spec:
  scope: Cluster
  group: demos.fearofoblivion.com
  names:
    plural: azurednscredentials
    singular: azurednscredential
    kind: AzureDnsCredential
    shortNames:
    - adc
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          description: "A set of credentials to access Azure DNS"
          properties:
            spec:
              type: object
              properties:
                resourceGroup:
                  type: string
                dnsZone:
                  type: string
                subscriptionId:
                  type: string
                tenantId:
                  type: string
                clientId:
                  type: string
                clientSecret:
                  type: string

As you can see, it is a CRD for a kind called AzureDnsCredential. It’s placed in the demos.fearofoblivion.com group at a cluster level. And it contains 6 properties. The name of the DNS Zone, and Resource Group it is located in is defined using resourceGroup and dnsZone. And the credentials is defined using subscriptionId, tenantId, ´clientId´ and clientSecret

Note: Storing your credentials in a resource like this is definitely not be the best solution from a security point of view. It would be much better to use Azure managed identities for this. But to keep it simple, I decided to take a little shortcut here. If you want more information about how to set up managed identities together with Kubernetes, have a look here.

The second CRD is called DnsRecordset, without any real reference to Azure. This allows us to treat DNS records as provider agnostic resources, which they are. And in the future we could go and build support for managing DNS records using several different providers, without having to modify the CRD.

It looks like this

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: dnsrecordsets.demos.fearofoblivion.com
spec:
  scope: Cluster
  group: demos.fearofoblivion.com
  names:
    plural: dnsrecordsets
    singular: dnsrecordset
    kind: DnsRecordset
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          description: "A set of credentials to access Azure DNS"
          properties:
            spec:
              type: object
              properties:
                dnsZone:
                  type: string
                aRecords:
                  type: array
                  items:
                    type: object
                    properties:
                      name:
                        type: string
                      ttlSecs:
                        type: integer
                        default: 3600
                      ipAddresses:
                        type: array
                        items:
                          type: string
                          pattern: "^([0-9]{1,3}.){3}[0-9]{1,3}$"
                txtRecords:
                  type: array
                  items:
                    type: object
                    properties:
                      name:
                        type: string
                      ttlSecs:
                        type: integer
                        default: 3600
                      values:
                        type: array
                        items:
                          type: string
                cnameRecords:
                  type: array
                  items:
                    type: object
                    properties:
                      name:
                        type: string
                      ttlSecs:
                        type: integer
                        default: 3600
                      alias:
                        type: string

It is a bit long, I know, but that’s because it has a lot more things being defined than the credentials. Not in the way of more configurable properties, but the properties are defined with limitations and defaults to make sure that the values we set are at least the right format.

The first property, dnsZone defines the DNS zone that the records should be added to. A zone can be for example fearofoblivion.com, or blog.fearofoblivion.com. Records are then attached to this.

The second, aRecords defines an array of A record objects that should be added. Each object containing a name, a TTL (Time To Live) with a default of 3600, and an array of IP addresses. The IP addresses in turn, is also limited by a pattern that should make sure they have reasonable values…

The third property, txtRecords defines an array of TXT record objects that should be added. Each object containing a name, a TTL (with a default of 3600), and an array of text values to add.

And the final one, cnameRecords defines an array of CName record objects. Each one containing a name, a TTL and an alias.

That’s all we need for now, so we can go ahead and add them to the cluster using kubectl apply.

Note: I have decided to add these resources at a cluster level, allowing them to defined per cluster instead of per namespace. The reason for this is that I found DNS entries to be pretty global anyway. But I guess it would be possible to put them at namespace level as well.

Creating the controllers

With the resource definitions in place, we can start tackling the controllers that are responsible for doing the actual work. And just as in the last post about CRDs, I have decided to build them using ASP.NET Core.

The actual ASP.NET application is very basic. All it really does, is registering a few services and a couple of IHostedService implementations. The reason to use ASP.NET part is pretty much just to get a long lived host for the hosted services in this case. The important parts are the services and the 2 IHostedService implementations.

Registering the Kubernetes API client

The first service we need, is a Kubernetes API client. Luckily, that is pretty easy to set up.

We just need to install a NuGet package called KubernetesClient, and add the KubernetesClient that it provides using the service collection. However, there is a caveat…we need to wire it up a bit differently depending on whether or not we are running it locally (while building and debugging), or if it is running inside a cluster. This can be figured out quite easily though, by calling KubernetesClientConfiguration.IsInCluster().

So to figure out what Kubernetes client configuration to use, I use the following code

KubernetesClientConfiguration config;
if (KubernetesClientConfiguration.IsInCluster())
{
    config = KubernetesClientConfiguration.InClusterConfig();
    services.AddSingleton<ILeaderSelector, KubernetesLeaderSelector>();
}
else
{
    config = new KubernetesClientConfiguration { Host = "http://localhost:8001" };
    services.AddSingleton<ILeaderSelector, DummyLeaderSelector>();
}

As you can see, I choose different configs based on the context

Note: I am also adding a “leader selector” service to my service collection. I will get back to this in just a second!

Once I have the configuration, I could add the Kubernetes client to the service collection. However, in this case I wanted to use it together with the HttpClientFactory. So the set up became a little bit more complicated. Like this

services.AddHttpClient("K8s")
        .AddTypedClient<IKubernetes>((httpClient, serviceProvider) => new Kubernetes(config, httpClient))
        .ConfigurePrimaryHttpMessageHandler(config.CreateDefaultHttpClientHandler)
        .AddHttpMessageHandler(KubernetesClientConfiguration.CreateWatchHandler);

This will make sure that when a KubernetesClient is being injected, it will piggyback on the HttpClientFactory and handle HttpClient creation etc for us automatically.

Leader Selection

As mentioned in my previous post, when running controllers that monitor CRDs, it is often very important that only a single controller monitors the resources. Otherwise we will end up with several controllers trying to do things every time a resource is added or modified. To solve this, I have simply decided to use a pre-existing image called leader-elector. This allows us to deploy a sidecar container that we can call to see if the current pod has been elected leader or not. If it hasn’t, we don’t need to monitor the resources.

Note: I am using a not so official version of it, because the official one has an issue that causes it to not work properly… Because of this, I’m using the image called fredrikjanssonse/leader-elector:0.6

The leader selector pod is added as a sidecar to our ASP.NET Core container by using a Kubernetes deployment that looks like this

apiVersion: apps/v1
kind: Deployment
metadata:
  name: azure-dns-controller
spec:
  replicas: 2
  selector:
    matchLabels:
      app: azure-dns-controller
  template:
    metadata:
      labels:
        app: azure-dns-controller
    spec:
      containers:
      - name: azure-dns-controller
        image: zerokoll.azurecr.io/azure-dns-controller:1.0
        ports:
        - containerPort: 80
        resources:
            limits:
              memory: 128Mi
              cpu: 250m
      - name: leader-election
        image: fredrikjanssonse/leader-elector:0.6
        args:
        - --election=foo-election
        - --http=0.0.0.0:4040
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 4040
        resources:
            limits:
              memory: 128Mi
              cpu: 250m

The leader election is then available in code by using an implementation of a very simple interface called ILeaderSelector. It looks like this

public interface ILeaderSelector
{
    Task<bool> IsLeader();
}

For local development and debugging, I have a dummy implementation that just returns true, and for in-cluster use, I have an implementation that that queries the sidecar using an HttpClient like this

public async Task<bool> IsLeader()
{
    var client = _httpClientFactory.CreateClient();
    string response;
    try
    {
        response = await client.GetStringAsync("http://localhost:4040");
    }
    catch (Exception ex)
    {
        return false;
    }
    return response.Contains(Environment.MachineName);
}

Very simple, but it works!

Talking to Azure DNS

To be able to talk to Azure DNS, I have created interface that looks like this

public interface IDnsClient
{
    Task AddARecordset(string name, int ttlSeconds, string[] ipAddresses);
    Task AddTxtRecordset(string name, int ttlSeconds, string[] values);
    Task AddCnameRecordset(string name, int ttlSeconds, string value);
    Task RemoveARecordset(string name);
    Task RemoveTxtRecordset(string name);
    Task RemoveCNameRecordset(string name);
}

However, there is a little caveat! The DNS client credentials will depend on what DNS zone we are trying to modify. So we can’t just get a DNS client injected, instead I’ve added factory service that can create one for us. It looks like this

public interface IDnsClientFactory
{
    bool TryGetClient(string dnsZone, out IDnsClient client);
}

This allows the system to create a new DNS client based on the DNS zone.

The implementation of the factory very much depends on what credentials that have been configured in the cluster. And to keep track of this, I decided to simply add a ConcurrentDictionary<string, AzureDnsCredentialSpec> to the service collection. This dictionary can then be used by the factory to figure out if there are any credentials for the requested DNS zone.

The implementation is pretty simple, and looks like this

public class AzureDnsClientFactory : IDnsClientFactory
{
    private readonly ConcurrentDictionary<string, AzureDnsCredentialSpec> _credentials;

    public AzureDnsClientFactory(ConcurrentDictionary<string, AzureDnsCredentialSpec> credentials)
    {
        _credentials = credentials;
    }

    public bool TryGetClient(string dnsZone, out IDnsClient client)
    {
        client = default;

        if (!_credentials.TryGetValue(dnsZone, out var creds))
        {
            return false;
        }

        client = new AzureDnsClient(creds);

        return true;
    }
}

As for the AzureDnsClient, that is not really the focus of this post. But in general it uses a couple of NuGet packages called Microsoft.Azure.Management.Dns and Microsoft.Rest.ClientRuntime.Azure.Authentication to talk to Azure to make the requested changes.

Note: If you are interested in the code for the DNS client, you can find it here

Ok, those are all the services I need to be honest! The only thing left now, are the two IHostedServices that will do the actual work!

Creating “generic” a CRD Controller

As there are 2 hosted services that will both talk to the Kubernetes API, and do a lot of VERY similar stuff, I decided to build a base class called KubernetesResourceController<T>. This is a base class that implements IHostedService, and allows any inheriting classes to easily listen to resource changes.

It has a constructor that takes a Kubernetes client, an ILeaderSelector and a logger, which are all stored for later use. Then is goes ahead and implements the StartAsync method from the IHostedService interface.

In this method, it starts a timer that goes off every 10 seconds. Inside the timer callback, it checks to see if it is the current leader. If not, it just returns, and waits for the timer to go off again. But if it is, it cancels the timer and calls a method called OnPromotedToLeader(). It looks like this

public Task StartAsync(CancellationToken cancellationToken)
{
    _leaderCheckTimer= new Timer(async x =>
    {
        var isLeader = await _leaderSelector.IsLeader();
        if (isLeader)
        {
            leaderCheckTimer.Dispose();
            leaderCheckTimer = null;
            OnPromotedToLeader();
        }

    }, null, TimeSpan.FromMilliseconds(0), TimeSpan.FromSeconds(10));

    return Task.CompletedTask;
}

The interesting stuff is all in the OnPromotedToLeader() method. And even that isn’t really that interesting to be honest. All it really does, is that it creates a Watcher<T> by using IKubernetesClient.ListClusterCustomObjectWithHttpMessagesAsync like this

private void OnPromotedToLeader()
{
    var response = Kubernetes.ListClusterCustomObjectWithHttpMessagesAsync(Group, Version, Plural, watch: true);
    _watcher = response.Watch((Action<WatchEventType, T>)(async (type, item) => await OnResourceChange(type, item)));
}

Ok…so is all the really cool and interesting stuff in the OnResourceChange method? Well, I guess so, but that method is abstract. So it is up to the inheriting classes to decide what should happen when a resource is added, deleted or changed.

Note: Where did the Group, Version, Plural properties come from in the previous code? Well, there are abstract string properties that the inheriting class defines to let the base class know what resource type it is interested in.

The final part is the StopAsync, which pretty much just stops the timer if needed, and disposes the Watcher<T> if it has been created.

public Task StopAsync(CancellationToken cancellationToken)
{
    if (_leaderCheckTimer != null)
        _leaderCheckTimer.Dispose();

    if (_watcher != null)
        _watcher.Dispose();

    return Task.CompletedTask;
}

Creating the AzureDnsCredentials controller

Now that the base class is in place, we can start focusing on the actual controllers. First up is the AzureDnsCredentialsController. It inherits from KubernetesResourceController<AzureDnsCredential>, and is notified whenever a AzureDnsCredential is changed in the cluster.

public class AzureDnsCredentialsController : KubernetesResourceController<AzureDnsCredential> {
	// Implemetation
}

So what is a AzureDnsCredential? Well, it is a simple DTO that we use to deserialize the JSON that comes from the API. It looks like this

public class AzureDnsCredential : KubernetesObject
{
    public const string Group = "demos.fearofoblivion.com";
    public const string Version = "v1";
    public const string Plural = "azurednscredentials";

    public V1ObjectMeta Metadata { get; set; }
    public AzureDnsCredentialSpec Spec { get; set; }

    public class AzureDnsCredentialSpec
    {
        public string SubscriptionId { get; set; }
        public string TenantId { get; set; }
        public string ClientId { get; set; }
        public string ClientSecret { get; set; }
        public string DnsZone { get; set; }
        public string ResourceGroup { get; set; }
    }
}

As you can see, it is a very simple object that exposes the values returned from the Kubernetes API. And to keep me from having to type too much, I am piggybacking on the KubernetesObject and V1ObjectMetaData types from the KubernetesClient library. Besides the built-in metadata property, I expose a type that corresponds to the spec property that was defined in the AzureDnsCredential CRD.

And to get away from “magic strings”, I expose the custom resource definition name values as constants. This allows the controller implement the abstract CRD info strings using typed values instead of strings. Like this

protected override string Group => AzureDnsCredential.Group;
protected override string Version => AzureDnsCredential.Version;
protected override string Plural => AzureDnsCredential.Plural;

The constructor for the AzureDnsCredentialsController accepts a few different services

public AzureDnsCredentialsController(IKubernetes kubernetes, ILeaderSelector leaderSelector, ILoggerFactory loggerFactory, ConcurrentDictionary<string, AzureDnsCredentialSpec> credentials)
    : base(kubernetes, leaderSelector, loggerFactory.CreateLogger<KubernetesResourceController<AzureDnsCredential>>())
{
    _logger = loggerFactory.CreateLogger<AzureDnsCredentialsController>();
    _credentials = credentials;
}

The only thing here that is of real interest is the ConcurrentDictionary<string, AzureDnsCredentialSpec>. This is the same dictionary that is passed to the AzureDnsClientFactory. This allows the controller to add AzureDnsCredentialsSpec objects to the dictionary, and have them picked up by the factory when creating IDnsClient instances.

The actual work in this controller is done in the OnResourceChange method. In here, the credentials dictionary is modified as needed when credentials are added, modified or deleted. Like this

protected override Task OnResourceChange(WatchEventType type, AzureDnsCredential item)
{
    switch (type)
    {
        case WatchEventType.Added:
        case WatchEventType.Modified:
            _credentials[item.Spec.DnsZone] = item.Spec;
            break;
        case WatchEventType.Deleted:
            _credentials.Remove(item.Spec.DnsZone, out var _);
            break;
    }

    return Task.CompletedTask;
}

That’s it! This controller will now watch the Kubernetes API for any changes among the AzureDnsCredential resources, and then add, remove or update the dictionary to keep it up to date at all times.

Creating the DnsRecordsetController

The final part is the most complicated one, and that is the DnsRecordsetController. This is another Kubernetes controller, just like the one we just looked at. However, this one is responsible for performing the required DNS changes caused by the addition or removal of DnsRecordset resources.

Once again, the controller inherits from KubernetesResourceController<T>, but this time the T is set to DnsRecordset. And instead of requesting an extra dictionary in the constructor, it requests an implementation if IDnsClientFactory, which is then used to create clients that enable it to communicate with the DNS back-end when needed.

The DnsRecordset is pretty much just a dumb DTO, just like AzureDnsCredential. However, it does have a little trick up it’s sleeve. Besides the values required to deserialize the JSON from the API, it also has a property called Status. This is used to keep track of the “status” of the current record set.

public const string StatusAnnotationName = Group + "/status";

public Statuses Status => Metadata.Annotations.ContainsKey(StatusAnnotationName) ? 
                                Enum.Parse<Statuses>(Metadata.Annotations[StatusAnnotationName]) : 
                                Statuses.Unknown;

public enum Statuses
{
    Unknown,
    RetryingCreate,
    RetryingRemove,
    Creating,
    Removing,
    Done
}

As you can see, it uses an annotation from the resource’s metadata to keep track of the current “status”. This allows us to keep track of the resource “status” between controller instances, in case the current leader for some reason is replaced, and a new controller needs to figure out what is happening.

In the OnResourceChange method, the change type and status is checked to figure out what needs to be done. Like this

protected override async Task OnResourceChange(WatchEventType type, DnsRecordset item)
{
    switch (type)
    {
        case WatchEventType.Added:
            switch (item.Status)
            {
                case DnsRecordset.Statuses.Unknown:
                case DnsRecordset.Statuses.Creating:
                case DnsRecordset.Statuses.RetryingCreate:
                    await RecordsetAdded(item);
                    return;
                case DnsRecordset.Statuses.Removing:
                case DnsRecordset.Statuses.RetryingRemove:
                    await RecordsetRemoved(item);
                    return;
            }
            break;
        case WatchEventType.Modified:
            _logger.LogWarning("Modifications are not supported at the moment");
            break;
        case WatchEventType.Deleted:
            await RecordsetRemoved(item);
            break;
    }
}

As you can see, if a record set is added, the status defines whether or not it still needs to be added to the DNS, or if it should potentially be removed.

Note: WatchEventType.Added doesn’t mean that it was just added to the cluster. Instead, it means that it was added at some point, and the current controller has not been notified of it yet. This means that any resource that is already in place when this controller comes on-line, will be sent to the controller with the type WatchEventType.Added. No matter how long ago it was really added…

Caveat: For simplicity, I have decided to not to implement the ability to modify a resource, and focused on add and remove. Modifications is much harder to figure out…

Ok, so now that we know how it figure out if a record set should be added or not, what does the implementation of it look like? Well, it is pretty mundane and repetitive. But let’s have a look anyway…

The RecordsetAdded method starts out by adding the new record set to an internal list of record sets. Then it goes ahead and checks to see if the Status has been set to Done. In that case, there is nothing that needs to be done, so it can just return…

private async Task RecordsetAdded(DnsRecordset recordset)
{
    _recordsets[recordset.Metadata.Name] = recordset;

    if (recordset.Status == DnsRecordset.Statuses.Done)
    {
        return;
    }

    …
}

Next, it uses the IDnsClientFactory.TryGetClient to see if it can get hold of a DNS client for the defined DNS Zone. If not, it sets up a timer to try a bit later, and then returns. There isn’t really a reason to do anything else if we can’t talk to the DNS back-end.

private async Task RecordsetAdded(DnsRecordset recordset)
{
    …
    if (!_dnsClientFactory.TryGetClient(recordset.Spec.DnsZone, out var client))
    {
        _logger.LogWarning($"Missing credentials for zone {recordset.Spec.DnsZone}. Retrying...");
        await SetUpAddRetry(recordset);
        return;
    }
    …
}

private async Task SetUpAddRetry(DnsRecordset recordset)
{
    await UpdateRecordsetStatus(recordset, DnsRecordset.Statuses.RetryingCreate);
    new Timer(async x => await RecordsetAdded(recordset), null, TimeSpan.FromSeconds(10), TimeSpan.FromMilliseconds(-1));
}

Oh, yeah… As you can see, it also sets the status of the record set to RetryingCreate so that we can keep track of what is happening!

If it does get a DNS client, it sets the status to Creating, and then loops through each one of the arrays of records to add, and adds them to the Azure DNS one at the time.

This code is VERY repetitive, so I will just show you the code for the A records, which looks like this

private async Task RecordsetAdded(DnsRecordset recordset)
{
    …
    await UpdateRecordsetStatus(recordset, DnsRecordset.Statuses.Creating);

    foreach (var record in recordset.Spec.ARecords)
    {
        try
        {
            await client.AddARecordset(record.Name, record.TtlSecs, record.IpAddresses);
        }
        catch (Exception ex)
        {
            await SetUpAddRetry(recordset);
            return;
        }
    }
    …
}

As you can see, if it fails to add a record, it sets up a retry timer and returns. This will cancel the addition of the remaining records for now, and start over in a little while again.

Comment: This might not be the most graceful way to sort of errors, but it does work pretty well for intermittent problems. Especially since the Azure client is idempotent.

Once all the records have been added, the status is set to Done

private async Task RecordsetAdded(DnsRecordset recordset)
{
    …
    await UpdateRecordsetStatus(recordset, DnsRecordset.Statuses.Done);
}

That’s it for the addition of record sets!

The RecordsetRemoved method is extremely similar! However, as it isn’t filtering by status in the OnResourceChange it has to start out with a little filter like this

private async Task RecordsetRemoved(DnsRecordset recordset)
{
    if (recordset.Status != DnsRecordset.Statuses.Done)
    {
        _recordsets.Remove(recordset.Metadata.Name, out var _);
        return;
    }
    ….
}

But as I said, the rest of it is almost identical to the RecordsetAdded method, except for it calling RemoveXXX instead of AddXXX on the DNS client. And because of this, I won’t add all the code here…

Conclusion

That’s it! Adding this to your Kubernetes cluster will allow you to manage your Azure DNS entries through Kubernetes! In a very crude way… So there is obviously potential for a lot more functionality, as well as some areas that should probably have a lot more defensive coding, but hopefully it shows how creating your own CRDs could be a really cool and useful way to extend your cluster’s functionality.

The full code is available at https://github.com/ChrisKlug/k8s.demos.crd.azuredns.controller if you want to have a look through it!

Azure Service Operator for Kubernetes

Microsoft recently (at the time of writing) announced Azure Service Operator for Kubernetes. This is an open source project for managing Azure resources using Kubernetes CRDs. I believe the implementation is very similar to what you have just seen. But hopefully with more defensive programming, better error handling, better security etc… And with support for a LOT more resource types. So if you want to manage Azure resources through Kubernetes, I recommend looking at that project before you build your own. However, it is still good to understand how it works under the hood. Which you should after reading this post!