Azure Key Vault + MSI = failing Web API integration tests

Introduction

In a number of our ASP.NET Core Web APIs we’re using Azure Key Vault for keeping secrets such as connection strings and authentication credentials out of source control.

Initially we were using Azure Key Vault with a clientid and clientsecret, which are stored in appsettings.json like so:

  "KeyVault": {
    "Name": "MyApplicationDev",
    "ClientId": "abcdefg123-b292-4177-ba53-858227a9143c",
    "ClientSecret": "5m9g9cpuNc31abcJZcjkfP9/pDwJgQ+T82t/qCey7Nc="
  },

But this begs the question – what happens if the Key Vault ClientId and ClientSecret get compromised? To prevent this you can setup Key Vault to use Managed Service Identity (MSI). With that in place you don’t need to have the ClientId and ClientSecret in your appsettings.json, instead you only need the KeyVault url:

  "KeyVaultSettings": {
    "Url": "https://myapplicationdev.vault.azure.net"
  },

This is usually configured in the WebApi’s Program.cs, via something like:

public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
    WebHost.CreateDefaultBuilder(args)
        .ConfigureAppConfiguration((context, builder) =>
        {
            builder.SetBasePath(context.HostingEnvironment.ContentRootPath)
                    .AddEnvironmentVariables();

            var config = builder.Build();
            var tokenProvider = new AzureServiceTokenProvider();
            var keyvaultClient = new KeyVaultClient((authority, resource, scope)
               => tokenProvider.KeyVaultTokenCallback(authority, resource, scope));
                    
            builder.AddAzureKeyVault(config["KeyVaultSettings:Url"], keyvaultClient, new DefaultKeyVaultSecretManager());
        })
        .UseStartup<Startup>();

Under the covers, this will add an AzureKeyVaultConfigurationSource to the registered list of IConfigurationBuilders (as well as do other things).

The problem

In ASP.NET 2.1 we are using the new WebApplicationFactory<T> for running integration tests against an in-memory TestServer. When you run these tests locally it’ll use your (i.e. you, the developer’s) AD credentials to authenticate against the Key Vault, and so the tests should pass if you have access to the Key Vault. However, when the tests run on the Build server (as part of your CI pipeline) then they’ll probably fail because the Build agent does not have access to the Key Vault.

My solution

Firstly, I didn’t want to change the code of the system under test (i.e. the Web API), i.e. by adding configuration to determine whether to use Key Vault or not. Although thinking about it, that might have been easier! But it feels a bit dirty to change the application code to make integration tests easier.

The approach I took was to remove the AzureKeyVaultConfigurationSource from the list of IConfigurationBuilders from the system under test, in my custom WebApplicationFactory<T>, i.e:

protected override void ConfigureWebHost(IWebHostBuilder builder)
{
    builder.ConfigureAppConfiguration((_, configurationBuilder) =>
    {
        var keyVaultSource = configurationBuilder.Sources.FirstOrDefault(cs => cs.GetType().Name == "AzureKeyVaultConfigurationSource");
        if (keyVaultSource != null)
        {
            configurationBuilder.Sources.Remove(keyVaultSource);
        }
    });
}

This way, the code in the system under test is unchanged, instead we are just changing the configuration of the TestServer prior to it starting. Hope that helps someone.