E2E Testing Azure Functions

Scenario

Recently I was working on a solution which included multiple Azure Function apps, with unit tests (xUnit) giving rapid feedback for discrete components across the solution in isolation. But the next step was to create some automated end-to-end (E2E) tests that exercised the behaviours of the solution as a whole.

The goal was to be able to get that feedback as early on in the development process as possible which meant being able to run those E2E tests against a local build on a dev machine when needed with minimal hassle, and in an Azure DevOps CI/CD pipeline, without requiring the Azure Functions to be deployed first. This means being able to run tests against a local build of the functions running via the Azure Functions Core Tools.

In my particular scenario, I wanted slightly different settings for the E2E tests from the ones in local.settings.json used during development.

Hat tip

First and foremost, I found a really helpful blog post series by David Guida (Twitter):
Testing Azure Functions on Azure DevOps Part 1 - Setup
Testing Azure Functions on Azure DevOps Part 2 - The Pipeline

This got me on the way to where I needed to be, so I’d recommend checking out those posts first of all.

To summarize the key points:

spawn a new process in an xUnit Collection Fixture that starts the Azure Function app up using the Azure Function Core Tools
have a separate version of local.settings.json file for the E2E tests, and replace the main dev local.settings.json file with it when before running the tests

Extra concerns

The main differences in my case, were:

I had multiple Azure Function apps in the same solution that needed to be running
I wanted to make it seamless for all devs in the team to be able to run the E2E tests locally without manually editing or swapping around local.settings.json files

I also encountered some other concerns I wanted to take care of, including:

waiting for the functions to be fully up and running before the tests start running
having the option to see the command windows for the functions to view the output just like during development (useful for debugging issues)

Full E2E test project example

I’ve created a sample solution in my AzureFunctions-E2ETest-Example GitHub repo that covers a full working example based on everything we’ll cover in this blog.

The repo contains a solution with 2 Azure Function app projects and an xUnit test project. To try it out for yourself, you just need to:

clone/download the repo
have the Azure Storage Emulator/Azurite running
set the variables at the top of the AzureFunctionsFixture class, to the appropriate local paths for your machine.

When you run the tests, the following workflow happens:

the xUnit Fixture starts up the 2 Azure Function apps (and waits for them to fully start)
the xUnit test sends an HTTP request to an endpoint running in SampleFunctionApp1, which will add the item to a queue (“AdaTheDevE2ETestQueue”)
a queue trigger function in SampleFunctionApp2 then consumes from that queue and stores the item in an Azure Table Storage (“AdaTheDevE2ETestTable”)
the xUnit test will check for the existence of that record in Table Storage

While the setup is based on being able to run against a local build for the purposes of this post, the E2E tests could still be run against a deployed environment, with some tweaks to the Fixture class to support extra settings to determine whether the processes need to be spun up or not for a local build.

In the rest of this blog post, we’ll dive into some of the key bits relating to the differences and extra concerns I mentioned above.

Code snippet 1: AzureFunctionProcess

To support my goals and make it easy to reuse, I wrapped the core functionality for spinning up an Azure Function app with the Azure Functions Core Tools, into a AzureFunctionProcess class:

using System;
using System.Diagnostics;
using System.Threading;

namespace AdaTheDev.AzureFunctionsE2ETests
{
    public class AzureFunctionProcess : IDisposable
    {
        private Process _funcHostProcess;
        private bool disposed;
        private bool _funcHostIsReady;        
        private readonly bool _useShellExecute;

        public AzureFunctionProcess(
            string dotnetExePath,
            string functionHostPath,
            string functionAppFolder,
            int port,
            bool useShellExecute = true)
        {
            _funcHostProcess = new Process
            {
                StartInfo =
                {
                    FileName = dotnetExePath,
                    Arguments = $"\"{functionHostPath}\" start -p {port}",
                    WorkingDirectory = functionAppFolder,                    
                    UseShellExecute = useShellExecute,
                    RedirectStandardOutput = !useShellExecute                    
                }
            };
            _useShellExecute = useShellExecute;
        }
        
        public void Start(int timeoutSeconds = 15)
        {
            _funcHostProcess.OutputDataReceived += _funcHostProcess_OutputDataReceived;

            try
            {
                _funcHostProcess.Start();

                var stopwatch = Stopwatch.StartNew();

                if (!_useShellExecute)
                {
                    _funcHostProcess.BeginOutputReadLine();

                    while (!_funcHostProcess.HasExited && !_funcHostIsReady && stopwatch.ElapsedMilliseconds < (timeoutSeconds * 1000))
                    {
                        Thread.Sleep(1000);
                    }
                }
                else 
                {
                    Thread.Sleep(timeoutSeconds * 1000);
                    _funcHostIsReady = true;
                }
            }
            finally
            {
                _funcHostProcess.OutputDataReceived -= _funcHostProcess_OutputDataReceived;
            }

            if (!_funcHostIsReady)
            {
                throw new InvalidOperationException("The Azure Functions host did not start up within an acceptable time.");
            }
        }
        
        private void _funcHostProcess_OutputDataReceived(object sender, DataReceivedEventArgs e)
        {
            if (e.Data?.Contains("For detailed output, run func with --verbose flag") ?? false)
            {
                _funcHostIsReady = true;
            }
        }

        protected virtual void Dispose(bool disposing)
        {
            if (!disposed)
            {
                if (disposing)
                {
                    if (_funcHostProcess != null)
                    {
                        if (!_funcHostProcess.HasExited)
                        {
                            _funcHostProcess.Kill();
                        }

                        _funcHostProcess.Dispose();
                    }
                }
                
                disposed = true;
            }
        }
        public void Dispose()
        {            
            Dispose(disposing: true);
            GC.SuppressFinalize(this);
        }
    }
}

There is an AzureFunctionProcessFactory class that can create AzureFunctionProcess instances.

Running multiple Azure Function Apps in E2E tests

Inside the AzureFunctionsFixture, we can now do the following to start both of the function apps up:

var processFactory = new AzureFunctionProcessFactory(dotnetExePath, functionHostPath);
						
try
{
    ...

    _functionApp1Process = processFactory.Create(functionApp1Folder, functionApp1Port);
    _functionApp1Process.Start();

    _functionApp2Process = processFactory.Create(functionApp2Folder, functionApp2Port);
    _functionApp2Process.Start();
}
catch
{
    _functionApp1Process?.Dispose();
    _functionApp2Process?.Dispose();
    ...
    throw;
}

Note, the paths to the function app folders (functionApp1Folder/functionApp2Folder) can either be:
a) the project directory - in which case the project will be built at test runtime as part of the Fixture
b) the build output directory under the bin\{configuration}\ - in which case, it just starts up the pre-built functions

I’m using option a) which means there can be a longer startup time for the functions, which leads us to the next section.

Automatically detecting when Azure Functions have started

Essentially, we need to introduce a delay in the xUnit Fixture to give the functions enough time to start up fully. But how much time? 1 second? 10 seconds?

The good news is we don’t need to take a guess at how long is long enough, as we can make use of the RedirectStandardOutput setting on the Process, and check for when a particular bit of output text from the Function App process is written out that indicates the functions are up and running.

In order to use this approach, you have to ensure UseShellExecute is false (as it is by default in this implementation).

This is all handled in AzureFunctionProcess class, as shown in the full class snippet further up.

Enabling command window for output debugging

I found on occassion, it was useful to be able to see the command windows for the functions in order to view the output just like when running them during normal development.

There is support for that by passing a third argument to the AzureFunctionProcessFactory.Create method:

_functionApp1Process =
    processFactory.Create(functionApp1Folder, functionApp1Port, useShellExecute: true);

In this scenario, as we can’t get access to the output, it will fall back to a cruder approach of just waiting for the full timeout duration to pass before carrying on.

Automatically swapping in E2E test specific local.settings.json for local runs

Firstly, we can create a local-e2etests.settings.json file alongside the main local.settings.json file for development in the function app project.

When we spawn the function app processes in the xUnit Fixture, we:

create a backup copy of its local.settings.json file with a unique name
overwrite local.settings.json with the contents of local-e2etests.settings.json

Then in the xUnit Fixture Dispose method we:

restore the original copy of local.settings.json from the backup
clean up the backup copy

This means that local.settings.json used for development aren’t lost - important because they’re not committed to source control (except in the case of my sample repo, for full example purposes). Plus, it removes the need to manually fiddle around and replace local.settings.json each time when you want to run the E2E tests locally.

In the sample repo, this is handled using the LocalSettingsSwapper class, which is used in the xUnit Fixture setup.

Closing remarks

While it is focused on xUnit, it shouldn’t really matter if you’re using a different test framework, as the fundamentals don’t change and can be adapted easily enough to suit the specifics of whichever framework you are using.

Hopefully the full example project will be useful to you, on your journey towards creating E2E tests for your Azure Functions that can be run on a local build via the Azure Functions Core Tools - whether that be in a CI/CD pipeline or on a local dev machine.