Differences in Azure DB and LocalDB in Integration Tests

We use Azure SQL databases in production and in our test environments, but for our integration tests, we use LocalDB by deploying a DACPAC to LocalDB and backing it up for reuse. The problem is that there are edge cases that are not caught due to differences between Azure SQL databases and LocalDB databases, particularly around collation.

We had a column in a table with a custom collation. With the wrong Entity Framework (EF) configuration, this didn’t work in Azure but worked in our integration tests. You do not want integration tests that work differently than production for obvious reasons. The solution was to set the LocalDB to Containment Partial. This way, collation is handled entirely by the database instance, and no fallback to default server collation will occur.

The code in our case:

public static async Task InitBackup(string systemName, IEnumerable<KeyValuePair<string, string>>? sqlCommandVariableValues = null)
{
    _systemName = systemName;
    var dacPackPath = GetDacPackPath();
    var dacMd5 = DacPacFingerprint(dacPackPath);

    _backup = $"{SystemName}_{dacMd5}.bak";

    try
    {
        SetBackupInfo();
    }
    catch (SqlException)
    {
        var backupFile = GetUserProfilePath(_backup);
        if (File.Exists(backupFile)) File.Delete(backupFile);

        var instance = new DacServices(string.Format(ConnectionStringTemplate, "master"));
        var options = new DacDeployOptions();
        sqlCommandVariableValues?.ForEach(kvp => options.SqlCommandVariableValues.Add(kvp.Key, kvp.Value));

        var tempDb = $"dacpac_{SystemName}_{Guid.NewGuid().ToString().Replace("-", "_")}";

        using (var dacPak = DacPackage.Load(dacPackPath))
            instance.Deploy(dacPak, tempDb, false, options);

        await ExecuteASync("sp_configure 'contained database authentication', 1;RECONFIGURE;");
        await ExecuteASync($"ALTER DATABASE [{tempDb}] SET CONTAINMENT = PARTIAL WITH NO_WAIT");

        var sql = $@"BACKUP DATABASE {tempDb}  
            TO DISK = '{_backup}'";
        await ExecuteASync(sql);
        SetBackupInfo();

        _ = Task.Run(() => Kill(tempDb));
    }
}

The important part is enabling contained authentication, which is required for partial containment, and then enabling partial containment for the database.

Parallel Test Execution Locally, Serial Execution on Build Agents

We encountered an issue where our tests were consuming too many resources on our build agents due to parallel execution. However, we wanted to keep the tests parallelized to reduce execution time locally. If you place a .runsettings file with parallelism enabled in the solution (.sln) folder, all solutions in that folder will automatically have parallelism enabled. You then create a buildagent.runsettings file, specified in the build pipeline, that executes tests sequentially.

The problem with this approach is that the auto settings apply to all solutions in the folder, which didn’t work well for us. We have legacy solutions that can’t run in parallel. So, my approach was to instead use an attribute in the test assemblies that supports parallelism:

[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.MethodLevel)]

This enables parallelism locally and uses all available logical CPUs. For the build agents, I created a custom buildagent.runsettings file:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <RunConfiguration>
    <MaxCpuCount>0</MaxCpuCount>
  </RunConfiguration>
  <MSTest>
    <Parallelize>
      <Workers>1</Workers>
      <Scope>MethodLevel</Scope>
    </Parallelize>
  </MSTest>
</RunSettings>

This setup overrides parallelism when called with dotnet test –settings buildagent.runsettings on the build agents.

Trace logging with exception in Application Insights

By default, Masstransit logs retries as warnings if they are resolved by the retry process; otherwise, they are logged as errors. Additionally, Masstransit provides the exception in both cases for tracing purposes.

However, there’s a potential confusion when using Application Insights: it logs every entry with a supplied exception as an exception with an Exception severity level. Unfortunately, the severity level isn’t a presented field in the AI user interface, making it easy to overlook. Both our sysops and we developers misinterpret these logged entries as actual exceptions, which led to unnecessary investigation and troubleshooting.

To mitigate this confusion, remember that you can filter out these non-exceptional logs by specifying the severity level. Specifically, filtering for “Error” severity level will narrow down the logs to actual exceptions.

I hope this blog post can save others some valuable time and prevent unnecessary confusion and alarm. Happy coding!

Getting rid of the slow Masstransit test harness

I wrote a blog about replacing the timeout based test harness with a semaphore driven one here. This made things much more robust when you want blackbox type testing, fire a number of events and wait until all have been digested and their spawned child events are digested.

This worked well and robust. But it still used the Masstransit harness for hosting. This made the InMemory bus more than twice as slow as hosting Masstransit in a service, including database I/O so probably a lot slower when only looking at bus performance.

But it’s pretty easy hosting Masstransit from a none service project like a test project. Instead of configuring with AddMassTransitTestHarness use the standard AddMasstransit extension method. Now events will not be consumed when you publish them, this is because the IHostedService haven’t been started. So that’s an easy fix. If we base the code on the IHarness from my previous blog post. 

public Harness(IEnumerable<IHostedService> services)
{
    _services = services;
}

public async Task Start()
{
    var source = new CancellationTokenSource();

    foreach (var service in _services)
        await service.StartAsync(source.Token);
}

public async Task Stop()
{
    var source = new CancellationTokenSource();

    foreach (var service in _services)
        await service.StopAsync(source.Token);
}

Call Start from your test setup and stop from your test teardown. This will start the background workers for Masstransit and make sure it listens and consumes events. The service will not work unless you add logging to your IoC config. 

new ServiceCollection()
    .AddLogging();

Coupled with the harness-code from previous blog post you now have a very robust and fast test harness. Full code below

(more…)

Test .NET 6 (Or any Core version) from legacy .NET Framework

I’m currently working on moving a large legacy system from .NET Framework 4.8 to .NET 6. Since this is a large system the move will take years and we need to work iteratively meaning both systems will co-exist over a few years.

This means that integration tests we already have for the legacy system now needs to execute code over two application domains running two completely different CLRs. I ended up making a little service that the legacy code can call to execute code in the new system.

First we create a new web API project with a single controller.

[ApiController]
[Route("[controller]")]
public class TestController : ControllerBase
{
    private static readonly Dictionary<Guid, IServiceProvider> ServiceProviders = new();

    public bool Get()
    {
        return true;
    }

    [HttpPost("SetupTest")]
    public async Task SetupTest(Guid testId, string connectionString)
    {

        var collection = new ServiceCollection();

        var provider = collection
            .AddCqs(configure => collection.AddMassTransitTestHarness(cfg =>
            {
                cfg.UsingInMemory((ctx, mem) =>
                {
                    mem.ConfigureTestHarness(ctx);
                    mem.AddOutbox(ctx);
                    mem.ConfigureEndpoints(ctx);
                });
                configure(cfg);
            }))
            .AddDbContext<PcDbContext>(b =>
            {
                b.UseSqlServer(connectionString);
            })
            .AddBusinessCore()
            .AddRepositories()
            .AddDomain()
            .AddTestHarness()
            .AddTestGuidFileRepository()
            .BuildServiceProvider();


        var harness = provider.GetRequiredService<IHarness>();
        await harness.Start();

        ServiceProviders.Add(testId, provider);
    }

    [HttpPost("TeardownTest")]
    public void TeardownTest(Guid testId)
    {
        ServiceProviders.Remove(testId);
    }


    private static readonly JsonSerializerOptions Options = new() { PropertyNameCaseInsensitive = true };

    [HttpPost("ExecuteCommand")]
    public async Task ExecuteCommand(Guid testId, string cmdType)
    {
        var provider = ServiceProviders[testId];

        var cmd = (await JsonSerializer.DeserializeAsync(HttpContext.Request.Body, Type.GetType(cmdType)!, Options))!;
        await provider.GetRequiredService<IBus>().Publish(cmd);
        await provider.GetRequiredService<IHarness>().WaitForBus();
    }
}
(more…)

A better Masstransit Test harness

At my latest customer project I choose to use Masstransit for events and Sagas. Its been a bumpy ride with outboxes and such, but now we have a pretty stable foundation to build upon. One problem have been testing. I like black box testing of our domain. Something like this.

[TestClass]
public class When_doing_a_complete_booking : BusinessTest
{
    private Booking _result;
    private DateTime _date;
  
    [TestInitialize]
    public void Context()
    {
        Guid bookingKey = Guid.Empty;
        _date = DateTime.Now.AddDays(5);
  
        _result = Given(db => /* Setup here */)
            .When(() => new SearchQuery{ Date = _date, ...})
            .And(result =>
            {
                bookingKey = result.First().BookingKey;
                return new ReserveCommand { BookingKey = bookingKey, ... };
            })
            .And(() => new ConfirmCommand
            {
                BookingKey = bookingKey, 
                ...
            })
            .Then(db => db.Set<booking>().FirstOrDefaultAsync(b => b.BookingKey == bookingKey));
    }
  
    [TestMethod]
    public void It_should_book_correctly ()
    {
        Assert.IsNotNull(_result);
        Assert.IsTrue(...);
    }
}

Masstransit harness really doesn’t support black box type testing. Chris Patterson favors a more unit testing-oriented approach, were you fire events and your Tests assert that events were consumed. You can await consumption with the built in harness, but its timeout oriented which makes it slow and unstable.

(more…)

Mock and callback a generic method with unavailable type using Moq

This is as much a note to self as a blog post. I needed to test a method that was dependent on a API with a callback action where T was scoped internal in the library that I was testing. Rather than making the type public I went ahead and tried to mock it anyway 😀

The code that we want to test is called from third party API like

await hubProxyFactory
            .Create(hubUrl, configureConnection, async p =>
                {
                    proxy = p;
                    await SendQueuedSubscriptions();
                    p.On<Message>("onEvent", OnEvent);
                },
                Reconnected, FaultedConnection, ConnectionComplete);

Message is a private type and OnEvent is of type Action<Message>. Its the OnEvent method we want to test. So we need to mock On<T> without knowing T and we need to save a reference to the OnEvent even though T is unknown. It looks like this

                    mock.Setup(x => x.On("onEvent", It.IsAny<Action<It.IsAnyType>>())).Callback(
                        (string e, MulticastDelegate callback) =>
                        {
                            onEvent = callback;
                        });
(more…)

SignalR event aggregation with Blazor WASM

As you probably know by now I love event aggregation and being able to seamless forward server side events to clients. My library SignalR.EventAggregatorProxy have been around for a while now and it enables seamless event aggregation between server and client. The .NET core client library just got updated to .NET 5. And as it turns out it works without any effort in Blazor WASM.

(more…)

Convention based Concurrency Management in Entity Framework Core

Who does not love convention over configuration? Whenever it makes sense I try to use it in my role as a system architect. It helps my programmers write more robust code out of the box.

Writing concurrency safe code is a corner stone in writing robust code today,  without it data quality can not be guaranteed. And when things go wrong you want to know who and when entities were updated so you can investigate what have gone wrong.

So what I did at my latest assignment was to force this onto the entities by convention using two markup interfaces ICreated and IUpdated.  (more…)

Using regex in Visual Studio

This is as much a note to self as a blog post :p

One developer in my team had done null == somevariable everywhere in our solution. I think unified code styles are important. So I wanted to remove them. Standard replace does not work here since we need to move the null after the variable name. So we need to use a regex.

Regex

We search for null and then any character as few as possible until we hit a space or a ending parenthesis. We then replace with group 1 ($1) which is the variable name == null and then group 2 ($2) which is either a space or a ending parenthesis.

This ensures above replace works with both if(null == somevar) and code like if(null == somevar && someotherstuff)

Update:
Here is a version on steroids that can do both equals and not equals
Regex v2