dotnet distributed redis cache with in memory fallback

Usually whenever we are talking about distributed cache we are talking about redis and this implementation

Imagine that you have two instances of your app and single instance of redis

Then it will be rather single point of failure rather than distributed cache

How about having local in memory fallback, so in case if Redis is down, application still may work

In general idea is following (pseudocode):

public class Cache {
    public T Get<T>(string key) {
        return _local.Get(key) ?? _remote.Get(key);
    }

    public void Set<T>(string key, T value) {
        _local.Set(key, value)
        _remote.Set(key, value)
    }

    public void Remove(string key) {
        _local.Remove(key);
        _remote.Remove(key);
    }
}

Where _remote is our redis and _local is our in memory cache.

With such approach our app may survive even if whole redis cluster will be down

Prerequisite Redis notifications

There is only two real problems: how to name things and how to invalidate caches

Redis has build in notification mechanism no notify its clients about changes in database

For it to work you should provide notify-keyspace-events with some value, in our case it will be AKE, more detail can be found here

Here is an sample of how it works

Start redis:

docker run -it --rm --name=redis -p 6379:6379 redis --notify-keyspace-events AKE

Listen for changes:

docker exec -it redis redis-cli psubscribe '__key*__:*'

Perform some actions:

docker exec redis redis-cli set foo bar EX 2
docker exec redis redis-cli set acme 42 EX 2

In subscriber you should receive events like:

1) "pmessage"
2) "__key*__:*"
3) "__keyevent@0__:expired"
4) "acme"

So redis will may notify us whenever something is expired, changed or removed

It is important for us, because imagine the following situation:

app instance 1 caches data and stores it in local memory and redis
app instance 2 also has local and remote cache
app 2 decides to invalidate cache and did remove it from local and remote caches
but app 1 still has it in local cache and will serve it

To fix this issue we may utilise built in redis subscriber, here is pseudo code of how it works:

public class RedisSubscriber {
    private readonly IMemoryCache _local;

    public RedisSubscriber(IConnectionMultiplexer connection, IMemoryCache cache)
    {
        var subscriber = connection.GetSubscriber();
        subscriber.Subscribe("__keyevent@0__:expired", (channel, key) => _local.Remove(key));
        subscriber.Subscribe("__keyevent@0__:del", (channel, key) => _local.Remove(key));
        subscriber.Subscribe("__keyevent@0__:set", (channel, key) => _local.Remove(key));
    }
}

So we are now syncing changes between all instances and whenever any app decide to invalidate cache it will be propagated to everyone

Drop in replacement

Here is challange and real fun, what if our implementation will be a drop in replacement for distributed cache, so can be easily swapped

To do so we need to pretend like we are yet another provider, but underneath instantiate both in memory and redis caches

using System.Threading;
using System.Threading.Tasks;

using Microsoft.Extensions.Caching.Distributed;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Caching.StackExchangeRedis;

namespace StackExchangeRedisWithFallback;

public class RedisDistributedCache : IDistributedCache
{
    private readonly RedisCache _remote;
    private readonly IMemoryCache _local;

    public RedisDistributedCache(RedisCache remote, IMemoryCache local)
    {
        _local = local;
        _remote = remote;
    }

    public byte[]? Get(string key)
    {
        // CONSIDER: if wound in remote cache - save to local?
        return _local.Get<byte[]?>(key) ?? _remote.Get(key);
    }

    public async Task<byte[]?> GetAsync(string key, CancellationToken token = default)
    {
        // CONSIDER: if wound in remote cache - save to local?
        return _local.Get<byte[]?>(key) ?? await _remote.GetAsync(key, token);
    }

    public void Set(string key, byte[] value, DistributedCacheEntryOptions options)
    {
        _local.Set(key, value, Convert(options));
        _remote.Set(key, value, options);
    }

    public async Task SetAsync(string key, byte[] value, DistributedCacheEntryOptions options, CancellationToken token = default)
    {
        _local.Set(key, value, Convert(options));
        await _remote.SetAsync(key, value, options, token);
    }

    public void Refresh(string key)
    {
        _remote.Refresh(key);
    }

    public async Task RefreshAsync(string key, CancellationToken token = default)
    {
        await _remote.RefreshAsync(key, token);
    }

    public void Remove(string key)
    {
        _local.Remove(key);
        _remote.Remove(key);
    }

    public async Task RemoveAsync(string key, CancellationToken token = default)
    {
        _local.Remove(key);
        await _remote.RemoveAsync(key, token);
    }

    // CONSIDER: add some jitter to timespan so local cache will be 0..30 percents lower than remote
    private static MemoryCacheEntryOptions Convert(DistributedCacheEntryOptions options) => new() { AbsoluteExpiration = options.AbsoluteExpiration, AbsoluteExpirationRelativeToNow = options.AbsoluteExpirationRelativeToNow, SlidingExpiration = options.SlidingExpiration };
}

Nothing fancy here the real fun begins now:

using System;
using System.Threading.Tasks;

using Microsoft.Extensions.Caching.Distributed;
using Microsoft.Extensions.Caching.StackExchangeRedis;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;

using StackExchange.Redis;

namespace StackExchangeRedisWithFallback;

public static class DistributedCacheExtensions
{
    public static void AddStackExchangeRedisCacheWithFallback(this IServiceCollection services, Action<RedisCacheOptions> setupAction)
    {
        services.AddMemoryCache(); // will be used as fallback
        services.Configure(setupAction);
        services.AddSingleton(p => Connection(p.GetRequiredService<IOptions<RedisCacheOptions>>().Value));
        services.AddSingleton(p => (RedisCache)new ServiceCollection().AddStackExchangeRedisCache(ReuseConnection(p, setupAction)).BuildServiceProvider().GetRequiredService<IDistributedCache>()); // the trick, we are instantiating Redis cache in its own service collection and adding it to our app as RedisCache instead of IDistributedCache
        services.AddSingleton<IDistributedCache, RedisDistributedCache>(); // whenever IDistributedCache is requested we will give our implementation instead of RedisCache one
        services.AddSingleton<RedisSubscriber>().AddHostedService(p => p.GetRequiredService<RedisSubscriber>()); // Subscribe to Redis notifications to invalidate local cache
    }

    /// <summary>
    /// Get connection from given options
    /// </summary>
    /// <param name="options"></param>
    /// <returns></returns>
    /// <exception cref="ArgumentNullException">Throws if no connection options awailable</exception>
    private static IConnectionMultiplexer Connection(RedisCacheOptions options)
    {
        if (options.ConnectionMultiplexerFactory != null)
        {
            return options.ConnectionMultiplexerFactory().GetAwaiter().GetResult();
        }

        if (options.ConfigurationOptions != null)
        {
            return ConnectionMultiplexer.Connect(options.ConfigurationOptions);
        }

        if (options.Configuration != null)
        {
            return ConnectionMultiplexer.Connect(options.Configuration);
        }

        throw new ArgumentNullException(nameof(options), "All connection options are null");
    }

    /// <summary>
    /// The trick to reuse existing connection in our and RedisCache implementations
    /// </summary>
    /// <param name="serviceProvider"></param>
    /// <param name="setupAction"></param>
    /// <returns></returns>
    private static Action<RedisCacheOptions> ReuseConnection(IServiceProvider serviceProvider, Action<RedisCacheOptions> setupAction) => options =>
    {
        setupAction(options);
        options.Configuration = null;
        options.ConfigurationOptions = null;
        options.ConnectionMultiplexerFactory = () => Task.FromResult(serviceProvider.GetRequiredService<IConnectionMultiplexer>());
    };
}

Note how we are hiding redis implementation in its own service provider and how we are reusing redis connection, that probably the only important things here

And for invalidation we are using:

using System;
using System.Threading;
using System.Threading.Tasks;

using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;

using StackExchange.Redis;

namespace StackExchangeRedisWithFallback;

public class RedisSubscriber: BackgroundService
{
    private readonly IConnectionMultiplexer _connection;
    private readonly IMemoryCache _cache;
    private readonly ILogger<RedisSubscriber> _logger;

    public RedisSubscriber(IConnectionMultiplexer connection, IMemoryCache cache, ILogger<RedisSubscriber> logger)
    {
        _connection = connection;
        _cache = cache;
        _logger = logger;
    }

    private void Handler(RedisChannel channel, RedisValue key)
    {
        if (key.IsNullOrEmpty)
        {
            return;
        }

        if (_logger.IsEnabled(LogLevel.Debug))
        {
            _logger.LogDebug("Removing '{Key}' from local cache because of '{Channel}'", key, channel);
        }

        _cache.Remove(key);
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var result = await _connection.GetDatabase().ExecuteAsync("config", "get", "notify-keyspace-events");
        if (!result.ToDictionary().TryGetValue("notify-keyspace-events", out var notifyKeyspaceEvents)|| notifyKeyspaceEvents == null || notifyKeyspaceEvents.IsNull || !Equals("AKE", notifyKeyspaceEvents.ToString()))
        {
            throw new ArgumentOutOfRangeException(nameof(notifyKeyspaceEvents), notifyKeyspaceEvents, "Redis is not configured to notify about changes, expected 'notify-keyspace-events' to be 'AKE', make sure you added '--notify-keyspace-events AKE'");
        }
        var subscriber = _connection.GetSubscriber();
        await subscriber.SubscribeAsync("__keyevent@0__:expired", Handler, CommandFlags.FireAndForget);
        await subscriber.SubscribeAsync("__keyevent@0__:del", Handler, CommandFlags.FireAndForget);
        await subscriber.SubscribeAsync("__keyevent@0__:set", Handler, CommandFlags.FireAndForget);
        if (_logger.IsEnabled(LogLevel.Debug))
        {
            _logger.LogDebug("Subscribed to notifications from Redis");
        }
    }
}

Having all that in our app we may just replace AddStackExchangeRedisCache with AddStackExchangeRedisCacheWithFallback and suddently everything will wire up, e.g.:

builder.Services.AddStackExchangeRedisCacheWithFallback(options => // drop in replacement for "builder.Services.AddStackExchangeRedisCache"
{
    options.Configuration = "localhost";
    options.InstanceName = "Api";
});

The cool thing is that nothing changed from perspective of consumers, no matter which realization you are using you code will stay the same, aka:

using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Caching.Distributed;

namespace Api.Controllers;

[ApiController]
[Route("[controller]")]
public class WeatherForecastController : ControllerBase
{
    private static readonly string[] Summaries = new[] { "Freezing", "Bracing", "Chilly", "Cool", "Mild", "Warm", "Balmy", "Hot", "Sweltering", "Scorching" };

    private readonly ILogger<WeatherForecastController> _logger;
    private readonly IDistributedCache _cache;

    public WeatherForecastController(ILogger<WeatherForecastController> logger, IDistributedCache cache)
    {
        _logger = logger;
        _cache = cache;
    }

    [HttpGet(Name = nameof(GetWeatherForecast))]
    public async Task<IEnumerable<WeatherForecast>?> GetWeatherForecast(CancellationToken token)
    {
        return await _cache.GetOrCreate(
            nameof(GetWeatherForecast),
            PretendWeAreDoingDatabaseCall,
            TimeSpan.FromSeconds(10),
            token
        );
    }

    private Task<IEnumerable<WeatherForecast>?> PretendWeAreDoingDatabaseCall()
    {
        var actual = Enumerable.Range(1, 5).Select(index => new WeatherForecast { Date = DateTime.Now.AddDays(index), TemperatureC = Random.Shared.Next(-20, 55), Summary = Summaries[Random.Shared.Next(Summaries.Length)] });
        _logger.LogInformation("Cache MISS");
        return Task.FromResult(actual)!;
    }
}

And if you will make your app setting somethin like this:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning",
      "StackExchangeRedisWithFallback": "Debug"
    }
  },
  "AllowedHosts": "*"
}

And run your endpoing, after a while you will see in logs:

dbug: StackExchangeRedisWithFallback.RedisSubscriber[0]
      Removing 'ApiGetWeatherForecast' from local cache because of '__keyevent@0__:expired'

Which tells you that inside redis, our ApiGetWeatherForecast was expired and we removed it from local cache as well - profit

Note: indeed, race conditions are possible here, but if you are not building banking system and eventual consistency is fine you should not care