dotnet polly policies for http resilient http client (retry, circuit breaker, timeout)

TLDR: http client with timeout, retry and circuit breaker

services
  .AddHttpClient("my api client", c => c.BaseAddress = new Uri("..."))
  .AddPolicyHandler(Policy
    .Handle<TimeoutRejectedException>()
    .OrTransientHttpError()
    .WaitAndRetryAsync(new[]
    {
      // number of retries and delay between them
      TimeSpan.FromMilliseconds(100),
      TimeSpan.FromMilliseconds(500),
      TimeSpan.FromSeconds(1),
      TimeSpan.FromSeconds(2),
      TimeSpan.FromSeconds(5),
    }))
  .AddPolicyHandler(Policy
    .Handle<TimeoutRejectedException>()
    .OrTransientHttpError()
    .CircuitBreakerAsync(
      5,                       // how much subsequant failures should open circuit
      TimeSpan.FromSeconds(30) // how long circuit should be opened before trying again
    ))
  .AddPolicyHandler(Policy
    .TimeoutAsync<HttpResponseMessage>(
      TimeSpan.FromSeconds(1) // timeout for each request
    ));

Notes:

DO NOT set httpClient.Timeout it will act as global timeout outside of polly policies
With this setup you may receive two more additional exceptions TimeoutRejectedException in case of timeouts, BrokenCircuitException in case of opened circuit breaker
For circuit breaker we should also add handling of 429 Too many requests
Package Microsoft.Extensions.Http.Polly

Polly ordering and chaining of policies

In this note I want just to backup samples of tests that were used to prove polly configured the way I want

In general there are few important thigs:

order of policies is important - they are kind of wrapping each other
for outer policies to respect inner - their exceptions should be added as well

The best example here is combination of retry and timeout policites, aka (pseudo code):

var retry = Polly.Retry(...);
var timeout = Polly.Timeout(...);

retry.Execute(() => doWork()); // will retry configured number of times

timeout.Execute(() => doWork()); // no retries, may timeout and throw TimeoutRejectedException


timeout.Execute( // global timeout, no matter how many retries inside
  retry.Execute( // will retry in case of failure
    () => doWork()
  )
);

retry.Execute( // will retry in case of failure
  timeout.Execute( // each attempt has its own timeout
    () => doWork()
  )
);

second example is about execption handling, one again pseudo code:

var retry = Polly.Handle<DivizionByZeroException>().Retry(2);
var timeout = Polly.Timeout(5);

retry.Execute(
  timeout.Execute(
    () => 42 / zero
  )
)

because retry does handle DivizionByZeroException it will continue to retry, but, if timeout TimeoutRejectedException will occur nothing wont be retried and exception will bubble up

to handle such cases you may want to describe all things that may go wrong and you want to retry on:

var retry = Polly
  .Handle<DivizionByZeroException>()
  .Or<TimeoutRejectedException>()
  .Retry(2);
var timeout = Polly.Timeout(5);

retry.Execute( // will retry on DivizionByZeroException and TimeoutRejectedException
  timeout.Execute(
    () => 42 / zero
  )
)

More examples may be found below

Here are tests I have ended up with:

using System;
using System.Diagnostics;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using FluentAssertions;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Hosting.Server.Features;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.DependencyInjection;
using Polly;
using Polly.CircuitBreaker;
using Polly.Extensions.Http;
using Polly.Timeout;
using Xunit;

namespace HttpPolly;

public class PollyHttp : IAsyncLifetime
{
    private const string Name = "FakeServerClient";
    private IWebHost _server = default!;
    private HttpClient _client = default!;
    private Uri _uri = default!;

    private int _handledEventsAllowedBeforeBreaking = 2;

    public void ConfigureServices(IServiceCollection services)
    {
        services
            .AddHttpClient(Name, c => c.BaseAddress = _uri)
            .AddPolicyHandler(Policy
                .Handle<TimeoutRejectedException>()
                .OrTransientHttpError()
                .WaitAndRetryAsync(new[]
                {
                    TimeSpan.FromSeconds(0),
                    TimeSpan.FromSeconds(0),
                    TimeSpan.FromSeconds(0),
                }))
            .AddPolicyHandler(Policy
                .Handle<TimeoutRejectedException>()
                .OrTransientHttpError()
                .CircuitBreakerAsync(_handledEventsAllowedBeforeBreaking, TimeSpan.FromSeconds(5)))
            .AddPolicyHandler(Policy
                .TimeoutAsync<HttpResponseMessage>(TimeSpan.FromSeconds(1)));
    }

    [Fact]
    public async Task ShouldRetryConfiguredNumberOfTimes()
    {
        // Arrange
        _handledEventsAllowedBeforeBreaking = 100;
        var counter = 0;
        await InitializeServer(context =>
        {
            counter += 1;
            context.Response.StatusCode = 500;
            return Task.CompletedTask;
        });

        // Act
        await _client.GetAsync("/");

        // Assert
        counter.Should().Be(4, "1 is our initial request, then 3 retry attempts");
    }
    
    [Fact]
    public async Task ShouldCircuit()
    {
        // Arrange
        var counter = 0;
        await InitializeServer(context =>
        {
            counter += 1;
            context.Response.StatusCode = 500;
            return Task.CompletedTask;
        });

        // Act
        var act = async () => await _client.GetAsync("/");

        // Assert
        await act.Should().ThrowAsync<BrokenCircuitException>();
        counter.Should().Be(2, "even so we have may expect 4 requests to be made, after second failed circuit opens and no followup requests were made");
    }
    
    [Fact]
    public async Task ShouldRetryOnTimeoutAsWell()
    {
        // Arrange
        _handledEventsAllowedBeforeBreaking = 100;
        var counter = 0;
        await InitializeServer(async context =>
        {
            counter += 1;
            await Task.Delay(TimeSpan.FromSeconds(2));
        });
        var timer = Stopwatch.StartNew();

        // Act
        var act = async () => await _client.GetAsync("/");

        // Assert
        await act.Should().ThrowAsync<TimeoutRejectedException>("all attempts failed because of timeout");
        counter.Should().Be(4, "1 is our initial request, then 3 retry attempts");
        Math.Round(timer.Elapsed.TotalSeconds).Should().Be(4, "we made 4 request, each has 2sec delay, so we may expect 8sec in total, but because of 1sec timeout policy we are not waiting for all that time and failing fast");
    }

    [Fact]
    public async Task ShouldOpenCircuitAfterConfiguredAmountOfTime()
    {
        // Arrange
        var counter = 0;
        await InitializeServer(context =>
        {
            counter += 1;
            if (counter < 3)
            {
                context.Response.StatusCode = 500;    
            }
            return Task.CompletedTask;
        });

        // Act
        var act = async () => await _client.GetAsync("/");

        // Assert
        await act.Should().ThrowAsync<BrokenCircuitException>();
        counter.Should().Be(2, "after second failed request circuit opens - no more requests send to backend");

        await Task.Delay(TimeSpan.FromSeconds(5));
        
        var response = await _client.GetAsync("/");
        response.Should().BeSuccessful("but after configured amount of time circuit half opens and allows for new attempts");
        counter.Should().Be(3, "we did landed to server and received successful response");
    }
    
    [Fact]
    public async Task ShouldCircuitOnTimeoutAsWell()
    {
        // Arrange
        var counter = 0;
        await InitializeServer(async context =>
        {
            counter += 1;
            await Task.Delay(TimeSpan.FromSeconds(2));
        });

        // Act
        var act = async () => await _client.GetAsync("/");

        // Assert
        await act.Should().ThrowAsync<BrokenCircuitException>("all attempts failed because of timeout but after second circuit opens");
        counter.Should().Be(2, "circuit does not allow followup requests to be send");

        await Task.Delay(TimeSpan.FromSeconds(5));
        await act.Should().ThrowAsync<BrokenCircuitException>("after 5sec circuit is half open and next request will define its state - in our case server is always failing because of timeout");
        counter.Should().Be(3, "we did only one attempt which immediately open circuit back");
    }

    private async Task InitializeServer(RequestDelegate action)
    {
        _server = new WebHostBuilder()
            .UseKestrel(o => o.Listen(IPAddress.Loopback, 0))
            .Configure(app => app.Run(action))
            .Build();

        await _server.StartAsync();

        _uri = new Uri(_server.ServerFeatures.Get<IServerAddressesFeature>()!.Addresses.First());

        var services = new ServiceCollection();
        ConfigureServices(services);
        
        var provider = services.BuildServiceProvider();
        var factory = provider.GetRequiredService<IHttpClientFactory>();
        _client = factory.CreateClient(Name);
    }

    public Task InitializeAsync()
    {
        return Task.CompletedTask;
    }

    public Task DisposeAsync()
    {
        _server.Dispose();
        return Task.CompletedTask;
    }
}

with them I can be sure that everything works as expected

Retrying on non expected non JSON responses

Imagine that your backend from time to time responses with statatus code 200 OK but with content like <h1>Cloudflare Captcha</h1> and you want to retry them

The catch here is that this happens outside of policies we have added (they are just simple message handlers inside, and parsing response happens outside of all that policies)

Here is an example of how it might be done

using System;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Net.Http.Json;
using System.Threading.Tasks;
using FluentAssertions;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Hosting.Server.Features;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.DependencyInjection;
using Newtonsoft.Json;
using Polly;
using Polly.Extensions.Http;
using Polly.Timeout;
using Xunit;

namespace HttpPolly;

public record Player
{
  public int Id { get; init; }
  public string Name { get; init; } = default!;
}

public class HandleNonJson
{
  [Fact]
  public async Task DemoSetupFirstResponseIsHtmlWithErrorSecondExpectedJson()
  {
    // Arrange
    var player = new Player { Id = 1, Name = "@mac" };
    var counter = 0;
    using var server = new WebHostBuilder().UseKestrel(o => o.Listen(IPAddress.Loopback, 0)).Configure(app =>
    {
      app.Run(async context =>
      {
        counter += 1;
        if (counter == 1)
        {
          await context.Response.WriteAsync("<h1>Cloudflare Error Page</h1>");
        }
        else
        {
          await context.Response.WriteAsJsonAsync(player);
        }
      });

    }).Build();
    await server.StartAsync();
    var uri = new Uri(server.ServerFeatures.Get<IServerAddressesFeature>()!.Addresses.First());
    var http = new HttpClient();
    http.BaseAddress = uri;

    // Act & Assert
    var act = async () => await http.GetFromJsonAsync<Player>("/");
    await act.Should().ThrowAsync<System.Text.Json.JsonException>("first response has HTML instead of JSON with error from Cloudflare");

    var response = await http.GetFromJsonAsync<Player>("/");
    response.Should().BeEquivalentTo(player);
  }

  [Fact]
  public async Task RetryOnJsonDecodeFailureWrongWay()
  {
    // Arrange
    var player = new Player { Id = 1, Name = "@mac" };
    var counter = 0;
    using var server = new WebHostBuilder().UseKestrel(o => o.Listen(IPAddress.Loopback, 0)).Configure(app =>
    {
      app.Run(async context =>
      {
        counter += 1;
        if (counter == 1)
        {
          await context.Response.WriteAsync("<h1>Cloudflare Captcha</h1>");
        }
        else
        {
          await context.Response.WriteAsJsonAsync(player);
        }
      });

    }).Build();
    await server.StartAsync();
    var uri = new Uri(server.ServerFeatures.Get<IServerAddressesFeature>()!.Addresses.First());
    var services = new ServiceCollection();
    services
      .AddHttpClient("demo", c => c.BaseAddress = uri)
      .AddPolicyHandler(Policy
        .Handle<System.Text.Json.JsonException>() // it wont work here because it happens outside
        .OrTransientHttpError()
        .RetryAsync(3));
    var provider = services.BuildServiceProvider();
    var factory = provider.GetRequiredService<IHttpClientFactory>();
    var http = factory.CreateClient("demo");

    // Act
    var act = async () => await http.GetFromJsonAsync<Player>("/");

    // Assert
    await act.Should().ThrowAsync<System.Text.Json.JsonException>("polly http extensions are just message handlers inside, and parsing response happens outside");
    counter.Should().Be(1, "that's why it was not catched and retried");
  }

  [Fact]
  public async Task RetryOnJsonDecodeFailureRightWay()
  {
    // Arrange
    var player = new Player { Id = 1, Name = "@mac" };
    var counter = 0;
    var innerRetriesCount = 0;
    var outerRetriesCount = 0;
    using var server = new WebHostBuilder().UseKestrel(o => o.Listen(IPAddress.Loopback, 0)).Configure(app =>
    {
      app.Run(async context =>
      {
        counter += 1;
        if (counter == 1)
        {
          await context.Response.WriteAsync("<h1>Cloudflare Captcha</h1>");
        }
        else
        {
          await context.Response.WriteAsJsonAsync(player);
        }
      });

    }).Build();
    await server.StartAsync();
    var uri = new Uri(server.ServerFeatures.Get<IServerAddressesFeature>()!.Addresses.First());
    var services = new ServiceCollection();
    services
      .AddHttpClient("demo", c => c.BaseAddress = uri)
      .AddPolicyHandler(HttpPolicyExtensions
        .HandleTransientHttpError() // even so we added this retry it will do nothing in this example
        .RetryAsync(3, (_, _) => innerRetriesCount += 1));
    var provider = services.BuildServiceProvider();
    var factory = provider.GetRequiredService<IHttpClientFactory>();
    var http = factory.CreateClient("demo");

    // outer policy, aka can be applied to anything, not only to http
    var outerRetry = Policy.Handle<System.Text.Json.JsonException>().RetryAsync(3, (_, _) => outerRetriesCount += 1);

    // Act
    var response = await outerRetry.ExecuteAsync(async () => await http.GetFromJsonAsync<Player>("/"));

    // Assert
    response.Should().BeEquivalentTo(player);
    counter.Should().Be(2, "we did 2 requests in total, first - failed because it was not JSON, second - succeed");
    innerRetriesCount.Should().Be(0, "http retry policy did nothig - it is not aware of what happens outside");
    outerRetriesCount.Should().Be(1, "first request was original and failed, we have 3 more retry attempts, first one was successful");
  }
}

Here is even more examples I was played with to understand how it works

Detailed examples

using System;
using System.Diagnostics;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using FluentAssertions;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Hosting.Server.Features;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.DependencyInjection;
using Polly;
using Polly.CircuitBreaker;
using Polly.Extensions.Http;
using Polly.Timeout;
using Xunit;

namespace HttpPolly;

public class PollyHttpExtensionsExperiments
{
    [Fact]
    public async Task step00_prepare_demo_server()
    {
        // Arrange
        var counter = 0;
        using var server = new WebHostBuilder().UseKestrel(o => o.Listen(IPAddress.Loopback, 0)).Configure(app =>
        {
            app.Run(async context =>
            {
                counter += 1;
                await context.Response.WriteAsync(counter.ToString());
            });

        }).Build();
        await server.StartAsync();
        var uri = new Uri(server.ServerFeatures.Get<IServerAddressesFeature>()!.Addresses.First());
        var http = new HttpClient();
        http.BaseAddress = uri;

        // Act
        var response1 = await http.GetStringAsync("/");
        var response2 = await http.GetStringAsync("/");
        
        // Assert
        response1.Should().Be("1", "we are incrementing counter on each request");
        response2.Should().Be("2", "we are incrementing counter on each request");
    }
    
    [Fact]
    public async Task step01_service_provider_and_http_client_factory()
    {
        // Arrange
        var counter = 0;
        using var server = new WebHostBuilder().UseKestrel(o => o.Listen(IPAddress.Loopback, 0)).Configure(app =>
        {
            app.Run(async context =>
            {
                counter += 1;
                await context.Response.WriteAsync(counter.ToString());
            });

        }).Build();
        await server.StartAsync();
        var uri = new Uri(server.ServerFeatures.Get<IServerAddressesFeature>()!.Addresses.First());
        const string name = "DemoClientName";
        var services = new ServiceCollection();
        services.AddHttpClient(name, c => c.BaseAddress = uri);
        var provider = services.BuildServiceProvider();
        var factory = provider.GetRequiredService<IHttpClientFactory>();
        var client = factory.CreateClient(name);

        // Act
        var response1 = await client.GetStringAsync("/");
        var response2 = await client.GetStringAsync("/");
        
        // Assert
        response1.Should().Be("1", "we are incrementing counter on each request");
        response2.Should().Be("2", "we are incrementing counter on each request");
    }

    [Fact]
    public async Task step02_extract_fake_server()
    {
        var counter = 0;
        var client = await FakeServer(async context =>
        {
            counter += 1;
            await context.Response.WriteAsync(counter.ToString());
        });
        
        // Act
        var response1 = await client.GetStringAsync("/");
        var response2 = await client.GetStringAsync("/");
        
        // Assert
        response1.Should().Be("1", "we are incrementing counter on each request");
        response2.Should().Be("2", "we are incrementing counter on each request");
    }
    
    private static async Task<HttpClient> FakeServer(RequestDelegate action, params IAsyncPolicy<HttpResponseMessage>[]? policies)
    {
        var server = new WebHostBuilder()
            .UseKestrel(o => o.Listen(IPAddress.Loopback, 0))
            .Configure(app => app.Run(action))
            .Build();

        await server.StartAsync();
        
        var uri = new Uri(server.ServerFeatures.Get<IServerAddressesFeature>()!.Addresses.First());
        
        var services = new ServiceCollection();
        var builder = services.AddHttpClient("FakeServerClient", c => c.BaseAddress = uri);
        if (policies != null)
        {
            foreach (var policy in policies)
            {
                builder.AddPolicyHandler(policy);
            }
        }

        var provider = services.BuildServiceProvider();
        var factory = provider.GetRequiredService<IHttpClientFactory>();
        
        var client = factory.CreateClient("FakeServerClient");

        return client;
    }

    [Fact]
    public async Task step03_prepare_fake_server_for_retry_test()
    {
        var counter = 0;
        var client = await FakeServer(async context =>
        {
            counter += 1;
            context.Response.StatusCode = counter < 3 ? 500 : 200; // first 2 failing
            await context.Response.WriteAsync(counter.ToString());
        });
        
        // Act
        var response1 = await client.GetAsync("/");
        var response2 = await client.GetAsync("/");
        var response3 = await client.GetAsync("/");
        
        // Assert
        response1.Should().HaveStatusCode(HttpStatusCode.InternalServerError, "first 2 requests filing");
        response2.Should().HaveStatusCode(HttpStatusCode.InternalServerError, "first 2 requests filing");
        response3.Should().HaveStatusCode(HttpStatusCode.OK, "all other succeed");
    }

    [Fact]
    public async Task step04_retry_policy_demo()
    {
        var policy = HttpPolicyExtensions
            .HandleTransientHttpError()
            .RetryAsync(3);

        var counter = 0;
        var client = await FakeServer(async context =>
        {
            counter += 1;
            context.Response.StatusCode = counter < 3 ? 500 : 200; // first 2 failing
            await context.Response.WriteAsync(counter.ToString());
        }, policy);
        
        // Act
        var response = await client.GetAsync("/");
        
        // Assert
        response.Should().HaveStatusCode(HttpStatusCode.OK, "we have retried");
        counter.Should().Be(3, "first request is original one, next two are from retry policy which has one more attempt before fail");
    }
    
    [Fact]
    public async Task step05_retry_policy_on_always_failing_server()
    {
        var policy = HttpPolicyExtensions
            .HandleTransientHttpError()
            .RetryAsync(3);

        var counter = 0;
        var client = await FakeServer(async context =>
        {
            counter += 1;
            context.Response.StatusCode = 500; // always failing
            await context.Response.WriteAsync(counter.ToString());
        }, policy);
        
        // Act
        var response = await client.GetAsync("/");
        
        // Assert
        response.Should().HaveStatusCode(HttpStatusCode.InternalServerError, "we have retried all 3 attempts");
        counter.Should().Be(4, "first request is original one, next 3 are retry count, all failed");
    }
    
    [Fact]
    public async Task step05_retry_with_callback_and_wait()
    {
        var now = DateTime.Now;
        var retries = 0;
        
        var policy = HttpPolicyExtensions
            .HandleTransientHttpError()
            .WaitAndRetryAsync(new[]
            {
                TimeSpan.FromSeconds(1),
                TimeSpan.FromSeconds(2),
                TimeSpan.FromSeconds(3)
            }, (exception, timeSpan, retryCount, context) =>
            {
                retries += 1;
            });

        var counter = 0;
        var client = await FakeServer(async context =>
        {
            counter += 1;
            context.Response.StatusCode = 500; // always failing
            await context.Response.WriteAsync(counter.ToString());
        }, policy);
        
        // Act
        var response = await client.GetAsync("/");
        
        // Assert
        response.Should().HaveStatusCode(HttpStatusCode.InternalServerError, "we have retried all 3 attempts");
        counter.Should().Be(4, "first request is original one, next 3 are retry count, all failed");
        retries.Should().Be(3, "we have retried all 3 attempts");
        DateTime.Now.Subtract(now).TotalSeconds.Should().BeGreaterThanOrEqualTo(6, "we are going to wait 6 seconds in total between attempts 1+2+3");
    }
    
    [Fact]
    public async Task step06_retry_does_not_handle_client_timeout()
    {
        var retries = 0;
        
        var policy = HttpPolicyExtensions
            .HandleTransientHttpError()
            .WaitAndRetryAsync(new[]
            {
                TimeSpan.FromSeconds(1),
                TimeSpan.FromSeconds(1),
                TimeSpan.FromSeconds(1)
            }, (exception, timeSpan, retryCount, context) =>
            {
                retries += 1;
            });

        var counter = 0;
        var client = await FakeServer(async context =>
        {
            counter += 1;
            context.Response.StatusCode = 500; // always failing
            await Task.Delay(TimeSpan.FromSeconds(2));
            await context.Response.WriteAsync(counter.ToString());
        }, policy);
        client.Timeout = TimeSpan.FromSeconds(1);
        
        // Act
        var act = async () => await client.GetAsync("/");

        // Assert
        await act.Should().ThrowAsync<TaskCanceledException>("we set client timeout to 1 second, and response has delay for 2 seconds");
        counter.Should().Be(1, "we did not retried because by default HandleTransientHttpError does not handle timeouts");
        retries.Should().Be(0, "we did not retried");
    }
    
    [Fact]
    public async Task step07_retry_wont_handle_client_timeout_no_matter_how_we_will_try()
    {
        var now = DateTime.Now;
        var retries = 0;

        var policy = Policy
            .Handle<TaskCanceledException>()
            .OrInner<TimeoutException>()
            .OrTransientHttpError()
            .WaitAndRetryAsync(new[]
            {
                TimeSpan.FromSeconds(1),
                TimeSpan.FromSeconds(1),
                TimeSpan.FromSeconds(1),
            }, (exception, timeSpan, retryCount, context) =>
            {
                retries += 1;
            });

        var counter = 0;
        var client = await FakeServer(async context =>
        {
            counter += 1;
            if (counter < 3)
            {
                context.Response.StatusCode = 500;
                await Task.Delay(TimeSpan.FromSeconds(2));
            }
            await context.Response.WriteAsync(counter.ToString());
        }, policy);
        
        // Act
        client.Timeout = TimeSpan.FromSeconds(5);
        var act = async () => await client.GetAsync("/");

        // Assert
        await act.Should().ThrowAsync<TaskCanceledException>("http client timeout property sets 'global' timeout for overall operation including all retries and wont be catched by poly");
        counter.Should().Be(2, "we did first request which took 2 seconds and timed out and because we catched that exception in polly second attempt was made");
        retries.Should().Be(2, "second attempt did not finished because of global timeout set to http client");
        DateTime.Now.Subtract(now).TotalSeconds.Should().BeLessThan(6, "no matter how many retries and policies we have global http client timeout will brake everything after 5 seconds");
    }
    
    [Fact]
    public async Task step08_the_right_way_to_handle_timeout()
    {
        var now = DateTime.Now;
        var retries = 0;
        var timeouts = 0;

        var timeout = Policy.TimeoutAsync<HttpResponseMessage>(TimeSpan.FromSeconds(1), (context, timespan, task, exception) =>
        {
            timeouts += 1;
            return Task.CompletedTask;
        });
        var retry = Policy
            .Handle<TimeoutRejectedException>()
            .OrTransientHttpError()
            .WaitAndRetryAsync(new[]
            {
                TimeSpan.FromSeconds(0),
                TimeSpan.FromSeconds(0),
                TimeSpan.FromSeconds(0),
            }, (exception, timeSpan, retryCount, context) =>
            {
                retries += 1;
            });

        var counter = 0;
        var client = await FakeServer(async context =>
        {
            
            counter += 1;
            if (counter < 3)
            {
                context.Response.StatusCode = 500;
                await Task.Delay(TimeSpan.FromSeconds(50));
            }
            await context.Response.WriteAsync(counter.ToString());
        }, retry, timeout);
        
        // Act
        // client.Timeout = TimeSpan.FromSeconds(5); // Don't do this it will set global timeout for all attempts and wont be handled
        var response = await client.GetAsync("/");

        // Assert
        response.Should().BeSuccessful("we did retried till success");
        counter.Should().Be(3, "first one is original and timed out, second is from retry but also timed out, third is ok");
        retries.Should().Be(2, "first one is original, then 2 retries first of which was timedout and second ok");
        timeouts.Should().Be(2, "first one is from original request, second is from first retry");
        DateTime.Now.Subtract(now).TotalSeconds.Should().BeLessOrEqualTo(3, "even so our two first requests has delay we are going to timeout much faster after configured 1 second, so first two attempts should take us max 2 seconds instead of 100 seconds");
    }
    
    [Fact]
    public async Task step09_order_of_policies_is_critical()
    {
        var now = DateTime.Now;
        var retries = 0;
        var timeouts = 0;

        var timeout = Policy.TimeoutAsync<HttpResponseMessage>(TimeSpan.FromSeconds(1), (context, timespan, task, exception) =>
        {
            timeouts += 1;
            return Task.CompletedTask;
        });
        var retry = Policy
            .Handle<TimeoutRejectedException>()
            .OrTransientHttpError()
            .WaitAndRetryAsync(new[]
            {
                TimeSpan.FromSeconds(0),
                TimeSpan.FromSeconds(0),
                TimeSpan.FromSeconds(0),
            }, (exception, timeSpan, retryCount, context) =>
            {
                retries += 1;
            });

        var counter = 0;
        var client = await FakeServer(async context =>
        {
            
            counter += 1;
            if (counter < 3)
            {
                context.Response.StatusCode = 500;
                await Task.Delay(TimeSpan.FromSeconds(2));
            }
            await context.Response.WriteAsync(counter.ToString());
        }, timeout, retry);
        
        // Act
        var act = async () => await client.GetAsync("/");

        // Assert
        await act.Should().ThrowAsync<TimeoutRejectedException>("because we have another order of policies now our timeout policy is working as global timeout");
        counter.Should().Be(1, "because of global timeout of 1 second and delay of 2 second for first response we wont have chance to retry anything");
        retries.Should().Be(0, "no chance we will have retry");
        DateTime.Now.Subtract(now).TotalSeconds.Should().BeLessThan(2, "no matter how are retries configured we have global timeout for 1 second so should definitely less than 2 second");
    }

    [Fact]
    public async Task step10_combining_timeouts_to_have_global_timeout_for_all_attempts_and_inner_timeout_for_each_attempt()
    {
        var now = DateTime.Now;
        var retries = 0;
        var innerTimeouts = 0;
        var outerTimeouts = 0;

        var outerTimeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(TimeSpan.FromSeconds(3), (context, timespan, task, exception) =>
        {
            outerTimeouts += 1;
            return Task.CompletedTask;
        });
        var innerTimeoutPolicy = Policy.TimeoutAsync<HttpResponseMessage>(TimeSpan.FromSeconds(1), (context, timespan, task, exception) =>
        {
            innerTimeouts += 1;
            return Task.CompletedTask;
        });
        var retry = Policy
            .Handle<TimeoutRejectedException>()
            .OrTransientHttpError()
            .WaitAndRetryAsync(new[]
            {
                TimeSpan.FromSeconds(0),
                TimeSpan.FromSeconds(0),
                TimeSpan.FromSeconds(0),
            }, (exception, timeSpan, retryCount, context) =>
            {
                retries += 1;
            });

        var counter = 0;
        var client = await FakeServer(async context =>
        {
            counter += 1;
            await Task.Delay(TimeSpan.FromSeconds(2));
            await context.Response.WriteAsync(counter.ToString());
        }, outerTimeoutPolicy, retry, innerTimeoutPolicy);
        
        // Act
        var act = async () => await client.GetAsync("/");

        // Assert
        await act.Should().ThrowAsync<TimeoutRejectedException>("we are expecting global timeout to fire because all our requests are failing to inner timeout and we are retrying till outer timeout triggers");
        counter.Should().Be(3, "first requests is original one and timed out after 1 second (1s), second requests is an retry attempt but also failed (2s), the same is true for second retry (3s) and here we hit outer timeout");
        retries.Should().Be(2, "we had a chance to make 2 retries before hitting outer timeout");
        Math.Round(DateTime.Now.Subtract(now).TotalSeconds).Should().Be(3, "no matter what overall time should not be more that 3 seconds because of outer timeout");
        innerTimeouts.Should().Be(2, "we did 2 retries and they all timedout");
        outerTimeouts.Should().Be(1, "outer timeout fired once and because it is outer there are no retries");
    }

    [Fact]
    public async Task step11_cancellation_token_is_respected()
    {
        var token = new CancellationTokenSource(TimeSpan.FromMilliseconds(2500));
        var now = DateTime.Now;
        var retries = 0;

        var policy = HttpPolicyExtensions
            .HandleTransientHttpError()
            .RetryAsync(100, (_, _) =>
            {
                retries += 1;
            });

        var counter = 0;
        var client = await FakeServer(async context =>
        {
            counter += 1;
            context.Response.StatusCode = 500;
            await Task.Delay(TimeSpan.FromSeconds(1));
            await context.Response.WriteAsync(counter.ToString());
        }, policy);
        
        // Act
        var act = async () => await client.GetAsync("/", token.Token);

        // Assert
        await act.Should().ThrowAsync<TaskCanceledException>("token cancellation happened sooner than policies");
        counter.Should().Be(3, "first request is original one, next two are our retry attempts from allowed 100");
        retries.Should().Be(2, "we had a chance to do only two retry attempts before token was cancelled");
        DateTime.Now.Subtract(now).TotalSeconds.Should().BeInRange(2,3, "we are canceling token after 2.5sec and taking int account we have 1 sec response delay and 100 retries test may go over 100sec if it is not working");
    }
    
    [Fact]
    public async Task step12_circuit_breaker_standalone()
    {
        var policy = HttpPolicyExtensions
            .HandleTransientHttpError()
            .CircuitBreakerAsync(2, TimeSpan.FromSeconds(5));

        var counter = 0;
        var client = await FakeServer(async context =>
        {
            counter += 1;
            context.Response.StatusCode = 500;
            await Task.Delay(TimeSpan.FromSeconds(1));
            await context.Response.WriteAsync(counter.ToString());
        }, policy);
        
        // Act & Assert
        var timer = Stopwatch.StartNew();
        var response = await client.GetAsync("/");
        counter.Should().Be(1, "we made first request");
        response.Should().HaveServerError("because all requests are failing");
        Math.Round(timer.Elapsed.TotalSeconds).Should().Be(1, "because all responses have 1 second delay");
        
        timer.Restart();
        response = await client.GetAsync("/");
        counter.Should().Be(2, "we made second request");
        response.Should().HaveServerError("because all requests are failing");
        Math.Round(timer.Elapsed.TotalSeconds).Should().Be(1, "because all responses have 1 second delay");
        
        timer.Restart();
        var act = async () => await client.GetAsync("/");
        await act.Should().ThrowAsync<BrokenCircuitException>("previous two attempts was unsuccessful so circuit is opened");
        counter.Should().Be(2, "circuit is open no more requests for next 5 seconds");
        Math.Round(timer.Elapsed.TotalSeconds).Should().Be(0, "because we did not make actual http call");
        
        timer.Restart();
        act = async () => await client.GetAsync("/");
        await act.Should().ThrowAsync<BrokenCircuitException>("followup requests will continue to throw BrokenCircuitException immediatelly");
        counter.Should().Be(2, "and no more actual requests will be sent");
        Math.Round(timer.Elapsed.TotalSeconds).Should().Be(0, "because we did not make actual http call");

        await Task.Delay(TimeSpan.FromSeconds(5));
        
        timer.Restart();
        response = await client.GetAsync("/");
        counter.Should().Be(3, "after configured time circuit closes to give next attemp");
        response.Should().HaveServerError("but requests are still failing");
        Math.Round(timer.Elapsed.TotalSeconds).Should().Be(1, "just to prove that we made requests");

        timer.Restart();
        act = async () => await client.GetAsync("/");
        await act.Should().ThrowAsync<BrokenCircuitException>("that's why circuit immediately opens again for next 5 seconds");
        counter.Should().Be(3, "and again we wont send requests");
        Math.Round(timer.Elapsed.TotalSeconds).Should().Be(0, "because we did not make actual http call");
    }
    
    [Fact]
    public async Task step13_circuit_first_retry_then_suggested_for_long_delays_between_retries_because_circuit_may_change_in_between_retries()
    {
        var retries = 0;
        var retry = HttpPolicyExtensions
            .HandleTransientHttpError()
            .WaitAndRetryAsync(new []
            {
                TimeSpan.FromSeconds(2), 
                TimeSpan.FromSeconds(2), 
            }, (_, _) =>
            {
                retries += 1;
                return Task.CompletedTask;
            });

        var circuit = HttpPolicyExtensions
            .HandleTransientHttpError()
            .CircuitBreakerAsync(2, TimeSpan.FromSeconds(5));

        var counter = 0;
        var client = await FakeServer(async context =>
        {
            counter += 1;
            context.Response.StatusCode = 500;
            await Task.Delay(TimeSpan.FromSeconds(1));
            await context.Response.WriteAsync(counter.ToString());
        }, retry, circuit);
        
        // Act & Assert
        var timer = Stopwatch.StartNew();
        var act = async () => await client.GetAsync("/");
        await act.Should().ThrowAsync<BrokenCircuitException>("we have retried few times and circuit opens");
        counter.Should().Be(2, "even so we have 2 retries (so it should be 3 requests in total) after second unsucessfull request circuit opened and no more requests made");
        retries.Should().Be(2, "indeed 3rd retry happened but immediately failed");
        Math.Round(timer.Elapsed.TotalSeconds).Should().Be(6, "because only two requests were, original took 1 sec, then 2 sec wait before first retry, then first retry attempt took also 1 sec, then 2 sec wait before second retry and tried to perform a call but circuit immediatelly throws, so in total 1+2+1+2");
        
        
        timer.Restart();
        act = async () => await client.GetAsync("/");
        await act.Should().ThrowAsync<BrokenCircuitException>("all next attempts for next 5 seconds will fail because of open circuit");
        counter.Should().Be(2, "no mo actual requests are made");
        retries.Should().Be(2, "no mo retries happened");
        Math.Round(timer.Elapsed.TotalSeconds).Should().Be(0, "that's why it took 0 seconds");

        await Task.Delay(TimeSpan.FromSeconds(5));
        
        timer.Restart();
        act = async () => await client.GetAsync("/");
        await act.Should().ThrowAsync<BrokenCircuitException>("but after 5 seconds we will try again");
        counter.Should().Be(3, "but only one time, once again got bad response, circuit immediately opens and throws");
        retries.Should().Be(3, "but technically there is one retry happened");
        Math.Round(timer.Elapsed.TotalSeconds).Should().Be(3, "request took 1sec, then we waited 1sec, then retry attempt which immediately failed because of circuit");
    }

    [Fact]
    public async Task step14_chaining_policies_or_order_is_matters()
    {
        var zero = 0;
        var retries = 0;
        var retry = Policy.Handle<DivideByZeroException>().RetryAsync(2, (_, _) =>
        {
            retries += 1;
            return Task.CompletedTask;
        });

        var result = await retry.ExecuteAsync(() => Task.FromResult(0 / 2));
        result.Should().Be(0, "policy is just an wrapper");
                    
        var act = () => retry.ExecuteAsync(() => Task.FromResult(1 / zero));
        await act.Should().ThrowAsync<DivideByZeroException>();
        retries.Should().Be(2, "because action is always failing we made all two attempts");


        var circuit = Policy.Handle<DivideByZeroException>().CircuitBreakerAsync(2, TimeSpan.FromSeconds(1));
        
        result = await circuit.ExecuteAsync(() => Task.FromResult(0 / 2));
        result.Should().Be(0, "same is true for any other handlers");
        
        act = () => circuit.ExecuteAsync(() => Task.FromResult(1 / zero));
        await act.Should().ThrowAsync<DivideByZeroException>();

        var policy = Policy.WrapAsync(retry, circuit);
        act = () => policy.ExecuteAsync(() => Task.FromResult(1 / zero));
        await act.Should().ThrowAsync<BrokenCircuitException>("policies can be chained");

        var counter = 0;
        act = () => Policy.WrapAsync(
            Policy.Handle<DivideByZeroException>().RetryAsync(100),
            Policy.Handle<DivideByZeroException>().CircuitBreakerAsync(2, TimeSpan.FromSeconds(1))
        ).ExecuteAsync(() =>
        {
            counter += 1;
            return Task.FromResult(1 / zero);
        });
        
        await act.Should().ThrowAsync<BrokenCircuitException>();
        counter.Should().Be(2, "even so we have 100 retries after second circuit open");


        retries = 0;
        counter = 0;
        act = () => Policy.WrapAsync(
            Policy.Handle<DivideByZeroException>().Or<BrokenCircuitException>().RetryAsync(100, (_, _) => { retries += 1; return Task.CompletedTask; }),
            Policy.Handle<DivideByZeroException>().CircuitBreakerAsync(2, TimeSpan.FromSeconds(1))
        ).ExecuteAsync(() =>
        {
            counter += 1;
            return Task.FromResult(1 / zero);
        });
        await act.Should().ThrowAsync<BrokenCircuitException>("but if we handle BrokenCircuitException it will be thrown instead");
        counter.Should().Be(2, "one time for original attempt and second is our first attempt, because all two failed circuit opened and no more requests made");
        retries.Should().Be(100, "but actually we did tried 100 times, this calls just not proceed by circuit");
    }

    [Fact]
    public async Task step15_retry_circuit_fine()
    {
        var zero = 0;
        var counter = 0;
        var retries = 0;
        var open = 0;

        var act = () => Policy.WrapAsync(
            Policy.Handle<DivideByZeroException>().RetryAsync(100, (_, _) => { retries += 1; return Task.CompletedTask; }),
            Policy.Handle<DivideByZeroException>().CircuitBreakerAsync(2, TimeSpan.FromSeconds(1), (_, _) => { open += 1; }, () => { })
        ).ExecuteAsync(() =>
        {
            counter += 1;
            return Task.FromResult(1 / zero);
        });
        
        await act.Should().ThrowAsync<BrokenCircuitException>();
        counter.Should().Be(2, "even so we have 100 retries after second circuit open and no more actual requests will be processed");
        retries.Should().Be(2, "because we are not handling BrokenCircuitException in retry policy it wont be retried - aka it is the same as if we will say - ok it is down, no need to do any more retries");
        open.Should().Be(1, "circuit opened");
    }
    
    [Fact]
    public async Task step16_retry_circuit_incorrect_but_still_working()
    {
        var zero = 0;
        var counter = 0;
        var retries = 0;
        var open = 0;

        var act = () => Policy.WrapAsync(
            Policy.Handle<DivideByZeroException>().Or<BrokenCircuitException>().RetryAsync(100, (_, _) => { retries += 1; return Task.CompletedTask; }),
            Policy.Handle<DivideByZeroException>().CircuitBreakerAsync(2, TimeSpan.FromSeconds(1), (_, _) => { open += 1; }, () => { })
        ).ExecuteAsync(() =>
        {
            counter += 1;
            return Task.FromResult(1 / zero);
        });
        
        await act.Should().ThrowAsync<BrokenCircuitException>();
        counter.Should().Be(2, "at the very end we made only 2 requests");
        retries.Should().Be(100, "because we are handling BrokenCircuitException it did retried 100 times, BUT it might be useful when our retry has big enough time waits between attempts");
        open.Should().Be(1, "circuit opened");
    }
    
    [Fact]
    public async Task step17_circuit_retry_critically_incorrect()
    {
        var zero = 0;
        var counter = 0;
        var retries = 0;
        var open = 0;

        var act = () => Policy.WrapAsync(
            Policy.Handle<DivideByZeroException>().CircuitBreakerAsync(2, TimeSpan.FromSeconds(1), (_, _) => { open += 1; }, () => { }),
            Policy.Handle<DivideByZeroException>().RetryAsync(100, (_, _) => { retries += 1; return Task.CompletedTask; })
        ).ExecuteAsync(() =>
        {
            counter += 1;
            return Task.FromResult(1 / zero);
        });
        
        await act.Should().ThrowAsync<DivideByZeroException>();
        counter.Should().Be(101, "circuit is closed, our initial request failed, then we made 100 retries, so 101 calls in total");
        retries.Should().Be(100, "from 101 requests, 100 are retries");
        open.Should().Be(0, "circuit is not opened because from its perspective it is very first attempt");
        
        await act.Should().ThrowAsync<DivideByZeroException>();
        open.Should().Be(0, "but what is fun even second attempt did not opens it");
        counter.Should().Be(202, "and requests are growing");
    }
    
    [Fact]
    public async Task step18_retry_circuit_timeout_fine_but_requires_to_handle_timeouts_up_by_chain__works()
    {
        var counter = 0;
        var retries = 0;
        var open = 0;
        var timeouts = 0;

        var act = () => Policy.WrapAsync(
            Policy.Handle<DivideByZeroException>().Or<TimeoutRejectedException>().RetryAsync(100, (_, _) => { retries += 1; return Task.CompletedTask; }),
            Policy.Handle<DivideByZeroException>().Or<TimeoutRejectedException>().CircuitBreakerAsync(2, TimeSpan.FromSeconds(1), (_, _) => { open += 1; }, () => { }),
            Policy.TimeoutAsync(TimeSpan.FromSeconds(1), TimeoutStrategy.Pessimistic, (_, _, _) => { timeouts += 1; return Task.CompletedTask; })
        ).ExecuteAsync(async () =>
        {
            counter += 1;
            await Task.Delay(TimeSpan.FromSeconds(2));
            return Task.FromResult(1 / 1);
        });
        
        await act.Should().ThrowAsync<BrokenCircuitException>();
        counter.Should().Be(2, "first request was send and timed out, second request is our first retry but it is also timedout, third one is not sent because of circuit even so we have 100 retries");
        retries.Should().Be(2, "after initial request our first retry did timedout and second received BrokenCircuitException which is not handled thats why no more retries");
        open.Should().Be(1, "circuit opened");
        timeouts.Should().Be(2, "and all that is because each time we did timedout");
    }
    
    [Fact]
    public async Task step19_retry_circuit_timeout_without_propagation_of_TimeoutRejectedException__makes_no_sence()
    {
        var counter = 0;
        var retries = 0;
        var open = 0;
        var timeouts = 0;

        var act = () => Policy.WrapAsync(
            Policy.Handle<DivideByZeroException>().RetryAsync(100, (_, _) => { retries += 1; return Task.CompletedTask; }),
            Policy.Handle<DivideByZeroException>().CircuitBreakerAsync(2, TimeSpan.FromSeconds(1), (_, _) => { open += 1; }, () => { }),
            Policy.TimeoutAsync(TimeSpan.FromSeconds(1), TimeoutStrategy.Pessimistic, (_, _, _) => { timeouts += 1; return Task.CompletedTask; })
        ).ExecuteAsync(async () =>
        {
            counter += 1;
            await Task.Delay(TimeSpan.FromSeconds(2));
            return Task.FromResult(1 / 1);
        });
        
        await act.Should().ThrowAsync<TimeoutRejectedException>();
        counter.Should().Be(1, "first request was send and timed out, no more actual requests landed which is technically fine");
        retries.Should().Be(0, "but because of lack of handling TimeoutRejectedException retry did nothing");
        open.Should().Be(0, "as result circuit did not opened");
        timeouts.Should().Be(1, "and because we did only one request counter of timeouts has nothing interesting");
    }
    
    [Fact]
    public async Task step20_retry_circuit_timeout_when_only_retry_is_aware_about_timeout__leads_to_unwanted_calls()
    {
        var counter = 0;
        var retries = 0;
        var open = 0;
        var timeouts = 0;

        var act = () => Policy.WrapAsync(
            Policy.Handle<DivideByZeroException>().Or<TimeoutRejectedException>().RetryAsync(3, (_, _) => { retries += 1; return Task.CompletedTask; }),
            Policy.Handle<DivideByZeroException>().CircuitBreakerAsync(2, TimeSpan.FromSeconds(1), (_, _) => { open += 1; }, () => { }),
            Policy.TimeoutAsync(TimeSpan.FromSeconds(1), TimeoutStrategy.Pessimistic, (_, _, _) => { timeouts += 1; return Task.CompletedTask; })
        ).ExecuteAsync(async () =>
        {
            counter += 1;
            await Task.Delay(TimeSpan.FromSeconds(2));
            return Task.FromResult(1 / 1);
        });
        
        await act.Should().ThrowAsync<TimeoutRejectedException>();
        counter.Should().Be(4, "first request was send and timed out, and because we have 3 retries we did 3 more requests");
        retries.Should().Be(3, "we did all possible 3 retries");
        open.Should().Be(0, "but circuit did not opened because it is not aware and we are still continue to try to talk to 'dead' service");
        timeouts.Should().Be(4, "all 4 times we had timeout");
    }

    [Fact]
    public async Task step21_final()
    {
        var retries = 0;
        var timeouts = 0;
        
        var retry = Policy
            .Handle<TimeoutRejectedException>()
            .OrTransientHttpError()
            .WaitAndRetryAsync(new[]
            {
                TimeSpan.FromSeconds(0),
                TimeSpan.FromSeconds(0),
                TimeSpan.FromSeconds(0),
            }, (exception, timeSpan, retryCount, context) =>
            {
                retries += 1;
            });

        var circuit = Policy
            .Handle<TimeoutRejectedException>()
            .OrTransientHttpError()
            .CircuitBreakerAsync(2, TimeSpan.FromSeconds(5));
        
        var timeout = Policy.TimeoutAsync<HttpResponseMessage>(TimeSpan.FromSeconds(1), (context, timespan, task, exception) =>
        {
            timeouts += 1;
            return Task.CompletedTask;
        });

        var alwaysFailingFastRequests = 0;
        var alwaysOkfFastRequests = 0;
        var alwaysOkOneSlowRequests = 0;
        var alwaysOkOneFailRequests = 0;
        var alwaysOkTimeoutRequests = 0;
        var client = await FakeServer(async context =>
        {
            if (context.Request.Path == "/")
            {
                await context.Response.WriteAsync("Hello World");
                return;
            }

            if (context.Request.Path == "/always-failing-fast")
            {
                alwaysFailingFastRequests += 1;
                context.Response.StatusCode = 500;
                await context.Response.WriteAsync(alwaysFailingFastRequests.ToString());
                return;
            }
            
            if (context.Request.Path == "/always-ok-fast")
            {
                alwaysOkfFastRequests += 1;
                await context.Response.WriteAsync(alwaysOkfFastRequests.ToString());
                return;
            }
            
            if (context.Request.Path == "/always-ok-one-slow")
            {
                alwaysOkOneSlowRequests += 1;
                if (alwaysOkOneSlowRequests == 1)
                {
                    await Task.Delay(TimeSpan.FromSeconds(2));
                }
                await context.Response.WriteAsync(alwaysOkOneSlowRequests.ToString());
                return;
            }
            
            if (context.Request.Path == "/always-ok-one-fail")
            {
                alwaysOkOneFailRequests += 1;
                if (alwaysOkOneFailRequests == 1)
                {
                    context.Response.StatusCode = 500;
                }
                await context.Response.WriteAsync(alwaysOkOneFailRequests.ToString());
                return;
            }
            
            if (context.Request.Path == "/always-ok-timeout")
            {
                alwaysOkTimeoutRequests += 1;
                await Task.Delay(TimeSpan.FromSeconds(2));
                await context.Response.WriteAsync(alwaysOkTimeoutRequests.ToString());
                return;
            }

            context.Response.StatusCode = 404;
            await context.Response.WriteAsync($"unknown '{context.Request.Path}' path");
        }, retry, circuit, timeout);

        
        var act = async () => await client.GetAsync("/always-failing-fast");
        await act.Should().ThrowAsync<BrokenCircuitException>();
        alwaysFailingFastRequests.Should().Be(2, "even so we have configured 3 retries and expecting 4 requests in total, after second - circuit opens - aka service down, no need to harm it");
        retries.Should().Be(2, "first request is original one, second - is our first of three retry attempts, then we make one more try but because of circuit throw BrokenCircuitException and we do not handle it retry aborted");
        circuit.CircuitState.Should().Be(CircuitState.Open, "circuit opens after second failed request");
        timeouts.Should().Be(0, "and there should be no timeouts");

        act = async () => await client.GetAsync("/always-ok-fast");
        await act.Should().ThrowAsync<BrokenCircuitException>();
        circuit.CircuitState.Should().Be(CircuitState.Open);
        alwaysOkfFastRequests.Should().Be(0, "because circuit is open, no requests will be sent to actual server");
        retries.Should().Be(2);
        timeouts.Should().Be(0);

        await Task.Delay(TimeSpan.FromSeconds(5));
        circuit.CircuitState.Should().Be(CircuitState.HalfOpen, "after 5 sec circuit becomes half open so next request will decide its state");
        var response = await client.GetAsync("/always-ok-fast");
        response.Should().BeSuccessful("after 5 secs circuit half opens and next attempt decide its state");
        circuit.CircuitState.Should().Be(CircuitState.Closed, "we made successfull request");

        retries = 0;
        timeouts = 0;
        response = await client.GetAsync("/always-ok-one-slow");
        response.Should().BeSuccessful("even so we have timeout of 1sec, because of 3 retries we hit response");
        alwaysOkOneSlowRequests.Should().Be(2, "first request is original one and timedout, second is our first retry attempt and succeed");
        retries.Should().Be(1, "we did only one of three retries and succeed");
        timeouts.Should().Be(1, "original request timedout");
        
        retries = 0;
        timeouts = 0;
        response = await client.GetAsync("/always-ok-one-fail");
        response.Should().BeSuccessful("even so first request failed, because of 3 retries we hit response");
        alwaysOkOneFailRequests.Should().Be(2, "first request is original one and failed, second is our first retry attempt and succeed");
        retries.Should().Be(1, "we did only one of three retries and succeed");
        timeouts.Should().Be(0, "no timeouts occurr");

        
        circuit.CircuitState.Should().Be(CircuitState.Closed, "even so two previous demos has in total two failed responses circuit still closed because there was successful responses as well");

        timeouts = 0;
        retries = 0;
        act = async () => await client.GetAsync("/always-ok-timeout");
        await act.Should().ThrowAsync<BrokenCircuitException>();
        circuit.CircuitState.Should().Be(CircuitState.Open);
        alwaysOkTimeoutRequests.Should().Be(2, "after two unsuccessful attempt circuit opened so no more retries");
        timeouts.Should().Be(2, "all two attempts timedout");
        retries.Should().Be(2, "we did only one retry, in total that was two failed requests so circuit opens, our second retry immediately received BrokenCircuitException which is not handled and stops");


        await Task.Delay(TimeSpan.FromSeconds(5));
        alwaysFailingFastRequests = 0;
        circuit.CircuitState.Should().Be(CircuitState.HalfOpen, "after 5 sec circuit becomes half open so next request will decide its state");
        act = async () => await client.GetAsync("/always-failing-fast");
        await act.Should().ThrowAsync<BrokenCircuitException>();
        circuit.CircuitState.Should().Be(CircuitState.Open, "circuit opens immediately");
        alwaysFailingFastRequests.Should().Be(1, "after first unsuccessful response");

    }
}

Polly Cache Policy

Unfortunately it seems that we can not add Cache policy into pipeline

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;

using FluentAssertions;

using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Hosting.Server.Features;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.DependencyInjection;

using Polly;
using Polly.Caching;
using Polly.Caching.Memory;
using Polly.Extensions.Http;

using Xunit;

namespace HttpPolly;

public class CacheExperiments
{
    [Fact]
    public async Task SetupSample()
    {
        // Arrange
        var counter = 0;
        using var server = new WebHostBuilder().UseKestrel(o => o.Listen(IPAddress.Loopback, 0)).Configure(app =>
        {
            app.Run(async context =>
            {
                counter += 1;
                await context.Response.WriteAsync(DateTime.Now.ToShortTimeString());
            });
        }).Build();
        await server.StartAsync();
        var uri = new Uri(server.ServerFeatures.Get<IServerAddressesFeature>()!.Addresses.First());
        var http = new HttpClient();
        http.BaseAddress = uri;

        // Act
        var response = await http.GetAsync("/");

        // Assert
        response.Should().BeSuccessful();
        counter.Should().Be(1);
    }

    [Fact]
    public async Task OuterCacheSample()
    {
        // Arrange
        var counter = 0;
        using var server = new WebHostBuilder().UseKestrel(o => o.Listen(IPAddress.Loopback, 0)).Configure(app =>
        {
            app.Run(async context =>
            {
                counter += 1;
                await context.Response.WriteAsync(context.Request.Query["name"].FirstOrDefault()?.ToUpper() ?? "ANONYMOUS");
            });
        }).Build();
        await server.StartAsync();
        var uri = new Uri(server.ServerFeatures.Get<IServerAddressesFeature>()!.Addresses.First());

        var services = new ServiceCollection();
        services
            .AddHttpClient("demo", c => c.BaseAddress = uri)
            .AddPolicyHandler(HttpPolicyExtensions
                .HandleTransientHttpError()
                .RetryAsync(3));
        var provider = services.BuildServiceProvider();
        var factory = provider.GetRequiredService<IHttpClientFactory>();
        var http = factory.CreateClient("demo");

        var memoryCache = new MemoryCache(new MemoryCacheOptions());
        var memoryCacheProvider = new MemoryCacheProvider(memoryCache);
        var cache = Policy.CacheAsync<string>(memoryCacheProvider, new AbsoluteTtl(DateTimeOffset.Now.Date.AddDays(1)));

        // Act & Assert
        var response = await http.GetStringAsync("/?name=Alex");
        response.Should().Be("ALEX");
        counter.Should().Be(1);

        var key = "Alex";
        response = await cache.ExecuteAsync(async context => await http.GetStringAsync($"/?name={key}"), new Context(key));
        response.Should().Be("ALEX");
        counter.Should().Be(2);

        response = await cache.ExecuteAsync(async context => await http.GetStringAsync($"/?name={key}"), new Context(key));
        response.Should().Be("ALEX");
        counter.Should().Be(2, "response from cache");
    }
    
    [Fact]
    public async Task InnerCacheSample__WontWorkBecauseWeCannotSerializeHttpResponseMessage()
    {
        // Arrange
        var counter = 0;
        using var server = new WebHostBuilder().UseKestrel(o => o.Listen(IPAddress.Loopback, 0)).Configure(app =>
        {
            app.Run(async context =>
            {
                counter += 1;
                await context.Response.WriteAsync(context.Request.Query["name"].FirstOrDefault()?.ToUpper() ?? "ANONYMOUS");
            });
        }).Build();
        await server.StartAsync();
        var uri = new Uri(server.ServerFeatures.Get<IServerAddressesFeature>()!.Addresses.First());

        var memoryCache = new MemoryCache(new MemoryCacheOptions());
        var memoryCacheProvider = new MemoryCacheProvider(memoryCache);
        var cache = Policy.CacheAsync<HttpResponseMessage>(memoryCacheProvider, new AbsoluteTtl(DateTimeOffset.Now.Date.AddDays(1)));
        
        var services = new ServiceCollection();
        services.AddSingleton<AsyncCachePolicy<HttpResponseMessage>>(cache);
        services.AddSingleton<CachePolicyDelegatingHandler>();
        services
            .AddHttpClient("demo", c => c.BaseAddress = uri)
            .AddHttpMessageHandler<CachePolicyDelegatingHandler>()
            .AddPolicyHandler(HttpPolicyExtensions
                .HandleTransientHttpError()
                .RetryAsync(3));
        var provider = services.BuildServiceProvider();
        var factory = provider.GetRequiredService<IHttpClientFactory>();
        var http = factory.CreateClient("demo");


        // Act & Assert
        var response = await http.GetStringAsync("/?name=Alex");
        response.Should().Be("ALEX");
        counter.Should().Be(1);
        
        response = await http.GetStringAsync("/?name=Alex");
        // response.Should().Be("ALEX");
        counter.Should().Be(1, "second request comes from cache");
        response.Should().Be("", "even so we grab HttpResponseMessage from cache it has no body anymore");

    }

    [Fact]
    public async Task HttpResponseMessageCanBeSerializedButContentWillBeEmpty()
    {
        // Arrange
        var counter = 0;
        using var server = new WebHostBuilder().UseKestrel(o => o.Listen(IPAddress.Loopback, 0)).Configure(app =>
        {
            app.Run(async context =>
            {
                counter += 1;
                await context.Response.WriteAsync("Hello World");
            });
        }).Build();
        await server.StartAsync();
        var uri = new Uri(server.ServerFeatures.Get<IServerAddressesFeature>()!.Addresses.First());

        var http = new HttpClient();
        http.BaseAddress = uri;

        var httpResponseMessage = await http.GetAsync("/");
        httpResponseMessage.Should().BeSuccessful();
        counter.Should().Be(1);

        var serialized = JsonSerializer.Serialize(httpResponseMessage);
        serialized.Should().NotBeEmpty();
        serialized.Should().Contain("\"Content\":{\"Headers\":[]}", "content is not serialized");

        var response = await httpResponseMessage.Content.ReadAsStringAsync();
        response.Should().Be("Hello World");
        
        serialized = JsonSerializer.Serialize(httpResponseMessage);
        serialized.Should().NotBeEmpty();
        serialized.Should().Contain("\"Content\":{\"Headers\":[]}", "even after we read response");

        var act = () => JsonSerializer.Deserialize<HttpResponseMessage>(serialized);
        act.Should().Throw<NotSupportedException>("even worse we can not deserialize because of content");

        httpResponseMessage.Content = null;
        var withoutContent = JsonSerializer.Serialize(httpResponseMessage);
        withoutContent.Should().NotBeEmpty();
        act = () => JsonSerializer.Deserialize<HttpResponseMessage>(withoutContent);
        act.Should().Throw<NotSupportedException>("even if we try to nullify content");

    }
    
    [Fact]
    public async Task HttpResponseMessageCanBeSerializedByParts()
    {
        // Arrange
        var counter = 0;
        using var server = new WebHostBuilder().UseKestrel(o => o.Listen(IPAddress.Loopback, 0)).Configure(app =>
        {
            app.Run(async context =>
            {
                counter += 1;
                await context.Response.WriteAsJsonAsync(new { message = "Hello World" });
            });
        }).Build();
        await server.StartAsync();
        var uri = new Uri(server.ServerFeatures.Get<IServerAddressesFeature>()!.Addresses.First());

        var httpClient = new HttpClient();
        httpClient.BaseAddress = uri;

        var httpResponseMessage = await httpClient.GetAsync("/");
        httpResponseMessage.Should().BeSuccessful();
        counter.Should().Be(1);

        var bytes = await httpResponseMessage.Content.ReadAsByteArrayAsync();
        bytes.Should().NotBeEmpty("we may read response as bytes");
        var contentType = httpResponseMessage.Content.Headers.ContentType;
        contentType.Should().NotBeNull("and header(s)");

        var cacheable = new CacheableHttpResponseMessage
        {
            Headers = httpResponseMessage.Headers,
            Version = httpResponseMessage.Version,
            ReasonPhrase = httpResponseMessage.ReasonPhrase,
            StatusCode = httpResponseMessage.StatusCode,
            Bytes = bytes,
            ContentType = contentType
        };
        var serialized = JsonSerializer.Serialize(cacheable);
        serialized.Should().NotBeEmpty("we may serialize it now");
        var deserialized = JsonSerializer.Deserialize<CacheableHttpResponseMessage>(serialized);
        deserialized.Should().NotBeNull();

        var restored = new HttpResponseMessage();
        foreach (var header in deserialized!.Headers)
        {
            restored.Headers.Add(header.Key, header.Value);
        }
        restored.Version = deserialized!.Version;
        restored.ReasonPhrase = deserialized!.ReasonPhrase;
        restored.StatusCode = deserialized.StatusCode;
        restored.Content = new ByteArrayContent(deserialized!.Bytes);
        restored.Content.Headers.ContentType = deserialized!.ContentType;

        var str = await restored.Content.ReadAsStringAsync();
        str.Should().Contain("{\"message\":\"Hello World\"}");
    }
}

public class CachePolicyDelegatingHandler : DelegatingHandler
{
    private readonly AsyncCachePolicy<HttpResponseMessage> _cache;

    public CachePolicyDelegatingHandler(AsyncCachePolicy<HttpResponseMessage> cache)
    {
        _cache = cache;
    }

    public CachePolicyDelegatingHandler(HttpMessageHandler innerHandler, AsyncCachePolicy<HttpResponseMessage> cache) : base(innerHandler)
    {
        _cache = cache;
    }

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        return await _cache.ExecuteAsync(
            async (ctx, ct) => await base.SendAsync(request, ct), 
            new Context(request.RequestUri?.ToString()), // FIXME: should be more preciese
            cancellationToken
        );
        //return base.SendAsync(request, cancellationToken);
    }
}

public class CacheableHttpResponseMessage
{
    public HttpStatusCode StatusCode { get; set; }
    public IEnumerable<KeyValuePair<string, IEnumerable<string>>> Headers { get; set; }
    public Version Version { get; set; }
    public string? ReasonPhrase { get; set; }

    public byte[] Bytes { get; set; }
    public MediaTypeHeaderValue? ContentType { get; set; }
}

If we will try to add it as yet another handler we wont be able to calculate cache key

But even so we can add cusome delegate handler and wrap send method into our cache policy how should we serialize and save response message

It will require some more work to be done which becomes not trivial

Also cache probably should rely on response headers