Auto retry with Polly - A resilience and transient-fault-handling library

Building distributed and micro-services is incomplete without the implementation of Circuit breaker and auto-retry. This post takes a stab on Auto retry with Poly.

While building microservices and distributed systems, usually multiple services talk to each other. In some cases through reactive channels, like a stream or queue. For other cases through HTTP calls.

In this kind of scenario, micro-services or single responsibility services are distributed across the network. They can be within the same organization or external service integration like Google or Facebook.

When we deal with calling external services, network issues happen all the time. Since services will fail one time or another, hence the implementation of Auto retry and circuit breakers are extremely important.

Why would we ever Auto retry?

Some of the Http exceptions sometimes just auto heals. You call a service it fails, but the subsequent call just works. And we have seen it more than enough times.

With the implementation of auto-retry of service call within our code, we can handle this kind of one-off failure.

Building an Auto retry service call

Firstly, I will create an ASP.Net Core Web API, with the default scaffolding of Visual Studio 2019. In the ValuesController class Get method, I will throw an Exception.

using System;
using System.Collections.Generic;
using Microsoft.AspNetCore.Mvc;

namespace ErrorApi.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class ValuesController : ControllerBase
    {
        // GET api/values
        [HttpGet]
        public ActionResult<IEnumerable<string>> Get()
        {
            throw new Exception("Error");
        }
    }
}

Secondly, I will create another ASP.Net Core Web API “Polly.Demo” to implement the client.

Once the project is created:

Firstly, I will update the Startup class’s ConfigureServices method to configure HttpClientFactory for the service call
Secondly, I will update the default implementation of the ValuesController class’s Get to call the faulty service.

using System;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;

namespace Polly.Demo
{
    public class Startup
    {
        public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
        }

        public IConfiguration Configuration { get; }

        // This method gets called by the runtime. Use this method to add services to the container.
        public void ConfigureServices(IServiceCollection services)
        {
            services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_1);

            services.AddHttpClient("errorApiClient", c => 
            {
                c.BaseAddress = new Uri("http://localhost:50937/");
            });
        }

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IHostingEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }

            app.UseMvc();
        }
    }
}

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Newtonsoft.Json;

namespace Polly.Demo.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class ValuesController : ControllerBase
    {
        private readonly IHttpClientFactory httpClientFactory;

        public ValuesController(IHttpClientFactory httpClientFactory)
        {
            this.httpClientFactory = httpClientFactory;
        }

        // GET api/values
        [HttpGet]
        public ActionResult<IEnumerable<string>> Get()
        {
            var httpClient = httpClientFactory.CreateClient("errorApiClient");

            var response = httpClient.GetAsync("api/values").Result;
            return JsonConvert.DeserializeObject<string[]>(response.Content.ReadAsStringAsync().Result);
        }
    }
}

Finally, when I will run this code and try to access the /api/values, it will fail due to the other service call failure, which is what is expected.

Implementing Auto retry

Now, I will update the implementation of the ValuesController of the “Polly.Demo” project to implement auto-retry.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Newtonsoft.Json;

namespace Polly.Demo.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class ValuesController : ControllerBase
    {
        private readonly IHttpClientFactory httpClientFactory;

        public ValuesController(IHttpClientFactory httpClientFactory)
        {
            this.httpClientFactory = httpClientFactory;
        }

        // GET api/values
        [HttpGet]
        public ActionResult<IEnumerable<string>> Get()
        {
            try
            {
                return GetValues();
            }
            catch
            {
                // Retry 3 times
                for (var count = 1; count < 3; count++)
                {
                    try
                    {
                        Thread.Sleep(1000 * count);
                        return GetValues();
                    }
                    catch { }
                }
            }
        }

        private string[] GetValues()
        {
            var httpClient = httpClientFactory.CreateClient("errorApiClient");

            var response = httpClient.GetAsync("api/values").Result;
            return JsonConvert.DeserializeObject<string[]>(response.Content.ReadAsStringAsync().Result);
        }
     }
}

In the above implementation, I will retry 3 exception cases with a wait time of 1, 2 and 3 seconds respectively before each try.

What is Polly?

Polly is a resilience and transient-fault-handling library. It provides an implementation of Auto retry, Circuit breaker, and more resilience features through fluent configuration.

Most importantly, Polly manages all this in a thread-safe manner. It is transparent to the application code. Meaning, the application does not have to change. Only the HttpClientFactory configuration changes can manage how Polly is configured.

Implementing Polly for Auto retry

Firstly I will add the Nuget package Microsoft.Extensions.Http.Polly. This will add all the necessary namespaces needed for Polly integration.

Secondly, I will update the Startup class to add AddTransientHttpErrorPolicy to the HttpClientFactory registration. And configure it to Auto retry on Http exception for three times.

Finally, I will update ValuesController to remove the custom code for auto retries.

using System;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;

namespace Polly.Demo
{
    public class Startup
    {
        public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
        }

        public IConfiguration Configuration { get; }

        // This method gets called by the runtime. Use this method to add services to the container.
        public void ConfigureServices(IServiceCollection services)
        {
            services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_1);

            services.AddHttpClient("errorApiClient", c => 
            {
                c.BaseAddress = new Uri("http://localhost:50937/");
            }).AddTransientHttpErrorPolicy(policy => policy.WaitAndRetryAsync(new[] 
            {
                TimeSpan.FromSeconds(1),
                TimeSpan.FromSeconds(5),
                TimeSpan.FromSeconds(15)
            }));
        }

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IHostingEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }

            app.UseMvc();
        }
    }
}

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Newtonsoft.Json;

namespace Polly.Demo.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class ValuesController : ControllerBase
    {
        private readonly IHttpClientFactory httpClientFactory;

        public ValuesController(IHttpClientFactory httpClientFactory)
        {
            this.httpClientFactory = httpClientFactory;
        }

        // GET api/values
        [HttpGet]
        public ActionResult<IEnumerable<string>> Get()
        {
            return GetValues();
        }

        private string[] GetValues()
        {
            var httpClient = httpClientFactory.CreateClient("errorApiClient");

            var response = httpClient.GetAsync("api/values").Result;
            return JsonConvert.DeserializeObject<string[]>(response.Content.ReadAsStringAsync().Result);
        }
    }
}

Conclusion

In conclusion, you can see how transparent Polly makes implementing Auto retries. In my next post, I will explore the Circuit Breaker with Polly.

Above all, the entire session is available on YouTube: https://youtu.be/oitGaSxIRVY