Graceful shutdown with Generic Host in .NET Core 2.1

.NET Core 2.1 introduced new Generic Host, which allows to host non-HTTP workloads with all benefits of Web Host. Currently, there is no much information and recipes with it, but I used following articles as a starting point:

https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/generic-host?view=aspnetcore-2.1

https://docs.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services?view=aspnetcore-2.1

https://docs.microsoft.com/en-us/dotnet/standard/microservices-architecture/multi-container-microservice-net-applications/background-tasks-with-ihostedservice

My .NET Core application starts, listens for new requests via RabbitMQ message broker and shuts down by user request (usually by Ctrl+C in console). However, shutdown is not graceful - application still have unfinished background threads while it returns control to OS. I see it by console messages - when I press Ctrl+C in console I see few lines of console output from my application, then OS command prompt and then again console output from my application.

Here is my code:

Program.cs

public class Program
{
    public static async Task Main(string[] args)
    {
        var host = new HostBuilder()
            .ConfigureHostConfiguration(config =>
            {
                config.SetBasePath(AppContext.BaseDirectory);
                config.AddEnvironmentVariables(prefix: "ASPNETCORE_");
                config.AddJsonFile("hostsettings.json", optional: true);
            })
            .ConfigureAppConfiguration((context, config) =>
            {
                var env = context.HostingEnvironment;
                config.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true);
                config.AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true, reloadOnChange: true);
                if (env.IsProduction())
                    config.AddDockerSecrets();
                config.AddEnvironmentVariables();
            })
            .ConfigureServices((context, services) =>
            {
                services.AddLogging();
                services.AddHostedService<WorkerPoolHostedService>();
                // ... other services
            })
            .ConfigureLogging((context, logging) =>
            {
                if (context.HostingEnvironment.IsDevelopment())
                    logging.AddDebug();

                logging.AddSerilog(dispose: true);

                Log.Logger = new LoggerConfiguration()
                    .ReadFrom.Configuration(context.Configuration)
                    .CreateLogger();
            })
            .UseConsoleLifetime()
            .Build();

        await host.RunAsync();
    }
}

WorkerPoolHostedService.cs

internal class WorkerPoolHostedService : IHostedService
{
    private IList<VideoProcessingWorker> _workers;
    private CancellationTokenSource _stoppingCts = new CancellationTokenSource();

    protected WorkerPoolConfiguration WorkerPoolConfiguration { get; }
    protected RabbitMqConfiguration RabbitMqConfiguration { get; }
    protected IServiceProvider ServiceProvider { get; }
    protected ILogger<WorkerPoolHostedService> Logger { get; }

    public WorkerPoolHostedService(
        IConfiguration configuration,
        IServiceProvider serviceProvider,
        ILogger<WorkerPoolHostedService> logger)
    {
        this.WorkerPoolConfiguration = new WorkerPoolConfiguration(configuration);
        this.RabbitMqConfiguration = new RabbitMqConfiguration(configuration);
        this.ServiceProvider = serviceProvider;
        this.Logger = logger;
    }

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        var connectionFactory = new ConnectionFactory
        {
            AutomaticRecoveryEnabled = true,
            UserName = this.RabbitMqConfiguration.Username,
            Password = this.RabbitMqConfiguration.Password,
            HostName = this.RabbitMqConfiguration.Hostname,
            Port = this.RabbitMqConfiguration.Port,
            VirtualHost = this.RabbitMqConfiguration.VirtualHost
        };

        _workers = Enumerable.Range(0, this.WorkerPoolConfiguration.WorkerCount)
            .Select(i => new VideoProcessingWorker(
                connectionFactory: connectionFactory,
                serviceScopeFactory: this.ServiceProvider.GetRequiredService<IServiceScopeFactory>(),
                logger: this.ServiceProvider.GetRequiredService<ILogger<VideoProcessingWorker>>(),
                cancellationToken: _stoppingCts.Token))
            .ToList();

        this.Logger.LogInformation("Worker pool started with {0} workers.", this.WorkerPoolConfiguration.WorkerCount);
    }

    public async Task StopAsync(CancellationToken cancellationToken)
    {
        this.Logger.LogInformation("Stopping working pool...");

        try
        {
            _stoppingCts.Cancel();
            await Task.WhenAll(_workers.SelectMany(w => w.ActiveTasks).ToArray());
        }
        catch (AggregateException ae)
        {
            ae.Handle((Exception exc) =>
            {
                this.Logger.LogError(exc, "Error while cancelling workers");
                return true;
            });
        }
        finally
        {
            if (_workers != null)
            {
                foreach (var worker in _workers)
                    worker.Dispose();
                _workers = null;
            }
        }
    }
}

VideoProcessingWorker.cs

internal class VideoProcessingWorker : IDisposable
{
    private readonly Guid _id = Guid.NewGuid();
    private bool _disposed = false;

    protected IConnection Connection { get; }
    protected IModel Channel { get; }
    protected IServiceScopeFactory ServiceScopeFactory { get; }
    protected ILogger<VideoProcessingWorker> Logger { get; }
    protected CancellationToken CancellationToken { get; }

    public VideoProcessingWorker(
        IConnectionFactory connectionFactory,
        IServiceScopeFactory serviceScopeFactory,
        ILogger<VideoProcessingWorker> logger,
        CancellationToken cancellationToken)
    {
        this.Connection = connectionFactory.CreateConnection();
        this.Channel = this.Connection.CreateModel();
        this.Channel.BasicQos(prefetchSize: 0, prefetchCount: 1, global: false);
        this.ServiceScopeFactory = serviceScopeFactory;
        this.Logger = logger;
        this.CancellationToken = cancellationToken;

        #region [ Declare ]

        // ...

        #endregion

        #region [ Consume ]

        // ...

        #endregion
    }

    // ... worker logic ...

    public void Dispose()
    {
        if (!_disposed)
        {
            this.Channel.Close(200, "Goodbye");
            this.Channel.Dispose();
            this.Connection.Close();
            this.Connection.Dispose();
            this.Logger.LogDebug("Worker {0}: disposed.", _id);
        }
        _disposed = true;
    }
}

So, when I press Ctrl+C I see following output in console (when there is no request processing):

Stopping working pool... command prompt Worker id: disposed.

How to shutdown gracefully?


You need IApplicationLifetime. This provides you with all the needed information about application start and shutdown. You can even trigger the shutdown with it via appLifetime.StopApplication();

Look at https://github.com/aspnet/Docs/blob/66916c2ed3874ed9b000dfd1cab53ef68e84a0f7/aspnetcore/fundamentals/host/generic-host/samples/2.x/GenericHostSample/LifetimeEventsHostedService.cs

Snippet(if the link becomes invalid):

public Task StartAsync(CancellationToken cancellationToken)
{
    appLifetime.ApplicationStarted.Register(OnStarted);
    appLifetime.ApplicationStopping.Register(OnStopping);
    appLifetime.ApplicationStopped.Register(OnStopped);

    return Task.CompletedTask;
}

c# - Graceful shutdown with Generic Host in .NET Core 2.1, NET Core 2.1 introduced new Generic Host, which allows to host non-HTTP workloads with all benefits of Web Host. Currently, there is no much information and  Generic Host is new in ASP.NET Core 2.1 and isn't suitable for web hosting scenarios. For web hosting scenarios, use the Web Host. Generic Host will replace Web Host in a future release and act as the primary host API in both HTTP and non-HTTP scenarios. View or download sample code (how to download)


I'll share some patterns I think works very well for non-WebHost projects.

namespace MyNamespace
{
    public class MyService : BackgroundService
    {
        private readonly IServiceProvider _serviceProvider;
        private readonly IApplicationLifetime _appLifetime;

        public QbdSkuVaultIntSchedulerService(
            IServiceProvider serviceProvider,
            IApplicationLifetime appLifetime)
        {
            _serviceProvider = serviceProvider;
            _appLifetime = appLifetime;
        }

        protected override Task ExecuteAsync(CancellationToken stoppingToken)
        {
            _appLifetime.ApplicationStopped.Register(OnStopped);

            return RunAsync(stoppingToken);
        }

        private async Task RunAsync(CancellationToken token)
        {
            while (!token.IsCancellationRequested)
            {
                using (var scope = _serviceProvider.CreateScope())
                {
                    var runner = scope.ServiceProvider.GetRequiredService<IMyJobRunner>();
                    await runner.RunAsync();
                }
            }
        }

        public void OnStopped()
        {
            Log.Information("Window will close automatically in 20 seconds.");
            Task.Delay(20000).GetAwaiter().GetResult();
        }
    }
}

A couple notes about this class:

  1. I'm using the BackgroundService abstract class to represent my service. It's available in the Microsoft.Extensions.Hosting.Abstractions package. I believe this is planned to be in .NET Core 3.0 out of the box.
  2. The ExecuteAsync method needs to return a Task representing the running service. Note: If you have a synchronous service wrap your "Run" method in Task.Run().
  3. If you want to do additional setup or teardown for your service you can inject the app lifetime service and hook into events. I added an event to be fired after the service is fully stopped.
  4. Because you don't have the auto-magic of new scope creation for each web request as you do in MVC projects you have to create your own scope for scoped services. Inject IServiceProvider into the service to do that. All dependencies on the scope should be added to the DI container using AddScoped().

Set up the host in Main( string[] args ) so that it shuts down gracefully when CTRL+C / SIGTERM is called:

IHost host = new HostBuilder()
    .ConfigureServices( ( hostContext, services ) =>
    {
        services.AddHostedService<MyService>();
    })
    .UseConsoleLifetime()
    .Build();

host.Run();  // use RunAsync() if you have access to async Main()

I've found this set of patterns to work very well outside of ASP.NET applications.

Be aware that Microsoft has built against .NET Standard so you don't need to be on .NET Core to take advantage of these new conveniences. If you're working in Framework just add the relevant NuGet packages. The package is built against .NET Standard 2.0 so you need to be on Framework 4.6.1 or above. You can find the code for all of the infrastructure here and feel free to poke around at the implementations for all the abstractions you are working with: https://github.com/aspnet/Extensions

.NET Generic Host, NET Core Generic Host, which is responsible for app startup and lifetime management: control over app startup and graceful shutdown. My new book, ASP.NET Core in Action, Second Edition is available now, and supports .NET Core 3.1! May 12, 2020 in ASP.NET Core Generic Host HostBuilder ~ 5 min read. Extending the shutdown timeout setting to ensure graceful IHostedService shutdown


In Startup.cs, you can terminate the application with the Kill() method of the current process:

        public void Configure(IHostApplicationLifetime appLifetime)
        {
            appLifetime.ApplicationStarted.Register(() =>
            {
                Console.WriteLine("Press Ctrl+C to shut down.");
            });

            appLifetime.ApplicationStopped.Register(() =>
            {
                Console.WriteLine("Shutting down...");
                System.Diagnostics.Process.GetCurrentProcess().Kill();
            });
        }

Program.cs

Don't forget to use UseConsoleLifetime() while building the host.

Host.CreateDefaultBuilder(args).UseConsoleLifetime(opts => opts.SuppressStatusMessages = true);

Background tasks with hosted services in ASP.NET Core, NET Core, background tasks can be implemented as hosted services. A hosted StopAsync(CancellationToken): Triggered when the host is performing a graceful shutdown. ShutdownTimeout when using Generic Host. As the application will be running on IIS, a pool recycle can happen after the controller has returned a response. I would like to implement a way to allow the application to do a graceful shutdown - finishing it's currently executing tasks, but not accepting any new. I am having difficulties having the graceful shutdown run to end.


Extending the shutdown timeout setting to ensure graceful , NET Core Generic Host HostBuilder ~ 5 min read. Extending the shutdown timeout setting to ensure graceful IHostedService shutdown. I wrote a .NET core console application to run as a task in AWS ECS Fargate. Basically, the console application starts a processing thread and then needs to just keep running (much like a Kestrel web host.) Initially, I drop in a Console.ReadKey() thinking it will never hit and the app will keep running. Well that leads to .NET exception


Lifecycle of Generic Host Background Services, NET Core 2.1 introduced something called the Generic Host, which is a this by providing discrete initialization, execution, and shutdown activities. which allows them to gracefully exit when the application is shutting down. When using the Generic Host, StartAsync is called before ApplicationStarted is triggered. StopAsync(CancellationToken): Triggered when the host is performing a graceful shutdown. StopAsync contains the logic to end the background task. Implement IDisposable and finalizers (destructors) to dispose of any unmanaged resources.


Having Fun with the .NET Core Generic Host · Jonathan's Blog, NET Core is cross-platform, hosting inside IIS isn't always an option. It is currently planned for the 2.1 release and although there is no preview Triggered when the application host is performing a graceful shutdown. The goal of the Generic Host is to decouple the HTTP pipeline from the Web Host API to enable a wider array of host scenarios. The Generic Host is new in ASP.NET Core 2.1 and isn’t suitable for web hosting, but is ideally used for Console Applications which are typically used to develop headless services. The Generic Host library is available