Start extremely long running processes through a REST request

rest api long-running process java
long running http request timeout
web application long running processes
long running apis
rest api callback c#
long time request
rest api asynchronous callback
asynchronous rest api

I'm working at an automation firm so we create processes for industrial automation. Previously this automation was done on the machine side of things, but we're slowly transitioning to controlling the machines with c#.

On my current project the production for one day takes about 2 hours. The operators of the factory have a web interface that we created in c# using asp.net core MVC in which they can start/pause/stop this production process.

When starting the process we await a function in our controller that is basically a while loop that controls this 2h long production process.

The problem is now that when I send out the REST request to start the production this request takes 2h to complete, I would prefer this request immediately completes and the production process starts on the background of my asp.net core application.

First I thought I could just leave out the await and simply do this in my controller (simplified code):

_ = _productionController.StartLongProcess(); // This contains the while loop
return Ok();

But since _productionController is scoped and all its dependencies are as well, these immediately get disposed of when the method returns and I can't access my database anymore for example.

The process should be able to continuously talk to our database to save information about the production process in case something fails, that way we can always pick off where we left off.

My question to you is now, are we tackling this the wrong way? I imagine it's bad practice to start these long running processes in the asp.net controller.

How do I make sure I always have access to my DatabaseContext in this long running process even though the REST request has already ended. Create a separate scope only for this method?

Starting ASP.NET Core 2.1, the right way to do this (within asp.net) is to extend BackgroundService (or implement IHostedService).

Incoming requests can tell the background service to start the long-running operation and return immediately. You'll of course need to handle cases where duplicate requests are sent in, or new requests are sent in before the existing request is completed.

The documentation page has an example where the BackgroundService reads commands of a queue and processes them one at a time.

How do I make sure I always have access to my DatabaseContext in this long running process even though the REST request has already ended. Create a separate scope only for this method?

Yes, create a separate scope.

My question to you is now, are we tackling this the wrong way? I imagine it's bad practice to start these long running processes in the asp.net controller.

We've done something similar in the past. As long as fault-tolerance (particularly w.r.t. app restarts) and idempotence are built into the long-running-operation's logic, you should be good to go.

Long running REST API with queues, You have a REST service, which receives requests from the client. Your problem that by your long running tasks the client connection timeouts, The only extra feature will be the subscription for the notification links, which� The long running operation itself is turned into a resource, created using the original request with a response telling the client where to find the results. ese will be available once the operation running in the background completes.

REST requests are expected to be short, a few seconds at maximum. So best practice here would be to offload a long running task to a background service and return a token where you can poll the service if the operation has already finished.

The background service could be a BackGroundWorker in Net Core. This is easy but not really fault tolerant, so some sort of db and retry logic could be good.

If you are in an intranet, you could also move to an inherently asynchronous protocol like RabbitMQ, where you send a StartOperation Message and then receive a Started Message when the process has completed.

Handling long Web Requests with Asynchronous Request Processing, The problems with long requests run directly off a Web application are many: The application server could be very, very simple and simply be directly fired then starts up a separate EXE file to process the event by passing the event id to it. I think it is the right decision to not include the long running process as part of the API request as long as it is not relevant to it. Before async and all, BackgroundWorker was most used for threading long running process. But now there are more options. The most easiest would be QueueBackgroundWorkItem that has been added with .NET 4.5.2.

Following is my understanding of the issue that you have posted:

  1. You want to initiate a long running call, via Rest api call
  2. You want to use the Async call, but not sure how to maintain the DB context for a long running call which is used for db communication on regular basis during the operation

Couple of important points:

  1. Mostly you are not clear regarding working of the Async calls
  2. When you make an Async call, then it stores the current thread synchronization context for the continuity using state machine, it doesn't block any thread pool thread, it utilize the hardware based concurrency
  3. Can use ConfigureAwait(false) on backend to avoid explicit reentry in the current synchronization context, which is better for performance
  4. Only challenge with Async calls to be genuine async the complete chain need to be Async enabled from the entry point, else the benefits can't be reaped, if you use Task.Wait or Task.Result anywhere, infact may even cause a deadlock in the ASP.Net

Regarding the long running operation, following are the options

  1. A Simple async call as suggested above, though it can help you can make large number of async calls (thus scalability) but context will be lost if the client goes away and no way to reap the status of operation back
  2. You can make a fire and forget call, and use a mechanism like ASP.Net SignalR, which is like IObservable over the network and can help in notifying the client when the processing finish
  3. Best option would be using a messaging queue like Rabbit MQ, which doesn't run the risk of client going down, it acts a producer consumer and can notify when the client comes up, in this case MQ can be notified when the process finish and thus client can be informed. MQ can be used for both incoming and response message in an async manner

In case, where client wants to periodically come up and check the status of the request, then DB persistence is important, which can be updated at regular intervals and it can be checked what's the status of the long running process.

Pattern: Long Running Operation with Polling, Use polling to avoid client timeouts when waiting long running operation results The server may need to perform expensive computations to process client requests Request cancellation: An explicit mechanism consistent with the REST or the POST-PUT Creation pattern where the job is started with the POST or the� An architect demonstrates the processes that a request to an API and response from an API go through in order to deliver data from the API to the client.

Another option would be to use Hangfire. It will allow you to Enqueue the work that you want to execute to a persistent store e.g. SQL Server, MSMQ, Redis depending on what you have in your infrastructure. The job will then be picked up by a worker which can also run in the ASP.NET process or a windows service. It's distributed too so you can have a number of instances of the workers running. Also supports retrying failed jobs and has a dashboard to view the jobs. Best of all, it's free!

var jobId = BackgroundJob.Enqueue(() => ExecuteLongRunningProcess(parameter1));

https://www.hangfire.io/

Trouble designing REST API that uses both long-running jobs, and , RESTful Casuistry has a pretty good discussion of restart; if you aren't already familiar with it, you should have a look at it. Usually for long� Note: The file content data object, FileScriptsFile, without suffix, TPM REST API processes the request as retrieving file content data object with .reference suffix. HTTP Message Type Example

Asynchronous Request-Reply Pattern, In most cases, APIs for a client application are designed to respond quickly, on the client applications also apply for server-to-server REST API calls in distributed and the action to be performed before starting the long running process. See Perform long-running tasks with the webhook action pattern. A typical customer enrollment may involve sending a KYC (Know Your Customer) request to an external agency, registering the customer, creating an account, printing debit cards, sending a mail, etc. These steps may be overlapping and the process would be long-running with several failure points.

Dealing with long-running HTTP Requests and Timeouts in Phoenix, In this article we see how to simulate a long-running request and Phoenix is fast and highly concurrent and it can process HTTP requests in less than a millisecond. It is our (if we need, for example, to upload a really large file to AWS S3) In this example we start two tasks and then await for the result. You can specify that a task will be long-running by supplying TaskCreationOptions.LongRunning to the constructor of the task (or the Factory.StartNew method). What this actually does internally is generate a new thread outside of the threadpool to run the task on. var task3 = new Task( () => MyLongRunningMethod(), TaskCreationOptions.LongRunning); task3.Start();

Long-Running Processes Behind REST APIs, The service allows a client to POST records which are stored in a SQL Server data was most used for threading long running process. For instance, for a long but linear process, you can send one hundred dots, flushing after each character. If the client side (such as a JavaScript application) knows that it should expect exactly 100 characters, it can match it with a progress bar to show to the user. Another example concerns a process which consists of several non-linear steps.

Comments
  • If you make an Async call using Db context, then its not a blocking operation for the system resources since the call goes into the background, and that's the way to make any async call
  • these immediately get disposed of when the method returns: Create a ProcessManager class, register it as a service, inject this service into controller, start the process by processMgr.StartLongProcess() . As this ProcessManager is managed by DI container, it won't be disposed when the controller action returns.
  • @itminus which DI container setting / configuration would you utilize to avoid the Dispose, it would certainly manage the object lifetime, but not post the call return, until and unless you make it singleton, but that's not a good idea here
  • Background service is a good addition in ASP.Net core, but I doubt it provides options for custom management, it is more for starting a background process when the application process is loaded in web server, it automatically proceeds in the background but in this case OP needs more control over the execution of the background process and want to know the running status
  • @MrinalKamboj what makes you think that's hard to achieve?
  • The example posted doesn't provide the suggested information clearly. From the documentation these are background services, which run without much of user intervention and if the use case needs continuous status feedback to the client, then there doesn't seems to be an out of box option. Would be great if you can point to an example or provide a code
  • @MrinalKamboj the linked example is clear enough IMO. If you have specific questions let me know and I'll try to answer them. As for an out-of-box option to report feedback -- one doesn't exist and that's a good thing IMO. The service could emit events, or expose a private-set property that can be polled, etc. Think of IHostedService as something similar to WebHost (but without the webserver). WebHost has the application loop, here we write our own, that's the only difference. The OP wanted to offload a task from the HTTP request and this answers it.
  • Rest calls can be long async operations, which are notified via SignalR, thus non blocking. Messaging queue is a good idea, but it needs to notify the completion to the client too, that's complete cycle
  • I would even suggest moving the process to an Azure function which can be scheduled or called via an http request. You can send a notification when it's done
  • @richardterris Azure functions cannot be running for hours, they are meant for short quick operations by design
  • @MrinalKamboj on the most basic level, HTTP calls are not completed, for the client until the request returns from the server. OfC you can build an asnyc protocl on top of that, but I often dislike this - using a full async protocl feels more natural.
  • @ChristianSauer that's for Synchronous calls, Async calls don't really need to wait they do get notified when response is available