Calling Azure Function calls fail with "Function host is not running"

azure function error handling
azure function a host error has occurred
azure function crash
logic app azure function queries
host lock lease acquired by instance id
host is not running; requests will be queued but not execute until the host is started
azure function job host stopped
azure function queue trigger

I've got a function which I've been playing with for a couple of months (even having some issue as can be seen here) and I've started running into issues when I start to scale up the usage from tests to "production".

The function takes in a 2 values, looks for a matching value in an Azure Table, removes it if it finds it, then adds the new values in together. This works fine in testing. As soon as I scale up, from a few calls every second, to 20-30 calls a second, it fails with the response mentioned above.

The actual issue, when I dive in using Insights, is that a System.InvalidOperationException exception is thrown. Here's the call stack:

System.InvalidOperationException:
   at Microsoft.Azure.WebJobs.Script.WebHost.SecretManager+<PersistSecretsAsync>d__27`1.MoveNext (Microsoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\SecretManager.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 440)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Microsoft.Azure.WebJobs.Script.WebHost.SecretManager+<GetHostSecretsAsync>d__12.MoveNext (Microsoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\SecretManager.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 104)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsSdkExtensionHookProvider+<GetOrCreateExtensionKey>d__6.MoveNext (Microsoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebHooks\WebJobsSdkExtensionHookProvider.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 71)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089)
   at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsSdkExtensionHookProvider.GetExtensionWebHookRoute (Microsoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebHooks\WebJobsSdkExtensionHookProvider.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 64)
   at Microsoft.Azure.WebJobs.Script.WebHost.WebJobsSdkExtensionHookProvider.GetUrl (Microsoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebHooks\WebJobsSdkExtensionHookProvider.csMicrosoft.Azure.WebJobs.Script.WebHost, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 49)
   at Microsoft.Azure.WebJobs.Host.Config.ExtensionConfigContext.GetWebhookHandler (Microsoft.Azure.WebJobs.Host, Version=2.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at Microsoft.Azure.WebJobs.Extensions.EventGrid.EventGridExtensionConfig.Initialize (Microsoft.Azure.WebJobs.Extensions.EventGrid, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
   at Microsoft.Azure.WebJobs.Host.Executors.JobHostConfigurationExtensions.InvokeExtensionConfigProviders (Microsoft.Azure.WebJobs.Host, Version=2.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at Microsoft.Azure.WebJobs.Host.Executors.JobHostConfigurationExtensions.CreateStaticServices (Microsoft.Azure.WebJobs.Host, Version=2.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at Microsoft.Azure.WebJobs.JobHost.InitializeServices (Microsoft.Azure.WebJobs.Host, Version=2.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at Microsoft.Azure.WebJobs.Script.Utility.CreateMetadataProvider (Microsoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Utility.csMicrosoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 362)
   at Microsoft.Azure.WebJobs.Script.ScriptHost.LoadBindingExtensions (Microsoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.csMicrosoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 966)
   at Microsoft.Azure.WebJobs.Script.ScriptHost.Initialize (Microsoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.csMicrosoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 299)
   at Microsoft.Azure.WebJobs.Script.ScriptHostManager.RunAndBlock (Microsoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=nullMicrosoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: C:\projects\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHostManager.csMicrosoft.Azure.WebJobs.Script, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: 178)

The message with it is:

Repository has more than 10 non-decryptable secrets backups (host). 

I unfortunately have no idea what that means. Any searches for this result in just a few threads which talk about regenerating the keys, but again, I don't really know what that means. Some threads mention moving back to V1 of the functions, but I'm already on V1, so that's no an option.

What is going on with this function and how do I fix it?

For any Azure employees looking at this, my function ID is:

2019-01-18T15:52:18.658 [Info] Function started (Id=fc6850e8-7554-46d8-81ec-4d1697c7b572)

Deleting D:\home\data\Functions\secrets solved the issue.

In general I learned that whatever strange behavior you get from Azure Function, Kudu is always your best tool for investigation.

Add and call Azure Functions from Azure Logic Apps, When working with logic apps, the same requirements apply to function apps and functions whether they are existing or new: Your function app  This could be a base for an alert. Since the function is supposed to run every 10 minutes, we should expect to see a successful request every 10 minute. If we don’t, it means the function didn’t run or failed. Azure Application Analytics takes a few minutes to make the telemetry available. It is usually done under 10 minutes.

The message indicated that this is something to do with the host-level keys (secrets) in your Function.

So even though I don't have a clear fix to this (as I have never experienced this issue), I would suggest that you check the host.json in D:\home\data\Functions\secrets folder and see if anything unusual there, e.g. there are more than 10 keys - as the error message indicated.

Azure Functions error handling guidance, Calls to APIs of underlying Azure services; Calls to REST endpoints; Calls to client libraries, packages, or third-party APIs. Following solid error  I am trying to call an Azure Function from ADF. Azure Function has Route implemented as shown below. [FunctionName("UpdateStatus_V1")] public static async Task<IActionResult> Run([HttpTrigger(AuthorizationLevel.Function, "post", Route = "V1/UpdateStatus")] HttpRequest req, ILogger log) Also the host file has a routePrefix in Azure Function

If you have a custom Startup, there might be an error that blocks the host to start.

In the portal, go to the function's page (the one where you can see the content of function.json), there will be (likely) an error message. From this, you can get an idea where to look for the error. In my case, an exception was thrown when I tried to connect to the Azure Key Vault from the startup class, but List permission was not given to the function.

Calling azure functions key management API fails in one environment, Calling azure functions key management API fails in one environment Does the key management API take hours to warm up and accept calls? 0. Share. In my case Azure Function name should be part of URL, but not function name. ADF pipeline Azure Function step throws: "message": "Call to provided Azure function '' failed with status-'NotFound' and message - 'Invoking Azure function failed with HttpStatusCode - NotFound.'.",

Little addition to what the solutions are mentioned above. I was facing this same issue and even after deleting secret files from D:\home\data\Functions\secrets folder on KUDU site, I was still getting this same error. I was able to fix this by deleting host.*.snapshot.*.json files from azure-webjobs-secrets folder, which was there in my blob storage on azure. Basically there were 10 such snapshot files.

External HTTP Request within Azure Functions Timeout in 21 secs , Within the function it calls an HTTP request to get RSS feed. However The first curl call was fine but any subsequent calls all failed. From the  You can set up an Azure Function Activity to call the Durable Function, which will return a response with a different URI, such as this example. Because statusQueryGetUri returns HTTP Status 202 while the function is running, you can poll the status of the function by using a Web Activity.

Durable Functions Fail to run · Issue #148 · Azure/azure-functions , Call ReadAsync or set AllowSynchronousIO to true instead.","ExceptionType":"​System.InvalidOperationException","StackTrace":" at Microsoft. To solve our function call issue, you can basically use Storage Bindings and Triggers to let them communicate. The function caller returns / creates a new storage entry. This entry contains the required data for processing. At this point the function will be triggered when new entries are added.

Creating a C# Azure Function to Call Microsoft Graph, Since we'll be calling Microsoft Graph and the MSAL .Net SDK libraries we'll add the supporting NuGet packages with the following: Run “dotnet  To run a non HTTP-triggered function, you need a way to send a request to Azure to run the function. The URL used to make this request takes a specific form. Host name: The function app's public location that is made up from the function app's name plus azurewebsites.net or your custom domain.

Azure Function call failing, Azure Function call failing. We were using CloudScript to directly call the Facebook API to send messages to players. However, because the  Calling an Azure Functions mean paying for the additional compute to a achieve the same behaviour which we are already paying for in Data Factory is used directly. Authentication needs to be handled from Data Factory to the Azure Function App and then from the Azure Function back to the same Data Factory.

Comments
  • By scale up the usage from tests to "production", do you mean swapping slot or what usage? Also is your function app on Consumption plan?
  • I have the same issue and it is a consumption plan.
  • What is the purpose of this folder? What are the effects of deleting it?
  • Good question. As I understood that is the place where KUDU tracks changes of app settings.
  • I tried deleting this (even though I only have a few keys in there, but there were a huge number of backups). This does seem to have fixed it (obviously the tokens changed though).
  • It's reverted to regenerating the key like it did before... stackoverflow.com/questions/53597614/…
  • My system had the secrets in Azure Storage Blob Storage named 'azure-webjobs-secrets'. With Storage Explorer, I wasted the entire subfolder to make my functions start again.
  • Lot of log-digging later, I found out that my host-log has the startup-exception in it. Apparently after failing to start number of time, a secondary error regarding more than 10 keys will replace the root cause.