Hello everyone!
We are currently experiencing an issue that has started happening more and more frequently recently and we don't really know how to proceed to track it down...
For some reason our application pool decides to restart about 1-2 a week nowadays and once it has sucessfully restarted once it only runs for 5-10 minutes before restarting again in a restart loop. It keeps doing this until we perform an IIS reset or restart the entire webserver. Once we do this the problem always stops.
In order to try and figure out why the application is restarting I have added some code to catch the reason in the "Application_End" event in Global.asax.cs:
protected void Application_End()
{
HttpRuntime runtime = (HttpRuntime) typeof(HttpRuntime).InvokeMember("_theRuntime",
BindingFlags.NonPublic
| BindingFlags.Static
| BindingFlags.GetField,
null,
null,
null);
if (runtime == null)
return;
string shutDownMessage = (string) runtime.GetType().InvokeMember("_shutDownMessage",
BindingFlags.NonPublic
| BindingFlags.Instance
| BindingFlags.GetField,
null,
runtime,
null);
string shutDownStack = (string) runtime.GetType().InvokeMember("_shutDownStack",
BindingFlags.NonPublic
| BindingFlags.Instance
| BindingFlags.GetField,
null,
runtime,
null);
Logger.Log(
Level.Critical," ==== APPLICATION END EVENT ====" + Environment.NewLine + shutDownMessage + Environment.NewLine
+ shutDownStack + " ==============================="
);
}
With this code in place we see this message in our logs each time the application restarts:
FATAL Piab.CMS.Global: ==== APPLICATION END EVENT ====
BuildResult change, cache key=cindexingservice.svc.1f180bcd
HostingEnvironment initiated shutdown
HostingEnvironment caused shutdown
It's always the same message each time. Even the "1f180bcd" part seems to be static. The only file called "Indexingservice.svc" in our project comes from the EPiServer.Search package and is located at the \IndexingService\IndexingService.svc path. I'm not sure if this means that our problem is somehow related to indexing/search and the EPiServer.Search package or if the restarts would still happen with a different message if we remove it.
We have a testing/staging server but unfortunately we've never seen this error there and we've never been able to reproduce it ourselves.
I have googled all I can for similar error messages but the only thing I find is the Microsoft.NET reference source code that decides to return the "BuildResult change, cache key=" part of the message I now see in the logs...
Has anyone of you guys seen this type of error and/or does anyone have any idea about how to proceeed troubleshooting something like this?
No modified dates are changing on this file itself or any other file in the "bin" folder... We've also tried to disable the IIS file system watcher completely (so dynamic recompilations should not be happening?):
<httpRuntime fcnMode="Disabled" ... />
Since we disabled this the application no longer restarts automatically if we manually edit a .cshtml file for example so the IIS should not be monitoring any files modified/created date and kick off any dynamic recompilations because of such things...