Tomcat shutting down by itself

tomcat stopping unexpectedly
a valid shutdown command was received via the shutdown port. stopping the server instance.
tomcat shutdown log
tomcat graceful shutdown
catalina shutdown
shutdown sh
centos restart tomcat
tomcat startup script

I'm running Tomcat 7.0.53 on CentOS 6.5 64 bit and OpenJDK 1.7 64.

I have several servers where occasionally - like randomly once per week - Tomcat will just gracefully shut itself down. I've been looking at this for months and cannot find the reason. The only pattern seems to be a long period of nothing going on then the shutdown.

  • I have upgraded to latest versions of the underlying tools (Tomcat, Java, etc)
  • I am running with the default memory settings
  • I have disabled the SHUTDOWN in the server.xml
  • I have moved MySQL to another box to ensure Tomcat is the only substantial application (server also runs nginx)
  • I have verified that I have no System.exit() in any of my code. I have not scanned the libs as I can't figure out how to do that. And I don't expect I will find anything there either. Why would a library exit anyway?
  • I have enabled logging of garbage collection. But they seem pretty quick - like full GC in 0.15s

Here is the memory usage of one of the servers. Lots of RAM free.

                 total       used       free     shared    buffers     cached
    Mem:          2006        771       1234          0        176        281
    -/+ buffers/cache:        313       1692
    Swap:         2047          0       2047

Below is one of the events in the Catalina.out. You can see it starts then does nothing exciting for several hours. Then gracefully shuts itself down as if it was told to.

I've researched this to death and have not bee able to get a handle on this.

Can someone please propose a plan of action for me?

Thanks

From the Server.xml:

    <Server port="-1" shutdown="__SHUTDOWN__">

From Catalina.out:

Apr 28, 2014 5:34:50 PM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Valve} Setting property 'remoteIpProxiesHeader' to 'x-forwarded-by' did not find a matching property.
Apr 28, 2014 5:34:50 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-apr-8080"]
Apr 28, 2014 5:34:50 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-apr-8009"]
Apr 28, 2014 5:34:50 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 3247 ms
Apr 28, 2014 5:34:50 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Apr 28, 2014 5:34:50 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.53
Apr 28, 2014 5:34:59 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-apr-8080"]
Apr 28, 2014 5:34:59 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-apr-8009"]
Apr 28, 2014 5:34:59 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 8278 ms
Apr 28, 2014 5:41:53 PM org.apache.jasper.compiler.TldLocationsCache tldScanJar
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Apr 28, 2014 10:32:32 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-apr-8080"]
Apr 28, 2014 10:32:32 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-apr-8009"]
Apr 28, 2014 10:32:32 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Apr 28, 2014 10:32:32 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-apr-8080"]
Apr 28, 2014 10:32:32 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-apr-8009"]
Apr 28, 2014 10:32:32 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-apr-8080"]
Apr 28, 2014 10:32:32 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-apr-8009"]
May 05, 2014 8:10:32 PM org.apache.catalina.core.AprLifecycleListener init
INFO: Loaded APR based Apache Tomcat Native library 1.1.29 using APR version 1.3.9.
May 05, 2014 8:10:32 PM org.apache.catalina.core.AprLifecycleListener init
INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
May 05, 2014 8:10:32 PM org.apache.tomcat.util.digester.SetPropertiesRule begin

Tomcat shutting down by itself automatically without logs, Keywords: Tomcat - Amazon Web Services - Technical issue - Other Description: We are running your apache-tomcat-sql (tomcat v8)  In my experience, Tomcat is flaky in its ability to log the OutOfMemory exception. Sometimes it does, sometimes it doesn't. In this case, while some messages like the ones you posted can be benign, the fact that Tomcat is shutting down while it "is still processing a request that has yet to finish" is disturbing.


There is a known (and fixed bug) in Tomcat versions prior to v7.0.55:

https://bz.apache.org/bugzilla/show_bug.cgi?id=56684

The thread is really good reading but I can summarize it for you. When you start Tomcat there is a wait state that starts waiting for the shutdown command. This is normal, but the bug in these versions is an implicit timeout on this wait of about 50 days. When the wait state timed out, the Tomcat would shutdown.

So there ARE cases of odd things happening in Tomcat and NO, you are not crazy. This was happening to a SOLR cluster that I maintained for more than a year before I found this bug and fixed it with an upgrade. No one ever noticed because it took so long to crash and with all that goes on it can be easy to brush past finding the root cause. The easiest thing is to just restart Tomcat and forget it ever happened.

I hope this answer helps you. I see that you are/were running a version with this bug.

Good luck!

Tomcat shuts down automatically every day at the same time , This is small application deployed on the same server (tomcat-7). If Tomcat shuts itself down because of an OutOfMemory Exception, then  Tomcat stops working without any logs, it doesn't hang, it just shut down after some hours or minutes. The problem is that the JVM is crashing. The JVM crash log typically has a name like hs_err_pid*.log and is created in the JVM's working directory.


How is your Tomcat started? If there is a parent process, a signal sent to the parent could be the cause of a clean shutdown.

My production server (tomcat-7) get shutdown automatically (Tomcat , HI i had installed tomcat server its works fine but its getting stop frequently.i had check out logs it doesint give anythings about error about. There's nothing in the above that indicates Tomcat is shutting down (Tomcat itself doesn't use Quartz). If one of your webapps is doing something antisocial such as calling System.exit(), that will cause a shutdown. What else is in the logs?


A case in DC/OS

Recently we had a similar incident. Our tomcat is running in a docker as a service in a DC/OS environment. The stderr just said tomcat was killed, and there was no useful traces in any log files of tomcat.

It turned out the reason was simply that we did not allocate enough memory to the service. More debugging suggestions are here, but the key technique we used is to check the kernel log:

# journalctl -f _TRANSPORT=kernel
...
Mar 29 15:09:09 host-17 kernel: Memory cgroup out of memory: Kill process 7935 (java) score 1021 or sacrifice child
Mar 29 15:09:09 host-17 kernel: Killed process 7906 (java) total-vm:11583840kB, anon-rss:1032860kB, file-rss:30924kB, shmem-rss:0kB
...

Once we found that out, we increased the memory allocation to the service in DC/OS and the problem was solved.

[SOLVED] Tomcat server getting shutdown frequently, The KVM virtualization kernel was killing the process because the VM was running out of memory. This is validated by checking out  Given that catalina.sh uses "&" to background the process, Cygwin is probably sending kill/HUP signals or something for some (buggy) reason, causing tomcat to shut itself down as requested (and quite orderly so).


How is your Tomcat started? If there is a parent process, a signal sent to the parent could be the cause of a clean shutdown. I suggest adding set -m option into start_script.sh.

14.10 - Tomcat 8.0.9 Shutting Down (no messages), As I remarked in the comments, my first idea would be that a shutdown command ( SHUTDOWN ) was sent on Tomcat's shutdown port (by  Tomcat unexpectedly shuts down. Hello, My Tomcat instance is unexpectedly shutting itself down every 8 to 72 hours. Can someone give me some advice on this? By all appearances this is a


Tomcat suddenly shuts down on its own, Tomcat randomly stops receiving requests and returns error 503. Im running a tomcat 7 on Ubuntu 12.04 TLS updated. In the apache error log  Tomcat does not gracefully shut down when it's "in trouble" (out of memory, or anything else). Linux does not gracefully shut down processes when it's low on memory, it stops them hard with a kill -9, which leave no trace in the Tomcat log.


Tomcat suddenly shutdown, This section details how to start up and shut down Tomcat, including specific This runs the java command to invoke the Tomcat startup and shutdown classes. Instead, Tomcat shuts down all of its own threads so that the VM can exit  -----Original Message----- From: Brown, Carlton Sent: Thursday, January 11, 2007 4:01 PM To: users@tomcat.apache.org Subject: Tomcat unexpectedly shuts down Hello, My Tomcat instance is unexpectedly shutting itself down every 8 to 72 hours. Can someone give me some advice on this? By all appearances this is a graceful shutdown. I


Starting, Stopping, and Restarting Tomcat, d directory to handle Tomcat's startup and shutdown. If you installed the standard RPM distribution, start Tomcat with the following command: # /etc/rc.d/init.d/  Jakarta Tomcat shuts down by itself 843836 Nov 13, 2003 8:44 PM Hi everybody, I'm having problems with Jakarta Tomcat. After some time up, it suddenly shuts down, with apparently no reason.