How to debug server crash

Enonic version: XP something
OS: UBUNTU

I have a server which just stops from time to time.
Where do I start looking for reasons why it stops?
I’ve checked /var/logs/xp and /home/xp/enonic/xp/logs but there’s no real java dump info to be found there.

First question - does it stop, as in process dies. Or does it just hang? If it hangs there might be a few things worth checking out:

  • Thread dump (kill -4) and analyze this
  • It might be out of memory

Using JvisualVM to connect to the java process to get diagnostics in terms of memory use and more

If it’s responding you could try out the /status pages. There you will find various VM info. For 6.8 there’s also some thread-dump info and a dead-lock detection page.

The server just flat-out dies.
We are using Ubuntu 14.04.4 LTS.
Enonic XP v 6.4.0.RC3

Server has 6GB ram. Should suffice.

Check if its out-of-mem with the command jstat -gcutil <pid number>. if the memory banks are up to a 100% the jvm is using all of its time trying to garbage collect.

And as always try do upgrade to the latest version of XP if possible.

I ran it now.
item@74968-INT-UB1:/var/log/xp$ sudo jstat -gcutil 2880
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
56.61 0.00 62.56 48.33 93.52 86.68 60 0.939 3 0.249 1.189

What am I looking at here? :slight_smile:

ok so the first four columns tell you how full in prosent each of the jvms memory banks ( survivor 1, survivor 2, Eden and Old Generation ) are. data in memory is first put into Survivor 1 (S0) space. when that space goes full, he jvm will then do a garbage collect on those objects and move in ise objects to Survivor 2 (S1) space and so on. Heres a nice description on the architecture: https://dzone.com/articles/java-memory-architecture-model-garbage-collection

When all banks are almost full, there is a possibility that the jvm vill spend all cpu on garbage collction and stop responding. The next time xp hangs, check if cpu usage is high and then run this command. If most of the memory banks are full, or you get an error message saying you cant connect to jvm, then my guess is that there is something in your app thats eating up your memory.

Thanks for this.
The problem is that when the server stops, it just stops. There´s no “hang”. No leftover java process to check whether or not the cpu usage has been high.
So… how can I backtrack this? Run a crontab on the command regularly and output the results to a logfile?

6GB ram on server, or assigned to the JVM? Remember to check your settings for Xmx and Xms in the service definition.

server.sh has this:
exec “$JAVACMD” $JAVA_OPTS -Dxp.install="$XP_INSTALL" -Dfile.encoding=UTF8 $XP_OPTS -classpath “$XP_INSTALL/lib/*” com.enonic.xp.launcher.LauncherMain $ARGS

Good idea to add the -Xmx4096M -Xms2048M ?
Or what values do you recommend?

If its an OOM, which sounds likely, add this to get dump when it happens:

-XX:+HeapDumpOnOutOfMemoryError

Server crashed again tonight.
I added the parameters in server.sh like this, looks ok? Just as a precaution, I put them in both of the java opts. I don´t know which one is actually in use.

setupDefaults() {
DEFAULT_JAVA_OPTS="-Djsse.enableSNIExtension=false -Xmx4096M -Xms2048M -XX:+HeapDumpOnOutOfMemoryError"
DEFAULT_JAVA_DEBUG_OPTS="-Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 -XX:+HeapDumpOnOutOfMemoryError -Xmx4096M -Xms2048M "
}

Looks good.

If you run with option “debug”, e.g xp debug it will use the debug-parameters, but I recon you dont do this on a production site :slight_smile:

It is recommended to specify same values for Xmx and Xms for production setups

Also - even if there are no known issues with 6.4.x you are running close to a year old software now, so consider upgrading. BTW: What version of Java are you using. Errors like these are typically more related to JVM than XP.

/usr/lib/jvm/java-8-oracle/bin/java -version
java version “1.8.0_72”
Java™ SE Runtime Environment (build 1.8.0_72-b15)
Java HotSpot™ 64-Bit Server VM (build 25.72-b15, mixed mode)

Ok, so set both to 4096M then. Should be enough?

Jup… Unless you have very much data…

Oki, I´ll keep it running with xms2048M and xmx4095M until the eventual next crash, and change the xmx and xms to be equal then.

Btw, the “-XX:+HeapDumpOnOutOfMemoryError” where will the dump, dump? And in what format will the heapdump be?