Alarm for tomcat service

[fsrodriguez]

I have set up an alarm to tell me when tomcat service is down.

hosts="server" source="ps" tomcat | stats latest(_time) as latest by host

That is what I have for my search. For the trigger I have set: search < 1.

Sometimes I get false alarms. Am I missing something?

Also at my job they used /var/logs directory... shouldn't we just use some tomcat directory just to monitor tomcat?

Thanks in advance!

[nickhills]

Tomcat can be installed in many ways, and whilst you are correct that the 'normal' location is /var/log/tomcat/catalina.out often this is symlinked to somewhere else such as /usr/share/tomcat8/log/catalina.out As long as you have the correct sourcetypes set for the inputs, I wouldn't worry about the paths too much unless its also your job to manage the servers and it bothers you.

In your query above you are monitoring the tomcat process from ps which from time to time (depending on config) may choose to restart itself (or crash and restart) whilst both of these are events you may be interested in, I have found that monitoring the catalina.out file over an x minute period provides a better indication of when the process stop because the log file approach is more forgiving of restarted processes.
It also would highlight if tomcat 'hangs'. In such a situation the process might still be running, but not servicing requests. This latter approach would catch that.

[fsrodriguez]

ah ok! yeah that makes a lot of sense just to monitor the catalina.out file. Are you using "ps" ? I have it to check every 5 minutes. Would you mind sharing your query?

I'm wondering if there is an alternative method to check other than running PS.

[nickhills]

Yes, I tend to monitor the tomcat sourcetype - since a running tomcat server is frequently writing logs (even when idle) I have found this a better method rather than ps.

that way even if tomcat hangs (as ours did from time to time) the lack of catalina logs is more telling than a running process in ps.