Alerting

Setting up Alert if jboss service went down

shakeel253
Explorer

I have 3 servers App-1, App-2 and App-3. The three application are running on Jboss. I need a query that Alerts me, if on any servers Jboss Service goes down.

Any help with this will be appreciated.

Thank You

0 Karma

niketn
Legend

@shakeel253, I am afraid you have duplicate questions opened for your JBOSS alerting issues with different description: https://answers.splunk.com/answers/597751/splunk-query-that-alert-if-services-on-a-jboss-ser.html

Please clarify if this is any different from the other post. Can you try the following? Where hosts will have list of servers you want to monitor and source is the JBOSS log path.

  | tstats latest(_time) as _time WHERE (host="ABC1" OR host="ABC2" OR host="ABC3") AND source="/opt/jboss-eap/standalone/log/server.log" by host
  | eval "downTime (in Min)"=round((now()-_time)/60,0)
  | append [
      | makeresults
      | eval host="ABC1", "downTime (in Min)"="999"]
  | append [
      | makeresults
      | eval host="ABC2", "downTime (in Min)"="999"]
  | append [
      | makeresults
      | eval host="ABC3", "downTime (in Min)"="999"]
  | dedup host
  | where 'downTime (in Min)'>5
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

MousumiChowdhur
Contributor

Hi @shakeel253,

You can run a script which will check the jboss status on the machines and index the output of the command in Splunk. Then you can set up an alert on when you get the status as "jboss is not running".

Thanks!

0 Karma

shakeel253
Explorer

Can you give an example of how this would be possible?

0 Karma

MousumiChowdhur
Contributor

Sure! You first create a script which is going to check the jboss status of your machines. You can refer to sample scripts in the below link which will check the jboss status.

https://stackoverflow.com/questions/8761374/how-to-check-if-jboss-is-running-on-unix-server

Save the script in the bin directory of your app and then create an index like you create index normally which will store the result of your script. Then create the inputs.conf in the local directory of your app and give the path to your script. This complete thing is going to index the output of your script. Post the indexing you can set up alerts which will be triggered once it finds "jboss is not running" in your events. I hope this sounds clear. Do let me know if you find any trouble in doing that.

Thanks!!

0 Karma

shakeel253
Explorer

@Mousumichowdhury i created a shell script and placed it inside /opt/splunkforwarder/bin with executable permission. I also made an entry inside inputs.conf, below is the how the entry looks like inside inputs.conf

[default]
host = svm
/opt/splunkforwarder/bin/tomcatscript.sh

how do you create an index?

0 Karma

MousumiChowdhur
Contributor

Hi,

you can create the index with the below stanza:

[yourindexname]
coldPath = $SPLUNK_DB/yourindexname/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB/yourindexname/db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB/yourindexname/thaweddb

If this is a single instance setup then you can place your indexes.conf stanza in /opt/splunk/etc/apps/search/local and in case of clustered environment you can place the indexes.conf file inside /opt/splunk/etc/master-apps//local

0 Karma

shakeel253
Explorer

@MousumiChowdhury i still dont see the new custom index in splunk search. I will describe how my splunk is setup, maybe, i am missing something.

1) tomcatscript.sh is inside /opt/splunkforwarder/bin

!/bin/bash

if [ -z "$(ps -ef | grep java | grep tomcat)" ]
then
echo "Tomcat is NOT running"
else
echo "Tomcat is running"
fi

2) inputs.conf inside /opt/splunkforwarder/etc/apps/search/local

[monitor:///opt/tomcat/logs/catalina.out]
disabled = false
index = tomcat

[monitor:///opt/splunkforwarder/bin/tomcatscript.sh]
disabled = false
index = tomcatindex

3)indexes.conf inside /opt/splunkforwarder/etc/apps/search/local

[tomcatindex]
coldPath = $SPLUNK_DB/tomcatindex/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB/tomcatindex/db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB/tomcatindex/thaweddb

when i do a search on splunk host="abcvm" but the only index i see is "os". I also did a restart as well for the splunkforwarder but didnt see the newly created index. Maybe i am doing something wrong, would appreciate if you could direct me, thank you

0 Karma

MousumiChowdhur
Contributor

Hi!

Follow the below steps for the single instance:

Create your script in /opt/splunk/etc/apps/search/bin/
Create your inputs.conf in /opt/splunk/etc/apps/search/local/
Create your indexes.conf in /opt/splunk/etc/apps/search/local/

Follow the below steps for clustered set up:

Create your script in /opt/splunk/etc/deployment-apps/<yourappname>/bin/
Create your inputs.conf in /opt/splunk/etc/deployment-apps/<yourappname>/local/
Create your indexes.conf in /opt/splunk/etc/master-apps/<yourappname>/local/

Do let me know if that's working for you.

Thanks!

0 Karma

shakeel253
Explorer

@MousumiChowdhury

firstoff thank you for assisting me with this, having said that bin directory is not inside /opt/splunk/etc/apps/search, so i had to manually create the bin folder and gave it splunk ownership and copy the tomcat script inside bin.

2) This is how the indexes.conf

[root@ip-xx.xxx.xx.xxx local]# cat indexes.conf
[tomcatindex]
coldPath = $SPLUNK_DB/tomcatindex/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB/tomcatindex/db
maxTotalDataSizeMB = 512000
thawedPath = $SPLUNK_DB/tomcatindex/thaweddb

[tomcatindex]
coldPath = /opt/splunkforwarder/etc/apps/search/bin/./tomcatscript.sh
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = /opt/splunkforwarder/etc/apps/search/bin/./tomcatscript.sh
maxTotalDataSizeMB = 512000
thawedPath = /opt/splunkforwarder/etc/apps/search/bin/./tomcatscript.sh

3)[root@ip-xx-xxx-xx-xx local]# cat inputs.conf
[monitor:///opt/tomcat/logs/catalina.out]
disabled = false
index = tomcat

[monitor:///opt/splunkforwarder/bin/tomcatscript.sh]
disabled = false
index = tomcatindex

[monitor:///opt/splunkforwarder/etc/apps/search/bin/./tomcatscript.sh]
disabled = false
index = tomcatindex

After making these changes, i have restarted splunkforwarder and search for the index, but i do not see the index still, what am i missing?

0 Karma

MousumiChowdhur
Contributor

Hi, may I know why are you creating the files in your splunkforwarder?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...