Splunk Search

Splunk Health Check (Warning, Info and N/A)

vino06
New Member

Hi Guys,

Good Day!

Regarding on our Splunk servers, we've performed a health check and we found some warning, info and n/a status. Just want to ask what this are and how this result impacts on our application. Please see below screenshot.

Thank you!
Kevin

0 Karma

mattymo
Splunk Employee
Splunk Employee

Hi Kevin!

The actions you should take are dependent on which nodes are involved and require some knowledge about your environment, but I'll do my best to set you in the right direction here.

ULIMITS & THP
For the System and Environment warnings, it is telling you that THP and ulimits are not optimally set for a Splunk Enterprise instance. I would only be worried about correcting this in your core Splunk nodes (Indexers, Search Heads, etc....anything other than Universal Forwarders, really).

Here is some documentation on THP and Ulimits. How you these depends on your system, so work with your sysadmins to ensure they are set persistently.

About Ulimit - https://docs.splunk.com/Documentation/Splunk/6.6.2/Troubleshooting/ulimitErrors
Example How to adjust - https://www.tecmint.com/increase-set-open-file-limits-in-linux/
* note in the versions of RHEL i have played with lately, setting the ulimits on boot-start can be tricky, consult your vendor docs if necessary.

About THP - https://docs.splunk.com/Documentation/Splunk/6.6.2/ReleaseNotes/SplunkandTHP
Example How to Disable THP in Centos 7 - https://newbiedba.wordpress.com/2015/09/07/disabling-transparent-huge-pages-in-centos-7-x/

Missing Forwarders

Missing forwarders is simply forwarders that have not been seen in the last 15 minutes by your indexers. You can see more on this in Forwarder Management dashboard and can be resolved by rebuilding your forwarder lookup if need be. This may or may not be impacting you depending on whether they are decommissioned servers, etc. You will need to

Skipped Searches

This is something you will need to look at in your search heads. You can use the Monitoring Console to analyze your search performance. There is a good break down on what is being skipped under Settings > Monitoring Console > Search > Search Activity: Instance

- MattyMo
Get Updates on the Splunk Community!

Join Us for Splunk University and Get Your Bootcamp Game On!

If you know, you know! Splunk University is the vibe this summer so register today for bootcamps galore ...

.conf24 | Learning Tracks for Security, Observability, Platform, and Developers!

.conf24 is taking place at The Venetian in Las Vegas from June 11 - 14. Continue reading to learn about the ...

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...