Splunk on Splunk doesn't collect resource consumption data after enabling ps_sos.sh script

[ehouse_splunk]

I installed the Splunk on Splunk app and followed the directions in the answer to How do I set up the SoS app to Monitor Splunk's System Resource Consumption to enable the ps_sos.sh script via the web interface. My Splunk instance is still not collecting CPU or memory usage data. What further things should I look into to get Splunk on Splunk working?

• OS: Mac OS X Mavericks 10.9
• Splunk 6.1
• Splunk on Splunk 3.2

[ehouse_splunk]

After looking at the raw index data, yes, there are events showing up.

[ehouse_splunk · Accepted solution]

I checked with another engineer that was experiencing the same issue. What we found was that the SoS index did contain data - but the default panel searches were not querying for the correct hostname. I.e. the events were being entered into the database under a different hostname than the one found in settings.

So for anybody else who has this problem, run the following search as hexx suggested:
index="sos" sourcetype="ps"

If events are showing up then SoS is working and changing the hostname in Settings -> System Settings -> General Settings may fix the problem.

[hexx]

Aha! So, I am curious:

• What is the value of "host" for the events written by ps_sos,sh to the "sos" index?
• What is the value of "host" in the global stanza of $SPLUNK_HOME/etc/system/local/inputs.conf?
• What value of "host" was S.o.S using to scope searches against the "sos" index?

[hexx]

Is there any data at all that is being indexed to the "sos" index?

Another thing to check is splunkd.log, and more particularly messages emitted by the ExecProcessor log channel. Is there anything that reports problems with the execution of the ps_sos.sh scripted input?

Finally, if you run the script manually like so, do you get any errors and/or any sane output?

$SPLUNK_HOME/bin/splunk cmd $SPLUNK_HOME/etc/apps/sos/bin/ps_sos.sh

[n5zap]

Broken for me on FreeBSD. Data is getting collected, but even though my server name is "voodoo", the host it is apparently searching for is voodoo, and the host in inputs.conf is voodoo, I get no data. However, the hostname on the SoS landing page says my server is voodoo.viewkeeper.org (the FQ name) so I suspect that is where the error is. How do I change the name that SoS thinks it should use?

[hexx]

You can edit the $SPLUNK_HOME/etc/apps/sos/lookups/splunk_servers_cache.csv file manually and set the value of the "sos_server" field to the value of the "host" field reflected by events recorded in the "sos" and "_internal" indexes for that particular instance.