I just loaded a brand new instance of Splunk Enterprise 6.3. I also loaded the Splunk S.o.S 3.2.1 app. While monitoring the splunkd.log file I noticed that I'm getting a ton of "empty splunk_forwarders_cache.csv" warnings.
10-09-2015 14:04:08.848 -0400 WARN SearchResults - /opt/app/splunk/etc/apps/sos/lookups/splunk_forwarders_cache.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-09-2015 14:04:09.848 -0400 WARN SearchResults - /opt/app/splunk/etc/apps/sos/lookups/splunk_forwarders_cache.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-09-2015 14:04:10.848 -0400 WARN SearchResults - /opt/app/splunk/etc/apps/sos/lookups/splunk_forwarders_cache.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
Now, these warnings only showed up when I tried to turn on SSL with the entry below in "./etc/system/local/web.conf"
[settings]
enableSplunkWebSSL = 1
But when I reset enableSplunkWebSSL back to 0 and restarted Splunk, the warnings still kept on coming.
Now I am having issues with SSL and my browser, but I didn't think there should be problems when I disabled that.
Any ideas?
Hi OldManEd, my expectation is that S.o.S. runs a job at some point to populate that lookup with forwarders it has found. It might be that this is a single instance without any forwarders, and so it has nothing to populate that lookup with.
If you do have forwarders sending in data, you could start to trace the config for any other mentions of this lookup. Go to $SPLUNKHOME/etc and run:
grep -i splunk_forwarders_cache ./*/*/*
for instance, or similar strings such as that to find other clues.
Now I'm getting the same thing when I tried to use the Management Console. The warning shows up every second and I can't turn it off:
10-12-2015 16:49:26.002 -0400 WARN SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-12-2015 16:49:27.002 -0400 WARN SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-12-2015 16:49:28.002 -0400 WARN SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-12-2015 16:49:29.002 -0400 WARN SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-12-2015 16:49:30.003 -0400 WARN SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-12-2015 16:49:31.002 -0400 WARN SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-12-2015 16:49:32.001 -0400 WARN SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-12-2015 16:49:33.002 -0400 WARN SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-12-2015 16:49:34.002 -0400 WARN SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
10-12-2015 16:49:34.998 -0400 WARN SearchResults - /opt/app/splunk/etc/apps/splunk_management_console/lookups/dmc_forwarder_assets.csv is empty, multi-line header is missing matching quotation, or could not parse CSV header
Hi OldManEd, my expectation is that S.o.S. runs a job at some point to populate that lookup with forwarders it has found. It might be that this is a single instance without any forwarders, and so it has nothing to populate that lookup with.
If you do have forwarders sending in data, you could start to trace the config for any other mentions of this lookup. Go to $SPLUNKHOME/etc and run:
grep -i splunk_forwarders_cache ./*/*/*
for instance, or similar strings such as that to find other clues.
Muebel,
You are correct. I'm just in the process of setting up this instance so I have no forwarders configured yet. When I looked at the "splunk_forwarders_cache.csv", it was there, but completely empty. I would think that even if this was the case, if the file is created, it would, at least, have the column "headers" by default.
Does anyone know what these column headers are for s.o.s? I was thinking that I could create a blank csv file, with the headers, to eliminate this repeating warning.
That particular lookup serves as an asset table for forwarder instances that are known to the S.o.S app. The columns it contains are described in $SPLUNK_HOME/etc/apps/sos/lookups/splunk_servers_cache.csv.spec
, which shares the same columns.
The main reason why we cannot ship that file by default with the S.o.S app is simply that it would overwrite what your instance has already generated, as there is no concept of "default" and "local" spaces for lookup files. By all means, do add a header line to the lookup file to make the message go away.
Hexx,
Hey thanks for the explanation. But your answer creates 3 more questions from me:
1, If I manually create the CSV file with the headers, will whatever process that created it in the first place overwrite it again? Or was the process that creates it a one-time run? I can't imagine it's a one-time run because the the follow-up question would be what about updates? What happens when forwarders are added or deleted? I still don't have any forwarders established yet so I know that's going to change the status of this issue.
If I add nothing to the file manually, will the warnings go away when I finally do add some forwarders? Will the issue clear itself?
Now I noticed that when I shut down S.o.S and started to work with Management Console, I saw similar warnings. Are these two applications related somehow? It appears that the Splunk Management Console offers similar data to S.o.S. Is MC a replacement app?
~Ed
Hexx,
Thanks for the quick reply. This explains everything I questions about. I think I'll just uninstall S.o.S an use Distributed Management Console, and hope I get access to some of the remote servers I need to add the Splunk forwarder to soon.
~Ed