Getting Data In

inputs.conf error

zservati1
New Member

I'm getting following error while starting splunkforwarder after updating inputs.conf under splunkforwarder. These are related to syntax issue with blacklist statements, although the file contains many statements like this but only few are erroring out.

[root@pprfefpba400 local]# /etc/init.d/splunk start
Starting Splunk...

Splunk> The IT Search Engine.

Checking prerequisites...
Checking mgmt port [8089]: open
Checking conf files for typos...
Possible typo in stanza [tail:///opt/splunk/var/log/splunk/searchhistory.log] in /opt/splunkforwarder/et c/system/local/inputs.conf, line 6: _blacklist = \.(gz)\$
Possible typo in stanza [tail:///opt/splunk/var/log/splunk/splunkd.log] in /opt/splunkforwarder/etc/syst em/local/inputs.conf, line 11: _blacklist = \.(gz)\$
Possible typo in stanza [tail:///opt/splunk/var/log/splunk/splunklogger.log] in /opt/splunkforwarder/etc /system/local/inputs.conf, line 16: _blacklist = \.(gz)\$
Possible typo in stanza [tail:///opt/splunk/var/log/splunk/web_access.log] in /opt/splunkforwarder/etc/s ystem/local/inputs.conf, line 21: _blacklist = \.(gz)\$
Possible typo in stanza [tail:///opt/splunk/var/log/splunk/web_service.log] in /opt/splunkforwarder/etc/ system/local/inputs.conf, line 26: _blacklist = \.(gz)\$
There might be typos in your conf files. For more information, run 'splunk btool check --debug'
All preliminary checks passed.
Starting splunk server daemon (splunkd)...

Here is the inputs.conf:

host = $web_server

[tail:///opt/splunk/var/log/splunk/searchhistory.log]
disabled = true
index = efepr
_blacklist = \.(gz)\$

[tail:///opt/splunk/var/log/splunk/splunkd.log]
disabled = true
index = efepr
_blacklist = \.(gz)\$

[tail:///opt/splunk/var/log/splunk/splunklogger.log]
disabled = true
index = efepr
_blacklist = \.(gz)\$

[tail:///opt/splunk/var/log/splunk/web_access.log]
disabled = true
index = efepr
_blacklist = \.(gz)\$

[tail:///opt/splunk/var/log/splunk/web_service.log]
disabled = true
index = efepr
_blacklist = \.(gz)\$

[monitor:///var/log/efe/audit.log]
disabled = false
sourcetype = log4j
index = efepr
_blacklist = \.(gz)\$
[monitor:///var/log/efe/audit.log]
disabled = false
sourcetype = log4j
index = efepr
_blacklist = \.(gz)\$

[monitor:///var/log/efe/boot.log]
disabled = false
sourcetype = log4j
index = efepr
_blacklist = \.(gz)\$

[monitor:///var/log/efe/cluster.log]
disabled = false
sourcetype = log4j
index = efepr
_blacklist = \.(gz)\$

[monitor:///var/log/efe/converter.log]
disabled = false
sourcetype = log4j
index = efepr
_blacklist = \.(gz)\$

[monitor:///var/log/efe/disaster-recovery/disaster-recovery.log]
disabled = true
sourcetype = log4j
index = efepr
_blacklist = \.(gz)\$

[monitor:///var/log/efe/filer-denied.log]
disabled = true
sourcetype = log4j
index = efepr
_blacklist = \.(gz)\$

[monitor:///var/log/efe/server.log]
disabled = false
sourcetype = log4j
index = efepr
_blacklist = \.(gz)\$

[monitor:///var/log/efe/clockSkew.log]
disabled = false
sourcetype = log4j
index = efepr
_blacklist = \.(gz)\$

[monitor:///var/log/efe/etxbridge.log]
disabled = true
sourcetype = log4j
index = efepr
_blacklist = \.(gz)\$

[monitor:///var/log/messages]
disabled = false
sourcetype = syslog
index = efepr
_blacklist = \.(gz)\$

[monitor:///opt/splunk/etc/system/local/inputs.conf]
sourcetype = splunk_inputs_conf
disabled = false
index = efepr
_blacklist = \.(gz)\$

[monitor:///usr/local/tomcat/logs/catalina.out]
disabled = false
sourcetype = log4j
index = efepr
_blacklist = \.(gz)\$

Tags (1)
0 Karma

GKC_DavidAnso
Path Finder

You should use the monitor:// input. If you only want to see new events (i.e. not read the history in the log) then add the "followTail = 1" parameter below the [monitor:// line.

0 Karma

zservati1
New Member

Thanks for your answer as I mentioned I was able to get it to work to have the file pathname to point to the right directory, in otherwords it seems the file should exist and also changed [tail...] to [monitor...]. Now I like to know what's the difference between monitor and tail directive, is this okay to have monitor for splunk logs. Here is an example.

[monitor:///opt/splunkforwarder/var/log/splunk/searchhistory.log]
disabled = true
index = efepr
blacklist = .(gz)$

0 Karma

zservati1
New Member

I think first the issue had to do with the file to be monitored not exist, and then I changed tail to monitor in few that had issue and was able to get get it to work. I am wondering what is the different between tail and monitor directives.

0 Karma

GKC_DavidAnso
Path Finder

Are you still getting the same number of errors? You will need to update all of the tail: lines to monitor:

Are the errors you are getting the same?

Are you trying to do some kind of variable substitution in the host setting? Can you use just web_server without the $?
host = $web_server

0 Karma

zservati1
New Member

I changed the file according to the suggestion but still get error. These files do not exist in the specified directory, can this be an issue.

0 Karma

GKC_DavidAnso
Path Finder

Try changing tail: to monitor: and setting followTail = 1, like below:

[monitor:///opt/splunk/var/log/splunk/searchhistory.log]
followTail = 1
disabled = true
index = efepr
_blacklist = .(gz)$

0 Karma

GKC_DavidAnso
Path Finder

I probably shouldn't have chosen a disabled input as the example.....

Make sure you don't disable all the inputs copying and pasting.

0 Karma

zservati1
New Member

So it seems the error is related to scripts related to splunk server
[script:///opt/splunk/etc/system/bin/addm.sh]
[script:///opt/splunk/etc/system/bin/awr.sh]
[script:///opt/splunk/etc/system/bin/tbspace.sh]
I checked the directory and I can't see these files. Where are these file should exist even checked under splunkforwarder.

0 Karma

zservati1
New Member

[root@pprfefpdb400 local]# /opt/splunkforwarder/bin/splunk -version
Splunk Universal Forwarder 4.2.1 (build 98164)
This is the splunk version I'm using.

0 Karma

zservati1
New Member

I tried removing the '' from ''blacklist but I sill see the error.

0 Karma

GKC_DavidAnso
Path Finder

First try removing the _. Splunk now prefers "blacklist = blah".

Also, is there a reason for your () brackets? In your pattern you don't really need them, try removing them.

What version of SplunkForwarder are you running? Try upgrading to the latest version.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...