Why are the new inputs.conf data is not added/searchable?

[kimche]

Hi all,

I started monitoring the splunk internal introspection logs. These seem to get logged. I also want to log some Linux logs. This is my inputs.conf.

[monitor:///opt/splunk/var/log/introspection]
sourcetype = splunk_internal

[monitor:///var/log/messages]
sourcetype = syslog

[monitor:///var/log/secure]
sourcetype = linux_secure

After restarting the /var/log/messages and /var/log/secure are not added to the list of data inputs though. They obviously also do not give results when I use search. When I add them manually (using the same path and sourcetype) it does work. It's not an option to do this manually for me though, so it really needs to be done by editing inputs.conf.

What is missing?

thanks!

[MuS]

Hi kimche,

does the user running Splunk have the permission to read those files?
What does splunkd.log report about those files?
Have to searched index=main earliest=0?
Is this a typo or is the closing ] missing in the secure stanza of your inputs.conf?

cheers, MuS

[kimche]

Hi MuS, thanks for your reply.
- The user is admin so then it does have permission, right? the last line in my log (see below) does complain about some authentication problem. But maybe that was because I logged in at splunkweb first with a wrong username..
- splunkd gives me a very long log. I copied some lines to pastebin that struck out to me but I can't puzzle the pieces together at what I should change http://pastebin.com/tZ9h6h1T
- When I search for that it shows no results at all.
- Sorry, that was a typo indeed. Just corrected it.

[MuS]

It looks like you hint a low disk space limit here:

11-03-2015 04:40:36.779 -0500 ERROR DiskMon - Cannot write data to index path "/opt/splunk/var/lib/splunk/audit/db" because you are low on disk space on partition "/". Indexing has been paused.  Will resume when free disk space rises above 5000MB.

either free up disk space or decrease the limit in server.conf:

[diskUsage]

minFreeSpace = <num>
* Specified in megabytes.
* The default setting is 5000 (approx 5GB)