Getting Data In

Monitor advice needed

kenison
New Member

After reading the docs and looking in forums, I thought I had a understanding of monitor and what it does...I guess not.
Is a monitor, set up to follow from tail, only supposed to index data that is written to a directory from the time of monitor creation? I manually made a monitor in my inputs.conf file after I saw that the monitor I set up in the manager was grabbing events that were pre-dated.

[monitor:///dir/path]
blacklist = dir/
followTail = 1

I didn't see events logging right away after restarting splunk, so I thought it was working properly...that is, only indexing events that are new. I came in today to find my license had exceeded limit over night and splunk has indexed events from last year.

Someone tell me what is wrong with this. Is there a way to set up a monitor that only indexes new events? Why is my monitor indexing the whole file?

Tags (1)
0 Karma

MuS
SplunkTrust
SplunkTrust

Hi kenison.vrabcak

well there is nothing wrong, this is the way splunk monitors directories: starting at the moment you add it in a monitor [stanza] splunk is 'eating' up any readable file in this directory. How should splunk know what you consider as old data?

if you really want to have only new data to be indexed, move the 'old logs' out of the way before you [monitor] the directory.

after that splunk will index only the new files coming in and will forget about the already indexed files.

regards

mzorzi
Splunk Employee
Splunk Employee


With the option followTail enabled Splunk is going to monitor only events being added into the monitored stanza after restarting Splunk. Maybe the old files are in compress format and their modification time has been changed. Or maybe you have an incorrect timestamp extraction problem, and the events are not really from last year.

You can find more information on how to troubleshoot this problem by reviewing the content of this twiki page:

http://www.splunk.com/wiki/Community:Troubleshooting\_Monitor_Inputs

and this answer:

http://splunk-base.splunk.com/answers/1162/is-there-some-way-to-see-the-current-tailing-status-of-my...

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...