Hello,
I have a hypothetical scenario which I hope someone can help me with.
Let's say I have a Linux server with a universal forwarder installed which is monitoring /var/log/messages.
Imagine the universal forwarder now fails for some reason - someone accidentally killed the process, etc.
Now when the universal forwarder has failed, /var/log/messages has been written to its maximum size and rolled-over to /var/log/messages-20120727, and a new /var/log/messages has been created.
We now discover that the universal forwarder has stopped, and we restart it, and it happily monitors /var/log/messages again.
Would I be right to say that whatever changes was written to the previous /var/log/messages file after the universal forwarder failed, is now lost, i.e. not sent to the indexer?
The Splunk forwarder maintains a pointer to what data it has forwarded to an indexer in the fishbucket index for /var/log/messages, but what happens now that /var/log/messages is essentially an entirely new file?
Thanks in advance for any help!
I think you'll find this docs page very useful, as it explains some things about how Splunk handles rotated log files.
http://docs.splunk.com/Documentation/Splunk/latest/Data/Howlogfilerotationishandled
Essentially, if you've told Splunk to monitor just /var/log/messages
, then obviously it won't see events in another file. If, however, you choose to monitor the whole /var/log
directory, or for that matter just all /var/log/messages*
files, Splunk will identify file contents based on the actual contents, not the filename, so if you move /var/log/messages
to /var/log/messages-20120727
it'll just pick up where it left off.
As it says in the docs page though, doing this with .gz compressed files is another thing, but you didn't specify that in your scenario so I'll omit that discussion 🙂
I think you'll find this docs page very useful, as it explains some things about how Splunk handles rotated log files.
http://docs.splunk.com/Documentation/Splunk/latest/Data/Howlogfilerotationishandled
Essentially, if you've told Splunk to monitor just /var/log/messages
, then obviously it won't see events in another file. If, however, you choose to monitor the whole /var/log
directory, or for that matter just all /var/log/messages*
files, Splunk will identify file contents based on the actual contents, not the filename, so if you move /var/log/messages
to /var/log/messages-20120727
it'll just pick up where it left off.
As it says in the docs page though, doing this with .gz compressed files is another thing, but you didn't specify that in your scenario so I'll omit that discussion 🙂
Could you explain what happens in the case where files are compressed when they are archived?
May be an old issue ...this question was about file.
how to handle data loss in case of TCP input ?
I want to handle universal forwarder failure in a multisite deployment scenario !!!
Thanks for the help 🙂 Guess I have to monitor all the files just in case.
May be an old issue ...this question was about file.
how to handle data loss in case of TCP input ?
I want to handle universal forwarder failure in a multisite deployment scenario !!!
[monitor:///var/log/messages*]
should do that.
Like I said, yes, if you only monitor /var/log/messages
then that's what will happen. The events that were written after the forwarder went down (though I must add that I've hardly ever seen forwarders do that) will no longer be in /var/log/messages
when the forwarder comes back up, so it has no way of seeing the events that it "missed". That is one of the reasons why you should make sure that if you are rotating your logfiles, you should monitor not just the non-rotated file but also the rotated ones.
Hello Ayn,
Thanks for the reply, please let me clarify.
Let's assume we are only monitoring one file, /var/log/messages, with the following sequence of events.
So, in this case, some events would not be indexed, because the forwarder was not active before /var/log/messages was rotated, right?