Getting Data In

Forwarder fails, monitored file is archived, what happens?

roychen
Path Finder

Hello,

I have a hypothetical scenario which I hope someone can help me with.

Let's say I have a Linux server with a universal forwarder installed which is monitoring /var/log/messages.

Imagine the universal forwarder now fails for some reason - someone accidentally killed the process, etc.

Now when the universal forwarder has failed, /var/log/messages has been written to its maximum size and rolled-over to /var/log/messages-20120727, and a new /var/log/messages has been created.

We now discover that the universal forwarder has stopped, and we restart it, and it happily monitors /var/log/messages again.

Would I be right to say that whatever changes was written to the previous /var/log/messages file after the universal forwarder failed, is now lost, i.e. not sent to the indexer?

The Splunk forwarder maintains a pointer to what data it has forwarded to an indexer in the fishbucket index for /var/log/messages, but what happens now that /var/log/messages is essentially an entirely new file?

Thanks in advance for any help!

1 Solution

Ayn
Legend

I think you'll find this docs page very useful, as it explains some things about how Splunk handles rotated log files.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Howlogfilerotationishandled

Essentially, if you've told Splunk to monitor just /var/log/messages, then obviously it won't see events in another file. If, however, you choose to monitor the whole /var/log directory, or for that matter just all /var/log/messages* files, Splunk will identify file contents based on the actual contents, not the filename, so if you move /var/log/messages to /var/log/messages-20120727 it'll just pick up where it left off.

As it says in the docs page though, doing this with .gz compressed files is another thing, but you didn't specify that in your scenario so I'll omit that discussion 🙂

View solution in original post

Ayn
Legend

I think you'll find this docs page very useful, as it explains some things about how Splunk handles rotated log files.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Howlogfilerotationishandled

Essentially, if you've told Splunk to monitor just /var/log/messages, then obviously it won't see events in another file. If, however, you choose to monitor the whole /var/log directory, or for that matter just all /var/log/messages* files, Splunk will identify file contents based on the actual contents, not the filename, so if you move /var/log/messages to /var/log/messages-20120727 it'll just pick up where it left off.

As it says in the docs page though, doing this with .gz compressed files is another thing, but you didn't specify that in your scenario so I'll omit that discussion 🙂

RichardCA
New Member

Could you explain what happens in the case where files are compressed when they are archived?

0 Karma

bkumarm
Contributor

May be an old issue ...this question was about file.
how to handle data loss in case of TCP input ?

I want to handle universal forwarder failure in a multisite deployment scenario !!!

0 Karma

roychen
Path Finder

Thanks for the help 🙂 Guess I have to monitor all the files just in case.

0 Karma

bkumarm
Contributor

May be an old issue ...this question was about file.
how to handle data loss in case of TCP input ?

I want to handle universal forwarder failure in a multisite deployment scenario !!!

0 Karma

ziegfried
Influencer

[monitor:///var/log/messages*] should do that.

0 Karma

Ayn
Legend

Like I said, yes, if you only monitor /var/log/messages then that's what will happen. The events that were written after the forwarder went down (though I must add that I've hardly ever seen forwarders do that) will no longer be in /var/log/messages when the forwarder comes back up, so it has no way of seeing the events that it "missed". That is one of the reasons why you should make sure that if you are rotating your logfiles, you should monitor not just the non-rotated file but also the rotated ones.

0 Karma

roychen
Path Finder

Hello Ayn,

Thanks for the reply, please let me clarify.

Let's assume we are only monitoring one file, /var/log/messages, with the following sequence of events.

  1. Forwarder fails.
  2. /var/log/messages is written to, and rotated to /var/log/messages1
  3. A new /var/log/messages is created.
  4. Forwarder is restarted.
  5. Forwarder now treats /var/log/messages as a new file because begin and end CRC have not been previously seen.

So, in this case, some events would not be indexed, because the forwarder was not active before /var/log/messages was rotated, right?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...