Getting Data In

how to index data from a log file which got generated when my forwarder was down?

saifuddin9122
Path Finder

Hello All,

i have a dumb question,

i have few servers which will have heavy traffic and with log files rotating for every few minutes. the problem i had, my forwarder was down for few mins in that down time log files got rotated and new file is generated and my forwarder is reading the current log file but when i search data for the time when my forwarder was down resulting in zero events. does it means i lost that data. if it is so then how should i read that data and for future purpose how should i address this problem.

here are my example log file names
webapp.log (currently written )
webapp.log.1 (rolled)
webapp.log.2 (rolled)
webapp.log.3 (rolled)
webapp.log.4 (rolled)

here are my inputs.conf

[monitor:///var/log/web/app/webapp.log]
index=main
sourcetype=web_application

Thanks for the help.

0 Karma
1 Solution

jtacy
Builder

Consider changing your input to:

[monitor:///var/log/web/app/webapp.log*]
index=main
sourcetype=web_application

If the logs are still in the folder this will capture them, but if the logs aren't on disk anymore I would imagine that the data is lost. The above configuration will help prevent this situation in the future. If you're concerned about duplicate events, consider that Splunk uses the data at the beginning of the file to determine if it's already read it, not the file name (by default).

There should be no negative effects from this change unless the star will match files you don't want to index. If that's the case you might want to review the whitelist and blacklist options on your input to fine tune what you capture:
http://docs.splunk.com/Documentation/Splunk/6.6.1/Admin/Inputsconf#MONITOR:

View solution in original post

0 Karma

jtacy
Builder

Consider changing your input to:

[monitor:///var/log/web/app/webapp.log*]
index=main
sourcetype=web_application

If the logs are still in the folder this will capture them, but if the logs aren't on disk anymore I would imagine that the data is lost. The above configuration will help prevent this situation in the future. If you're concerned about duplicate events, consider that Splunk uses the data at the beginning of the file to determine if it's already read it, not the file name (by default).

There should be no negative effects from this change unless the star will match files you don't want to index. If that's the case you might want to review the whitelist and blacklist options on your input to fine tune what you capture:
http://docs.splunk.com/Documentation/Splunk/6.6.1/Admin/Inputsconf#MONITOR:

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...