Splunk Dev

Splunk 6 Can it index old rotated log files?

aliimami
Explorer

Hi.

I am testing out splunk and splunk storm for our cluster deployment. In our pilot, I have set up a single host quite well and am receiving data on splunk storm for now.

However, we have old log files that go back almost 4 years. Is it possible to add those files to splunk, by using the same sourcetypes as the ones I have designated for the live data? We have a lot of custom applications logging to /var/log and we have given them all custom types.

Thanks in advance.

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Sure.

Point a Splunk forwarder at the old files like you do with new files, and it'll eat them all up. Two things to keep in mind: Try to do it all in one day, so you only cause one license violation... and, if you have data older than five years, you need to set your sourcetypes to allow such old data, by default Splunk drops data older than 2000 days.

As for Splunk Storm, I'm not quite sure if you need to do additional changes or if there are other restrictions. The above approach is based on Splunk Enterprise.

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

Sure.

Point a Splunk forwarder at the old files like you do with new files, and it'll eat them all up. Two things to keep in mind: Try to do it all in one day, so you only cause one license violation... and, if you have data older than five years, you need to set your sourcetypes to allow such old data, by default Splunk drops data older than 2000 days.

As for Splunk Storm, I'm not quite sure if you need to do additional changes or if there are other restrictions. The above approach is based on Splunk Enterprise.

aliimami
Explorer

Yeah. Storm is space based. Thanks for the Licensing tip off.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Yeah, see http://docs.splunk.com/Documentation/Splunk/6.0.1/Admin/HowSplunklicensingworks for reference.

When bulk-backfilling old data it's best to do it all in one go. That's Splunk Enterprise licensing of course, not sure how Splunk Storm works. Isn't that space-based? Maybe also with restrictions on the retention time?

0 Karma

aliimami
Explorer

License violation???

And yes, Enterprise helps because I will eventually be deploying splunk enterprise at $WORK. Thanks.

0 Karma

aliimami
Explorer

To clarify, I can manually upload old files. But can Splunk automatically load old logrotated files?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...