Getting Data In

How to configure Splunk to NOT repeatedly read and index old data?

LuiesCui
Communicator

New to splunk...i need the splunk server to get data from a .log file which is constantly being inserted data. I hope the index will read the new data in the .log file once the new data is inserted. But what happens is the index also read the old data and it makes the index volume too large. How could I solve this problem?

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

The difference is that Splunk expects logfiles to be appended to the end. If the file changes anywhere else it's considered to be a new file, and therefore reindexed entirely.

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

The difference is that Splunk expects logfiles to be appended to the end. If the file changes anywhere else it's considered to be a new file, and therefore reindexed entirely.

LuiesCui
Communicator

Is it possible to make splunk read the new data only by changing any configurations?

0 Karma

Ayn
Legend

No. Splunk works on streams of log data, where it is assumed that whatever data is added comes at the end. Otherwise Splunk would have to somehow keep a record of what the file looked like, do a diff and then grab only stuff that's been added, which would be very resource intensive and lead to all kinds of weird problems.

martin_mueller
SplunkTrust
SplunkTrust

That should by default lead to Splunk only reading in new lines from the end.

Do post the configuration you used to read this file.

0 Karma

LuiesCui
Communicator

sorry the new data are inserted in the middle of the whole file's data cuz there are different parts of data in the file. What's the difference between these situations?

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Are you appending to the end of the logfile or inserting data somewhere else?

0 Karma

LuiesCui
Communicator

Appending to the end of the .log

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...