Monitoring Splunk

whole file is re-indexed on addition of new events

newbie2tech
Communicator

Hi All,

I need to monitor a log file which is pipe delimited, events get written onto log as the day progresses.

On my sandbox am trying to mimic the addition of new events by adding some events manually using vi editor, vi the file , add some lines at the bottom and save it [:wq!] , however this process is re-indexing the complete file.

I am using initCrcLength = 2500 in my props.conf. I am not using crcSALT and my log does not have any header

Any suggestions on how to avoid the file from completely re-indexing

Below is the log sample.

hostname|cluster_name|11/26/17 00:43:19|AB- 1|INFO| Retail.getCategoryListCodesFromProperties() retail Code List to show the link ::[02756, 2127]
hostname|cluster_name|11/26/17 00:49:28|AB-No Memory|object|||||||123467|123123123|01
hostname|cluster_name|11/26/17 00:51:42|AB-No Memory|object|||||||123455|123123123|00
hostname|cluster_name|11/26/17 01:04:28|AB-No Memory|object|||||||111111|123123123|01
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Header><wsse:Security xmlns:wsse="http://docs.basis-open.org/wss/2004/01/basis-2011-wss-wss-secext-1.0.xsd"><wsse:UsernameToken xmlns:wsse="http://docs.basis-open.org/wss/2004/01/basis-2011-wss-wss-secext-1.0.xsd" xmlns:wsu="http://docs.basis-open.org/wss/2004/01/basis-2011-wss-wss-utility-1.0.xsd"></soapenv:Body></soapenv:Envelope>
hostname|cluster_name|11/26/17 01:06:42|AB-No Memory|object|||||||222222|123123123|00
hostname|cluster_name|11/26/17 01:19:28|AB-No Memory|object|||||||333333|123123123|01
hostname|cluster_name|11/26/17 01:21:42|AB-No Memory|object|||||||555555|123123123|10
hostname|cluster_name|11/26/17 01:34:28|AB-No Memory|object|||||||777777|123123123|11
hostname|cluster_name|11/26/17 01:36:42|AB-No Memory|object|||||||111111|123123123|10
Tags (1)
0 Karma

sandyIscream
Communicator

Could you please share your inputs.conf

But my suggestion is first change the CrcSalt text into something else and then push the changes again from the deployment server.

If this also doesn't work then delete your fishbucket of your forwarder and check

0 Karma

skoelpin
SplunkTrust
SplunkTrust

The issue is with your fishbucket on the forwarder..

https://www.splunk.com/blog/2008/08/14/what-is-this-fishbucket-thing.html

0 Karma

traxxasbreaker
Communicator

The initCrcLength setting is known to cause data to be reindexed when it is applied. Did you only see it reindexed once right after applying that update, or do you see it reindexed every time an event is added?

0 Karma

newbie2tech
Communicator

i was having same issue even before introducing the initCrclength, as it was re-indexing whole file i tried initCrclength to avoid it but it did not help.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...