Getting Data In

Why won't Splunk re-index my data?

Mick
Splunk Employee
Splunk Employee

I wanted to see how Splunk would index my data, so I configured it to index a few files into a 'test' index. Now that I have it configured properly, I want to re-index that same data into the 'main' index. I cleaned the test index - ./splunk clean eventdata index test - and removed the index = test from inputs.conf, but Splunk doesn't automatically re-index the files - why not?

Tags (3)
1 Solution

amrit
Splunk Employee
Splunk Employee

removing index=test probably tripped you up.

the logic for this feature is:
if:
- indexing a file F
- splunkd's last record of F is from time T1
- the creation time of the destination index is T2
- T1 < T2
then:
- begin reading F from position 0 again.

i'm assuming you didn't clean index main here, so its creation date is well before our last fishbucket record for that file - thus T1<T2 is false, and we don't re-read the file.

View solution in original post

amrit
Splunk Employee
Splunk Employee

removing index=test probably tripped you up.

the logic for this feature is:
if:
- indexing a file F
- splunkd's last record of F is from time T1
- the creation time of the destination index is T2
- T1 < T2
then:
- begin reading F from position 0 again.

i'm assuming you didn't clean index main here, so its creation date is well before our last fishbucket record for that file - thus T1<T2 is false, and we don't re-read the file.

gkanapathy
Splunk Employee
Splunk Employee

i forget one very. very.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

that would be a very very very good feature to have.

0 Karma

amrit
Splunk Employee
Splunk Employee

maybe you should RTFM and get it right the first time. jk!! ❤️ ...did you clean before you had index=test in the conf? if you cleaned test, restarted, added index=test, and restarted again, you'll encounter the above fail. anyway, good point about finalizing in a different index. right now this isn't possible (although, you could add a bogus crcSalt and leave it there forever...), but we can add something. the idea would be something like "splunk reset filepos /path/to/file", which would use btprobe to zero out our record of the file. this feature does not exist currently.

Mick
Splunk Employee
Splunk Employee

even if I leave 'index = test' the files don't get re-read, but based on what you're saying, I would have to create a brand new index for the data to get re-read? That doesn't make sense to me, I want to add it to my existing index once I'm happy that it will be indexed correctly. What if I have to tweak settings several times before I get it right, do I need to create a new 'test' index each time?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...