Splunk Search

Why are historical events not showing up in searches?

amoore12
Explorer

I needed to restart my Splunk instance on our heavy forwarder the other day. After restarting, I am unable to search on anything before the restart on our main indexer. I am not seeing anything in the logs to indicate an issue. If I browse out to the DB directories, I see the older data still there. The search summary shows the stats correctly. I have restarted the Splunk process several times in the past with no issues. Any ideas how to correct this issue? Any help would be great. Thanks.

Labels (1)
Tags (2)

lukejadamec
Super Champion

If you have made no changes to your index retention policy, then you should have no local/index.conf files with policy settings. If you do have local/index.conf files, they should not contain DEFAULT settings.
Here is some info on index.conf settings:
http://wiki.splunk.com/Deploy:BucketRotationAndRetention

0 Karma

lukejadamec
Super Champion

What are the permissions on the hot buckets?

0 Karma

amoore12
Explorer

Now that I stopped Splunk and started it back up again. I can only search back to a few minutes ago. Very strange.

Yes permissions are the same

0 Karma

amoore12
Explorer

I ran the splunk fsck --all. No results were returned.

0 Karma

lukejadamec
Super Champion

Those permissions seem a bit restrictive. Are they the same on the searchable and non-searchable buckets?

0 Karma

amoore12
Explorer

db_1374688147_1374668673_3725_79F4ED3B-4785-4ECF-8AE6-AA7FE75EEAA9
Wednesday, July 24, 2013 1:49:07 PM to Wednesday, July 24, 2013 8:24:33 AM

db_1375807093_1375787134_3841_79F4ED3B-4785-4ECF-8AE6-AA7FE75EEAA9
Tuesday, August 06, 2013 12:38:13 PM to Tuesday, August 06, 2013 7:05:34 AM

*Data is contained in that file*
[root@feprsplunk01 db_1375807093_1375787134_3841_79F4ED3B-4785-4ECF-8AE6-AA7FE75EEAA9]# ls -l
total 622264
-rw------- 1 root root 451868154 Aug 6 09:34 1375807093-1375787134-2736230854333308492.tsidx

0 Karma

lukejadamec
Super Champion

You can also run a check on your buckets.

splunk stop
splunk fsck --all
splunk start

Here are the details on "splunk fsck"
http://wiki.splunk.com/Community:PostCrashFsckRepair

0 Karma

lukejadamec
Super Champion

You know, the buckets are named 'earliesttimestamp_latesttimestamp_uniquenumber', and the times are epoch times. Convert one or two of the non-searchable ones: http://www.epochconverter.com/

0 Karma

amoore12
Explorer

Yes, if I go to say the windows DB directory via ssh, I can see all the new and old files.

Yes, I can search all indexes and get only the new events since restarting.

0 Karma

lukejadamec
Super Champion

Let me see if I have this straight.

1) You can see the searchable and non-searchable buckets in the Same index/database.
2) When you search that index/database you only see events since the restart?

0 Karma

wagnerbianchi
Splunk Employee
Splunk Employee

Maybe the buckets configuration. Could you check configurations?

0 Karma

wagnerbianchi
Splunk Employee
Splunk Employee

Check that in indexes.conf

0 Karma

amoore12
Explorer

Is there a global setting or is that done per index? If global, what file am I looking for?

0 Karma

amoore12
Explorer

Yes the unix servers are not going to the heavy forwarder. All events since the restart have been indexed. Yes, splunk has done something crazy. I was wondering if there is a corrupt DB file or something since splunk starts with newer events and goes backwards.

0 Karma

lukejadamec
Super Champion

You win the splunk did something weird award.

I'm gonna have to think about that one.

0 Karma

amoore12
Explorer

It is indexing new events as it should from both unix and windows. What I lost were events prior to restarting the service. I can search on all events since the restart.

0 Karma

lukejadamec
Super Champion

So, you're not getting data from either unix (not passing through the heavy forwarder, or from windows?

0 Karma

amoore12
Explorer

On the main indexer, I have syslog and Splunk forwarders from unix. On the heavy forwarder, it is just windows servers from Splunk forwarders that I am filtering out some events before forwarding on.

0 Karma

lukejadamec
Super Champion

What type of inputs?

0 Karma

dhimanv
Loves-to-Learn Lots

What solution found in this case?

Was issue sorted out because I am also facing the same kind of issue.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...