We’re having an issue with splunk where our logs are only being partially indexed. are there any logs in splunk where we can check to see what was actually indexed or some way to force a reindexing of said logs.
Basically when we search for specific parts of the log it does not show up in the splunk search results, but when we look at the actual log files we see the information we are looking for is there.
Is this a multi-line log?
If it is, check out the documentation on multi-line events.
for example i will search for 'isnwmkflbndd15eyhwtn0rk1' and splunk will return this snippet,
2012-03-06 08:50:04,177 [41] [sid=isnwmkflbndd15eyhwtn0rk1] INFO OfferHistoryProcessor - Sending request to service bus...
and that's all there is when i show source in splunk. when in reality the actual log has a lot more information in it.
it looks like splunk is setting a timestamp for the complete logs for some date in the future and i'll get all the results i'm looking for if i set my time range to all time.
So, it sounds like the logs 'are' there, but your search term isn't catching. Can you share 1) what you are searching on 2) what you are expecting to get and 3) what you are actually getting?