Hi All,
My splunk has indexed some data today. However, I am not able to search the previously indexed data anymore. For example, I am doing a search source="log.2015-05-31"
, it didn't show up any events, but it was able to show events on my previous report. When I change a search to source="log.2015-06-01"
, it does show the events, but not in my report. Thus my report can only show the result until 31-05-2015.
Is there any permission issue during search? I only made changes to admin role to inherit can_delete.
Given this screenshot:
The problem is clear, Splunk assumes the date format is day/month/year until it realizes that this cannot be correct because the month is greater than 12 so it swaps and uses month/day/year.
You need to add this to props.conf
[YourSourcetypeHere]
TIME_FORMAT = %m/%d/%Y %H:%M:%S
Then all will be well for FUTURE events (events in the past will stay broken).
I understand, I only deleted source="log.2015-05-22"
, but other source likesource="log.2015-05-23"
or source="log.2015-06-01"
should not be deleted and able to display the event, am I right?
If I want to re-index back, what should I do? I have already tried to re-index the source="log.2015-05-22"
, but there is no event showing anymore for this source.
If you edit the file and swap the first 2 lines (move the top line down 1 line), it should re-index the file. The rest of what you are saying makes no sense unless you accidentally deleted more than you think you did.
I know it sound weird, but it actually happen to me. For example, I put in a new log file(/home/user/cdr/chat.log.2015-06-02), when I try to do a search source="/home/user/cdr/chat.cdr.2015-06-02"
, there is no result at all. Any comments?
Do this search for "All Time" just to make sure the events are not timestamped "in the future" or something way off from what you expect:
... | eval lagSecs=(_indextime - _time) | stats count avg(lagSecs) BY source
Hi Woodcock,
I have tried the command you provided, and it's able to show some of the index files. The result only show log.2015-05-22 until log.2015-05-31. Since my oldest log file is log.2015-05-22, hence the result display is correct. However, my latest indexed file should display log.2015-06-02, unfortunately, it doesn't show up.
Did you run it for "All Time"? This is very important (otherwise "future" events will not be found).
Hi Woodcock,
Yes, after I did a "All Time", it does show all my logs with the latest log display(log.2015-06-02). But it is weird when I look on the lagSecs column, for the log from 2015-05-22 until 2015-05-31 (legSec2 is around 200000~1000000) but lagSecs for log 2015-06-01 until 2015-06-02 is very huge (12000000~10000000). Is this the reason that caused the Splunk can't show the event of 2015-06-01 onward?
Yes, after I run the command for "All time", the source display all the log which start from log.2015-05-22 until log.2015-06-02. Since the log file of 2015-06-02 has been indexed, why I can't see the statistic display on my report? My report only show the statistic start from 2015-05-22 until 2015-05-31 only.