Hi All,
My splunk has indexed some data today. However, I am not able to search the previously indexed data anymore. For example, I am doing a search source="log.2015-05-31"
, it didn't show up any events, but it was able to show events on my previous report. When I change a search to source="log.2015-06-01"
, it does show the events, but not in my report. Thus my report can only show the result until 31-05-2015.
Is there any permission issue during search? I only made changes to admin role to inherit can_delete.
Given this screenshot:
The problem is clear, Splunk assumes the date format is day/month/year until it realizes that this cannot be correct because the month is greater than 12 so it swaps and uses month/day/year.
You need to add this to props.conf
[YourSourcetypeHere]
TIME_FORMAT = %m/%d/%Y %H:%M:%S
Then all will be well for FUTURE events (events in the past will stay broken).
Given this screenshot:
The problem is clear, Splunk assumes the date format is day/month/year until it realizes that this cannot be correct because the month is greater than 12 so it swaps and uses month/day/year.
You need to add this to props.conf
[YourSourcetypeHere]
TIME_FORMAT = %m/%d/%Y %H:%M:%S
Then all will be well for FUTURE events (events in the past will stay broken).
Hi Woodcock,
I can't find my sourcetype on the /opt/splunk/etc/system/default/props.conf file, last time it was automatic naming by the Splunk for setting the sourcetype name. Is there anyway I can change the setting instead of changing props.conf?
You have to change props.conf
to fix this; the sourcetype that is given to you is call.cdr-too_small
so just use that like this:
[call.cdr-too_small]
TIME_FORMAT = %m/%d/%Y %H:%M:%S
Hi Woodcock,
I did added the TIME_FORMAT = %m/%d/%Y %H:%M:%S inside the /opt/splunk/etc/apps/learned/local/props.conf which have the [call.cdr-too_small], and restart the Splunk. However it doesn't change the format on my event. Am I configure the correct props.conf?
It will not fix previously indexed events; only events that are indexed after you make the change and restart the indexers.
Hi woodcock,
Unfortunately, it still not changing the time format that I defined after I index some new files.
Then your configuration settings are not in scope for your data. You should start over, specify your own MyApp in$SPLUNK_HOME/etc/apps/MyApp/default/
, define your owninputs.conf
(assigning your own sourcetype
) and props.conf
files and work out from there, The problem at this point is that you relying too much on Splunk learning about your input instead of you telling it about it. I have never tried to work with any of the learned
sourcetypes and there may be quirks to doing so which is why I always start out as my own app.
Another thing that you can do that will probably work is to force Splunk to "forget" what it has learned about your source. Go into the learned
directory and look for settings that conflict with yours or, alternatively, disable the learned app and see what that does (as a test, though; do not leave it disabled).
Thanks, I have defined another sourcetype and get into props.conf to configure the time format, and it's working fine now.
OK, based on what you just wrote, the problem is now clear: you have a tomestamp
peoblem that is putting nowish/newish events into the future. Such events are only searchable with All time
. The problem is probably timezone related. This search will help you focus in on and track the problem. You need to get the numbers in the range of 100-1000 (typical):
... | eval lagSecs=(_indextime - _time) | stats count avg(lagSecs) BY source
Hi Woodcock,
Yes, after I did a "All Time", it does show all my logs with the latest log display(log.2015-06-02). But it is weird when I look on the lagSecs column, for the log from 2015-05-22 until 2015-05-31 (legSec2 is around 200000~1000000) but lagSecs for log 2015-06-01 until 2015-06-02 is very huge (12000000~10000000).
On my search command, I always search as a "All Time" period. For example, my log file has a keyword "Call_successful", then I put this keyword on the search bar and click "All Time". My first page of the first event is showing the log event of 2015-05-31 but the last page is showing log event of 2015-06-02. I noticed on the Time column the time format is different, for log event of 2015-05-31 the time format is 5/22/15 12:46:04.000 PM but for log event of 2015-06-02 is 2/6/15 11:49:23.000 PM, I believe this caused the time order mismatch, is there anyway to fix it?
Yes, you need to explicitly tell Splunk where the correct timestamp is using the TIME_PREFIX
directive. You can create a RegEx
for it that is flexible (e.g. use the 5th field if it exists but if not, use the 4th field).
Hi Woodcock,
In this case if I want to change the _time format to 5/22/15, as I highlighted as red color, please refer the screenshot (http://imgur.com/cGtFMdu). Some of my event has a different _time format, that's is the reason why it can't show the 1st Jun 2015 event, it is because the format is different from May, when you look at the screenshot then you will have a clear picture for my issue. I want to have _time with a same format, so it can display correct date when I do a timechart span=1d command.
Just to be safe, try this:
earliest=-90d@d source="log.2015-05-31"
If that works, your issue is just the time range of your search.
I have tried this, but still the same, no result at all.
How big is your index space (in indexes.conf
)? Splunk will automatically expire data (FIFO) and if you have a very small amount of space for you index, it could have already aged out. Use this search to check your retentions:
index=_internal sourcetype=splunkd bucketmover "will attempt to freeze" | rex "/splunk(?:/[^/]*)?/(?<indexname>[^/]*)/db/db_(?<newestTime>[^_]*)_(?<oldestTime>[^_]*)_.*" | dedup indexname | eval retentionDays=(now()-oldestTime)/(60*60*24) | stats values(retentionDays) as retentionDays by indexname
Hi Woodcock,
Below is my Splunk indexes.conf, and when I run your search command that you provided, the retention days is around 34.857905.
maxTotalDataSizeMB = 500000
By any chance, are you using | delete
somewhere to delete your events?
Hi Woodcock,
Yes, I did a |delete
before to delete one event.
I did a source="log.2015-05-22" | delete
, will this affect my whole result? I thought I only delete this event, and the other event should be able to display.
Whatere events were returned by the same command without the | delete
command will all be deleted when yoi tack that on so all events from that source are now gone. So that explains everything, right?