Splunk Search

Why am I unable to search previously indexed data?

cykuan
New Member

Hi All,

My splunk has indexed some data today. However, I am not able to search the previously indexed data anymore. For example, I am doing a search source="log.2015-05-31", it didn't show up any events, but it was able to show events on my previous report. When I change a search to source="log.2015-06-01", it does show the events, but not in my report. Thus my report can only show the result until 31-05-2015.

Is there any permission issue during search? I only made changes to admin role to inherit can_delete.

Tags (2)
0 Karma
1 Solution

woodcock
Esteemed Legend

Given this screenshot:
alt text

The problem is clear, Splunk assumes the date format is day/month/year until it realizes that this cannot be correct because the month is greater than 12 so it swaps and uses month/day/year.

You need to add this to props.conf

[YourSourcetypeHere]
TIME_FORMAT = %m/%d/%Y %H:%M:%S

Then all will be well for FUTURE events (events in the past will stay broken).

View solution in original post

woodcock
Esteemed Legend

Given this screenshot:
alt text

The problem is clear, Splunk assumes the date format is day/month/year until it realizes that this cannot be correct because the month is greater than 12 so it swaps and uses month/day/year.

You need to add this to props.conf

[YourSourcetypeHere]
TIME_FORMAT = %m/%d/%Y %H:%M:%S

Then all will be well for FUTURE events (events in the past will stay broken).

cykuan
New Member

Hi Woodcock,

I can't find my sourcetype on the /opt/splunk/etc/system/default/props.conf file, last time it was automatic naming by the Splunk for setting the sourcetype name. Is there anyway I can change the setting instead of changing props.conf?

0 Karma

woodcock
Esteemed Legend

You have to change props.conf to fix this; the sourcetype that is given to you is call.cdr-too_small so just use that like this:

[call.cdr-too_small]
TIME_FORMAT = %m/%d/%Y %H:%M:%S
0 Karma

cykuan
New Member

Hi Woodcock,

I did added the TIME_FORMAT = %m/%d/%Y %H:%M:%S inside the /opt/splunk/etc/apps/learned/local/props.conf which have the [call.cdr-too_small], and restart the Splunk. However it doesn't change the format on my event. Am I configure the correct props.conf?

0 Karma

woodcock
Esteemed Legend

It will not fix previously indexed events; only events that are indexed after you make the change and restart the indexers.

0 Karma

cykuan
New Member

Hi woodcock,

Unfortunately, it still not changing the time format that I defined after I index some new files.

0 Karma

woodcock
Esteemed Legend

Then your configuration settings are not in scope for your data. You should start over, specify your own MyApp in$SPLUNK_HOME/etc/apps/MyApp/default/, define your owninputs.conf (assigning your own sourcetype) and props.conf files and work out from there, The problem at this point is that you relying too much on Splunk learning about your input instead of you telling it about it. I have never tried to work with any of the learned sourcetypes and there may be quirks to doing so which is why I always start out as my own app.

0 Karma

woodcock
Esteemed Legend

Another thing that you can do that will probably work is to force Splunk to "forget" what it has learned about your source. Go into the learned directory and look for settings that conflict with yours or, alternatively, disable the learned app and see what that does (as a test, though; do not leave it disabled).

0 Karma

cykuan
New Member

Thanks, I have defined another sourcetype and get into props.conf to configure the time format, and it's working fine now.

0 Karma

woodcock
Esteemed Legend

OK, based on what you just wrote, the problem is now clear: you have a tomestamp peoblem that is putting nowish/newish events into the future. Such events are only searchable with All time. The problem is probably timezone related. This search will help you focus in on and track the problem. You need to get the numbers in the range of 100-1000 (typical):

... | eval lagSecs=(_indextime - _time) | stats count avg(lagSecs) BY source
0 Karma

cykuan
New Member

Hi Woodcock,

Yes, after I did a "All Time", it does show all my logs with the latest log display(log.2015-06-02). But it is weird when I look on the lagSecs column, for the log from 2015-05-22 until 2015-05-31 (legSec2 is around 200000~1000000) but lagSecs for log 2015-06-01 until 2015-06-02 is very huge (12000000~10000000).

On my search command, I always search as a "All Time" period. For example, my log file has a keyword "Call_successful", then I put this keyword on the search bar and click "All Time". My first page of the first event is showing the log event of 2015-05-31 but the last page is showing log event of 2015-06-02. I noticed on the Time column the time format is different, for log event of 2015-05-31 the time format is 5/22/15 12:46:04.000 PM but for log event of 2015-06-02 is 2/6/15 11:49:23.000 PM, I believe this caused the time order mismatch, is there anyway to fix it?

0 Karma

woodcock
Esteemed Legend

Yes, you need to explicitly tell Splunk where the correct timestamp is using the TIME_PREFIX directive. You can create a RegEx for it that is flexible (e.g. use the 5th field if it exists but if not, use the 4th field).

0 Karma

cykuan
New Member

Hi Woodcock,

In this case if I want to change the _time format to 5/22/15, as I highlighted as red color, please refer the screenshot (http://imgur.com/cGtFMdu). Some of my event has a different _time format, that's is the reason why it can't show the 1st Jun 2015 event, it is because the format is different from May, when you look at the screenshot then you will have a clear picture for my issue. I want to have _time with a same format, so it can display correct date when I do a timechart span=1d command.

0 Karma

jacobwilkins
Communicator

Just to be safe, try this:

earliest=-90d@d source="log.2015-05-31"

If that works, your issue is just the time range of your search.

0 Karma

cykuan
New Member

I have tried this, but still the same, no result at all.

0 Karma

woodcock
Esteemed Legend

How big is your index space (in indexes.conf)? Splunk will automatically expire data (FIFO) and if you have a very small amount of space for you index, it could have already aged out. Use this search to check your retentions:

index=_internal sourcetype=splunkd bucketmover "will attempt to freeze" | rex "/splunk(?:/[^/]*)?/(?<indexname>[^/]*)/db/db_(?<newestTime>[^_]*)_(?<oldestTime>[^_]*)_.*" | dedup indexname | eval retentionDays=(now()-oldestTime)/(60*60*24) | stats values(retentionDays) as retentionDays by indexname
0 Karma

cykuan
New Member

Hi Woodcock,

Below is my Splunk indexes.conf, and when I run your search command that you provided, the retention days is around 34.857905.

maxTotalDataSizeMB = 500000

0 Karma

woodcock
Esteemed Legend

By any chance, are you using | delete somewhere to delete your events?

0 Karma

cykuan
New Member

Hi Woodcock,

Yes, I did a |delete before to delete one event.

I did a source="log.2015-05-22" | delete, will this affect my whole result? I thought I only delete this event, and the other event should be able to display.

0 Karma

woodcock
Esteemed Legend

Whatere events were returned by the same command without the | delete command will all be deleted when yoi tack that on so all events from that source are now gone. So that explains everything, right?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...