Why are the Preset Times in Splunk Web not displaying results for a recently added log file?

[JDukeSplunk]

I recently added a .log file for an app called solr. When searching using the presets like "Today" i get no results. However, if I change this to a date range for today (3/10/17) I get results. I suspect this is because the log is not picking up a timestamp?

My inputs.conf for this.

#####################
#solr.log           #
#####################

[monitor:///var/solr/logs/solr.log]
disabled = 0
index = application
sourcetype = apollo:dev:solr
ignoreOlderThan = 7d

[monitor:///var/solr2/logs/solr.log]
disabled = 0
index = application
sourcetype = apollo:dev:solr
ignoreOlderThan = 7d

And a sample of the file.

2017-03-09 19:22:57.190 INFO  (searcherExecutor-9-thread-1-processing-n:ATLAPDSOLR02:8983_solr x:SearchAllParticipants_shard1_replica2 s:shard1 c:SearchAllParticipants r:core_node2) [c:SearchAllParticipants s:shard1 r:core_node2 x:SearchAllParticipants_shard1_replica2] o.a.s.c.QuerySenderListener QuerySenderListener done.
2017-03-09 19:22:57.191 INFO  (searcherExecutor-9-thread-1-processing-n:ATLAPDSOLR02:8983_solr x:SearchAllParticipants_shard1_replica2 s:shard1 c:SearchAllParticipants r:core_node2) [c:SearchAllParticipants s:shard1 r:core_node2 x:SearchAllParticipants_shard1_replica2] o.a.s.c.SolrCore [SearchAllParticipants_shard1_replica2] Registered new searcher Searcher@2820ae1e[SearchAllParticipants_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_qif(6.4.1):C222001/17584:delGen=3479) Uninverting(_uyp(6.4.1):C31058/1487:delGen=1029) Uninverting(_14o3(6.4.1):C64670/7052:delGen=440) Uninverting(_1b18(6.4.1):C74056/5073:delGen=51) Uninverting(_1c5i(6.4.1):c6962/2:delGen=1) Uninverting(_1c5s(6.4.1):c6968/1) Uninverting(_1c5t(6.4.1):C8/4:delGen=1) Uninverting(_1c5u(6.4.1):C1) Uninverting(_1c5v(6.4.1):C15/6:delGen=1) Uninverting(_1c5w(6.4.1):C4/1:delGen=1)))}
2017-03-09 19:23:10.176 INFO  (commitScheduler-26-thread-1) [c:SearchAllParticipants s:shard1 r:core_node2 x:SearchAllParticipants_shard1_replica2] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2017-03-09 19:23:10.176 INFO  (commitScheduler-26-thread-1) [c:SearchAllParticipants s:shard1 r:core_node2 x:SearchAllParticipants_shard1_replica2] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@463d1773
2017-03-09 19:23:10.241 INFO  (commitScheduler-26-thread-1) [c:SearchAllParticipants s:shard1 r:core_node2 x:SearchAllParticipants_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@13754916[SearchAllParticipants_shard1_replica2] realtime]
2017-03-09 19:23:10.242 INFO  (commitScheduler-26-thread-1) [c:SearchAllParticipants s:shard1 r:core_node2 x:SearchAllParticipants_shard1_replica2] o.a.s.u.DirectUpdateHandler2 end_commit_flush

[woodcock · Accepted solution]

You have not shown us the props.conf entry for this (and there may not be one) that shows how you are telling splunk about the timestamp. In any case, Splunk should be able to find that timestamp easily. The problem is likely TimeZone based. Do you have your indexer on NTP? Make sure that you do. In props.conf on your Indexers you need to tell it what TZ to use for each host value. This is probably your problem.

[JDukeSplunk]

You bonked me the right way.. Props lives in the indexer, not with the app. I had it in the wrong spot.

[apollo:dev:solr]
SHOULD_LINEMERGE=false
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3N
TZ = GMT