Splunk Search

Events not found, even though dashboard says there are

gregwilliams
Path Finder

I have a universal forwarder pulling in a log file from a linux server. It has been working just fine up until the other day. The Summary dashboard shows events are coming in, however when I search for anything related to the source, sourcetype, anything within the event, I get "No Matching Events found". I can see events from other servers no problem. Nothing in the infrastructure nor Splunk config has changed. I am an admin, so I should be seeing everything. Any ideas?

Tags (1)
0 Karma
1 Solution

gregwilliams
Path Finder

Problem solved. The problem was Splunk was reading the log timestamp incorrectly. What's odd however is that the log timestamp format never changed, but Splunk was just now unable to read it correctly. I was able to configure the program sending the log to include the year instead of just the date.

View solution in original post

0 Karma

gregwilliams
Path Finder

Problem solved. The problem was Splunk was reading the log timestamp incorrectly. What's odd however is that the log timestamp format never changed, but Splunk was just now unable to read it correctly. I was able to configure the program sending the log to include the year instead of just the date.

0 Karma

gregwilliams
Path Finder

I was able to change the timestamp being produced by the program and Splunk now identifies it correctly. It's very odd however that it would just stop reading the correct timestamp from one minute to the next when nothing changed in the program, the logs themselves nor splunk

0 Karma

gregwilliams
Path Finder

I added this in my props.conf but without luck.

[sourcetype]
TIME_FORMAT=%m/%d-%h:%m:%s

The event comes in as:

07/05-10:01:01

0 Karma

sowings
Splunk Employee
Splunk Employee

Check TIMESTAMP_FORMAT in props.conf.

0 Karma

gregwilliams
Path Finder

That actually helped me figure it out. Apparently Splunk is reading the timestamp on the logs differently now and putting them in completely different days. So for example 07/05-08:25:43 is being read as May 8 8:25:43. It's ignoring month. I'll try to figure out how to change that. Thanks for the help.

0 Karma

Ayn
Legend

Perhaps it's writing to another index than the one(s) you're searching?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...