Splunk Search

How to troubleshoot why I'm missing events in my search results?

omuelle1
Communicator

Hi,

I have an issue with a search, that I also use as an alert, which is not finding current events:

alt text

So the search shows me the latest entry at 12/15/15 when in fact it also occurred last night 12/22/15.

2015 Dec 22 03:46:56:844 GMT -0500 BW.BusinessConnect-Interior_Server-2 Debug [bw.logger] BW-EXT-LOG-300002  {standardID=X12, errorInfoList={1={errorDataParent=CTB*AA*Mr. Simon          xxx-xxx-xxxx, errorSeverity=Normal, errorSegmentCount=11, errorData=Mr. Simon   xxx-xxx-xxxx, errorCategory=Rejecting,

As I see above in the log.

Splunk is not showing me any malfunctiong regarding the index, and is showing me events around the timeframe even within the same minute and even second. However, it didn't index this event.

I didn't change anything regarding the indexes or excluded anything since 12/15/15. So I am a little confused why it wouldn't find the string.

The only thing I noticed is that the log file that splunk searches is generating probably around 1000 lines per second. Might that be too much for Splunk to index?

Thank you,

Oliver

0 Karma

jkat54
SplunkTrust
SplunkTrust

Hi, you're doing a full text search on the data. "key=value" is not the same in Splunk as key=value. See this for more details: http://docs.splunk.com/Documentation/Splunk/6.3.1511/Search/Usethesearchcommand

1st off, you need to properly "handle" your events by breaking these multiline events into separate events. For this you will need props.conf settings. You'll need should line merge set to false, and a proper line breaker or break only before, etc. configured. Please read the docs on getting data into splunk. http://docs.splunk.com/Documentation/Splunk/6.2.0/Data/IndexMulti-lineEvents

Once you've added a line breaker and set should line merge to false, you'll then need to extract this field called "errorCategory".

This way your search will look like this index=tibco errorCategory=Rejecting versus index=tibco "errorCategory=Rejecting". If you do what I say... you'll find it solves your problem.

jplumsdaine22
Influencer

When you say you have events around the missing one, are they all from the same BusinessConnect-Interior_Server.log ? Or do the events in the same timeframe come from different logs?

Do you get any events if you search the splunkd.log on the forwarder for "BusinessConnect-Interior_Server.log"?

Are the values for _indextime and _time identical for events in the BusinessConnect-Interior_Server.log ?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...