Splunk Search

Why are some expected events missing when applying a filter in our search?

mdelwaide
Path Finder

We recently onboarded some applications' logs, and at our client request, we had to put a custom field to have the application name in the fields. Since those weren’t in the log file, we hard-coded the application name in the log with a “_meta” field in the inputs.conf stanza. Since each application has its own stanza, it was easy.

One of the analysts discovered this week that some events were discarded when a filter was applied for a certain application. We were able to reproduce the problem at a smaller scale.

alt text

Here the picture shows that there are 11 events for IAM, if we apply the filter for that application...

alt text

This is only a 5 minutes window, at a "All Time" time range it's more than 10K events that goes missing

Here is our Inputs.conf for that application

[monitor://D:\APPLogs\IAM\IAM.log]
index = grc
sourcetype = GRC:APP
alwaysOpenFile = 1
disabled = false
_meta = application::IAM
0 Karma
1 Solution

mdelwaide
Path Finder

Thanks somesoni2 for your help, your last fix helped us out with our problem.

Steps:

On our SH and Indexers we deployed a TA with a fields.conf file with this stanza :

[application]
INDEXED = true

(Reference: https://answers.splunk.com/answers/389567/why-is-a-search-for-fields-added-with-meta-in-inpu.html)

Restarted and the problem was gone.

Thanks again

M.

View solution in original post

0 Karma

mdelwaide
Path Finder

Thanks somesoni2 for your help, your last fix helped us out with our problem.

Steps:

On our SH and Indexers we deployed a TA with a fields.conf file with this stanza :

[application]
INDEXED = true

(Reference: https://answers.splunk.com/answers/389567/why-is-a-search-for-fields-added-with-meta-in-inpu.html)

Restarted and the problem was gone.

Thanks again

M.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Do you get a correct count when you use either of these searches?

index=grc application=*IAM

index=grc | regex application="IAM"

index=grc application!=IAM:IWSS
0 Karma

mdelwaide
Path Finder

index=grc application=*IAM

  • yes (Only IAM)

index=grc | regex application="IAM"

  • We get all events (IAM and IAM:IWSS)

index=grc application!=IAM:IWSS

  • yes
0 Karma

somesoni2
SplunkTrust
SplunkTrust

Ok.. how about this?

index=grc application="IAM"
0 Karma

mdelwaide
Path Finder

No, showing only 10 Events

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Ok lets check if there are additional character coming in the application name.

index=grc application=*IAM | eval applength=len(application) | table applength
0 Karma

mdelwaide
Path Finder

I've got only 3's

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Try this fix. On your search head and Indexers, add following to fields.conf (preferred to be kept under some app). Should work (Reference: https://answers.splunk.com/answers/389567/why-is-a-search-for-fields-added-with-meta-in-inpu.html)

[application]
 INDEXED = true

mdelwaide
Path Finder

Hi, sorry for the late reply. It fixed our problem. I'll add an answer to the question with your fix. Thanks for your help

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...