We recently onboarded some applications' logs, and at our client request, we had to put a custom field to have the application name in the fields. Since those weren’t in the log file, we hard-coded the application name in the log with a “_meta” field in the inputs.conf stanza. Since each application has its own stanza, it was easy.
One of the analysts discovered this week that some events were discarded when a filter was applied for a certain application. We were able to reproduce the problem at a smaller scale.
Here the picture shows that there are 11 events for IAM, if we apply the filter for that application...
This is only a 5 minutes window, at a "All Time" time range it's more than 10K events that goes missing
Here is our Inputs.conf for that application
[monitor://D:\APPLogs\IAM\IAM.log]
index = grc
sourcetype = GRC:APP
alwaysOpenFile = 1
disabled = false
_meta = application::IAM
Thanks somesoni2 for your help, your last fix helped us out with our problem.
Steps:
On our SH and Indexers we deployed a TA with a fields.conf file with this stanza :
[application]
INDEXED = true
(Reference: https://answers.splunk.com/answers/389567/why-is-a-search-for-fields-added-with-meta-in-inpu.html)
Restarted and the problem was gone.
Thanks again
M.
Thanks somesoni2 for your help, your last fix helped us out with our problem.
Steps:
On our SH and Indexers we deployed a TA with a fields.conf file with this stanza :
[application]
INDEXED = true
(Reference: https://answers.splunk.com/answers/389567/why-is-a-search-for-fields-added-with-meta-in-inpu.html)
Restarted and the problem was gone.
Thanks again
M.
Do you get a correct count when you use either of these searches?
index=grc application=*IAM
index=grc | regex application="IAM"
index=grc application!=IAM:IWSS
index=grc application=*IAM
index=grc | regex application="IAM"
index=grc application!=IAM:IWSS
Ok.. how about this?
index=grc application="IAM"
No, showing only 10 Events
Ok lets check if there are additional character coming in the application name.
index=grc application=*IAM | eval applength=len(application) | table applength
I've got only 3's
Try this fix. On your search head and Indexers, add following to fields.conf (preferred to be kept under some app). Should work (Reference: https://answers.splunk.com/answers/389567/why-is-a-search-for-fields-added-with-meta-in-inpu.html)
[application]
INDEXED = true
Hi, sorry for the late reply. It fixed our problem. I'll add an answer to the question with your fix. Thanks for your help