Splunk Search

Why is Splunk suddenly returning incorrect results with unchanged field extraction?

wpreston
Motivator

Something strange is going on. I have fields extracted via regex in transforms.conf that have been working fine for over a year. Today I was searching on one of them and it started returning incorrect results. For example, if I search for

field1=AA

over the previous 24 hours, Splunk returns 6 results. I know that there were several hundred events with field1=AA today alone, so something is off. So to prove this, I searched for a string that is in the same events where field1=AA would be true. This time, several hundred events show up and, if I click on field1 in the fields sidebar and look at the statistics they are correct, showing a count of several hundred for the value AA. I try my basic search again and it returns only 6 events.

I tried some other popular values for field1 and found mixed results. One value that should return several thousand results only returned 10. Another one returned the correct results.

I have verified the field extraction, it remains unchanged from when it was first defined over a year ago. Props.conf is also unchanged. This search worked up until around 3:30 pm today, then it just started giving back wrong results. I tried restarting Splunk but nothing changed, and I even tried cleaning and rebuilding my index, but the problem persists. Any ideas, 'O great and noble Splunkers?

EDIT: Tried creating a new field with the same extraction regex as field1 but it shows the same problems.

0 Karma

woodcock
Esteemed Legend

You have 2 options: Change from using search-time extraction (REPORT-) to index-time extraction (TRANSFORMS-) or add this to fields.conf:


[field1]
INDEXED_VALUE = false

0 Karma

grijhwani
Motivator

How many of the results you do you get for

field1=A*

Is there a possibility of other hidden characters in the field which are present, but not displayed?

What do you get if you try charting a count?

field1=A* | chart count(field1) by field1

That may give you some indication if what is going on.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...

They're back! Join the SplunkTrust and MVP at .conf24

With our highly anticipated annual conference, .conf, comes the fez-wearers you can trust! The SplunkTrust, as ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...