Why does this indexed field search give the wrong results when looking for specific values?

[socconsulting]

Hi,

We have an heavy forwarder in every location.
At the HF have an indexed field (meta) called "site-id" that gets added to each event via props/transforms (Regex = .*)

If I now do a search:

index="my_index" site-id="*" (verbose, 24 h)

As a result, I get a count of 122565 events and if I clock on the field "site-id" it shows a distribution of 100% and only 1 value "my_value"

Now, the strange behavior starts:

If I click on the field and add it to the search with the one value that exists

index="my_index" site-id="my_value" (verbose, 24 h)

I only get 47 results

If I do

index="my_index" site-id="my_value*" (verbose, 24 h)

I get the 122565 results again

There are no hidden character or anything at that values I exported it and looked at the character coding only "LF"

I even tried the following two searches to see if there is any difference:

index="my_index" site-id="*"| strcat site-id ":TEST" new_site_id | search new_site_id="my_value:TEST" | stats count by new_site_id
gives me the result result: count 122565
index="my_index" site-id="my_value"| strcat site-id ":TEST" new_site_id | search new_site_id="my_value:TEST" | stats count by 
new_site_id

gives me the result: count 47

search ... | fieldsummary site-id gives a count of 122565 a singe value of my_value and a dc=1

Why can't I search for site-id="my_value" and get the 122565 results?

Please any ideas?

Best
Michael

[socconsulting · Accepted solution]

SOLUTION:

The field was not known to the search head in the context of the search app / system
So we added a metadata export description on the search head inside our fields.conf

my_fields_app/metadata/default.meta
[]
access = read : [ * ], write : [ admin ]
export = system

Now everything works like expected

[prakash007]

When you run this search, did you see any other additional sourcetypes/sources/hosts...??

index="my_index" site-id="my_value*" (verbose, 24 h)

[socconsulting]

No just the expected one.
We now even tried to change the added metafield from site-id to site_id to see if the "-" was not accepted by splunk but that did not change anything. We than added a fields.conf for the search head and the indexer cluster like:

[site_id]
INDEXED = true
INDEXED_VALUE = false

[site-id]
INDEXED = true
INDEXED_VALUE = false

We tried with and without the "INDEXED_VALUES" attribute without any difference.