Why am I having trouble searching exact field values indexed via an HTTP Event Collector?

[jensguenther]

Hi Splunkers,

I've got a strange problem over here: I got events indexed via the http event collector which behave strange when searched for exact field values.

Let's say I got events with a field "domain" with value "example.org". Splunk reports those field/values correctly.

Now, if I search for

domain="example.org"

Splunk returns no events at all. However, if I search for

domain="*example.org*"

Splunk returns all matching events. More funny, if I do

* | eval dd=domain | search dd="example.org"

Splunk returns all matching events.

We do have this effect only on a few fields for this particular http collector stream. Here is an example of the (stripped) payload we send over

{"event":"EventName","fields":{"domain":"example.org"}}

Any ideas?

[appneta_tray]

I have the same issue. We are submitting several fields via HTTP collector. The fields all seem to be parsed correctly, all my events are showing up in the index. When I try something as simple as clicking on one of the fields and selecting 'Add to search', the exact value that I've just clicked is added to the search and the search returns nothing.

The fact this thread is 2 years old is a little disheartening. . . ,I have this same issue. The fact that this issue is 2 years old isn't very promising . . .

[493669]

@jensguenther,
It seems there is a space before your example.org in fieldname domain so try to remove these spaces using trim command like below-

|eval domain=trim(domain)

It will remove spaces /tabs and now search for your value , you will get as expected.

[jensguenther]

thanks 493669 ;).

We thought about that double / tripple checking it and, well, nope :/. We thought also there's some crazy hidden char, we even recommitted our javascript generating the events - nothing. It is like it is.

Is there any way to access the low level event data at the below data storage? Not _raw, we obviously tried that already. Maybe I've to wireshark it just before the indexer...

[marycordova]

There was another question like that here recently, I couldn't find that post, but essentially it had to do with "major" and "minor" key value delimiters and what Splunk calls search "terms".

It might not actually be a space as suggested in this answer, but there is something there that is mucking up what your expected behavior is.

[jensguenther]

thanks Mary!

I checked that major and minor but got stuck somehow.

However, we have this problem also on another field where we just put in plain simple single words with the exact same behavior.

I checked the source generating the events: they look solid, no additional chars, all's OK.

Honestly, I currently think of throwing the "fields" : {...} section away and just put everything into "event": "key=value"...