Why am I seeing inconsistent results when specifying fields from an index time extraction in my searches?

[myandow]

We have an index time extraction that pulls out the facility and severity from syslog. This extraction occurs prior to another extraction that removes the values from _raw before indexing.

What I am seeing is that when I search using one of those two fields I get very odd, inconsistent search results.

For instance, for a set period of time (1 minute):

index=test "test search string11223344" | search facility=daemon

yields 27 results.

while the search:

index=test "test search string11223344" facility=daemon

yields 0 results.

And the search:

index=test "test search string11223344" facility=*daemon

yields 27 results.

It is almost like there is an invisible character at the beginning of the fields, but if there was I wouldn't expect the first example to work as it does.

[somesoni2 · Accepted solution]

Can you try putting the fields.conf on your Search Head, as suggested by below link?

http://docs.splunk.com/Documentation/Splunk/6.4.3/Data/Configureindex-timefieldextraction#Where_to_p...

[myandow]

This lead me down the right path, although it wasn't adding them to the fields.conf on the search heads that fixed the problem, it was adding them to the fields.conf on the indexers. We are running 6.2 for reference.

[s2_splunk]

What results do you get if you searched for index=test "test search string11223344" facility::daemon?
Also, try to specify facility=TERM(daemon) and see what you get.

If you want to see what terms are actually indexed, you can use the walklex command on the tsidx file associated with a bucket that contains your indexed data.

[myandow]

The results when I use facility::daemon match the results of facility=*daemon, in that I get 27 results returned.

The results when I use facility=TERM(daemon) match the results of facility=daemonin that I get 0 results returned.

Is there a reason that the format of key::value works, but key=value does not work as expected in the searches?

[s2_splunk]

In your case it should not matter, I think. It is possible that the indexed field is not extracted as you expect, so looking at the contents of an index file using walklex maybe able to shed some light.
Can you provide a sample message along without configuration settings for the index-time extraction?

[myandow]

Using the walklex command as suggested, and searching for facility* seems to return expected results:

my needle: facility*
836948 19333 facility::auth
836949 14587 facility::authpriv
836950 5172 facility::cron
836951 1264740 facility::daemon
836952 6634 facility::kern
836953 173438 facility::local0
836954 54 facility::local1
836955 394973 facility::local4
836956 2306 facility::local6
836957 242 facility::local7
836958 526506 facility::mail
836959 84882 facility::news
836960 1344 facility::syslog
836961 173157 facility::user

Configuration settings on the indexers

props.conf

[unix-syslog]
SHOULD_LINEMERGE = False
TRANSFORMS-strip-usyslog = usyslog-priority-facility, usyslog-host, usyslog-header-stripper-ts-prio-host

transforms.conf

[usyslog-priority-facility]
REGEX = ^<(\w+)\.(\w+)> [A-Z][a-z]+\s+\d+\s+\d+:\d+:\d+\s[^\s]*\s.*$
FORMAT = facility::$1 severity::$2
WRITE_META = true

[usyslog-host]
REGEX = ^<\w+\.\w+> [A-Z][a-z]+\s+\d+\s+\d+:\d+:\d+\s([^\s]*)\s.*$
FORMAT = host::$1
DEST_KEY = MetaData:Host

[usyslog-header-stripper-ts-prio-host]
REGEX = ^<\w+\.\w+> [A-Z][a-z]+\s+\d+\s+\d+:\d+:\d+\s[^\s]*\s(.*)$
FORMAT = $1
DEST_KEY = _raw

Sample message (with ips and hostnames redacted)

<daemon.info> Sep  7 13:37:19 XXXXXX.XXXXXXX.XXXXXX.XXXXX.com named[1147]: zone XXXXXX.XXXXXX.XXXXXX.com/IN/default: refused notify from non-master: XXX.XX.XXX.XXX#43538