Search time indexing failing with certain field name

[anderssv]

We are doing search time indexing, and the following stanza is added to props.conf on the search heads:

[log4j]
EXTRACT-loglevel = \s(?P<loglevel>(FATAL|ERROR|WARN|INFO|DEBUG|TRACE))\s

This was working, but stopped. No changes to config, and I suspect the only thing that changed was the indexed data.

Is there any chance that smart field extraction or something like that is interfering with our extraction? I can see some log statements containing:

....
### loglevel=warn ...
....

Searches for loglevel gives zero results:

loglevel=ERROR

But... If we change the name of the extracted field it starts working... (notice the j in front of loglevel)...

[log4j]
EXTRACT-loglevel = \s(?P<jloglevel>(FATAL|ERROR|WARN|INFO|DEBUG|TRACE))\s

Any clue as to where we should start digging? We can of course live with a different name, but seems a bit unnecessary too. 🙂

[Runals]

For S&Gs I would change your extract statement to something like the following. Changing the field name shouldn't matter but IRC you want to make the EXTRACT name unique. As Luke mentioned since this is a search time extraction you can make changes to the props and then run your search over historical data.

EXTRACT-log4j_loglevel = \s(?P<loglevel>(FATAL|ERROR|WARN|INFO|DEBUG|TRACE))\s

The telling thing really is does an inline rex work as expected?

sourcetype=log4j | rex "\s(?P<loglevel>(FATAL|ERROR|WARN|INFO|DEBUG|TRACE))\s" | stats count by loglevel

[Runals]

Very strange. By chance did you restart the search head after making the change? You don't have to really. If you are running probably v5.0.4 or so Splunk will re-read in search time extractions automagically. Prior to about that version you can run "| extract reload=t". It has been a while since I've used that. Want to say you just run it as is with the leading pipe. The other thing you should do when making changes like this is reload the app (probably not right term). I usually just copy my query, click the Splunk logo and paste the search back in the bar.

[anderssv]

Test query works as expected:

sourcetype=log4j | rex "\s(?P<loglevel>(FATAL|ERROR|WARN|INFO|DEBUG|TRACE))\s" | stats count by loglevel

I run a more detailed query to filter that works:

sourcetype=log4j | rex "\s(?P<loglevel>(FATAL|ERROR|WARN|INFO|DEBUG|TRACE))\s" | search loglevel="ERROR"

Then I move stuff to props.conf...

The count query works:

sourcetype=log4j | stats count by loglevel

The detailed query doesn't???

sourcetype=log4j loglevel=INFO

INFO did occur in the table for stats, and is present in the timeframe...

Have the faintest idea what's going on here?

[anderssv]

Sorry, thought the proxy was blocking stuff. But trying to post an answer from a different computer/network doesn't work either. I'll just split for the comments here.

[anderssv]

Thanks. So I updated to unique names. But it doesn't seem to help. 😞 Updated answer below because of length limitation on comments.

[anderssv]

Ah, sorry about that. Extraction of course.

I did searches in timeframes that did not have the "### loglevel=warn" data, but it still didn't work...

[lukejadamec]

If it is a problem with the data, then running the original props.conf against a timeframe where it was working should still work.
Also, you're talking about search time extraction, not indexing.