key value begins with newline not searchable

[jeastman]

We have a key value pair where the value begins with a newline '\n'. It used to not have that newline and old searched did the following to access the field:
index=test MSGTXT="JOBNAME*"

This would return all events with MSGTXT value starting with JOBNAME. A new release of the data changed things to start with a newline "\n" character. So I naturally just tried all of the following scenarios thinking one of them would work:

index=test MSGTXT="\nJOBNAME*"
index=test MSGTXT="\\\\nJOBNAME*"

The following works:

index=test MSGTXT="*JOBNAME*" 

But the performance is so bad its not reasonable.

I set up a field extraction

TXT\":\"\\n(?<Job_Name>\w{1,8})

thinking that would fix the issue. So I try the following:

index=test Job_Name="JOBNAME" 

Which returns no results. So I try:

index=test | where Job_Name="JOBNAME" 

and that works.

So I believe that there is something about a newline character that is the first character of a key value and that is being searched for on initial search criteria.

Here is the raw text I am testing with:
{"DATETIME":"2015-07-15 11:13:05.46","SYSLOGSYSTEMNAME":"XXX2","JOBID":"","MSGNUM":"SE","MSGTXT":"\nJOBNAME ENDED AT N1 MAXCC=0000',LOGON,USER=(WWCXXX)"}

Any ideas on what might be wrong or if this is potentially a bug?

[martin_mueller]

As a quick bandaid, you can set this in fields.conf and hit http://your_host:8000/en-US/debug/refresh?entity=admin/fields to load it:

[MSGTXT]
INDEXED_VALUE=false

Then search using literal newlines (shift+return) like this:

index=bar sourcetype=foo MSGTXT="
JOBNAME*" | ...

As for the actual issue, it seems \n at the beginning of a token is confusing Splunk. I'm not 100% certain it's a bug, but it's certainly worth opening a support case to have it looked at.
Obviously, the best solution would be to not start field values with newlines in the first place 🙂
You could consider using props.conf SEDCMD-foo or transforms.conf to replace this at index time if you can't change it at the source.

[martin_mueller]

I would strongly recommend not changing your data to influence the event listing in Splunk. Instead, make your queries smarter or use custom event renderers to do what you need.

The INDEXED_VALUE=false disables Splunk's default behaviour of translating "field=value" into "look for 'value' token in lexicon, retrieve events, apply field extraction, filter for field=value" and instead makes Splunk do "retrieve all events, apply field extraction, filter for field=value". Apparently the token handling is confused by your newline, or your newline is confused by the token handling... the search filter itself works as you already discovered by piping to where.

To open a support case, go to http://www.splunk.com/ and hit Support & Services -> Support Portal. Make sure you log in using an account that is registered as a support contact under your support entitlement.

[jeastman]

Thank you for the response.

The reason that the '\n' is in there is because of formatting columns of data in the event for easy reading as an event. My example does not show this but a lot of the MSGTXT messages have multiple lines with many '\n'. Without the first newline in the value, that first line is offset by the length of the key (MSGTXT:). Inserting the newline character makes the first line of text show up with the remaining.

I realize a quick table of MSGTXT would make this readable but its an extra step that the newline character fixed. The idea is to not have to modify .conf files to make this visible and handle it in the JSON. But perhaps, that is not possible at this time.

What is it that the INDEX_VALUE=false provides that allows it to accept a newline (shift+return)?

How does one open a support case?