Why Raw events table (populated using tokens) displays raw events for some events but not the others?

[aina_sloan]

Hi,

Would really appreciate if someone could help me with this issue:

1. I have a Table that displays Host and "Error Message" and Count
   The "Error Message" field is a shortened version of an original Log Message. The search is as follows:

   (index=WebSphere OR index=Pega) (log_level=Error OR wasLogLevel=E OR wasLogLevel=R)
   |eval logmessage_short=subst(logmessage,1, 146)
   |stats count by host, logmessage_short
   | rename count as "Number of Occurrences", host as Host, logmessage_short as "Error Message"
   | sort -"Number of Occurrences"

   1. I have then have specified drilldown:

 cell
    <drilldown>
        <set token="Error_Messages">$row.Error Messages$</set>

1. When a user clicks on an error message, a new dashboard panel will open showing the Raw errors

   (index=WebSphere OR index=Pega) (log_level=Error OR wasLogLevel=E OR wasLogLevel=R)
   |search ("$Error_Messages$" OR $Error_Messages|s$

The result

User clicks on the Error Message and in most cases the Raw Events list shows below. However, in some cases it doesn't. Although when I open the Raw events in the search it then displays raw events for the selected field.

Here is an example:

This Error Message works :

-Exception com.pega.pegarules.pub.services.ConnectorException: com.pega.pegarules.pub.PRRuntimeException: No such Directory or Folder: /data/PEGA

This Error Message doesn't work:

-PEGA-OUTBOUND_PP_007: Transfer Error during Omni Outbound Response: java.lang.Throwable at com.pegarules.generated.activity.ra_action_generate

I have no idea why the list of Raw Events wouldn't display for this error message since the search is working. Anyone has experienced anything like this before?

[Richfez · Accepted solution]

I'd guess that messages that got trimmed will only show up if the trimming landed on a normal separator character, like a space.

If you temporarily remove the truncation I believe they might all work. This would be the fastest/easiest way to reverse just that for a test:

(index=WebSphere OR index=Pega) (log_level=Error OR wasLogLevel=E OR wasLogLevel=R)
|eval logmessage_short=logmessage
|stats count by host, logmessage_short
| rename count as "Number of Occurrences", host as Host, logmessage_short as "Error Message"
| sort -"Number of Occurrences"

If that works OK, then this is your problem.

I'll give you two possible solutions (and here's where there may be lots of suggestions on what to do)

One: use rex to remove the last space-delimited piece of the string. So, substr can trim it to 146 characters then rex can take off the trailing word so that it ends on a logical boundary to split on. That would look something like

(index=WebSphere OR index=Pega) (log_level=Error OR wasLogLevel=E OR wasLogLevel=R)
|eval logmessage_short=subst(logmessage,1, 146)
| rex field=logmessage_short "(?<logmessage_short>.*)[ /:][^ :/]+$"
|stats count by host, logmessage_short
| rename count as "Number of Occurrences", host as Host, logmessage_short as "Error Message"
| sort -"Number of Occurrences"

Another method might be to adjust the drill-down in the dashboard to add a wildcard to the end, so your drill down looks like this

(index=WebSphere OR index=Pega) (log_level=Error OR wasLogLevel=E OR wasLogLevel=R)
|search ("$Error_Messages*$" OR $Error_Messages*|s$

I can't remember off hand if those need quoting or escaping, so if it doesn't work go look that up in the advanced dashboard drilldown topics.

Give one of those a try and let us know how it went! Happy splunking!
-Rich

[aina_sloan]

Hi Rich,

Thank you so much for your help. By adding

(?<logmessage_short>.*)[ /:][^ :/]+$"

it worked but the issue was that it trimmed some of the "Error Messages" to, for example just User ID. This does not say a lot about what has happened and doesn't provide much value to the end user. So I looked at how it worked and it seemed to pick up only second line on the short message. So I made a small change to the rex you gave me:

(?<logmessage_short>(.*\A)(.*)[ /:][^ :/]+$"

The (.*\A) ensured that I have the first line. It seems to work now.

Thank you very much, I was really struggling with this.

[gcusello]

Hi aina_sloan,
I think that in the secondary search you have to use brachets also for the second tokens.
Why do you used two times the same token?
In addition I see that there's a missed parenthesis but probably it's a copy error.
In addition, you don't need to use | search command, you can write:

(index=WebSphere OR index=Pega) (log_level=Error OR wasLogLevel=E OR wasLogLevel=R)
"$Error_Messages$"

Bye.
Giuseppe

[aina_sloan]

Hi Giuseppe,

Thank you for contributing to my question. It seems that the issue was related to the way logmessage_short field value was defined.

Thanks,

Aina