Splunk Search

Why is my search pulling the wrong events when selecting an appended column on a graph?

Spiere
Path Finder

Hey guys,

I recently created a graph using the search:

sourcetype=testing PhpFatal="PHP Fatal error" 
| stats count by PhpFatal
| append [ search sourcetype=testing DrupalPHPFatal="Error: PHP FATAL Error" 
    | stats count as DrupalPHPFatal ]

I ran into an issue with the appended graph that I am unsure how to fix. Firstly, it does not actually show the events in different colors unless I make the appended graph stats count as DrupalPHPFatal instead of stats count by DrupalPHPFatal. Although it pulls up the correct number of events using both syntaxes, when I select the DrupalPHPFatal section, it does not pull up the right events. It pulls up the events for the first search, which in this case was for PhpFatal, not DrupalPHPFatal.

Is there something wrong with the appending syntax, or do I have to have another specific command in there for Splunk to pull up the right events when I select the events sections on the bar graph?

0 Karma
1 Solution

lguinn2
Legend

I believe that there must not be a field named DrupalPHPFatal in your testing sourcetype. Also, you could make the results easier to understand this way

sourcetype=testing PhpFatal="PHP Fatal error" 
 | stats count as "Count of PHP Fatal"
 | append [ search sourcetype=testing DrupalPhpFatal="Error: PHP FATAL Error" 
     | stats count as "Count of Drupal PHP Fatal" ]

If the data set is very large and the two conditions do not overlap, you could optimize the search like this:

sourcetype=testing PhpFatal="PHP Fatal error" OR DrupalPhpFatal="Error: PHP FATAL Error" 
| eval ErrorType = if(PhpFatal=="PHP Fatal error",PhpFatal,DrupalPhpFatal)
| stats count by ErrorType

Note that you need to figure out the right spelling for the field name; I just guessed at DrupalPhpFatal

View solution in original post

lguinn2
Legend

Okay. First, field names are case-sensitive, DrupalPHPFatal and DrupalPhpFatal are not the same thing. Second, when you look at your data with a simple search:

 sourcetype=testing

What fields appear on the left panel? Does either PhpFatal or DrupalPhpFatal or some variant appear? And if not, if you click the link for "all fields," do either of these fields appear in the list? If yes, the fields exist and at least one of the searches in the answers should work, if the field names are spelled correctly. Or, you could use the search below, removing lines 2 and/or 3, and putting in the right field names.

My guess is that at least one of the fields are missing. And while you may want to fix that by creating a permanent field, you can create fields on-the-fly. The following search creates both fields and then reports:

sourcetype=testing "php fatal error"
| rex "\]\s+(?<PhpFatal>PHP Fatal error)\:"
| rex "\|(?<DrupalFatal>Error: PHP FATAL Error)"
| stats count(PhpFatal) as "PHP Fatal Errors" count(DrupalFatal) as "Drupal Fatal Errors"

Note that the base search is case insensitive so you can simply search for the phrase "php fatal error".
The stats command simply counts the number of events that where the fields are present.

Spiere
Path Finder

When I run sourcetype=testing, both the fields DrupalPHPFatal and PhpFatal appear. When I select them, they pull up the correct events/correct number of events.

The first query I run as sourcetype=testing PhpFatal="PHP Fatal error" | stats count as "Count of PHP Fatal" | append [ search sourcetype=testing DrupalPHPFatal="Error: PHP FATAL Error" | stats count as "Count of Drupal PHP Fatal" ]

It returns the correct number of events when I split the search and run each query on its own, but when combined using append it will return DrupalPHPFatal, but not PhpFatal(it sets the yaxis to the number of PhpFatal events). This isnt super important since one of those queries works, I was just curious as to why it works with the second one query, but not the first.

Thanks you for all the help. I appreciate it.

0 Karma

lguinn2
Legend

The reason for your problem is that the data is not in a format that Splunk can chart properly. If you want to see the data in a chart, try this

 sourcetype=testing  PhpFatal="PHP Fatal error" OR DrupalPHPFatal="Error: PHP FATAL Error"
 | eval ErrorType=if(isnotnull(DrupalPHPFatal),"DrupalPHPFatal","PhpFatal")
 | stats count by ErrorType

If you run this versus one of the other searches, take a look at the statistics tab and you can see the difference in the results...

0 Karma

Spiere
Path Finder

What about the data makes it unchartable? Is it just the way the fields are being extracted, or is it the way that the logs are written? Seems like the big difference is the addition of that isnotnull argument, so the isssue was that one of the fields would come up as null and throw off the charting?

0 Karma

lguinn2
Legend

Okay then - your search could simply be

sourcetype=testing  PhpFatal="PHP Fatal error" OR DrupalPHPFatal="Error: PHP FATAL Error"
| stats count(PhpFatal) as "PHP Fatal Errors" count(DrupalFatal) as "Drupal Fatal Errors"

It looks like we went around the world to get back to something simple 🙂

0 Karma

lguinn2
Legend

I believe that there must not be a field named DrupalPHPFatal in your testing sourcetype. Also, you could make the results easier to understand this way

sourcetype=testing PhpFatal="PHP Fatal error" 
 | stats count as "Count of PHP Fatal"
 | append [ search sourcetype=testing DrupalPhpFatal="Error: PHP FATAL Error" 
     | stats count as "Count of Drupal PHP Fatal" ]

If the data set is very large and the two conditions do not overlap, you could optimize the search like this:

sourcetype=testing PhpFatal="PHP Fatal error" OR DrupalPhpFatal="Error: PHP FATAL Error" 
| eval ErrorType = if(PhpFatal=="PHP Fatal error",PhpFatal,DrupalPhpFatal)
| stats count by ErrorType

Note that you need to figure out the right spelling for the field name; I just guessed at DrupalPhpFatal

Spiere
Path Finder

If that is the case, how can I have it filter the events properly? Without a field specifying what a drupal error looks like in the sourcetype, how will it know to filter the events from those errors correctly?

0 Karma

lguinn2
Legend

Fields are not made up on the spot! They aren't created by the search.

You have to extract the data into the fields before you can use them. You can read up on Splunk field extraction.

0 Karma

Spiere
Path Finder

Unfortunately, both the Drupal errors and the PHP errors are inside of the same file, which is already being mapped to the testing sourcetype. Splunk does not allow me to have the same file input for another sourcetype when it is already being used as the input for the previous one. What is the next step now, or is this not possible based on the circumstances?

0 Karma

Spiere
Path Finder

So looking at your original answer again, the first query written exhibits the same behavior as mine, and only displays one field after I change the field name to DrupalPHPFatal. The second query you entered does work properly after I change the field name. Why is that the case?

PhpFatal - [Mon Jan 11 09:52:35 2016] PHP Fatal error: Call to undefined method i18n_object_wrapper::strings_update() in /.../sites/all/modules/i18n/i18n_string/i18n_string.module on line 851

DrupalFatal = Jan 12 16:50:11 someServerName drupal: https://edit.fletc.gov|1452635411|php|155.555.555.55|https://edit.fletc.gov/node/2193/draft|https://... PHP FATAL Error - Function name must be a string in (line 40 of /var/www/html/wwwroot/fletc_gov_build/Rel11/profiles/dhs_wcm/modules/contrib/workbench_moderation/workbench_moderation.node.inc).

0 Karma

lguinn2
Legend

It would really, really help if you showed a couple of same events - some Drupal and some PHP.

It is not a problem that both of these events are in the same file. I think we are just having some communication problems because I don't really understand the data.

In fact, I think this may work:

sourcetype=testing "PHP Fatal error"
| eval Drupal_error = if(match(_raw,"Error: PHP FATAL Error",1,0)
| stats count as PHP_fatal_error sum(Drupal_error) as Drupal_PHP_fatal_error

The first line simply retrieves all the testing events that have the string.
The second line sets the field Drupal_error to 1 for each event that matches the string.
The third line counts all the events and assigns the value to PHP_fatal_error (because they all match the PHP pattern). It also sums up all the ones that were assigned to events matching the Drupal pattern and names that value Drupal_PHP_fatal_error.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...