Splunk Search

Search and Alert produce different results

jlbark
Explorer

When I type this search in the normal Splunk search app, I get normal expected results:

"usb" | transaction host startswith="New USB device found" endswith="USB disconnect" | search NOT keyboard NOT mouse NOT host=xyz

However when I create an alert with the same search pattern, I constantly get results for the host xyz.

I am using the newest Splunk 5.0.3. Is this a bug or is there something happening behind the scenes I don't understand?

The settings for the alert are thus:
time range: real time
alert mode: once per search
condition: always
alert action: send email

Any hints?

0 Karma
1 Solution

jlbark
Explorer

So, through trial an error, I fixed the problem. There was nothing wrong with the search syntax.

I used the same query, on the search app, and then clicked create -> alert. And created the alert that way. Instead of the previous way, which was to go into the Manager -> Searches and Reports and click "New". For some weird reason I know get the correct alerts. I don't know why this made such a big difference, must be a bug.

View solution in original post

0 Karma

gordo32
Communicator

I've seen this same question come up a couple times, and my solution is different, so thought I'd share on a few of these in case others have the same problem I did.

The problem was that the query in my Alert was "search index=myindex sourcetype=waf httpstatus=400".

As soon as I removed the keyword "search" from the beginning of this query in the alert, it produced results consistent with manually issuing the search (index=myindex sourcetype=waf httpstatus=400). The rationale behind this (if I understood the support engineer correctly) is that the Alert passes the query to the CLI (i.e. /bin/splunk search ), so the CLI interprets the "search" item in my query as a searchable word, not a function.

0 Karma

jlbark
Explorer

So, through trial an error, I fixed the problem. There was nothing wrong with the search syntax.

I used the same query, on the search app, and then clicked create -> alert. And created the alert that way. Instead of the previous way, which was to go into the Manager -> Searches and Reports and click "New". For some weird reason I know get the correct alerts. I don't know why this made such a big difference, must be a bug.

0 Karma

jtrucks
Splunk Employee
Splunk Employee

Perhaps quote the server name and, also, try putting your NOT parameters in the beginning with the "usb" search like this:

NOT keyboard NOT mouse NOT host="xyz" "usb" | transaction host startswith="New USB device found" endswith="USB disconnect"

Unless you need to have the keyboard and mouse negations at the end to make the transaction work. Then do:

NOT host="xyz" "usb" | transaction host startswith="New USB device found" endswith="USB disconnect" | NOT keyboard NOT mouse

Another option is to use xyz* in case it's catching FQDN entries of some sort, which looks like:

"usb" | transaction host startswith="New USB device found" endswith="USB disconnect" | search NOT keyboard NOT mouse NOT host="xyz*"

Anything like these work?

--
Jesse Trucks
Minister of Magic
0 Karma

jlbark
Explorer

thanks for your reply jtrucks, but neither of those seemed to do the trick. This is a very weird problem, because the search query works fine in the search app, but the alert seems like it is having the problem.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...