Splunk Search

Why am I not getting all expected results when using time modifiers in my search in Splunk 5.0?

shengcow
Explorer

I have a little confusion about how time stamp actually works. I want to do a very simple query to combine the result for two searches into one single table. I am using a simple OR to achieve that. A simplified version of my query will be:

(index=firewall1 dst=8.8.8.8) OR (index=firewall2 starttime="03/23/2015:12:13:45" endtime="03/23/2015:13:13:55" dst=8.8.8.8)

I know this is a very inefficient search and I am supposed to use subsearches. However, I find that the Splunk server I am using does not have enough memory for the millions of results returned by a subsearch in this case.

The tricky part is that, the time modifier should only be applied to the second index. When running the search, I find the results for the first search are missing. I know it's missing because I ran the first search alone and there are a lot of results, yet, nothing shows up for the search after the OR clause. If I remove the time modifier in the second part, I get all the expected results.

What is creepy is that this does not happen for all the IPs, only for some of the IP addresses, I got all the results I want and for some I got nothing for the first search. The time modifier cannot be global because I can sometimes see results from the first search that does not fall in the time range. Same thing happened with earliest/latest keyword and I double checked I don't have any format issue with IPs.

The version of my Splunk is 5.0. Can anyone help me explain the logic behind that? Or help me come up with another solution without using any subsearches?

0 Karma

chimell
Motivator

Hi shengcow
Use simply the following search code :

|set union[ search index=firewall1 dst=8.8.8.8 ][search index=firewall2 starttime="03/23/2015:12:13:45" endtime="03/23/2015:13:13:55" dst=8.8.8.8]

shengcow
Explorer

set union/intersect will not return the result for sub-searches if they have millions of results(over the sub-search limitation), it will just return empty. And there is usually no notification on that. It is not working in my case here since there are not enough memory on Splunk server to increase the sub-search limitation.

0 Karma

stephane_cyrill
Builder

Hi I'm not sur i understand you very well.but this is a way to do what you want without using OR.

1-
|multisearch [search index=firewall1
dst=8.8.8.8 ]
[search index=firewall2
starttime="03/23/2015:12
:13:45"
endtime="03/23/2015:13:
13:55" dst=8.8.8.8]

2-you can pipe all an continue your search with: table,chart,..........

0 Karma

stephane_cyrill
Builder

You can also control the subsearch with
settings in limits.conf for the runtime and
maximum number of results returned:
[subsearch]
maxout =
Maximum number of results to return
from a subsearch.
This value cannot be greater than or
equal to 10500.
Defaults to 10000.
maxtime =
Maximum number of seconds to run a
subsearch before finalizing
Defaults to 60.
ttl =
Time to cache a given subsearch's
results, in seconds.
Do not set this below 120 seconds.
Defaults to 300.

docs.splunk.com/Documentation/Splunk/6.2.2/Search/Aboutsubsearches

0 Karma

shengcow
Explorer

Let me explain more. Our team do not control the Splunk server and because of the limited memory, we are not allowed to change the subsearch configuration. It's a shared server and I should not consume all the resources. It will be great if we can focus on the problem itself. I am sorry for my current situation and limitations.

0 Karma

shengcow
Explorer

Thanks for the quick answer. As I metioned in my question, I can not use any subsearches because of the massive number results for both queries and the limited memory my Splunk server. And it will be nice if you can explain my observation first.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...