I have a little confusion about how time stamp actually works. I want to do a very simple query to combine the result for two searches into one single table. I am using a simple OR to achieve that. A simplified version of my query will be:
(index=firewall1 dst=8.8.8.8) OR (index=firewall2 starttime="03/23/2015:12:13:45" endtime="03/23/2015:13:13:55" dst=8.8.8.8)
I know this is a very inefficient search and I am supposed to use subsearches. However, I find that the Splunk server I am using does not have enough memory for the millions of results returned by a subsearch in this case.
The tricky part is that, the time modifier should only be applied to the second index. When running the search, I find the results for the first search are missing. I know it's missing because I ran the first search alone and there are a lot of results, yet, nothing shows up for the search after the OR clause. If I remove the time modifier in the second part, I get all the expected results.
What is creepy is that this does not happen for all the IPs, only for some of the IP addresses, I got all the results I want and for some I got nothing for the first search. The time modifier cannot be global because I can sometimes see results from the first search that does not fall in the time range. Same thing happened with earliest/latest keyword and I double checked I don't have any format issue with IPs.
The version of my Splunk is 5.0. Can anyone help me explain the logic behind that? Or help me come up with another solution without using any subsearches?
... View more