Splunk Search

Why am I getting different results for essentially the same search?

tdiestel
Path Finder

Hi All;

Here's my issue. I'm trying to search data where a single event appears as below. When I use the search:

index=*mobile Action_Name=Page_View user_id="1159314c-25c6-11e4-aa0c-069a72358463" 

I am able to see my data, but when I do the search

index=*mobile Action_Name=Page_View | search user_id="1159314c-25c6-11e4-aa0c-069a72358463" 

I do not see my data.

Has anyone experienced this before? This has a lot of impact on my search as generally my search would be able to aggregate the data by OS counting the distinct user_id's, and this problem causes a few user_id's not to be counted.

2015-10-18T08:48:32-07:00
Category="Lifecycle"
Action_Name="Page_View"
Current_Page="ParkingListPanel"
vn_app_version="2.3.63"
device_ip="192.168.137.49"
OS="Android"
OS_version="2.3.4"
user_id="1159314c-25c6-11e4-aa0c-069a72358463"
location=""
connection_type="wifi"
battery_level=0.890000
page_name="ParkingDetailPanel"
previous_page_name="ParkingListPanel"
Tags (2)

martin_mueller
SplunkTrust
SplunkTrust

Running search | search is a terrible pattern, including all your filters in the initial search gives you not only accurate results but also great performance.

If you have statistics over all data you want to add to detailed searches, consider storing them in a lookup and adding this to your detailed searches as needed. Schedule a search to update the lookup, for example once a day depending on your data. Then you won't have to go over your entire data set again and again, but still get information from your entire data set at ludicrous speed.

tdiestel
Path Finder

Understand this is a terrible idea, but would like to know what is the reason why this would produce inaccurate data. The actual search that I'm running, which this problem is occurring is this:

 index=*mobile Action_Name=Page_View |stats count by user_id

But when i do this, the user_id "1159314c-25c6-11e4-aa0c-069a72358463" is missing from the results. Then when i run

 index=*mobile Action_Name=Page_View user_id="1159314c-25c6-11e4-aa0c-069a72358463" |stats count by user_id

I DO receive the result. That's the real issue.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Well that's an entirely different issue... Do pastebin your job inspector and search.log somewhere so we can have a look.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

That being said, I tried to reproduce the issue with data everyone has in their Splunk. Compare these two searches:

index=_audit TERM(action=splunkStarting)
index=_audit | search TERM(action=splunkStarting)

If the second index=_audit only yielded 10k events to the following search I should see no events at all, but I see events a year back - for me on Enterprise 6.3.0 standalone at least.

0 Karma

ltrand
Contributor

Base setting to return only 10k results could be limiting you. So the primary search brings back 10k results, stops, then then subsearches that and finds no match.

Meanwhile in the first search attempt, you are bringing out only items with the qualifying uid, thus staying under the 10k limit (limit on search results, not events searched).

woodcock
Esteemed Legend

I do not see any way that a base search can be limited at all. Please educate us on this limit!

0 Karma

tdiestel
Path Finder

I have had more than 10K results returned before, but i tried narrowing down the time range i was looking at and the events started to show up, which is GREAT.

But, (and this is an odd 'but'), when i set the date range back to what i originally had, the results were now showing the correct number. It was as if splunk finally recognized these events when i narrowed in, so when i now have larger ranges it continues to recognize them.

Any thoughts on this?

0 Karma

ltrand
Contributor

Great question, I'm not sure. I'd have to look at a similar dataset & search conditions to see what's happening.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...