Splunk Search

No results on first run of Search - correct results on second run

ferdbiffle
Explorer

I have been modifying searches to accommodate Windows data in the CIS Top 20 Critical Controls app. The following search does not return results when invoked by the visualization on the Dashboard or on the first run when opening the Search window. If I run it again in the Search window, it runs and delivers the correct result.


Control #1 - Inventory of Unauthorized Devices - Count


tag=dhcp signature=DHCPREQUEST OR signature="A lease was renewed by a client"
| lookup approved_device_inventory clientip AS dest_ip
| eval approval_status = if(is_approved==1,"1","0")
| where approval_status = 0
| dedup dest_ip
| stats count by dest_ip
| stats sum(count)
| rename sum(count) AS Unauthorized_Devices


The Search works fine and populates the visualization on the installation I created on a Sandbox instance. The only difference on my Prod install is that I have added the OR clause: OR signature="A lease was renewed by a client"

Has anyone else encountered this "second run works" issue?

Tags (2)
0 Karma

ferdbiffle
Explorer

Turns out the issue was the assigned "Owner" of the saved search. For this search the Owner was set to "Nobody".

When the search is run for the first time in either the Visualization or the Search window, Splunk uses the "Splunk_System_User" in the "dispatchRunner - search context" if the owner is not set to Admin. Then when the search is initiated in the Search window, it uses the current User account (Admin in this case) for the search context.

We navigated to the /opt/splunk/etc/apps/CIS Top 20 Critical Controls/metadata/local.meta file and changed the owner for this search to Admin.

Bingo! It works as expected now 🙂

ferdbiffle
Explorer

We examined the search.log files for differences between failed first run and successful second run... No apparent errors or glaring differences 😞

0 Karma

cmerriman
Super Champion

I have run into the issue before and it is normally when my search is very large and times out or runs into a memory issue. when i run it again, it generally finishes because it has cached the previous data. i'm not sure if that's what's happening here, when you run it, does it come back with any errors in the Job dropdown?

0 Karma

ferdbiffle
Explorer

Search size doesn't appear to be the issue... It's looking for the last 60 minutes against a sourcetype with only 264K events. No errors in the "Inspect Job" panel after first run - just "did not return any data".

One other clue... the Search completes with results only if run in the edit Search window accessed through the visualization Search icon. If I try to Refresh the visualization, the search does not return results.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...