I have been modifying searches to accommodate Windows data in the CIS Top 20 Critical Controls app. The following search does not return results when invoked by the visualization on the Dashboard or on the first run when opening the Search window. If I run it again in the Search window, it runs and delivers the correct result.
Control #1 - Inventory of Unauthorized Devices - Count
tag=dhcp signature=DHCPREQUEST OR signature="A lease was renewed by a client"
| lookup approved_device_inventory clientip AS dest_ip
| eval approval_status = if(is_approved==1,"1","0")
| where approval_status = 0
| dedup dest_ip
| stats count by dest_ip
| stats sum(count)
| rename sum(count) AS Unauthorized_Devices
The Search works fine and populates the visualization on the installation I created on a Sandbox instance. The only difference on my Prod install is that I have added the OR clause: OR signature="A lease was renewed by a client"
Has anyone else encountered this "second run works" issue?
Turns out the issue was the assigned "Owner" of the saved search. For this search the Owner was set to "Nobody".
When the search is run for the first time in either the Visualization or the Search window, Splunk uses the "Splunk_System_User" in the "dispatchRunner - search context" if the owner is not set to Admin. Then when the search is initiated in the Search window, it uses the current User account (Admin in this case) for the search context.
We navigated to the /opt/splunk/etc/apps/CIS Top 20 Critical Controls/metadata/local.meta file and changed the owner for this search to Admin.
Bingo! It works as expected now 🙂
We examined the search.log files for differences between failed first run and successful second run... No apparent errors or glaring differences 😞
I have run into the issue before and it is normally when my search is very large and times out or runs into a memory issue. when i run it again, it generally finishes because it has cached the previous data. i'm not sure if that's what's happening here, when you run it, does it come back with any errors in the Job dropdown?
Search size doesn't appear to be the issue... It's looking for the last 60 minutes against a sourcetype with only 264K events. No errors in the "Inspect Job" panel after first run - just "did not return any data".
One other clue... the Search completes with results only if run in the edit Search window accessed through the visualization Search icon. If I try to Refresh the visualization, the search does not return results.