- Splunk Answers
- :
- Splunk Administration
- :
- Monitoring Splunk
- :
- Issues with Saved searches not running
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Issues with Saved searches not running
I have multiple saved searches running at different times and strangely none of them are running in the weekend . When I checked the scheduler logs I dont see a single log for the saved searches on weekends bit there are logs before and after.
I am using the below search to see if searches are running
index=_internal sourcetype=scheduler savedsearch_name=*
I checked the internal logs of the search head and I see many logs with error as below
03-30-2019 11:16:43.983 -0400 WARN DispatchSearchMetadata - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/rsa_scheduler__nobody_c3BsdW5rX2FwcF9kYl9jb25uZWN0__RMD54b8a4bb1225327ae_at_1528935000_10807_122777D6-EEF1-4A1A-BEB3-DAD58F68D73C/metadata.csv
When I ran the search manually I see results .Can anyone help me in this
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You are probably using either LDAP or SSO, in other words, non-local splunk accounts, and your Search Head has been rebooted. What happens in this case, is that splunk cannot validate the user because the user has not logged in since the Search Head reboot. For LDAP, you only need any user to login and all user's scheduled searches will run, but for SSO, you need each user to login before any of his scheduled searches will run. This is why many people deliberately orphan searches to be owned by nobody because that (non-)user always (sort of) exists enough for saved searches to run.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We are using SAML authentication and most of the searches are running under user nobody
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Is this one?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
No.The user nobody is not using SAML. but there are multiple searches running on my name those also did not run. The strange thing is there are not logs in sourcetype=scheduler in the weekends (whole two days) .Last saturday I ran a saved search manually ,then all of sudden the other searches started running and from then own the logs in scheduler started to show
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What is your schedule definition? it is probably something like 0 0 * * 1-5 and you need it to be 0 0 * * *.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello @woodcock
Yes the searches are in the correct format .Strangely the scheduler does not run in weekends
Detecting Remote Code Executions With the Splunk Threat Research Team
Observability | Use Synthetic Monitoring for Website Metadata Verification
More Ways To Control Your Costs With Archived Metrics | Register for Tech Talk
