wanted to reach out for help regarding an issue we have been experiencing on one of our customers.  We build an app that exports events from a standalone customer using the Splunk Enterprise instance.  We have that box gather the logs and hold them until it can be exported out of the box manually. We used the savedseaches.conf file to schedule a search query script (export.py) to pull events.  The problem is that on this particular customer he is only getting like 11 minutes worth of logs, but the file is scheduled to pull all index events from lets say 3:30pm-4:30pm, but the events start loading only from 4:19pm-4:30pm.   It does this across all times consistently. example, missing the first like 49 minutes of events:  4:19pm-430pm 5:19pm-5:30pm 6:19pm-6:30pm  We have a export.py script that goes out and gathers all index=* events according to the cron specified. savedsearches.conf cron_schedule = 30 */1 * * *  enablesched = 1 dispatch.ttl = 1800 allow_skew = 10m search = | export disable = 0  To compensate for lags, we build into the |export.py script to pull the events 1 hour prior so like.  This is part of the script dealing with the specific search. now = str(time.time()-3600).split(".")[0] query = "search index=* earliest=" + last_scan + "  lastest=" + now + "  once script is done, it creates a timestamp in a file of the now in epoch time, which is used for the next schedule time. Any help would be appreciated ... View more

We have standalone environment and are getting error "the percentage of non-high priority searches skipped (61%) over the last 24 hours is very high and exceeded the red threshold (20%) on this splunk instance."  The environment: Customer has standalone where we created an app with a savedsearch script that pulls all indexed events every 1 hour and bundles them into a  .json file, customer then compresses it into a .gz file for transfer into our production environment.   What we are seeing is this skipped searches message and when we check the specific job, we see that every time it runs there are 2 things that come up as jobs, the export app started by python calling the script and then the actual search job activity with our SPL search, both jobs are 1 second apart and stays in the jobs page for 10 minutes each, customer states that it takes ~2.5 minutes for this job to complete.   The python script seems to stay longer for some reason, even after its job  Not sure how to proceed, since we had it scheduled every 4 hours and it was doing the same thing, so we lowered it to 1 hour, no difference. Our search looks at the last completed .json file epoch time and current epoch time to grab those events in that range, so not sure if that message is like a false positive by the way we are catching events (timestamps).  How can i remove the skipped searches error message.  Tips??       ... View more

I'ved been having issues with getting "CPU utilization" to up on the Windows infrastructure dashboard.  I found that when i click on the Windows entities and move onto a single windows machine itself i see all pertinent data for CPU utilization, but for some reason i cannot get it to show on the dashboards as a graph.    I have it set as the key indicator on the overview dashboard, it shows nothing, whereas the other key values show data (memory, network, and disk utilization). key info - 4 windows hosts (all same issue, shows N/A for CPU utilization) - i have manipulated the search job schedule  - entity discovery search is enabled, and manipulated the savedsearches.conf, gave it more time. - set correct Index in macros in SA-ITOA - checked on the _meta field in the windows stanza and the entity_type::windows_host is all there.  - perfmon::CPU is all there. So its weird why i am getting N/A for cpu utilization on the windows entity overview page and the infrastructure overview dashboard.  Any ideas would be greatly appreciated.    ... View more