Thank you Nick, Yes, that is one of the files that I had verified that also had correct Server Name in it in format below: [default] host = NewServerName.fqdn ... View more

Thank you so much.  Upgrading to 2.2.5 resolved the issue. ... View more

Job was just a coincidence in scheduling.  Crash still occurring.  See that it matches the Web Data Model Acceleration Cron Schedule.  Decided to try to perform a rebuild but crash still persists.  Does anyone have any other suggestions.... short of not using the WebAnalytics App?  This is a 'no-go' as customer uses this very extensively.   Configuration Settings These settings can be changed by going to actions and selecting edit acceleration. Learn More allow_old_summaries = true allow_skew = 0 backfill_time = - cron_schedule = 2,12,22,32,42,52 * * * * earliest_time = -3mon hunk.compression_codec = - hunk.dfs_block_size = 0 hunk.file_format = - manual_rebuilds = true max_concurrent = 6 max_time = 3600 poll_buckets_until_maxtime = false schedule_priority = highest workload_pool = - ... View more

Ok... I have finally isolated the individual Job that is Crashing the thread.  I would like to disable a scheduled job under Web Analytics called "Generate Goal summary - Scheduled"  as it is the culprit.  I ran it manually and it crashed.  I checked under Goals and do not see any configured and the WA_Goas.csv lookup is empty.  Under "Goals" "Setup" I do not see any Goals configured for any of our sites so I don't think it is used.   Here is the code for that job.... in case any of you can figure out if it is a bug with this App and 8.1.0.1 or if it is just bad code! | inputlookup WA_goals | map maxsearches=10000 search="search tag=web eventtype=pageview site=\"$site$\" | eval goal_id=\"$goal_id$\" | eval goal_start=\"$start$\" | eval goal_end=\"$end$\" | rex field=goal_start mode=sed \"s/\*/%/g\" | rex field=goal_end mode=sed \"s/\*/%/g\" | lookup WA_sessions user AS user OUTPUT http_session,http_session_start,http_session_end,http_session_pageviews,http_session_duration,http_referer,http_referer_domain AS http_session_referrer_domain,http_referer_hostname AS http_session_referrer_hostname,http_session_channel | where isnotnull(http_session) | lookup user_agents http_user_agent | eval ua_mobile=if(eventtype==\"ua-mobile\",'ua_device', \"\") | bucket _time span=10m | stats dc(http_session) AS Sessions,dc(user) AS Users,count(eval(like(uri,goal_start))) AS Entries,count(eval(like(uri,goal_end))) AS Completed by _time, site, goal_id, http_session_channel, http_session_referrer_domain,ua_family,ua_mobile,ua_os_family,goal_id | eval goal_start=\"$start$\" | eval goal_end=\"$end$\" | collect index=goal_summary " ... View more

Found where similar issue was fixed in 8.0.2....  Can this be similar in any way to previous bug-fixes? "https://docs.splunk.com/Documentation/Splunk/8.0.2/ReleaseNotes/Fixedissues","Search issues","2020-01-06","SPL-180268, SPL-177675","Crash in BucketSummaryActorThread for a specific summary directory, persists after removing" ... View more

May have Identified the source of the crash as SplunkAppForWebAnalytics Acceleration Job that 'fires' every ten minutes crashing the "BucketSummaryActorThread" but now need to figure out how to address it. ERROR SavedSplunker - savedsearch_id="nobody;SplunkAppForWebAnalytics;_ACCELERATE_DM_SplunkAppForWebAnalytics_Web_ACCELERATE_", message="The search job ... View more

I have verified that the Secpol does not have "Lock Pages in Memory" enabled on this system. ... View more

Fixed my second issue by using "OR Powershell*" vs. "OR Powershell.exe".   (not as clean as I would like but it works again) Can these two issues be related to some sort of 'Injection Protection' or similar?  Just odd that these SPL queries used to work just fine as Saved Alerts before. Thanks, Greg ... View more

Ok... more strange behavior from a second Alert which says is working in Scheduler Logs.  This one is our "Alert Suspicious/Administrative Processes" Alert.  I used Open Search and got Server Error.  Noticed an obvious typo with "sysprep.exe ORsysteminfo.exe ".  "OR" needs a space but when fixed I still see Server Error. So this means what is scheduled vs. what we open are different copies...  yes? Same procedure followed to identify culprit.  "OR powershell.exe".  When removed the SPL compiles and executes properly!  I confirmed this is correct exe name and no other issues with SPL.   I can provide code but at this point...   ... View more

All, Since no one has provided an answer or any feedback on this incident here, I thought I'd share the information that we received from one of our partners. The gist of the information is that this a generic and benign error. "...*Splunk PS Slack channel. I was able to find this answer, which is apparently what Splunk support had previously sent to a customer. "This is an error we have come across with some of our Windows customers, and seems more common of virtualized instances. The splunk process will periodically check the time of the OS system and will show this error if there is a difference (~15 ms) as an indication of the time progress internally. This is really an internal ERROR that should not be reported." Can you confirm that that OS on that Splunk server does have the correct time? Another thought would be that since you are currently running a X.0.0 version of Splunk, to upgrade Splunk to the latest version and see if that will clear up the issue for you. *... We have found no delta in time although we are operating in a virtuaized environment. I hope this information helps you! Best regards, Greg ... View more