Thank you for the quick take.   I am confident that i searched for and invoked deletion of my jobs across all apps.  I’m thinking it takes splunk a while to act on or confirm deletion and until so it jobs remain searchable in job activity manager.   I have experienced this sort of problem in multiple clustered on-prem splunk implementations  over the years and am frustrated on behalf of users for their non deterministic experience recovering from queuing through prescribed actions on job activity monitor.  ... View more

if it makes you feel any better I am having the same problem.. Any luck? ... View more

thanks for the input.   i had already checked the spec but was hoping someone was aware of a creative workaround.   ... View more

unfortunately indeed.  please do not include such without overrides in future versions. ... View more

good enough for me.  if operating any older of a version id have bigger problems to fixate on.  thanks! ... View more

Any idea what version of Splunk Universal forwarder introduced the $xmlRegex option? ... View more

you are my hero. now I need need to go back and update about 20 inefficient dashboards I have made over time knowing that this should have been a thing all along. #SmartAnswers ... View more

heres my powershell way of getting large results through rest https://github.com/dstaulcu/SplunkTools/blob/master/Splunk-SearchLargeJobs-Example.ps1 ... View more

I would not hold your breath waiting on improvements in TA-microsoft-sysmon to be published to Splunkbase. Instead I'd recommend forking the project (in github) and tuning it as needed for implementation in your environment. It would be nice to see innovations submitted as pull requests but I think that baseline is frozen for the foreseeable future. If I recall correctly, the last merge made TA-microsoft-sysmon dependent on "breaking" versions of TA-microsoft-windows. Upgrading to TA-microsoft-windows v5+ requires a tedious transition of references to sourcetype in saved searches and dashboards. Defaulting of all windows inputs to render as XML further delays search time field extraction processing which may result in increased hosting costs to compensate. ... View more

Thanks for the suggestion. Have to revisit this problem for another event type in today. Will give your suggestion a shot. Meanwhile, here are some fill-in-the-blank blacklist patterns for all event types from Sysmon: # Sample blacklist entries for Sysmon schemaversion 4.21 blacklist1 = EventCode="^255$" Message="(?i)^UtcTime:\s+(.*)\s+ID:\s+(.*)\s+Description:\s+(.*)\s+Description:\s+(.*)$" blacklist2 = EventCode="^1$" Message="(?i)^RuleName:\s+(.*)\s+UtcTime:\s+(.*)\s+ProcessGuid:\s+(.*)\s+ProcessId:\s+(.*)\s+Image:\s+(.*)\s+FileVersion:\s+(.*)\s+Description:\s+(.*)\s+Product:\s+(.*)\s+Company:\s+(.*)\s+OriginalFileName:\s+(.*)\s+CommandLine:\s+(.*)\s+CurrentDirectory:\s+(.*)\s+User:\s+(.*)\s+LogonGuid:\s+(.*)\s+LogonId:\s+(.*)\s+TerminalSessionId:\s+(.*)\s+IntegrityLevel:\s+(.*)\s+Hashes:\s+(.*)\s+ParentProcessGuid:\s+(.*)\s+ParentProcessId:\s+(.*)\s+ParentImage:\s+(.*)\s+ParentCommandLine:\s+(.*)\s+ParentCommandLine:\s+(.*)$" blacklist3 = EventCode="^2$" Message="(?i)^RuleName:\s+(.*)\s+UtcTime:\s+(.*)\s+ProcessGuid:\s+(.*)\s+ProcessId:\s+(.*)\s+Image:\s+(.*)\s+TargetFilename:\s+(.*)\s+CreationUtcTime:\s+(.*)\s+PreviousCreationUtcTime:\s+(.*)\s+PreviousCreationUtcTime:\s+(.*)$" blacklist4 = EventCode="^3$" Message="(?i)^RuleName:\s+(.*)\s+UtcTime:\s+(.*)\s+ProcessGuid:\s+(.*)\s+ProcessId:\s+(.*)\s+Image:\s+(.*)\s+User:\s+(.*)\s+Protocol:\s+(.*)\s+Initiated:\s+(.*)\s+SourceIsIpv6:\s+(.*)\s+SourceIp:\s+(.*)\s+SourceHostname:\s+(.*)\s+SourcePort:\s+(.*)\s+SourcePortName:\s+(.*)\s+DestinationIsIpv6:\s+(.*)\s+DestinationIp:\s+(.*)\s+DestinationHostname:\s+(.*)\s+DestinationPort:\s+(.*)\s+DestinationPortName:\s+(.*)\s+DestinationPortName:\s+(.*)$" blacklist5 = EventCode="^4$" Message="(?i)^UtcTime:\s+(.*)\s+State:\s+(.*)\s+Version:\s+(.*)\s+SchemaVersion:\s+(.*)\s+SchemaVersion:\s+(.*)$" blacklist6 = EventCode="^5$" Message="(?i)^RuleName:\s+(.*)\s+UtcTime:\s+(.*)\s+ProcessGuid:\s+(.*)\s+ProcessId:\s+(.*)\s+Image:\s+(.*)\s+Image:\s+(.*)$" blacklist7 = EventCode="^6$" Message="(?i)^RuleName:\s+(.*)\s+UtcTime:\s+(.*)\s+ImageLoaded:\s+(.*)\s+Hashes:\s+(.*)\s+Signed:\s+(.*)\s+Signature:\s+(.*)\s+SignatureStatus:\s+(.*)\s+SignatureStatus:\s+(.*)$" blacklist8 = EventCode="^7$" Message="(?i)^RuleName:\s+(.*)\s+UtcTime:\s+(.*)\s+ProcessGuid:\s+(.*)\s+ProcessId:\s+(.*)\s+Image:\s+(.*)\s+ImageLoaded:\s+(.*)\s+FileVersion:\s+(.*)\s+Description:\s+(.*)\s+Product:\s+(.*)\s+Company:\s+(.*)\s+OriginalFileName:\s+(.*)\s+Hashes:\s+(.*)\s+Signed:\s+(.*)\s+Signature:\s+(.*)\s+SignatureStatus:\s+(.*)\s+SignatureStatus:\s+(.*)$" blacklist9 = EventCode="^8$" Message="(?i)^RuleName:\s+(.*)\s+UtcTime:\s+(.*)\s+SourceProcessGuid:\s+(.*)\s+SourceProcessId:\s+(.*)\s+SourceImage:\s+(.*)\s+TargetProcessGuid:\s+(.*)\s+TargetProcessId:\s+(.*)\s+TargetImage:\s+(.*)\s+NewThreadId:\s+(.*)\s+StartAddress:\s+(.*)\s+StartModule:\s+(.*)\s+StartFunction:\s+(.*)\s+StartFunction:\s+(.*)$" blacklist10 = EventCode="^9$" Message="(?i)^RuleName:\s+(.*)\s+UtcTime:\s+(.*)\s+ProcessGuid:\s+(.*)\s+ProcessId:\s+(.*)\s+Image:\s+(.*)\s+Device:\s+(.*)\s+Device:\s+(.*)$" blacklist11 = EventCode="^10$" Message="(?i)^RuleName:\s+(.*)\s+UtcTime:\s+(.*)\s+SourceProcessGUID:\s+(.*)\s+SourceProcessId:\s+(.*)\s+SourceThreadId:\s+(.*)\s+SourceImage:\s+(.*)\s+TargetProcessGUID:\s+(.*)\s+TargetProcessId:\s+(.*)\s+TargetImage:\s+(.*)\s+GrantedAccess:\s+(.*)\s+CallTrace:\s+(.*)\s+CallTrace:\s+(.*)$" blacklist12 = EventCode="^11$" Message="(?i)^RuleName:\s+(.*)\s+UtcTime:\s+(.*)\s+ProcessGuid:\s+(.*)\s+ProcessId:\s+(.*)\s+Image:\s+(.*)\s+TargetFilename:\s+(.*)\s+CreationUtcTime:\s+(.*)\s+CreationUtcTime:\s+(.*)$" blacklist13 = EventCode="^12$" Message="(?i)^RuleName:\s+(.*)\s+EventType:\s+(.*)\s+UtcTime:\s+(.*)\s+ProcessGuid:\s+(.*)\s+ProcessId:\s+(.*)\s+Image:\s+(.*)\s+TargetObject:\s+(.*)\s+TargetObject:\s+(.*)$" blacklist14 = EventCode="^13$" Message="(?i)^RuleName:\s+(.*)\s+EventType:\s+(.*)\s+UtcTime:\s+(.*)\s+ProcessGuid:\s+(.*)\s+ProcessId:\s+(.*)\s+Image:\s+(.*)\s+TargetObject:\s+(.*)\s+Details:\s+(.*)\s+Details:\s+(.*)$" blacklist15 = EventCode="^14$" Message="(?i)^RuleName:\s+(.*)\s+EventType:\s+(.*)\s+UtcTime:\s+(.*)\s+ProcessGuid:\s+(.*)\s+ProcessId:\s+(.*)\s+Image:\s+(.*)\s+TargetObject:\s+(.*)\s+NewName:\s+(.*)\s+NewName:\s+(.*)$" blacklist16 = EventCode="^15$" Message="(?i)^RuleName:\s+(.*)\s+UtcTime:\s+(.*)\s+ProcessGuid:\s+(.*)\s+ProcessId:\s+(.*)\s+Image:\s+(.*)\s+TargetFilename:\s+(.*)\s+CreationUtcTime:\s+(.*)\s+Hash:\s+(.*)\s+Hash:\s+(.*)$" blacklist17 = EventCode="^16$" Message="(?i)^UtcTime:\s+(.*)\s+Configuration:\s+(.*)\s+ConfigurationFileHash:\s+(.*)\s+ConfigurationFileHash:\s+(.*)$" blacklist18 = EventCode="^17$" Message="(?i)^RuleName:\s+(.*)\s+EventType:\s+(.*)\s+UtcTime:\s+(.*)\s+ProcessGuid:\s+(.*)\s+ProcessId:\s+(.*)\s+PipeName:\s+(.*)\s+Image:\s+(.*)\s+Image:\s+(.*)$" blacklist19 = EventCode="^18$" Message="(?i)^RuleName:\s+(.*)\s+EventType:\s+(.*)\s+UtcTime:\s+(.*)\s+ProcessGuid:\s+(.*)\s+ProcessId:\s+(.*)\s+PipeName:\s+(.*)\s+Image:\s+(.*)\s+Image:\s+(.*)$" blacklist20 = EventCode="^19$" Message="(?i)^RuleName:\s+(.*)\s+EventType:\s+(.*)\s+UtcTime:\s+(.*)\s+Operation:\s+(.*)\s+User:\s+(.*)\s+EventNamespace:\s+(.*)\s+Name:\s+(.*)\s+Query:\s+(.*)\s+Query:\s+(.*)$" blacklist21 = EventCode="^20$" Message="(?i)^RuleName:\s+(.*)\s+EventType:\s+(.*)\s+UtcTime:\s+(.*)\s+Operation:\s+(.*)\s+User:\s+(.*)\s+Name:\s+(.*)\s+Type:\s+(.*)\s+Destination:\s+(.*)\s+Destination:\s+(.*)$" blacklist22 = EventCode="^21$" Message="(?i)^RuleName:\s+(.*)\s+EventType:\s+(.*)\s+UtcTime:\s+(.*)\s+Operation:\s+(.*)\s+User:\s+(.*)\s+Consumer:\s+(.*)\s+Filter:\s+(.*)\s+Filter:\s+(.*)$" blacklist23 = EventCode="^22$" Message="(?i)^RuleName:\s+(.*)\s+UtcTime:\s+(.*)\s+ProcessGuid:\s+(.*)\s+ProcessId:\s+(.*)\s+QueryName:\s+(.*)\s+QueryStatus:\s+(.*)\s+QueryResults:\s+(.*)\s+Image:\s+(.*)\s+Image:\s+(.*)$" ... View more

Keep a careful eye on CPU utilization changes in your IDX tier as you transition. It was a very sad day when we had to scale back our perfmon to metrics deployment to fall within financial constraints. ... View more

When we upgraded to Splunk 6.3 providing initial support for the feature our implementation did not have the desired effect. With the crossServerCheckesum enabled amid DNS load balanced deployment servers, clients still exhibited recurring checksum mismatches, app downloads, and deployment client restarts. I'm wide open to the idea that our implementation was flawed. The intent of my Question was to determine if anyone else actually had success with the feature in 6.3 or later versions. Based on ejenson's input as well as myriad upgrades we have implemented since 6.3, a retry is certainly worth a shot! I was unaware of the endpoint setting. Now I am curious whether that channel provides for secure communication and to what extent delays in phone home request responses are influenced by app download activity. ... View more

didnt know about makeresults at the time. thatd work too! ... View more

Thank you again. That approach would certainly function but also shift costs to computation of regexes. Ideally we would filter on the client side to avoid network and computation costs in addition to storage. ... View more

Thank you. We are already reducing the size of the index by adjusting retention. The problem with that strategy is that there is valuable stuff in the _internal index, sourced from splunk servers, which we do want to retain/review over longer periods of time. ... View more

Seems odd to me, too, that that the event in the audit index do not describe the nature of the role change. ... View more

Knowledge objects (searches, dashboards, workflow actions, automated lookups, etc.) shipped in apps you would download from splunkbase should not reference indexes by name. Instead, best practice is to query events by source or perhaps sourcetype. Index names are yours to set as you see fit in your environment and splunkbase app developers should not make assumptions about what those index names will be. Changing an index (into which perfmon-based inputs route) to a metrics data store would likely break things in the windows-add on unless you also customize the app to handle index-time transforms which fit the perfmon data into a metrics mold. You would also need to replace search commands in windows-add-on knowledge objects to metrics based search commands. I imagine you probably shouldn't go there just yet. 🙂 ... View more

Some of the things to consider in creation/configuration of indexes and routing of windows data among them include access control, retention, performance, and cost recovery objectives. Indexes are a point of administration where you can separate search access based on roles. For example, you might route sensitive data such as windows security event logs into an index which is separate from non-sensitive windows logs. You can also set retention and tiered storage strategies on an per-index basis.. Maybe you want to keep security related indexes searchable for years and maybe you want to keep performance/metrics logs searchable for months. Among those logs you want to keep searchable for a long time, maybe you want to keep more recent events in faster storage and older events on cheaper/slower storage. Regarding performance, many of the events produced by the windows add-on are derived from performance counters. You might want to transform/reroute those outputs from a standard index to an index optimized for metrics to reduce search costs. Last, if you provide Splunk as a service in your organization having multiple business units feeding you windows-based data, you might want to consider routing windows data into indexes that are specific to each business unit so that you are better positioned recover costs based on storage displacement and search history. Definitely tracking this to see where the conversation goes. ... View more

Gotta love positional based strategies for masking passwords. We have something like 200 masking rules which process against every event for sourcetypes where passwords are probable. Next strat is user education. Hopefully Stream processor will give more logic control for sensitive data masking. ... View more

Good technique to know. Sounds like that condition would be faster to diagnose with powershell get-winevent cmdlet. Also sounds like a useful warning for winevent log component to write to splunkd in future versions. Eg. warn when getevent feature takes more than, say 10 seconds, to return a result. ... View more