True custom alert_actions.conf will make you pull your hair out. Unfortunately based on the documentation though it seems as though you can deploy alert_actions.conf with the deployer inside an app, push with deployer, and it should update the search heads, but we've been unable to get this function to work. Changing in the GUI 'works', but if you're like us in our environment, we use a mixture of Chef for initial Splunk builds and the deployer for search head configuration updates, so using the GUI is less than ideal. ... View more

It picks up some pieces of he appropriate alert_actions.conf file and it looks like it's picking up my file for some parts but logs indicate it's still trying to use localhost in the mail server hostname. Here are the results from the --debug - Also thanks! I didn't know this was an option but is very helpful. /opt/splunk/etc/system/default/alert_actions.conf [default] /opt/splunk/etc/system/default/alert_actions.conf hostname = /opt/splunk/etc/system/default/alert_actions.conf maxresults = 10000 /opt/splunk/etc/system/default/alert_actions.conf maxtime = 5m /opt/splunk/etc/system/default/alert_actions.conf track_alert = 0 /opt/splunk/etc/system/default/alert_actions.conf ttl = 10p /opt/splunk/etc/apps/configuration/default/alert_actions.conf [email] /opt/splunk/etc/system/default/alert_actions.conf auth_password = /opt/splunk/etc/system/default/alert_actions.conf auth_username = /opt/splunk/etc/system/default/alert_actions.conf bcc = /opt/splunk/etc/system/default/alert_actions.conf cc = /opt/splunk/etc/system/default/alert_actions.conf command = $action.email.preprocess_results{default=""}$ | sendemail "results_link=$results.url$" "ssname=$name$" "graceful=$graceful{default=True}$" "trigger_time=$trigger_time$" maxinputs="$action.email.maxresults{default=10000}$" maxtime="$action.email.maxtime{default=5m}$" results_file="$results.file$" /opt/splunk/etc/system/default/alert_actions.conf footer.text = If you believe you've received this email in error, please see your Splunk administrator. splunk > the engine for machine data /opt/splunk/etc/system/default/alert_actions.conf format = table /opt/splunk/etc/apps/configuration/default/alert_actions.conf from = SplunkEmail@mydomain.com /opt/splunk/etc/apps/configuration/default/alert_actions.conf hostname = splunksearch.mydomain.com /opt/splunk/etc/system/default/alert_actions.conf include.results_link = 1 /opt/splunk/etc/system/default/alert_actions.conf include.search = 0 /opt/splunk/etc/system/default/alert_actions.conf include.trigger = 0 /opt/splunk/etc/system/default/alert_actions.conf include.trigger_time = 0 /opt/splunk/etc/system/default/alert_actions.conf include.view_link = 1 /opt/splunk/etc/system/default/alert_actions.conf inline = 0 /opt/splunk/etc/apps/configuration/default/alert_actions.conf mailserver = mailrelay.mydomain.com /opt/splunk/etc/system/default/alert_actions.conf maxresults = 10000 /opt/splunk/etc/system/default/alert_actions.conf maxtime = 5m /opt/splunk/etc/system/default/alert_actions.conf message.alert = The alert condition for '$name$' was triggered. /opt/splunk/etc/system/default/alert_actions.conf message.report = The scheduled report '$name$' has run. /opt/splunk/etc/system/default/alert_actions.conf pdfview = /opt/splunk/etc/system/default/alert_actions.conf preprocess_results = /opt/splunk/etc/system/default/alert_actions.conf priority = 3 /opt/splunk/etc/system/default/alert_actions.conf reportCIDFontList = gb cns jp kor /opt/splunk/etc/system/default/alert_actions.conf reportIncludeSplunkLogo = 1 /opt/splunk/etc/system/default/alert_actions.conf reportPaperOrientation = portrait /opt/splunk/etc/system/default/alert_actions.conf reportPaperSize = letter /opt/splunk/etc/system/default/alert_actions.conf reportServerEnabled = false /opt/splunk/etc/apps/configuration/default/alert_actions.conf reportServerURL = /opt/splunk/etc/system/default/alert_actions.conf sendcsv = 0 /opt/splunk/etc/system/default/alert_actions.conf sendpdf = 0 /opt/splunk/etc/system/default/alert_actions.conf sendresults = 0 /opt/splunk/etc/system/default/alert_actions.conf subject = Splunk Alert: $name$ /opt/splunk/etc/system/default/alert_actions.conf subject.alert = Splunk Alert: $name$ /opt/splunk/etc/system/default/alert_actions.conf subject.report = Splunk Report: $name$ /opt/splunk/etc/system/default/alert_actions.conf to = /opt/splunk/etc/system/default/alert_actions.conf track_alert = 1 /opt/splunk/etc/system/default/alert_actions.conf ttl = 86400 /opt/splunk/etc/system/default/alert_actions.conf useNSSubject = 0 /opt/splunk/etc/system/default/alert_actions.conf use_ssl = 0 /opt/splunk/etc/system/default/alert_actions.conf use_tls = 0 /opt/splunk/etc/system/default/alert_actions.conf width_sort_columns = 1 /opt/splunk/etc/system/default/alert_actions.conf [populate_lookup] /opt/splunk/etc/system/default/alert_actions.conf command = copyresults dest="$action.populate_lookup.dest$" sid="$search_id$" /opt/splunk/etc/system/default/alert_actions.conf dest = /opt/splunk/etc/system/default/alert_actions.conf hostname = /opt/splunk/etc/system/default/alert_actions.conf maxresults = 10000 /opt/splunk/etc/system/default/alert_actions.conf maxtime = 5m /opt/splunk/etc/system/default/alert_actions.conf track_alert = 0 /opt/splunk/etc/system/default/alert_actions.conf ttl = 120 /opt/splunk/etc/system/default/alert_actions.conf [rss] /opt/splunk/etc/system/default/alert_actions.conf command = createrss "path=$name$.xml" "name=$name$" "link=$results.url$" "descr=Alert trigger: $name$, results.count=$results.count$ " "count=30" "graceful=$graceful{default=1}$" maxtime="$action.rss.maxtime{default=1m}$" /opt/splunk/etc/system/default/alert_actions.conf hostname = /opt/splunk/etc/system/default/alert_actions.conf maxresults = 10000 /opt/splunk/etc/system/default/alert_actions.conf maxtime = 1m /opt/splunk/etc/system/default/alert_actions.conf track_alert = 0 /opt/splunk/etc/system/default/alert_actions.conf ttl = 86400 /opt/splunk/etc/system/default/alert_actions.conf [script] /opt/splunk/etc/system/default/alert_actions.conf command = runshellscript "$action.script.filename$" "$results.count$" "$search$" "$search$" "$name$" "Saved Search [$name$] $counttype$($results.count$)" "$results.url$" "$deprecated_arg$" "$search_id$" "$results.file$" maxtime="$action.script.maxtime{default=5m}$" /opt/splunk/etc/system/default/alert_actions.conf filename = /opt/splunk/etc/system/default/alert_actions.conf hostname = /opt/splunk/etc/system/default/alert_actions.conf maxresults = 10000 /opt/splunk/etc/system/default/alert_actions.conf maxtime = 5m /opt/splunk/etc/system/default/alert_actions.conf track_alert = 1 /opt/splunk/etc/system/default/alert_actions.conf ttl = 600 /opt/splunk/etc/system/default/alert_actions.conf [summary_index] /opt/splunk/etc/system/default/alert_actions.conf name = summary /opt/splunk/etc/system/default/alert_actions.conf command = summaryindex spool=t uselb=t addtime=t index="$action.summary_index._name{required=yes}$" file="$name_hash$$#random$.stash_new" name="$name$" marker="$action.summary_index*{format=$KEY=\\"$VAL\\", key_regex="action.summary_index.(?!(?:command|inline|maxresults|maxtime|ttl|track_alert|(?:_.))$)(.)"}$" /opt/splunk/etc/system/default/alert_actions.conf hostname = /opt/splunk/etc/system/default/alert_actions.conf inline = 1 /opt/splunk/etc/system/default/alert_actions.conf maxresults = 10000 /opt/splunk/etc/system/default/alert_actions.conf maxtime = 5m /opt/splunk/etc/system/default/alert_actions.conf track_alert = 0 /opt/splunk/etc/system/default/alert_actions.conf ttl = 120 ... View more

Yes the file gets deployed to /etc/apps/configuration/default/alert_actions.conf. Local directories on the SHC members reveal just alert_actions.conf files inside /etc/apps/configuration/default and the default.old directories that shc seems to create. Is this problem in how we are handling files on the deployer perhaps? apps no longer seem to have /local folders and just have /default now which seems a little strange. ... View more

I deploy alert_actions.conf to /etc/shcluster/apps/configuration/ folder on the Deployer. Then I apply shcluster-bundle to the shc members. This puts alert_actions.conf in /etc/apps/configuration/default/alert_actions.conf on the shc members, in the same manner as we use it for authentication, outputs, and web conf files, however doesn't seem to take effect. When looking at python.log I continue to get failures saying connection refused connecting to localhost to send these messages which would leave me to believe some alert_actions.conf exists from default that has higher precedence than an app potentially. using find here are the files which match alert_actions.conf. I would expect /opt/splunk/etc/apps/configuration/default/alert_actions.conf to take precedence here. /opt/splunk/etc/apps/configuration/default.old.20150813-163109/alert_actions.conf /opt/splunk/etc/apps/configuration/default.old.20150813-155352/alert_actions.conf /opt/splunk/etc/apps/configuration/default.old.20150813-163835/alert_actions.conf /opt/splunk/etc/apps/configuration/default/alert_actions.conf /opt/splunk/etc/system/default/alert_actions.conf I then take a look using btool to determine what's getting used for alert_actions and the following appears: splunk cmd btool alert_actions list [default] hostname = maxresults = 10000 maxtime = 5m track_alert = 0 ttl = 10p [email] auth_password = auth_username = bcc = cc = command = $action.email.preprocess_results{default=""}$ | sendemail "results_link=$results.url$" "ssname=$name$" "graceful=$graceful{default=True}$" "trigger_time=$trigger_time$" maxinputs="$action.email.maxresults{default=10000}$" maxtime="$action.email.maxtime{default=5m}$" results_file="$results.file$" footer.text = If you believe you've received this email in error, please see your Splunk administrator. splunk > the engine for machine data format = table from = SplunkEmail@mydomain.com hostname = splunksearch.mydomain.com include.results_link = 1 include.search = 0 include.trigger = 0 include.trigger_time = 0 include.view_link = 1 inline = 0 mailserver = mailrelay.mydomain.com maxresults = 10000 maxtime = 5m message.alert = The alert condition for '$name$' was triggered. message.report = The scheduled report '$name$' has run. pdfview = preprocess_results = priority = 3 reportCIDFontList = gb cns jp kor reportIncludeSplunkLogo = 1 reportPaperOrientation = portrait reportPaperSize = letter reportServerEnabled = false reportServerURL = sendcsv = 0 sendpdf = 0 sendresults = 0 subject = Splunk Alert: $name$ subject.alert = Splunk Alert: $name$ subject.report = Splunk Report: $name$ to = track_alert = 1 ttl = 86400 useNSSubject = 0 use_ssl = 0 use_tls = 0 width_sort_columns = 1 [populate_lookup] command = copyresults dest="$action.populate_lookup.dest$" sid="$search_id$" dest = hostname = maxresults = 10000 maxtime = 5m track_alert = 0 ttl = 120 [rss] command = createrss "path=$name$.xml" "name=$name$" "link=$results.url$" "descr=Alert trigger: $name$, results.count=$results.count$ " "count=30" "graceful=$graceful{default=1}$" maxtime="$action.rss.maxtime{default=1m}$" hostname = maxresults = 10000 maxtime = 1m track_alert = 0 ttl = 86400 [script] command = runshellscript "$action.script.filename$" "$results.count$" "$search$" "$search$" "$name$" "Saved Search [$name$] $counttype$($results.count$)" "$results.url$" "$deprecated_arg$" "$search_id$" "$results.file$" maxtime="$action.script.maxtime{default=5m}$" filename = hostname = maxresults = 10000 maxtime = 5m track_alert = 1 ttl = 600 [summary_index] name = summary command = summaryindex spool=t uselb=t addtime=t index="$action.summary_index._name{required=yes}$" file="$name_hash$$#random$.stash_new" name="$name$" marker="$action.summary_index*{format=$KEY=\\"$VAL\\", key_regex="action.summary_index.(?!(?:command|inline|maxresults|maxtime|ttl|track_alert|(?:_.))$)(.)"}$" hostname = inline = 1 maxresults = 10000 maxtime = 5m track_alert = 0 ttl = 120 ... View more

frobinson - Thank you for your help and input. I have to say your response times on answers.splunk.com put our Enterprise support agreement support times to shame. ... View more

bmacias84 thank you very much. This seems to be the exact strategy as of 6.2 for scripted inputs. ... View more

I did make the changes to the search heads that you recommended to not allow them to index data. This then fixed the issue of the data not being forwarded to the indexers. However each search head runs the scripted input on it's interval of 10 minutes. I'm assuming scripted inputs don't act like scheduled searches in 6.2, where 1 head executes the input and forwards the data to index. It does appear that if this is the case the best approach would be to use a separate box as our "data collection api" box and let it forward the data on to the indexers. I was hoping this could be done in the search heads to keep complexity down but this is a workable solution. ... View more

Yes we do use search head clustering by utilizing the deployer and clustering ... View more

This took care of it. I put this in the props.conf on the heavy forwarder and it didn't change, but then I restarted splunk on the heavy forwarder and it worked like a champ. That now makes me wonder if my other changes would have worked too but I'll take your faster approach 🙂 ... View more

Yes the sourcetype matches, the regex for ^\s*Date is something I've tried as well as ^Date and Date itself. ... View more

head 3 will grab the last 3 events and then do the stats command on those results. Next logical step there for me would be head 3 by X_Forwarded_For but unfortunately you can't do that with the head command. ... View more

Hope it's not too late to chime in here. It's mostly in the ASA logging configuration. Enable Logging, set a logging host, and set your list. Example configuratino here: logging enable logging timestamp logging host [interface] [forwarder address or indexer] - Example[logging host inside 10.0.0.5] logging list cisco message 111009 logging list cisco message 111008 logging list cisco message 304009 logging trap cisco[cisco is the logging list name] And that is pretty much it. Make sure you've got an input for udp514 traffic and you should be seeing data provided the Splunk side of things is in shape. ... View more

I modified both heavy forwarders configs to this: [tcpout:autolbgroup1] server = 192.168.101.22:9997,192.168.101.23:9997,192.168.101.33:9997 autoLB = true useACK = true [tcpout] defaultGroup = autolbgroup1 disabled = 0 the key appears to be autoLB = true. I previously understood that it was always true but didn't appear so. Anyway setting this to true fixes the entire problem. I'm assuming it was sending all indexers all copies of the data and they all thought they were new primary copies and then returning those results. Now it is all working properly. ... View more

WOW iunderwood this works perfectly. Thank you very much. ... View more

Message itself contains all of ApplicationName through StackTrace inside that field in the eventlog. A splunk search on this yields that the Severity, Error, and StackTrace Fields show up in the 22 interesting fields, I assume being autoextracted, but ApplicationName itself does not get extracted. ... View more