I am trying to configure the distributed monitoring console without the UI (for automation purposes). It seems that I have gotten most things right - all instances show up the way I want them to, however they are all marked as "unreachable". It seems that I must do the step where I provide credentials for the mc host to login to the monitored host. However I cannot figure out what this step actually does. Also I cannot find anything that hints to the credentials being stored anywhere. So what does this login process actually do, and how can I mimic that bevhaviour for the mc from the commandline/ via config files? Any insight on how the setup process works behind the scene would be appreciated. ... View more

My splunk indexer stopped with lots of ERROR messages which ended in "Too many open files". ulimit -n shows 65536 for the splunk user. I was able to just start splunk again, and it is running fine now for two days. Also, when I count current open files with lsof -u splunk | awk 'BEGIN { total = 0; } $4 ~ /^[0-9]/ { total += 1 } END { print total }' it shows about 2600 open files for user splunk. So nothing serious. So obviously increasing the nofile limit would not help to prevent this in the futur. Any ideas on what could be done to further analyse this? I am currently monitoring the number of open files over time. Maybe during crash time (0:03 am) nofiles was different form now in the afternoon. Edit after comments: OS is RedHat 7.3. It is a cluster of two indexers, only one went down. Number of forwarders, as estimated by | metadata type=hosts | where now()-recentTime < (7*24*60*60) |stats dc(host) is 1381. Ulimit at startup, as determined by index=_internal host=myhost source="/opt/splunk/var/log/splunk/splunkd.log" Limit open files is 65536, as stated bevor. My nofile over times shows a destinct increase about the time the server crashed, however still far from that 65536. However, this increase might already be the explanation, because the crash happened Monday morning, a time where a lot more scheduled searches might take place than on a Thursday. Here is the graph: ... View more

I would like to modify my search term before I actually search for it. Background: I want to see how the MX for a certain domain connects to my MX. Say I want to know how (e.g. with TLS) the mail relay for acme.org connects - If I search for acme.org in my maillog, I find nothing, because the MX record for acme.org might be coolmailer.com. Currently do the lookup by hand before I search, but it would to cool to integrate this step into the splunk search. I tried using a scripted lookup, but I fail using a lookup before getting any results from search. Something along the lines of sourcetype=whatever give_mx_record_of("acme.org") |stats count by encryption_level where give_mx_record_of("acme.org") would return coolmailer.com , which is what would be searched for, so while I enter acme.org into my (saved)search (because that is what I know), splunk actually searches for sourcetype=whatever coolmailer.com |stats count by encryption_level This particular example would probably yield just one line, or no result. I have a python script that does the conversion of acme.org to coolmailer.com , currently in the form of a lookup script (scripted lookup), but if it helps I can transform it to any other format. Can splunk do such a thing, and if yes, how? ... View more