Alert Messages Coming from localhost@localdomain

[I_am_Jeff]

I've recently brought up one additional pooled search head to join my original two. All my search head are version 4.3.4, build 136012.

Splunk e-mail alerts coming from the new search head have the format "From: splunk@localhost.localdomain", while the other two show "From: Splunk Daemon User [splunk@searchhead01.full.name.com]". (Or '@searchhead02.' depending on source. You get the idea.)

I do not like localhost@localdomain in the header. I would rather have the more informative, real name included in the mail header. My users agree with me, for once. 😉

I've looked through the GUI and not found differences in Manager > System Setting > {General Setting or Email alert settings}. I've run "find . -type f -exec grep localdomain {} ; -print" and nothing obvious has jumped out at me. I'll admit I haven't compared everything in all the files, however.

All three have the same content in .../etc/system/local/alert_actions.conf (Non-pooled directory, no such file in the pooled area).

Additionally, I brought up a new search head, that is not pooled, at the same time as the new pooled search head. (I have a total of four search heads.) Same version and build. It also uses the unpreferred "From: splunk@localhost.localdomain". Same settings and version as the new pooled search head. It is just not pooled.

All are RHEL 5.7. All respond correctly to the hostname(1). hosts(5) files are correct. /etc/sysconfig/network have "HOSTNAME=" set correctly.

I've used mailx(P) on all four to send test messages, from the Splunk user, using the command line. The headers all come through with the preferred format of "From: Splunk Daemon User [splunk@searchhead01.full.name.com]" so I'm convinced it's not sendmail(8) or the operating system causing the problem. I am not convinced some interaction between the OS and Splunk couldn't cause the problem, however. But I really believe I've missed something in the Splunk configuration on the new boxes.

What am I missing?

[the_wolverine · Accepted solution]

Try setting it on your Searchhead as an admin user:

[I_am_Jeff]

I have no doubt this will work. But still would like to know where I went wrong with my original install.

[rsennett_splunk]

I believe that will either be set in the conf file or with scripting. So I'd check to see if there is a $SPLUNK_HOME/etc/system/local/alert_actions.conf on the SH where it works as you prefer.

I haven't tested this much, but that would be the first thing I'd check...

[I_am_Jeff]

All four do not have $LOCALHOST set. (echo $LOCALHOST returns nothing on any of them.) All four alert_actions.conf look like this, just 3 lines.

[email]
reportServerEnabled = 1
reportServerURL =