Why is my Splunk universal forwarder monitor hostname key not working?

[vgolof]

Splunk Forwarder monitor hostname key is not working.

Amazon Linux AMI release 2015.03 3.14.48-33.39.amzn1.x86_64

Splunk Universal Forwarder 6.2.5 (build 272645)

Forwards:

input-prd-p-t865bwklrqxn.cloud.splunk.com:9997 (ssl)

I want to monitor log files from one host with Splunk Light.
Used command:

/opt/splunkforwarder/bin/splunk add monitor /var/log/messages -hostname <node hostname>
/opt/splunkforwarder/bin/splunk add monitor /var/log/audit/audit.log -hostname <node hostname>
/opt/splunkforwarder/bin/splunk add monitor /var/log/docker -hostname <node hostname>

But there are no hostnames when I check:

/opt/splunkforwarder/bin/splunk list monitor
Monitored Files:
        /var/log/audit/audit.log
        /var/log/docker
        /var/log/messages

And as result i got 3(!) different Hosts on my cloud.splunk.com

ip-10-1-0-82        201 9/17/15 9:36:08.000 AM
_node hostname_     14,843  9/17/15 9:44:03.000 AM
_local hostname_        39  9/14/15 1:42:59.000 PM

What is the solution to this problem?

[bosburn_splunk]

Couple of questions:
Are you running any applications in the cloud?
Do you have any field extractions that may be overwriting the hostname?

While you are setting the hostname at input time, you can overwrite it when it gets to the indexer..

[vgolof]

No custom apps, jobs, filters or field extractions.
Field extractions and Field transformations in cloud.splunk.com is stored by defaults.

I just install splunkforwarder, add you cluster:

/opt/splunkforwarder/bin/splunk install app /opt/splunkforwarder/etc/splunkclouduf.spl -auth admin:changeme
/opt/splunkforwarder/bin/splunk restart

splunkclouduf.spl from https://.cloud.splunk.com/en-US/app/search/splunkclouduf

add 3 logs (see topic):

/var/log/audit/audit.log
/var/log/docker
/var/log/messages

And change some hosts in a disorderly heap of configs for tests.

All what i want - have a same host field or something other field

[bmacias84]

You shouldn't have to specify hostname in the command. Splunk's default is to use the system name. You can use btool to debug your configs. This command below will also show which apps each setting is coming from.

$SPLUNK_HOME/bin/splunk cmd btool --debug inputs list

[vgolof]

< hostname > - for example, in config used real hostname.

Ok, output:

...
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf [monitor:///var/log/cron]
/opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf disabled = false
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf host = vastest3.<hostname>
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf ignoreOlderThan = 14d
/opt/splunkforwarder/etc/system/default/inputs.conf index = default
...
/opt/splunkforwarder/etc/system/local/inputs.conf host = vastest.<hostname>
opt/splunkforwarder/etc/apps/search/local/inputs.conf host = <hostname>
...

In logs

host = ip-10-1-0-41 source = /var/log/cron sourcetype = syslog

And i think it is worst way to use something like this for every new log record.
http://answers.splunk.com/answers/23507/why-is-host-localhost-when-inputs-conf-set-up-to-use-custom-...

How can I override host for ALL LOGS on first install ?

Ok... let see props.conf

[linux_messages_syslog]
TRANSFORMS = syslog-host
...

[syslog]
TRANSFORMS = syslog-host
...

Created transforms.conf:

[syslog-host]
REGEX = vastest4.<hostname>
DEST_KEY = MetaData:Host
FORMAT = host::$1

not work too:
REGEX = \s(\w*)$
DEFAULT_VALUE = vastest5.qdoba-mera.seed1)

Result:
host = ip-10-1-0-41 source = /var/log/cron sourcetype = syslog

That next ?

p. s. And how can I delete "bad" Hosts events from *.cloud.splunk.com ?

[bmacias84]

Probably need to contact Support.

[vgolof]

I submitted CASE [271586] 17.09.15, but don't got any answers and can't find any link of case for track result.

[piebob]

@vgolof: i looked up your case--you do not appear to have an active Support entitlement, which is why your case was not responded to. you must have a paid Support plan to receive a response from our Support team. i'll see if anyone has a minute to look at this.

[vgolof]

Are you think it's not a bug ?