How to add _meta to all entries from forwarder?

[mestuleslv]

We have couple development environments that have Windows servers with same names (i.e. HOSTNAME1) and would like to add meta "env" to every log file sent from local forwarder to splunk server.

I can achieve this by adding "_meta = env::env01" to every log type in inputs.conf i.e.:

[WinEventLog://Application]
_meta = env::env01

[WinEventLog://Security]
_meta = env::env01

BUT I when I try to add it to [default] at the top of inputs.conf, to save repeating _meta=env01 for every log type, nothing happens, i.e.:

[default]
host = HOSTNAME
_meta = env::env01

host= gets applied, but _meta doesn't.

What am I doing wrong? How could I achieve apply meta to every entry going out of server, if [default] is not supported?

[mhoogcarspel_sp]

For anyone finding this, as this came up on Slack:

[default] doesn't apply to modular inputs, most notably wineventlog.

You'll need to add it to the top level stanza for that input type:

[WinEventLog] 
_meta = hf_proxy::meta_test

https://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorWindowseventlogdata#Specify_global_s...

[woodcock]

Definitely open a support case.

[ntankersley_spl]

Would a better global _meta make life easier for everyone?

[woodcock]

That would cover the vast majority of the use-cases, yes.

[jjensenyahoo]

For Splunk team,
I think a better/easier way for us to add indexed "tags" (i.e. additional fields) to all events of a certain sourcetype/source, or even globally, (all events sent from this host), from a Universal Forwarder is something which needs first class support and documentation. Seems like most folks are having to hack on _meta (and then on fields.conf and possibly elsewhere).

My use case for example. We run hosts in AWS EC2 and we have several tags on our instances which identify things like server-type (api, worker, etc), environment (prod, staging, dev, etc), and additionally things like instance_id, aws region, etc. Would be awesome if we can either provide this tags from a file. ( What I'm currently doing is running a cloud-init script at start to assemble all of the tags and inject them into the _meta field on my inputs.conf of forwarder. Also, I had to deploy a systemd override/drop-in config for splunkd to ensure splunk forwarder is started only after my cloud-final, as I'm using cloud-init per-instance start scripts for this instance-start-time configuration hack, but YMMV for other folks.)

[DATEVeG]

Also _meta in [default] doesn't seem to work for http inputs

[danbrook]

I too am having the same issue. I'd like to understand exactly how this works.

[srmorrison]

@mestuleslv did you end up finding a workaround for you issue? I am currently having the same problem.

[dflodstrom]

If you able to search for your new field using index=* env::env01 ? If so then it is working and you just need to add this new field to fields.conf on your search head(s) to make it searchable. You should be able to add something like ... [env]
INDEXED=true to your fields.conf

[nickhills]

The documentation is a little confusing on this:

Under the global settings it says:

# Pipeline Key defaulting.

* Pipeline keys in general can be defaulted in inputs stanzas.
* The list of user-available modifiable pipeline keys is described in
  transforms.conf.spec; see transforms.conf.spec for further information on
  these keys.
* The currently-defined keys which are available literally in inputs stanzas
  are as follows:

queue = <value>
_raw = <value>
_meta = <value>
_time = <value>
* Inputs have special support for mapping host, source, sourcetype, and index
  to their metadata names such as host -> Metadata:Host
* Defaulting these values is not recommended, and is
  generally only useful as a workaround to other product issues.
* Defaulting these keys in most cases will override the default behavior of
  input processors; but this behavior is not guaranteed in all cases.
* Values defaulted here, as with all values provided by inputs, can be
  altered by transforms at parse-time.

This is the bit which I think is unclear:
The currently-defined keys which are available literally in inputs stanzas
are as follows...
Which almost suggests they can only be applied to a specific stanza (other than default)