Getting Data In

Combined logs, props.conf transforms.conf and inputs.conf

bpgoodm
New Member

I have a centralized syslog server which I forward all other server logs to. All of those logs are combined per log type such as /var/log/messages, /var/log/secure, etc. I've setup my transforms.conf file to take the hostname out of each line and assign it as the host per log file line. I've associated the transform to the various logs. After restarting the splunk forwarder I'm still not seeing my logs being broken out by host. I'm wondering if it's because of the host value in inputs.conf and if that's overriding the dynamic host value I'm creating via props and transforms. Here is what I have so far:

etc/apps/search/local/transforms.conf

[hostname]
DEST_KEY = MetaData:Host
REGEX = (?:\S+ ){2}(\S+)
FORMAT = host::$1

etc/system/local/props.conf

[source::.../messages]
TRANSFORMS-bg = hostname
SHOULD_LINEMERGE = False

Example from splunk show config inputs

[monitor:///var/log/messages*]
_rcvbuf=1572864
disabled=false
host=[splunk_forwarder_hostname]
index=[index_name]
sourcetype=[source_type]

My concern is the host= attribute associated with every monitor entry. Is this the source of my issues or do I need to be looking somewhere else?

Bert

Tags (1)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi bpgoodm,
let me understand:

  • you have a server with Universal Forwarders that receives syslogs,
  • then this server sends logs to Indexer without intermediate forwarders,

Is it correct?

if this is you situation, the problem is on the location of props.conf and transforms.conf: they must be on Indexers not on Universal Forwarders.

There are only two exceptions to this rule:

  • when there's an intermediate Heavy Forwarder that they must be here,
  • if you have to ingest csv or xml or json, in other words indexed extractions, in this case they must be on Universal Forwarders.

Bye.
Giuseppe

View solution in original post

0 Karma

bpgoodm
New Member

Hi Giuseppe,

Ah, thank you. I didn't realize. Your assumptions are correct. I'm setting this stuff up on the Universal Forwarder and the Universal Forwarder is sending the logs to the Indexer. The exceptions you mention don't apply to this situation. Thank you again for the information.

Bert

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi bpgoodm,
let me understand:

  • you have a server with Universal Forwarders that receives syslogs,
  • then this server sends logs to Indexer without intermediate forwarders,

Is it correct?

if this is you situation, the problem is on the location of props.conf and transforms.conf: they must be on Indexers not on Universal Forwarders.

There are only two exceptions to this rule:

  • when there's an intermediate Heavy Forwarder that they must be here,
  • if you have to ingest csv or xml or json, in other words indexed extractions, in this case they must be on Universal Forwarders.

Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...