Events from second-tier Windows Universal Forwarders not appearing in Splunk

[grahampoulter]

We are failing to get events indexed with the following topology: Splunk 4.2 receiving compressed events over the internet from a single primary Universal Forwarder, in turn receiving uncompressed events from local-area secondary Universal Forwarders. All installations are on Windows Server 2008R2 (64-bit).

In our test the secondary forwarders have a simple inputs.conf:

[WinEventLog:Application]
disabled = 0

Events from the primary forwarder show up in central Splunk, and in another test we sucessfully forwarded events from a Linux secondary forwarder through the primary forwarder to the central Splunk. The problem is thus windows-specific.

But, with a windows secondary forwarder, the application eventlog and splunk index::_internal events do not appear in the central splunk. This we ascertained from viewing the summary dashboard looking for the hosts and searching for the hosts. However the forwarders metrics.log reportis successful forwarding and there are no errors in splunkd.log:

• secondary forwarders metrics.log reports connecting to primary forwarder on 9997 and sending events
• primary forwarder metrics.log reports receiving the events on 9997, and reports sending events to central splunk on 9998
• central splunk metrics.log reports receiving events from primary forwarder.

I would like to get these twice-forwarded windows events to appear in the main splunk.

[grahampoulter]

We have worked around the issue by having the primary Universal Forwarder remotely collect WMI logs (wmi.conf) from the other machines on the local network. This does away with second-tier forwarders entirely.

We do however encounter occasional missing events or missing "Message" fields with this setup.

[gr]

Possibly OP was encountering SPL-39592 (see http://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/WorkaroundtoaddWindowseventloginputs... )?

I don't have that problem, I do have WinEventLog:Application and :Security stanzas enabled on the systems running the Universal Forwarder (and, for systems with a Universal Forwarder sending data directly to my indexing cluster, I do receive WinEventLog events).

[gr]

Did anyone ever come to a conclusion on the underlying problem here?

I'm seeing similar behavior under 4.2.2, with a light-weight forwarder as the middle-tier system and universal forwarders as the bottom tier (those two tiers are all Windows; Splunk indexing cluster is Linux).

The middle-tier's inputs.conf:

[udp://514]
sourcetype = syslog
connection_host = dns

[tcp://514]
sourcetype = syslog
connection_host = dns

[splunktcp-ssl://9997]
disabled = 0
connection_host = dns
_TCP_ROUTING = indexCluster

[SSL]
serverCert=$SPLUNK_HOME\etc\auth\server.pem
password=password
rootCA=$SPLUNK_HOME\etc\auth\ca.pem
requireClientCert=false

Middle-tier's outputs.conf:

[tcpout]
maxQueueSize = 1000
forwardedindex.0.whitelist = .*
forwardedindex.1.whitelist = _.*
forwardedindex.filter.disable = false
disabled=false
defaultGroup=indexCluster

[tcpout:indexCluster]
server=10.128.81.31:9997,10.128.81.32:9997
autoLB = true

[tcpout-server://10.128.81.31:9997]
sslCertPath=$SPLUNK_HOME/etc/auth/server.pem
sslPassword=password
sslRootCAPath=$SPLUNK_HOME/etc/auth/ca.pem
sslVerifyServerCert=false
[et cetera]

Second-tier outputs.conf:

[tcpout]
maxQueueSize = 1000
forwardedindex.0.whitelist = .*
forwardedindex.1.whitelist = _.*
forwardedindex.filter.disable = false
disabled=false
defaultGroup=Collector

[tcpout:Collector]
server=middle-tier:9997
autoLB = true

[tcpout-server://middle-tier:9997]
sslCertPath=$SPLUNK_HOME/etc/auth/server.pem
sslPassword=password
sslRootCAPath=$SPLUNK_HOME/etc/auth/ca.pem
sslVerifyServerCert=false

With this configuration, I do see regular file-monitor logs coming through from the second-tier systems (for example, I see metrics.log come to my central _internal index; note that I explicitly whitelisted the _ indexes in my outputs.conf above), but I never see any WinEventLog: events get passed through the forwarder (except those generated by the forwarder itself).

WMI isn't really a viable workaround here, as I expect it wouldn't be for many customers.

Any thoughts, Gerald?

[gkanapathy]

Interesting. It would be useful to have some more information about your configuration. First, what type of forwarders are each kind? I assume the "secondary" source forwarders are UF, but are the "primary" ones LWF, UF, or standard? It would also be helpful to see the outputs.conf from each tier, and the inputs.conf from the middle tier, and whether you are applying any TRANSFORMS to _TCP_ROUTING.

[grahampoulter]

All the forwarders Universal Forwarder 4.2 on Windows Server 2008R2. Only the central destination is a full-fledged Splunk instance.

We are not applying any transforms to _TCP_ROUTING

Monitored file inputs forward fine from source UF->intermediate UF->Splunk

Only Windows Event Log and WMI events fail to appear in Splunk when passed on by an intermediate UF.

Therefore, we have worked around the problem by uninstalling the second-tier forwarders and using WMI on the old "intermediate" UF to pull WMI events from other machines and forward direct to Central Splunk.