Getting Data In

can't receive events from fowarders

dbutch1976
Explorer

Hello,

I have a brand new install of a splunk indexer and several clients running forwarders. To install the clients I used the following command:

msiexec.exe /i \\dc1.butcher.local\Splunkd\splunk-4.2.1-98164-x86-release.msi SPLUNK_APP="SplunkLightForwarder" FORWARD_SERVER="splunk.domain.local:9997" RBG_LOGON_INFO_USER_CONTEXT=2 IS_NET_API_LOGON_USERNAME="Domain\splunk-svc" IS_NET_API_LOGON_PASSWORD="Password1" WINEVENTLOGAPPCHECK=0 WINEVENTLOGSECCHECK=1 WINEVENTLOGSYSCHECK=1 WINEVENTLOGFWDCHECK=1 WINEVENTLOGSETCHECK=1 /quiet

The installer completes with no errors and I can see splunk running in services.

The account I'm using (Domain\splunk-svc)has all the required permissions to run splunk.
I have installed splunk on the server I would like to use as my indexer/search head. (splunk.domain.local)
I have configured splunk.domain.local to receive on port 9997. There is no firewall blocking this port and I can telnet to port 9997 from any of the clients.

When I go to Search there are simply no events to display and it does not see any hosts. I am able to add data by logging into the web page and adding remote data sources, but I know this should not be required and it's not the way I want to roll out splunk to my domain.

Here are excerpts of my splunkd log on the client machines:

05-06-2011 10:39:15.559 -0400 WARN  IndexProcessor - received event for unconfigured/disabled index='_audit' with source='source::audittrail' host='host::EXCH1' sourcetype='sourcetype::audittrail' (1 missing total)
05-06-2011 10:39:15.559 -0400 WARN  pipeline - Empty pipeline (no processors): scheduler, exiting pipeline
05-06-2011 10:39:15.559 -0400 INFO  loader - Server supporting SSL v2/v3
05-06-2011 10:39:15.559 -0400 INFO  loader - Using cipher suite ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
05-06-2011 10:39:15.574 -0400 INFO  TPool - initializing BatchReaderTPool with 1 workers
05-06-2011 10:39:15.731 -0400 INFO  TailingProcessor - TailWatcher initializing...
05-06-2011 10:39:15.731 -0400 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME\var\spool\splunk.
05-06-2011 10:39:15.746 -0400 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME\var\spool\splunk\...stash_new.
05-06-2011 10:39:15.746 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\etc\splunk.version.
05-06-2011 10:39:15.746 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\splunk.
05-06-2011 10:39:15.746 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\splunk\metrics.log.
05-06-2011 10:39:15.746 -0400 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME\var\log\splunk\splunkd.log.
05-06-2011 10:39:15.746 -0400 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
05-06-2011 10:39:15.902 -0400 INFO  WatchedFile - Will begin reading at offset=1762894 for file='C:\Program Files\Splunk\var\log\splunk\metrics.log'.
05-06-2011 10:39:15.996 -0400 INFO  WatchedFile - Will begin reading at offset=24996713 for file='C:\Program Files\Splunk\var\log\splunk\metrics.log.1'.
05-06-2011 10:39:16.621 -0400 INFO  WatchedFile - Will begin reading at offset=635848 for file='C:\Program Files\Splunk\var\log\splunk\splunkd.bak'.
05-06-2011 10:39:16.762 -0400 INFO  TcpOutputProc - Connected to idx=192.168.1.117:9997

On the server side the metrics log says:

05-06-2011 10:44:36.010 -0400 INFO  StatusMgr - destPort=9997, eventType=connect_done, sourceHost=192.168.1.111, sourceIp=192.168.1.111, sourcePort=2428, statusee=TcpInputProcessor
05-06-2011 10:44:36.010 -0400 INFO  StatusMgr - sourcePort=9997, ssl=false, statusee=TcpInputProcessor
05-06-2011 10:44:41.728 -0400 INFO  StatusMgr - destPort=9997, eventType=connect_close, sourceHost=192.168.1.111, sourceIp=192.168.1.111, sourcePort=2428, statusee=TcpInputProcessor
05-06-2011 10:44:45.603 -0400 INFO  StatusMgr - destPort=9997, eventType=connect_done, sourceHost=192.168.1.136, sourceIp=192.168.1.136, sourcePort=10659, statusee=TcpInputProcessor
05-06-2011 10:44:45.603 -0400 INFO  StatusMgr - sourcePort=9997, ssl=false, statusee=TcpInputProcessor92

I can see from this log that my client machine (192.168.1.136) has clearly connected, and yet I can't see any events!

Can anyone tell me what step I've missed????

`

92

Tags (1)
0 Karma

dbutch1976
Explorer

Thanks, that's got me working again. It's unfortunate that there's no backward compatibility because I had been planning to roll this out across my network when the splunk forwarders I was installing stopped doing anything after I began to reference teh new version. This has taken me a week to figure out so I appreciate your help. For anyone else installing splunkforwarders via Msiexec they may want to note the syntax change:

msiexec.exe /i \dc1.butcher.localSplunkdsplunk-4.2.1-98164-x86-release.msi SPLUNK_APP="SplunkLightForwarder" FORWARD_SERVER="splunk.domain.local:9997" RBG_LOGON_INFO_USER_CONTEXT=2 IS_NET_API_LOGON_USERNAME="Domainsplunk-svc" IS_NET_API_LOGON_PASSWORD="Password1" WINEVENTLOG_APP_ENABLE=1 WINEVENTLOG_SEC_ENABLE=1 WINEVENTLOG_SYS_ENABLE=1 /quiet

Since I'm already rolling out with this method I'm very curious to know why you don't recommend rolling out this way?

0 Karma

MarioM
Motivator

what i mean is that in your forwarder installation it should have an inputs.conf with the following(if not you can create one in splunk\etc\system\local\inputs.conf):

[WinEventLog:Security]
disabled = false
start_from = oldest
current_only = 1
checkpointInterval = 5

[WinEventLog:Application]
disabled = false
start_from = oldest
current_only = 1
checkpointInterval = 5

[WinEventLog:System]
disabled = false
start_from = oldest
current_only = 1
checkpointInterval = 5
0 Karma

dbutch1976
Explorer

Sorry perhaps I'm not understanding the question, you're asking which inputs have been configured. My goal is to forward Windows Security, Application, and System logs. Isn't that what's been configured by the install syntax?

msiexec.exe /i \dc1.butcher.local\Splunkd\splunk-4.2.1-98164-x86-release.msi SPLUNK_APP="SplunkLightForwarder" FORWARD_SERVER="splunk.domain.local:9997" RBG_LOGON_INFO_USER_CONTEXT=2 IS_NET_API_LOGON_USERNAME="Domain\splunk-svc" IS_NET_API_LOGON_PASSWORD="Password1" WINEVENTLOGAPPCHECK=0 WINEVENTLOGSECCHECK=1 WINEVENTLOGSYSCHECK=1 WINEVENTLOGFWDCHECK=1 WINEVENTLOGSETCHECK=1 /quiet

I believe what you're saying is that the forwarder is forwarding events however the particular events will not show up in a default search. How do I make these events appear?

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

No, those flags don't work in version 4.2.x (and did not work in 4.1.x). They don't do anything, and I recommend not using MSI or CLI flags to set apps, forwarders, or inputs anyway. Nevertheless, the valid flags for 4.2.x are here: http://www.splunk.com/base/Documentation/latest/Installation/InstallonWindowsviathecommandline

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

What inputs have you configured? The forwarder log indicates that it's sending internal logs, but those will show up in the _internal index, which isn't searched by a default search.

0 Karma

MarioM
Motivator

do you see any inputs.conf with configuration in your LWF installation?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...