Getting Data In

Forwarder to splunkstorm is timing out?

SeanKilleen
Engager

Hi all,

I did the following:

  • Set up a splunk forwarder
  • Obtained my SplunkStorm Credentials
  • Installed splunk credentials SPL credentials (though I'm not sure that I did this 100% correctly)
  • Edited inputs.conf to add windows system log events
  • Started the forwarder.

In the log, I see several lines of:

Line 272: 10-11-2013 11:53:30.478 -0400 WARN  TcpOutputProc - Raw connection to ip=107.20.29.58:9997 timed out
Line 276: 10-11-2013 11:54:30.479 -0400 WARN  TcpOutputProc - Cooked connection to ip=54.224.46.188:9997 timed out

I ran the command splunk cmd btool outputs list --debug and got the result (sslpassword and project id has been changed from its value to [redacted]:

C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        [tcpout]
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               autoLBFrequency = 30
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               blockOnCloning = true
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               blockWarnThreshold = 100
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               compressed = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               connectionTimeout = 20
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf defaultGroup = storm_indexers
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               disabled = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               dropClonedEventsOnQueueFull = 5
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               dropEventsOnQueueFull = -1
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               forceTimebasedAutoLB = false
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.0.whitelist = .*
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.1.blacklist = _.*
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.2.whitelist = _audit
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.filter.disable = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               heartbeatFrequency = 30
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               indexAndForward = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               maxConnectionsPerIndexer = 2
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               maxFailuresPerInterval = 2
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               maxQueueSize = auto
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               readTimeout = 300
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               secsInFailureInterval = 1
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               sendCookedData = true
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               useACK = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               writeTimeout = 300
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\local\outputs.conf   [tcpout:storm_indexers]
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf autoLB = true
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf server = forwarder.d9bw-e6eh.data.splunkstorm.com:9997
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\local\outputs.conf   sslPassword = [redacted]
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf sslRootCAPath = $SPLUNK_HOME/etc/apps/stormforwarder_[redacted]/ssl/star.splunkstorm.com.chain
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf useACK = true
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        [tcpout]
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               autoLBFrequency = 30
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               blockOnCloning = true
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               blockWarnThreshold = 100
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               compressed = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               connectionTimeout = 20
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf defaultGroup = storm_indexers
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               disabled = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               dropClonedEventsOnQueueFull = 5
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               dropEventsOnQueueFull = -1
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               forceTimebasedAutoLB = false
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.0.whitelist = .*
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.1.blacklist = _.*
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.2.whitelist = _audit
C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\outputs.conf                        forwardedindex.filter.disable = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               heartbeatFrequency = 30
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               indexAndForward = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               maxConnectionsPerIndexer = 2
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               maxFailuresPerInterval = 2
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               maxQueueSize = auto
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               readTimeout = 300
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               secsInFailureInterval = 1
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               sendCookedData = true
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               useACK = false
C:\Program Files\SplunkUniversalForwarder\etc\system\default\outputs.conf                                               writeTimeout = 300
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\local\outputs.conf   [tcpout:storm_indexers]
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf autoLB = true
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf server = forwarder.d9bw-e6eh.data.splunkstorm.com:9997
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\local\outputs.conf   sslPassword = [redacted]
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf sslRootCAPath = $SPLUNK_HOME/etc/apps/stormforwarder_[redacted]/ssl/star.splunkstorm.com.chain
C:\Program Files\SplunkUniversalForwarder\etc\apps\stormforwarder_[redacted]\default\outputs.conf useACK = true
0 Karma
1 Solution

yannK
Splunk Employee
Splunk Employee

The timeout is simply caused when the forwarder rotating across the ip of the dns load balancer.
Or if a maintenance if happening.

View solution in original post

akoeplinger
New Member

I'm seeing exactly the same behavior, my Windows event logs and Perfmon data isn't showing up in Splunk Storm but a normal text-based logfile does. I've already opened a ticket a couple days ago, but got no response yet.

0 Karma

akoeplinger
New Member

@pbradfordkc: Windows Event Logs don't work yet when using Universal Forwarder 6 with Storm, see http://answers.splunk.com/answers/123027/splunk-storm-universal-forwarder

0 Karma

pbradfordkc
Explorer

I am running into the same issue. I am seeing text based logs but no windows event logs?

Does splunk storm support windows event logs. I thought it was because i was running server 2012 r2 but 2012 r2 is now supported with 6.1. Is there any trouble shooting steps out there?

0 Karma

yannK
Splunk Employee
Splunk Employee

The timeout is simply caused when the forwarder rotating across the ip of the dns load balancer.
Or if a maintenance if happening.

yannK
Splunk Employee
Splunk Employee

the backlog effect does not produce the same messages, it does like "cannot sent dagta to the output queue, parsing queue full".

0 Karma

yannK
Splunk Employee
Splunk Employee

yes, please open a ticket from the storm portal (help page), and authorize the support team to check your project.

0 Karma

SeanKilleen
Engager

Related: I guess I'm also asking, Should I start another question based on the fact that I still can't seem to receive any log entries into SplunkStorm from splunk despite it not showing any errors in the logs besides the TcpOutputProc messages?

0 Karma

SeanKilleen
Engager

Thanks. So I shouldn't be worried that I see hundreds of those entries in splunkd.log? And one last follow-up: I also see no entries being uploaded to splunk, but could this be because it is processing the first giant backlog of windows event log events?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...