Getting Data In

How to troubleshoot why my heavy forwarder is unable to keep up with processing log data?

ronj_clark
Explorer

I have a heavy forwarder running on a RHEL 6 server that has 16 processors and 16GB. This heavy forwarder has usually kept up with all of the logs that were sent to it, but a few months ago, I am pretty sure I overwhelmed it. Now, I have moved all of the extra logs off of this server to another server and I am back to the original set of logs that I started with. However, it will not keep up.

The logs are Cisco ASA firewall logs, WSA logs, and some other small volume syslogs. In the inputs.conf file, the WSA logs are set to "batch" mode and all of the rest are in monitor mode.

inputs.conf:

####  WSA Logs  ####
[batch:///logs/sawmill]
disabled = 0
# followTail = 0
index = wsa
initCrcLength = 1024
sourcetype = cisco:wsa:squid
whitelist = aclog.*
move_policy = sinkhole
crcSalt = <SOURCE>

Any one having this same issue? Any help is appreciated. I can post snippets of conf files if it will help.

Thank you in advance,
Ron

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Your box is still receiving the other logs? Its a bit hard to trouble shoot this without more information. Here are some typical things to start looking at though:

1) System resource utilization [ CPU / Memory / HD (I/o ] - Any of these constrained?
2) Look for errors in _internal, turn up debuging level -- https://wiki.splunk.com/Community:Troubleshooting_Monitor_Inputs
3) Have you enable input parallelization?

You should be able to open a support ticket also and send them a diag, this might provide more information to Support..

0 Karma

ronj_clark
Explorer

esix,

It is version 6.1.1. Yes the box is still receiving other logs.

  1. system resource utilization is what I would call normal. 2 of the 16 procs are over 50%, but not above 75%. all other procs are under 40%. Less than half of the 16GB RAM is being used.
  2. The only errors in the logs are some time parsing errors, but they are intermittent. Most of the logs are just entries of this box connecting to my index cluster to deliver logs.
  3. No, I have not enabled input parallelization. I will look this up.

I cannot open a ticket with splunk as we do not have support contract.

Thanks!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...