Getting Data In

High cpu usage on splunk forwarder

remy06
Contributor

Hi,

I've installed splunk and configured it as a forwarder on one of our windows DC/file server last week and has been experiencing high cpu usage as reported by our administrator..we had to disable splunk services..

I've configured it to send wineventlogs for system,security and application and has no issues with other DC/file servers with the same settings.

Any idea what could be the problem?

Could it be due to the low disk space on the server thats causing it?

Tags (2)
0 Karma
1 Solution

jkerai
Splunk Employee
Splunk Employee

I am assuming that forwarder is running on a windows server. If so, can you please run "procdump -h splunkd.exe -c 80 -n 3 -o splunk c:\dump\splunk". This will create dump files in c:\dump\splunk that you can send to splunk support. This should help developers find the root cause. You can download procdump from http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx.

If this is happening on Linux or similar platform, you can run 'pstack <pid_of_splunkd>' couple of times when the issue is seen and send output to splunk support. you can find pid_of_splunkd by running 'splunk status'. The first pid reported is pid_of_splunkd.

View solution in original post

etwilegar
Engager

Stop blaming the boxes. Splunkd is a background process. You could make the effort to add CPU throttling into your splunkd app. Database servers have internal governors, so could you. Your app is an observer, it should be like a referee and not be seen like this.

cmaier
Explorer

Since this is still the first hit for "forwarder cpu," I'll add my situation and solution as well...

+1 for making sure you are aware of how many files you're monitoring. I was looking at 50GB+ of logs under one folder (~6k total files) and CPU was getting out of hand. Pruned some of the older stuff out (down to ~1600 files) and it's averaging about 3% now.

I have a few systems monitoring single 20GB+ files with no problem, so file size doesn't seem to be much of an issue - at least with those.

LewisWheeler
Communicator

Agreed - Its the quantity of files and the directory structure which makes more of a difference. You can have 50GB of data in 10 files and it'll work fine, if you then have 50GB of data in 5000 files across 50 directories your going to see CPU going high.

0 Karma

t9445
Path Finder

Interesting sidenote: v6.4.1 Windows UF --- if at least one matching directory does not exist, the UF will peg the CPU on the windows server - as soon as you create one that will match it is fine again.

0 Karma

mvierling
Engager

Naive use of '...' can cause CPU problems. The splunkd was using 80 to 90% of the CPU on our Forwarders. After debugging the issue we found that monitor folder traversals looking for new log files is very CPU expensive. We tracked our CPU issue to the following inputs.conf stanza:

[monitor://C:\Windows...LogFiles] disabled = false sourcetype = iis crcSalt =

This replacement fixed our CPU issue:

[monitor://C:\WINDOWS\system32\LogFiles] disabled = false sourcetype = iis crcSalt =

bman2k2k
Engager

had the same problem on linux servers with the 6.0.1 Universal Forwarder with the input
[monitor:///var/.../messages]
disabled = 0
sourcetype = syslog
index = linevents

0 Karma

LewisWheeler
Communicator

This was my issue, I had accidentally used a full tree traversal when I thought I had told Splunk to only search for a particular file under a path. Was killing CPU on my Windows box.

0 Karma

jkerai
Splunk Employee
Splunk Employee

I am assuming that forwarder is running on a windows server. If so, can you please run "procdump -h splunkd.exe -c 80 -n 3 -o splunk c:\dump\splunk". This will create dump files in c:\dump\splunk that you can send to splunk support. This should help developers find the root cause. You can download procdump from http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx.

If this is happening on Linux or similar platform, you can run 'pstack <pid_of_splunkd>' couple of times when the issue is seen and send output to splunk support. you can find pid_of_splunkd by running 'splunk status'. The first pid reported is pid_of_splunkd.

rsanders30
Path Finder

Had the same issue. Would have been nice to have Splunk update the forum with the possible cause found from the dump file.

0 Karma

remy06
Contributor

Hi, I'm using a normal forwarder but I've set Forwarding defaults not to store local copy of forwarded events which shouldn't take up disk space?

0 Karma

jmadi
Splunk Employee
Splunk Employee

remy06 where you ever able to resolve the high CPU issue on your DC?

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

disk space usage won't affect CPU usage, but running a standard (vs light) forwarder certainly will. does the machine have a smaller/slower CPU, or does it forward more data than others?

0 Karma

Joffer
Path Finder

Do you use the LightForwarder or the normal Forwarder. The normal Forwarder can/will index your data and use diskspace etc. It will then take a bigger hit on your system. How much I don't know.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...