Getting Data In

Why does an AIX 6.5.2 forwarder have high swap/memory and cpu consumption?

ddrillic
Ultra Champion

We see the following -

sh-4.2$ ps avwx | head -1; ps avwx | sort +4n -r | head -10
      PID    TTY STAT  TIME PGIN  SIZE   RSS   LIM  TSIZ   TRS %CPU %MEM COMMAND
  7274610      - A    51121:15 3427531 1739848 749560    xx 100303  9692  1.0  8.0 splunkd -p 8089 start

What can it be?

Tags (2)
0 Karma
1 Solution

gjanders
SplunkTrust
SplunkTrust

I suspect you are misinterpreting the stats if the question is correct, I would suggest you use svmon in AIX to accurately determine the memory in use.
Reading your question you appear to be using 1% CPU.

Here's how to measure memory use in AIX:
svmon -P 7274610 -O unit=MB

I checked two production forwarders, a 6.5.2 instance was:
Pid Command Inuse Pin Pgsp Virtual
22937622 splunkd 1570.29 39.6 0 300.84

Another instance I checked 7.0.0:
Pid Command Inuse Pin Pgsp Virtual
57540776 splunkd 1185.07 256.23 4.55 558.29

Both show 1% CPU in the ps command, you might like to open topas in AIX and see if you are seeing high CPU by Splunk.

If you are looking for a more comprehensive monitoring solution I use the Nmon application for Splunk on AIX servers (and Linux) , the official Splunk add on for unix is here and the app is here

View solution in original post

0 Karma

gjanders
SplunkTrust
SplunkTrust

I suspect you are misinterpreting the stats if the question is correct, I would suggest you use svmon in AIX to accurately determine the memory in use.
Reading your question you appear to be using 1% CPU.

Here's how to measure memory use in AIX:
svmon -P 7274610 -O unit=MB

I checked two production forwarders, a 6.5.2 instance was:
Pid Command Inuse Pin Pgsp Virtual
22937622 splunkd 1570.29 39.6 0 300.84

Another instance I checked 7.0.0:
Pid Command Inuse Pin Pgsp Virtual
57540776 splunkd 1185.07 256.23 4.55 558.29

Both show 1% CPU in the ps command, you might like to open topas in AIX and see if you are seeing high CPU by Splunk.

If you are looking for a more comprehensive monitoring solution I use the Nmon application for Splunk on AIX servers (and Linux) , the official Splunk add on for unix is here and the app is here

0 Karma

MuS
SplunkTrust
SplunkTrust

Try accessing this REST endpoint on your UF https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus to see how may files are being monitored. High numbers of monitored files can cause such behaviour ...

ddrillic
Ultra Champion

@MuS - only two files are being monitored ...

0 Karma

MuS
SplunkTrust
SplunkTrust

How many directories needs to be scanned by the UF to reach those two files? Also can you try truss the process and see what it actually does?

0 Karma

ddrillic
Ultra Champion

Barely five directories and explicit two files to monitor ; - ) maybe an AIX specific issue?

0 Karma

MuS
SplunkTrust
SplunkTrust

Actually looking at the numbers are 1% CPU and 8% memory usage really that high? Does vmstat provide some hints where the potential bottleneck could be?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...