Our 6.6.2 search head (linux 2.6.32-573.18.1.el6.x86_64) is constantly low on free swap space,
I tried swapoff -a (clears swap), swapon -a
but after some time, splunk eats up almot 4 gigs of swap. Its also consistently maxing out CPU use
where would bea good place to troubleshoot this? Its not an indexer so not sure why such high CPU use here.
Facing the same high swap usage on SH here. Have you managed to bring down the beast ?
checked any running jobs, have a few searches, but nothing major,
/opt/splunk/bin/splunk list jobs
Asynchronous jobs:
Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5452ecaa3d81bade9_at_1517928000_19793, ttl: 438
Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5452ecaa3d81bade9_at_1517927700_19745, ttl: 139
Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5f91236005de9cb9c_at_1517927400_19669, ttl: 1042
Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5f91236005de9cb9c_at_1517926500_19506, ttl: 141
Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5f91236005de9cb9c_at_1517855400_10230, ttl: 13641
Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5f91236005de9cb9c_at_1517853600_9906, ttl: 11845
Job id: scheduler_bWlrZS5yZWlkZXI__search__RMD5452ecaa3d81bade9_at_1517846400_8549, ttl: 4641
Job id: 1517428177.10568377, ttl: 0