Deployment Architecture

Splunk 6.x and Linux? not so fast?

abonuccelli_spl
Splunk Employee
Splunk Employee

New 6.0.x/6.1.x installation and both Indexer and Search Head seem to have latency and not performing as expected!

CPU/Mem/IOPS requirements all met - What's wrong?

1 Solution

abonuccelli_spl
Splunk Employee
Splunk Employee

Read this
http://docs.splunk.com/Documentation/Splunk/6.1.1/ReleaseNotes/SplunkandTHP

Check if your OS is affected:

cat /sys/kernel/mm/transparent_hugepage/enabled 
[always] madvise never

Then fix on OS level:

echo never > /sys/kernel/mm/transparent_hugepage/enabled

Validate on OS level:

cat /sys/kernel/mm/transparent_hugepage/enabled 
always madvise [never]

Please note the above instructions may vary from distro to distro.
This setting will need to persist at each restart of the box of course.

After a restart of Splunk, you can validate the setting as seen by Splunkd process by checking $SPLUNK_HOME/var/log/splunk/splunkd.log

bad

06-17-2014 12:08:05.854 +0200 INFO  ulimit - Linux transparent hugetables support, enabled="always" defrag="always"

good

   06-17-2014 14:03:05.991 +0200 INFO  ulimit - Linux transparent hugetables support, enabled="never" defrag="never"

You should then notice a substantial increase in performance.
This is required on Search Heads and Indexer version 6.x and greater on Linux only.

View solution in original post

anilyelmar
Explorer

If you are setting transparent_hugepage=never in grub.conf (disable THP at boot time) then
"Linux transparent hugetables support, enabled="never" defrag="always"" in splunkd.log is also a GOOD condition.
Please see - https://access.redhat.com/solutions/46111
and section below. Splunk must reword their logging message
NOTE: Some third party application install scripts check value of above files and complain even if THP is disabled at boot time using transparent_hugepage=never, this is due to the fact when THP is disabled at boot time, the value of /sys/kernel/mm/redhat_transparent_hugepage/defrag will not be changed, however this is expected and system will never go in THP defragmentation code path when it is disabled at boot and THP defrag need not to be disabled separately.

abonuccelli_spl
Splunk Employee
Splunk Employee

Read this
http://docs.splunk.com/Documentation/Splunk/6.1.1/ReleaseNotes/SplunkandTHP

Check if your OS is affected:

cat /sys/kernel/mm/transparent_hugepage/enabled 
[always] madvise never

Then fix on OS level:

echo never > /sys/kernel/mm/transparent_hugepage/enabled

Validate on OS level:

cat /sys/kernel/mm/transparent_hugepage/enabled 
always madvise [never]

Please note the above instructions may vary from distro to distro.
This setting will need to persist at each restart of the box of course.

After a restart of Splunk, you can validate the setting as seen by Splunkd process by checking $SPLUNK_HOME/var/log/splunk/splunkd.log

bad

06-17-2014 12:08:05.854 +0200 INFO  ulimit - Linux transparent hugetables support, enabled="always" defrag="always"

good

   06-17-2014 14:03:05.991 +0200 INFO  ulimit - Linux transparent hugetables support, enabled="never" defrag="never"

You should then notice a substantial increase in performance.
This is required on Search Heads and Indexer version 6.x and greater on Linux only.

jenipherc
Splunk Employee
Splunk Employee

What if the log says Linux transparent hugetables support, enabled="always" defrag="never" ? Would it be still alright (in between bad and good?)

0 Karma

dwaddle
SplunkTrust
SplunkTrust

Nope. The correct configuration is never/never

0 Karma

Glenn
Builder

Sorry, that's not always correct dwaddle. And Splunk is monitoring this incorrectly for cases where hugepages was disabled since boot. The issue is addressed in the RedHat doc itself (https://access.redhat.com/solutions/46111 "To disable THP at run time" 2nd Note). If hugepages is disabled in grub.conf at boot time (as per the RedHat instructions), then defrag is not set and so may still show "always". In fact hugepages defrag is never used if hugepages have been disabled since boot time. All you need to monitor for, in the common case where hugepages are disabled at boot, is enabled != "never".

dwaddle
SplunkTrust
SplunkTrust

Hey Glenn, agreed. If THP is disabled, then defrag on or off is meaningless. Splunk's health check code is misreporting the combination of enabled=never, defrag=always as an issue when it is not, and there is a bug opened to resolve that. Good catch!

0 Karma

ppeterson
Path Finder

As rc.local runs these commands after Splunk starts I put this in init.d so upon restart Splunk logs the correct/current status on reboot.

0 Karma

alanwill
Explorer

Actually I think I see what's missing, in addition to:

echo never > /sys/kernel/mm/transparent_hugepage/enabled

we also need to run:

echo never > /sys/kernel/mm/transparent_hugepage/defrag

Also, I have to add these commands to /etc/rc.local so that the settings persist across reboots.

0 Karma

alanwill
Explorer

What does defrag="always" mean? After disabling THP that value still remains as "always"

10-13-2014 21:51:56.613 +0000 INFO ulimit - Linux transparent hugetables support, enabled="never" defrag="always"

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...