Table of Contents
June 7, 2024
|9m
How to Fix Linux Kernel Transparent Huge Pages Warning in Splunk?
If you're running Splunk Enterprise on a Linux server such as Red Hat, CentOS, or Ubuntu, you may have encountered the Linux Kernel Transparent Huge Pages (THP) warning. This warning is specific to Linux deployments of Splunk and can significantly impact the performance of your Splunk instance.
Transparent Huge Pages is an advanced memory management scheme in the Linux kernel that allows the Memory Management Units (MMUs) to work with huge memory pages, which can help manage large amounts of memory more efficiently. However, THP has been associated with degradation of Splunk Enterprise performance in certain Linux kernel versions.
When enabled, THP can be overly aggressive in coalescing memory pages for short-lived processes, such as many Splunk searches. It can also prevent the jemalloc memory allocation implementation from releasing memory back to the operating system after use, leading to performance issues. Splunk has observed a minimum of a 30% degradation in indexing and search performance on Linux systems where THP is active, with a similar percentage increase in latency. In this blog post, we will explore how to identify and fix the Linux Kernel Transparent Huge Pages warning on your Linux deployment of Splunk Enterprise server.
What is Transparent Huge Pages (THP)?
Transparent Huge Pages (THP) is a memory management feature introduced in the Linux kernel to improve system performance by reducing the overhead of memory management. It works by allocating memory in larger chunks, called "huge pages," which can range in size from 2MB to 1GB, depending on the system's architecture and configuration.
The primary goal of THP is to reduce the number of page table entries required to map virtual memory to physical memory. By using larger page sizes, the system can map more memory with fewer page table entries, which can lead to faster memory access and reduced memory management overhead.
Why is THP Implemented?
THP is implemented in the Linux kernel to address some of the performance challenges associated with managing large amounts of memory. As systems continue to grow in size and complexity, the overhead of memory management can become a significant bottleneck, especially for applications that require frequent memory allocation and deallocation.
By using larger page sizes, THP can help reduce the number of page faults and improve memory access times. This can be particularly beneficial for applications that have large memory footprints, such as databases, scientific simulations, and big data processing frameworks.
In addition to improving performance, THP can also help reduce memory fragmentation. When memory is allocated in smaller chunks, it can become fragmented over time, leading to wasted space and reduced performance. By allocating memory in larger chunks, THP can help reduce fragmentation and improve overall system efficiency.
Is THP a Curse or a Blessing?
While THP can provide significant performance benefits in some scenarios, it can also be a curse for certain applications, particularly those that are sensitive to memory allocation and deallocation patterns. Splunk Enterprise is one such application that has been known to suffer from performance degradation when THP is enabled.
The problem with THP in the context of Splunk is that it can be overly aggressive in coalescing memory pages for short-lived processes, such as many Splunk searches. This can lead to increased memory usage and reduced performance, as Splunk may not be able to efficiently allocate and deallocate memory as needed.
Moreover, THP can also interfere with the jemalloc memory allocator used by Splunk, preventing it from releasing memory back to the operating system after use. This can result in memory leaks and further performance degradation over time.
In summary, while THP can be a blessing for some applications, it can be a curse for others, particularly those that rely on efficient memory management and allocation patterns. In the case of Splunk Enterprise, it is generally recommended to disable THP to ensure optimal performance and stability.
How to Disable Transparent Huge Pages in Linux?
Whatever may be the reason: to fix the warning message in your Splunk Health Check or to increase the performance of search results in Splunk Enterprise, Splunk recommend to disable the THP on your Linux server. Well, you can disable it temporary or for forever.
To disable Transparent Huge Pages (THP) on your Linux system running Splunk Enterprise, follow these step-by-step instructions:
Step 1: Check the current status of THP
Before making any changes, it's a good idea to check the current status of THP on your system. You can do this by running the following commands:
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defragIf THP is enabled, the output of these commands will show either [always] madvise never or always [madvise] never.
If THP is disabled, the output will show always madvise [never].
Step 2: Disable THP temporarily
If you want to try disabling THP temporarily, you can run the following commands:
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defragThese commands will set the enabled and defrag files to never, effectively disabling THP. However, this change will not persist across system reboots.
Step 3: Disable THP permanently
To disable THP permanently, you need to create a script that runs at boot time. Here's how to do it:
1. Create a new file called disable-thp.sh in the /etc/init.d directory.
sudo nano /etc/init.d/disable-thp.sh2. Copy and paste the following contents into the file:
#!/bin/bash
### BEGIN INIT INFO
# Provides: disable-thp
# Required-Start: $local_fs
# Required-Stop:
# X-Start-Before: couchbase-server
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Disable THP
# Description: Disables transparent huge pages (THP) on boot, to improve
# Couchbase performance.
### END INIT INFO
case $1 in
start)
if [ -d /sys/kernel/mm/transparent_hugepage ]; then
thp_path=/sys/kernel/mm/transparent_hugepage
elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
thp_path=/sys/kernel/mm/redhat_transparent_hugepage
else
return 0
fi
echo 'never' > ${thp_path}/enabled
echo 'never' > ${thp_path}/defrag
re='^[0-1]+$'
if [[ $(cat ${thp_path}/khugepaged/defrag) =~ $re ]]
then
# RHEL 7
echo 0 > ${thp_path}/khugepaged/defrag
else
# RHEL 6
echo 'no' > ${thp_path}/khugepaged/defrag
fi
unset re
unset thp_path
;;
esacThis script will disable THP on boot.
3. Save the file and exit the text editor.
4. Make the script executable by running the following command:
sudo chmod 755 /etc/init.d/disable-thp.sh5. Enable the script to run at boot time by running the following command:
sudo update-rc.d disable-thp.sh defaultsThis command will create the necessary symlinks to run the script at boot time.
Step 4: Reboot your system
To apply the changes and ensure that THP is disabled permanently, reboot your Linux system:
sudo rebootStep 5: Verify that THP is disabled
After your system has rebooted, you can verify that THP is disabled by running the following commands:
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defragThe output should show always madvise [never], indicating that THP is disabled.
Additionally, you can check the Splunk logs to confirm that THP is disabled. Look for a message similar to the following in the splunkd.log file:
Linux transparent huge pages support, enabled="never" defrag="never"This message indicates that Splunk has detected that THP is disabled on your system.
By following these steps, you can ensure that Transparent Huge Pages are disabled on your Linux system running Splunk Enterprise, which will help optimize Splunk's performance and avoid potential issues related to memory management.
How to Enable Transparent Huge Pages in Linux?
If you've disabled Transparent Huge Pages (THP) on your Linux system and want to re-enable it for any reasons, follow these step-by-step instructions:
Step 1: Remove the THP disable script
If you followed the steps to permanently disable THP, you'll need to remove the disable-thp.sh script from the /etc/init.d directory. To do this, run the following command:
sudo rm /etc/init.d/disable-thp.shStep 2: Remove the script from the boot sequence
Next, remove the script from the boot sequence by running the following command:
sudo update-rc.d -f disable-thp.sh removeThis command will remove the symlinks created earlier to run the script at boot time.
Step 3: Enable THP temporarily
To enable THP temporarily, you can run the following commands:
echo always > /sys/kernel/mm/transparent_hugepage/enabled
echo always > /sys/kernel/mm/transparent_hugepage/defragThese commands will set the enabled and defrag files to always, effectively enabling THP. However, this change will not persist across system reboots.
Step 4: Enable THP permanently
To enable THP permanently, you can modify the GRUB configuration file. Here's how to do it:
1. Open the GRUB configuration file in a text editor:
sudo nano /etc/default/grub2. Locate the line that starts with GRUB_CMDLINE_LINUX_DEFAULT and add the following parameters to the end of the line:
transparent_hugepage=alwaysFor example, the line might look like this after the modification:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash transparent_hugepage=always"3. Save the file and exit the text editor.
4. Update the GRUB configuration by running the following command:
sudo update-grubThis command will regenerate the GRUB configuration file with the new parameters.
Step 5: Reboot your system
To apply the changes and ensure that THP is enabled permanently, reboot your Linux system:
sudo rebootStep 6: Verify that THP is enabled
After your system has rebooted, you can verify that THP is enabled by running the following commands:
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defragThe output should show [always] madvise never, indicating that THP is enabled.
Keep in mind that enabling THP may impact the performance of Splunk Enterprise and other applications that rely on efficient memory management. If you experience performance issues after enabling THP, you may want to consider disabling it again using the steps outlined in the previous section.
In this tutorial, we've explored the concept of Transparent Huge Pages (THP) in the Linux kernel and its potential impact on Splunk Enterprise performance. We've seen how THP can be a double-edged sword, providing benefits in some scenarios while causing performance degradation in others, particularly for applications like Splunk that rely on efficient memory management.
By following the step-by-step instructions provided, you can easily disable THP on your Linux system running Splunk Enterprise, ensuring optimal performance and stability. Remember to verify that THP is disabled after rebooting your system and keep an eye on your Splunk logs to confirm that the change has taken effect.
That's all for now, we will cover more informative topic about the Splunk in the up coming articles. Please keep visiting thesecmaster.com for more such technical information. Visit our social media page on Facebook, Instagram, LinkedIn, Twitter, Telegram, Tumblr, & Medium and subscribe to receive information like this.
Arun KL
Arun KL is a cybersecurity professional with 15+ years of experience in IT infrastructure, cloud security, vulnerability management, Penetration Testing, security operations, and incident response. He is adept at designing and implementing robust security solutions to safeguard systems and data. Arun holds multiple industry certifications including CCNA, CCNA Security, RHCE, CEH, and AWS Security.
