How to Fix GameOver(lay)- Two Local Privilege Escalation Vulnerabilities in Ubuntu Linux Kernel?

Sagi Tzadik, Shir Tamari from Wiz disclosed two easy-to-exploit local privilege escalation vulnerabilities in the OverlayFS module of the Ubuntu Linux Kernel. These two vulnerabilities assigned with the identifiers CVE-2023-2640 and CVE-2023-32629 could be severe than it appears as no configuration changes are needed to exploit these vulnerabilities. These issues are unique, although they look similar to the two-year-old Overlayfs vulnerability (CVE-2021-3493).  Let’s dive deep into the issue and understand how these vulnerabilities are unique, although they share the same functionality in terms of exploitation.

Before going to the technical details, it’s good to know about the OverlayFS that has existed since 2009 as part of the Linux Kernel.

What is OverlayFS and How Does it Work?

OverlayFS is a union mount filesystem implementation for Linux that allows combining multiple directories and mount points into one merged directory. It was initially proposed in 2010 and merged into the Linux kernel in version 3.18 in 2014.

OverlayFS allows combining multiple directory trees and mount points into one merged view by overlaying a writable filesystem on top of a read-only base filesystem.

This simple example shows how OverlayFS is being used to overlay a read-write upper directory over a read-only lower directory. Here is an explanation of this scenario:

Image Source: embeddedcomputing.com

The lower directory contains files and folders and is mounted read-only. The upper directory is a separate directory mounted read-write. Let’s see how OverlayFS works in a real-world scenario.

When OverlayFS is used to merge these two directories, it creates a single unified directory structure with the combined contents of both. Any files that exist in both the upper and lower directories will be accessed from the upper directory when overlayed.

Now any writes to files will happen in the upper directory and not modify the lower read-only directory. If a file exists only in the lower directory, it will be copied up to the upper directory on first write access through the overlay.

Deletions are handled by OverlayFS, creating “whiteout” files in the upper directory to mask files that exist only in the lower directory. This makes it appear deleted in the overlay view.

So the key benefit is that the lower directory remains unchanged and read-only, while the upper directory provides a writable layer for modifications. The overlay merge allows presenting a single unified view of the two directories together.

A Real World Scenario- Docker Container

In Docker, container images start from a base “layer,” which contains the root filesystem with all the binaries, libraries, and files needed to run the container. This base layer is read-only and immutable.

When a container is started from the image, Docker uses OverlayFS to overlay a writable “container layer” on top of the base layer. This writable layer captures any changes made while the container is running, such as writing new files, modifying existing files, or deleting files.

The OverlayFS combines the base layer and container layer into one unified view so the running container sees it as a single filesystem. But under the hood, the underlying base image remains unchanged in its original read-only state.

When the container is deleted, the writable container layer is discarded, leaving the base image untouched. This helps optimize storage usage with multiple containers from the same image. The overlay allows each container to have a private writable layer without needing a full copy of the base image storage for every container.

OverlayFS is a key part of what enables Docker’s copy-on-write architecture. All modifications by running containers are captured in disposable overlay layers. This leaves the underlying base image pristine and immutable, allowing containers to share common base image layers. OverlayFS is an integral filesystem technology for efficient storage and image management in Docker.

What are the Two Local Privilege Escalation Vulnerabilities in Ubuntu Linux Kernel?

Local Privilege Escalation Vulnerabilities in Ubuntu Linux Kernel (Image source: wiz.io)

GameOver(lay) refers to two vulnerabilities: CVE-2023-2640 and CVE-2023-32629. Both are local privilege escalation vulnerabilities in the Ubuntu Linux kernel’s OverlayFS module. They arise from Ubuntu’s modifications to OverlayFS that allow unintended handling of file capabilities under certain conditions, enabling unprivileged users to gain root privileges. Both issues can be exploited locally to escalate privileges by leveraging OverlayFS behavior related to copying file metadata and extended attributes. Wait, you have read all the similarities. That doesn’t mean that they are same. Both vulnerabilities are unique, although they look similar. Before we go into the differences, let’s go little bit of deep into technical things to understand both of them.

CVE-2023-2640

CVE-2023-2640 (Image source: wiz.io)

The vulnerability tracked as CVE-2023-2640 arises due to the OverlayFS module in certain Ubuntu kernels calling the ovl_do_setxattr function to copy extended attributes when files are modified. This function directly calls __vfs_setxattr_noperm, which skips performing permission checks and namespace conversions when setting file capabilities. As a result, privileged file capabilities can be copied from lower directories to upper directories in an OverlayFS mount. This enables crafting executables with capabilities that grant root privileges when executed by an unprivileged user. Exploiting this requires establishing a user namespace and mounting the OverlayFS filesystem, making remote attacks unlikely.

CVE-2023-32629

The vulnerability CVE-2023-32629 also stems from insufficient permission checking in Ubuntu’s modifications to the OverlayFS module. Specifically, the ovl_copy_up_meta_inode_data function uses ovl_do_setxattr to copy file capabilities when an OverlayFS filesystem is mounted with the metacopy flag enabled. This results in file capabilities being copied to the upper directory when only metadata changes occur without proper namespace conversion. By creating a skeleton file with capabilities in the upper directory and then modifying metadata, the full privileged executable can be reconstructed in the overlayed view. This flaw enables privilege escalation from low-privilege users by leveraging OverlayFS behavior.

What Makes the Vulnerabilities Different from One Another?

While CVE-2023-2640 and CVE-2023-32629 are similar in that they are local privilege escalation vulnerabilities stemming from insufficient permission checking in Ubuntu’s modifications to OverlayFS, there are some key differences:

Both lead to privilege escalation via mishandling file capabilities in OverlayFS. The specific mechanisms differ in terms of the functions involved, triggering conditions, and prerequisites for exploitation. But the end result is ultimately the same – gaining root access from an unprivileged local user.

Kernel Versions Vulnerable to CVE-2023-2640 and CVE-2023-32629

Researchers wrote in their blog that they accidentally caught these vulnerabilities when Ubuntu made the changes in the Kernel to patch CVE-2023-0386. In their research, they found that these vulnerabilities, CVE-2023-2640 and CVE-2023-32629, affect only Ubuntu distributions and not other Linux distributions because they originate from specific changes made by Ubuntu to the OverlayFS module in their kernel code.

In 2018, Ubuntu modified its kernel to use an internal function __vfs_setxattr_noperm instead of the standard vfs_setxattr function for setting extended attributes when using OverlayFS. Later in 2020, Linux added security mitigations to vfs_setxattr to prevent privilege escalation attacks related to OverlayFS capabilities. However, since Ubuntu was calling __vfs_setxattr_noperm due to its earlier code change, it remained vulnerable even after the upstream Linux fix.

The __vfs_setxattr_noperm function skips performing permission checks and namespace conversions when copying file capabilities between OverlayFS layers. Whereas vfs_setxattr properly restricts capabilities before copying to upper layers. This difference due to Ubuntu’s modifications is the root cause of the two privilege escalation flaws.

Later additional code changes made only to Ubuntu’s OverlayFS implementation introduced a second vulnerable path with the same underlying issue. These Ubuntu-specific changes resulted in the flaws not being present in the upstream Linux kernel or other Linux distributions.

They also left a note that research is still underway. There could be a new update in the affected version as they progress in the research.

Kernel Versions Vulnerable to CVE-2023-2640 and CVE-2023-32629 (Image source: wiz.io)

How to Test the GameOver(lay) Vulnerabilities?

Initially you can check the Kernel and distribution versions to ensure that your machines are running the versions marked as vulnerable. If you are really interested in testing the flaw, you can use the exploit developed to test the CVE-2021-3493 vulnerability. The exploit is available for public in GitHub. Download the exploit and execute as a non root user.

Step 1: To check the version of the kernel running on your Ubuntu server. Open the terminal and issue this command to check the version of Ubuntu:

$ cat /etc/os-release

in our case, we are running 20.04 LTS which is vulnerable to Ubuntu overlayfs vulnerability.

Step 2: Download the exploit from the git using the wget command.

$ wget https://github.com/briskets/CVE-2021-3493/blob/main/exploit.c

Step 3: Change the directory to Desktop.

$ cd Desktop/CVE-2021-3493-main/

Step 4: Set executable permission for ‘exploit.c‘ using ‘chmod’ command:Enter the root password to execute the command.

$ sudo chmod +x exploit.c

First compel the ‘exploit.c’ file and store the compelled file to ‘execure.out‘ using a gcc compiler.

$ gcc exploit.c -o exploit

As soon as you execute the compelled file. You will see the root command prompt without authentication.

How to Fix GameOver(lay)- Two Local Privilege Escalation Vulnerabilities in Ubuntu Linux Kernel?

On July 24, 2023, Ubuntu publicly released security updates that address CVE-2023-2640 and CVE-2023-32629 across affected Ubuntu kernels. Users of vulnerable Ubuntu versions should update their kernel packages to the latest patched releases as soon as possible to mitigate these flaws.

We recommend upgrading Kernel to version 6.2 and above. Follow this How-To procedure to update your kernel to the desired version.

Note: Before you download and install it on your production server, we recommend testing this on a test machine. Don’t forget to take the full VM snapshot if are upgrading the kernel on a Virtual Image. Or, take the filesystem backup if you have a physical server.

Step 1. Check the kernel version

Before we start upgrading the kernel, it is a good practice to determine the version of the kernel running on the system. To check the kernel version, open the terminal and type the following command:

$ uname -rs

The output of the command will show the current kernel version installed on the system.

Step 2. Download kernel packages v6.4

Download the kernel packages directly from the kernel.ubuntu.com website. Download the latest version available (At the bottom) from the website to a dedicated directory. Change the permission of the files to execute.
Create a directory in your path:

$ mkdir /home/arunkl/kernel-6.4.1

Change the directory:

$ cd /home/arunkl/kernel-6.4.1/

Download these two files (where X.Y.Z is the patched version):
1. linux-image-*X.Y.Z*-generic-*.deb2. linux-modules-X.Y.Z*-generic-*.deb
Commands to download the kernel v6.4.1

$ wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v6.4.1/amd64/linux-image-unsigned-6.4.1-060401-generic_6.4.1-060401.202307071129_amd64.deb

$ wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v6.4.1/amd64/linux-modules-6.4.1-060401-generic_6.4.1-060401.202307071129_amd64.deb

Run this command to set the file’s permission to execution mode:

$ chmod +x *.deb

Step 3. Install kernel Packages v6.4

Install the downloaded packages using the default dpkg utility then reboot the server.

$ sudo dpkg –-install *.deb

$ reboot

Step 4. Check the kernel version after the reboot

Use the same command used in the first step. You will see an upgraded kernel version if everything goes well.

How to Mitigate GameOver(lay)?

For systems that cannot be immediately upgraded, the vulnerabilities can be temporarily mitigated by restricting access to unprivileged user namespaces. This can be done by setting the kernel parameter kernel.unprivileged_userns_clone to 0.

$ sudo sysctl -w kernel.unprivileged_userns_clone=0

The above command will not resist the reboot. Making it persistent across reboots involves adding “kernel.unprivileged_userns_clone=0” to /etc/sysctl.d/99-disable-unpriv-userns.conf.

$ echo kernel.unprivileged_userns_clone=0 | \

$ sudo tee /etc/sysctl.d/99-disable-unpriv-userns.conf

We hope this post helps you know how to fix GameOver(lay), two local privilege escalation vulnerabilities in Ubuntu Linux Kernel. Thanks for reading this post. Please share this post and help secure the digital world. Visit our website thesecmaster.com, and our social media page on Facebook, LinkedIn, Twitter, Telegram, Tumblr, Medium, and Instagram and subscribe to receive updates like this.