The linux-2.4.21-mlock.patch in Red Hat Enterprise Linux 3 does not properly maintain the mlock page count when one process unlocks pages that belong to another process, which allows local users to mlock more memory than specified by the rlimit.
Local privilege escalation is possible on vulnerable Red Hat Enterprise Linux 3 systems due to a flaw in the mlock functionality. This allows a malicious user to bypass memory limits, potentially leading to denial-of-service or the ability to execute arbitrary code with elevated privileges by exhausting system resources.
Step 1: Initial Setup: A user with local access to the vulnerable system logs in.
Step 2: Resource Limit Check: The user verifies the existing rlimit for locked memory using ulimit -l or similar tools.
Step 3: Memory Locking: The malicious user attempts to lock a large amount of memory, approaching the rlimit.
Step 4: Triggering the Vulnerability: The user then unlocks the locked memory. Due to the flaw in the mlock patch, the kernel may not correctly decrement the locked page count.
Step 5: Repeated Locking and Unlocking: The user repeatedly locks and unlocks memory, exploiting the incorrect page count tracking to bypass the rlimit.
Step 6: Memory Exhaustion (DoS): The user continues to lock memory beyond the intended limit, potentially exhausting system resources like physical RAM or swap space, leading to a denial-of-service condition.
Step 7: Potential Code Execution (Less Likely): In a highly complex scenario, the attacker might attempt to store malicious code in the locked memory, although this is less likely and depends on other vulnerabilities. This is more of a theoretical possibility.
The vulnerability lies within the linux-2.4.21-mlock.patch applied in Red Hat Enterprise Linux 3. The patch fails to correctly track the number of pages locked by each process. Specifically, when one process unlocks memory pages that were previously locked by another process, the kernel's internal page count is not properly decremented. This leads to an inconsistent state where the kernel believes less memory is locked than is actually the case. A malicious user can then exploit this by repeatedly locking and unlocking memory, effectively bypassing the rlimit (resource limit) imposed on the amount of memory a process can lock. This can lead to memory exhaustion, potentially causing a denial-of-service (DoS) condition. Furthermore, in a worst-case scenario, the attacker could potentially use the extra locked memory to store malicious code, although this is less likely and depends on other vulnerabilities.