Heap-based buffer overflow in rfbproto.c in LibVNCClient in LibVNCServer before 0.9.11 allows remote servers to cause a denial of service (application crash) or possibly execute arbitrary code via a crafted FramebufferUpdate message containing a subrectangle outside of the client drawing area.
LibVNCServer is vulnerable to a critical heap-based buffer overflow, allowing attackers to potentially execute arbitrary code or cause a denial-of-service (DoS). Exploitation involves sending a malicious FramebufferUpdate message, triggering memory corruption within the server's handling of subrectangles. This vulnerability poses a significant risk to systems using vulnerable VNC implementations.
Step 1: Connection Establishment: The attacker establishes a VNC connection to a vulnerable LibVNCServer instance.
Step 2: FramebufferUpdate Request: The attacker crafts a malicious FramebufferUpdate message. This message contains one or more subrectangles.
Step 3: Malformed Subrectangle Definition: The attacker defines a subrectangle with dimensions that, when combined with its position, extend beyond the bounds of the client's drawing area (framebuffer).
Step 4: Server Processing: The LibVNCServer receives the malicious FramebufferUpdate message and begins processing the subrectangles.
Step 5: Out-of-Bounds Write: Due to the lack of proper bounds checking, the server attempts to copy data for the oversized subrectangle into the client's framebuffer, resulting in a heap-based buffer overflow.
Step 6: Memory Corruption: The overflow corrupts heap metadata, potentially overwriting other allocated memory blocks, function pointers, or other critical data.
Step 7: Exploitation (Optional): Depending on the extent of the corruption and the attacker's skill, the attacker may be able to achieve arbitrary code execution by overwriting a function pointer or other control data. Alternatively, the corruption leads to a crash and DoS.
The vulnerability lies within the rfbproto.c file of LibVNCClient (and LibVNCServer) versions prior to 0.9.11. Specifically, the flaw occurs when processing a FramebufferUpdate message. The server fails to properly validate the dimensions of subrectangles within the update, allowing an attacker to specify a subrectangle that extends beyond the allocated drawing area (client's framebuffer). This leads to a heap-based buffer overflow when the server attempts to copy data for the oversized subrectangle. The lack of bounds checking on the subrectangle's dimensions allows for an out-of-bounds write, corrupting the heap. This corruption can lead to a crash (DoS) or, with careful crafting of the malicious message, potentially allow for arbitrary code execution by overwriting critical data structures or function pointers.