Heap-based buffer overflow in ultra.c in LibVNCClient in LibVNCServer before 0.9.11 allows remote servers to cause a denial of service (application crash) or possibly execute arbitrary code via a crafted FramebufferUpdate message with the Ultra type tile, such that the LZO payload decompressed length exceeds what is specified by the tile dimensions.
LibVNCServer is vulnerable to a heap-based buffer overflow, allowing a malicious VNC server to crash the client or potentially execute arbitrary code. This vulnerability, triggered by a crafted FramebufferUpdate message, exploits a flaw in the handling of LZO-compressed data within the Ultra tile type, leading to a memory corruption condition. Successful exploitation can result in a denial-of-service (DoS) or, in a worst-case scenario, complete system compromise.
Step 1: Connection Establishment: The attacker establishes a VNC connection to a vulnerable LibVNCClient instance. This can be initiated by the attacker or by tricking a user into connecting to a malicious VNC server.
Step 2: FramebufferUpdate Message Injection: The attacker, acting as a malicious VNC server, sends a specially crafted FramebufferUpdate message. This message contains an Ultra tile with an LZO-compressed payload.
Step 3: LZO Decompression: The LibVNCClient attempts to decompress the LZO payload using the LZO algorithm.
Step 4: Size Mismatch: The decompressed size, as determined by the LZO algorithm, exceeds the buffer allocated based on the tile dimensions specified in the FramebufferUpdate message.
Step 5: Heap Overflow: The LZO decompression process writes data beyond the allocated buffer, causing a heap-based buffer overflow.
Step 6: Exploitation (DoS or Code Execution): The buffer overflow corrupts memory. This can lead to a crash (DoS) or, if the attacker has carefully crafted the malicious payload, the execution of arbitrary code.
The vulnerability lies within the ultra.c file of LibVNCClient, specifically in the handling of the Ultra tile type within FramebufferUpdate messages. The flaw stems from an improper calculation or validation of the decompressed size of LZO-compressed data. When a malicious VNC server sends a crafted FramebufferUpdate message with an Ultra tile, the client attempts to decompress the LZO payload. If the decompressed length, as determined by the LZO algorithm, exceeds the buffer allocated based on the tile dimensions, a heap-based buffer overflow occurs. This buffer overflow corrupts adjacent memory, potentially overwriting critical data structures or control flow information. This can lead to a crash (DoS) or, with careful crafting of the malicious payload, the execution of arbitrary code. The root cause is a lack of bounds checking or incorrect size calculation when allocating memory for the decompressed data, allowing the LZO decompression process to write beyond the allocated buffer.