Segfault when connecting to a desktop with a different resolution

[shanock01]

xrdp version

0.10.1, 0.10.4.1

Detailed xrdp version, build options

xrdp 0.10.4.1
  A Remote Desktop Protocol Server.
  Copyright (C) 2004-2025 Jay Sorg, Neutrino Labs, and all contributors.
  See https://github.com/neutrinolabs/xrdp for more information.

  Configure options:
      --enable-ipv6
      --enable-jpeg
      --enable-fuse
      --enable-rfxcodec
      --enable-opus
      --enable-painter
      --enable-pam
      --enable-pam-config=debian
      --enable-vsock
      --build=x86_64-linux-gnu
      --prefix=/usr
      --includedir=${prefix}/include
      --mandir=${prefix}/share/man
      --infodir=${prefix}/share/info
      --sysconfdir=/etc
      --localstatedir=/var
      --disable-silent-rules
      --libdir=${prefix}/lib/x86_64-linux-gnu
      --libexecdir=${prefix}/lib/x86_64-linux-gnu
      --disable-maintainer-mode
      --disable-dependency-tracking
      --with-socketdir=/run/xrdp/sockdir
      build_alias=x86_64-linux-gnu
      CFLAGS=-g -O2 -Werror=implicit-function-declaration
-ffile-prefix-map=/build/reproducible-path/xrdp-0.10.4.1=.
-fstack-protector-strong -fstack-clash-protection -Wformat
-Werror=format-security -fcf-protection 
      LDFLAGS=-Wl,-z,relro -Wl,-z,now -Wl,--as-needed
      CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2 
      CXXFLAGS=-g -O2
-ffile-prefix-map=/build/reproducible-path/xrdp-0.10.4.1=.
-fstack-protector-strong -fstack-clash-protection -Wformat
-Werror=format-security -fcf-protection
      PKG_CONFIG_PATH=/build/reproducible-path/xrdp-0.10.4.1/pkgconfig

  Compiled with OpenSSL 3.5.4 30 Sep 2025

Operating system & version

Debian 13.2

Installation method

dnf / apt / zypper / pkg / etc

Which backend do you use?

x11vnc 0.9.17-1

What desktop environment do you use?

LXQT/xfwm4

Environment xrdp running on

Tested on VMs, physical machines, nvidia, amd, and intel graphics

What's your client?

remmina 1.4.29, remmina 1.4.39, Win10Pro's mstsc.exe

Area(s) with issue?

Crashes such as segfault

Steps to reproduce

Summary:
Using x11vnc and xrdp on Debian 13, if the client resolution is different from the physical desktop resolution, xrdp crashes.

Background:
Works fine in Debian 12; if a client requests a different resolution, it gets rightfully overridden and set to the server's desktop resolution. Upgraded to Debian 13, xrdp crashes unless the client's requested resolution exactly matches the server's. If it matches, connection succeeds, and I get a perfectly working desktop. However, on some resolutions like 1366x760 or 1680x1050, even if I explicitly set the resolution in the client, the client (both remmina and mstsc do this) will decide on its own to request a resolution that is a few pixels off (1368 or 1364 for example) and still trigger the resize crash.

Reproducible:
Very consistent across different systems with different graphics cards.
0.9.21.1-1+deb12u1 (from Debian 12, no crash)
0.10.1-3.1 (from Debian 13, crash)
0.10.4.1-2 (from Debian experimental, crash)

Log snippet from Debian 12, working

Nov 19 10:58:26 ttutil01 xrdp[3401]: [WARN ] Received [MS-RDPBCGR] TS_UD_HEADER type 0xc006 is unknown (ignored)
Nov 19 10:58:26 ttutil01 xrdp[3401]: [WARN ] Received [MS-RDPBCGR] TS_UD_HEADER type 0xc00a is unknown (ignored)
Nov 19 10:58:26 ttutil01 xrdp[3401]: [WARN ] xrdp_caps_process_codecs: unknown codec id 5
Nov 19 10:58:26 ttutil01 xrdp[3401]: [WARN ] local keymap file for 0x00000409 found and doesn't match built in keymap, using local keymap file
Nov 19 10:58:26 ttutil01 xrdp[3401]: [WARN ] Received a message for the disabled channel drdynvc (1007)
Nov 19 10:58:29 ttutil01 x11vnc[2161]: 19/11/2025 10:58:29 Got connection from client ::1
Nov 19 10:58:29 ttutil01 x11vnc[2161]: 19/11/2025 10:58:29   0 other clients
Nov 19 10:58:30 ttutil01 x11vnc[2161]: 19/11/2025 10:58:30 Normal socket connection
Nov 19 10:58:30 ttutil01 x11vnc[2161]: 19/11/2025 10:58:30 incr accepted_client=1 for ::1:38106  sock=10
Nov 19 10:58:30 ttutil01 x11vnc[2161]: 19/11/2025 10:58:30 Client Protocol Version 3.3
Nov 19 10:58:30 ttutil01 x11vnc[2161]: 19/11/2025 10:58:30 Protocol version sent 3.3, using 3.3
Nov 19 10:58:30 ttutil01 x11vnc[2161]: Libgcrypt warning: missing initialization - please fix the application
Nov 19 10:58:30 ttutil01 x11vnc[2161]: 19/11/2025 10:58:30 copy_tiles: allocating first_line at size 44
Nov 19 10:58:30 ttutil01 x11vnc[2161]: 19/11/2025 10:58:30 Pixel format for client ::1:
Nov 19 10:58:30 ttutil01 x11vnc[2161]: 19/11/2025 10:58:30   16 bpp, depth 16, little endian
Nov 19 10:58:30 ttutil01 x11vnc[2161]: 19/11/2025 10:58:30   true colour: max r 31 g 63 b 31, shift r 11 g 5 b 0
Nov 19 10:58:30 ttutil01 x11vnc[2161]: 19/11/2025 10:58:30 Enabling full-color cursor updates for client ::1
Nov 19 10:58:30 ttutil01 x11vnc[2161]: 19/11/2025 10:58:30 Enabling NewFBSize protocol extension for client ::1
Nov 19 10:58:30 ttutil01 x11vnc[2161]: 19/11/2025 10:58:30 Enabling ExtDesktopSize protocol extension for client ::1
Nov 19 10:58:30 ttutil01 x11vnc[2161]: 19/11/2025 10:58:30 Using raw encoding for client ::1
Nov 19 10:58:31 ttutil01 x11vnc[2161]: 19/11/2025 10:58:31 Sending rfbEncodingExtDesktopSize for size (1366x768)
Nov 19 10:58:31 ttutil01 x11vnc[2161]: 19/11/2025 10:58:31 Client requested resolution change to (1364x768)
Nov 19 10:58:31 ttutil01 x11vnc[2161]: 19/11/2025 10:58:31 Sending rfbEncodingExtDesktopSize for size (1366x768) resize prohibited
Nov 19 10:58:31 ttutil01 xrdp[3401]: [WARN ] VNC server resize failed - error code 1 [Resize is administratively prohibited]
Nov 19 10:58:31 ttutil01 xrdp[3401]: [WARN ] Resizing client to server 1366x768
Nov 19 10:58:31 ttutil01 x11vnc[2161]: 19/11/2025 10:58:31 client_set_net: ::1  0.0001
Nov 19 10:58:31 ttutil01 x11vnc[2161]: 19/11/2025 10:58:31 Sending rfbEncodingExtDesktopSize for size (1366x768)
Nov 19 10:58:31 ttutil01 xrdp[3401]: [WARN ] xrdp_caps_process_codecs: unknown codec id 5
Nov 19 10:58:31 ttutil01 x11vnc[2161]: 19/11/2025 10:58:31 client 1 network rate 39438.8 KB/sec (39438.8 eff KB/sec)
Nov 19 10:58:31 ttutil01 x11vnc[2161]: 19/11/2025 10:58:31 client 1 latency: 0.5 ms
Nov 19 10:58:31 ttutil01 x11vnc[2161]: 19/11/2025 10:58:31 dt1: 0.0044, dt2: 0.0490 dt3: 0.0005 bytes: 2098188
Nov 19 10:58:31 ttutil01 x11vnc[2161]: 19/11/2025 10:58:31 link_rate: LR_LAN - 1 ms, 39438 KB/s
Nov 19 10:58:39 ttutil01 x11vnc[2161]: 19/11/2025 10:58:39 created selwin: 0x1e0002f
Nov 19 10:58:39 ttutil01 x11vnc[2161]: 19/11/2025 10:58:39 called initialize_xfixes()

Log snippet from Debian 13, crashing

Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 Got connection from client ::1
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10   0 other clients
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 Normal socket connection
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 incr accepted_client=1 for ::1:38250  sock=10
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 Client Protocol Version 3.3
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 Protocol version sent 3.3, using 3.3
Nov 19 11:01:10 ttutil01 x11vnc[1596]: Libgcrypt warning: missing initialization - please fix the application
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 Pixel format for client ::1:
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10   16 bpp, depth 16, little endian
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10   true colour: max r 31 g 63 b 31, shift r 11 g 5 b 0
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 Enabling full-color cursor updates for client ::1
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 Enabling NewFBSize protocol extension for client ::1
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 Enabling ExtDesktopSize protocol extension for client ::1
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 Using raw encoding for client ::1
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 Sending rfbEncodingExtDesktopSize for size (1366x768)
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 Client requested resolution change to (1364x768)
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 Sending rfbEncodingExtDesktopSize for size (1366x768) resize prohibited
Nov 19 11:01:10 ttutil01 xrdp[2667]: [WARN ] VNC server resize failed - error code 1 [Resize is administratively prohibited]
Nov 19 11:01:10 ttutil01 kernel: xrdp[2667]: segfault at 8 ip 00007f43386515ae sp 00007ffe6851f080 error 4 in libcommon.so.0.0.0[b5ae,7f433864f000+12000] likely on CPU 2 (core 1, socket 0)
Nov 19 11:01:10 ttutil01 kernel: Code: 83 c4 08 48 89 ef 5b 5d e9 1f e7 ff ff 0f 1f 80 00 00 00 00 c3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 89 f9 48 83 ec 28 49 89 f0 <8b> 57 08 48 8b 3f 3b 51 0c 74 17 48 63 c2 83 c2 01 4c 89 04 c7 b8
Nov 19 11:01:10 ttutil01 xrdp[2667]: [WARN ] Resizing client to server
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 Sending rfbEncodingExtDesktopSize for size (1366x768)
Nov 19 11:01:10 ttutil01 xrdp[1438]: [ERROR] Child 2667 terminated unexpectedly with signal SIGSEGV
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 rfbSendUpdateBuf: write: Connection reset by peer
Nov 19 11:01:10 ttutil01 x11vnc[1596]: *** rfbSendFramebufferUpdate *FAILED* #1
Nov 19 11:01:10 ttutil01 x11vnc[1596]: *** fb_push ublen not zero: 4
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 client_count: 0
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 Client ::1 gone
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 Statistics events    Transmit/ RawEquiv ( saved)
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 FramebufferUpdate   :      2 |         0/        0 (  0.0%)
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10  raw :      1 |        14/       14 (  0.0%)
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10  RichCursor :      1 |       726/      726 (  0.0%)
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 ExtendedDesktopSize :      3 |        96/       96 (  0.0%)
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10  TOTALS :      7 |       836/      836 (  0.0%)
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 Statistics events    Received/ RawEquiv ( saved)
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10  SetDesktopSize :      1 |        24/       24 (  0.0%)
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10  ClientCutText :      1 |        33/       33 (  0.0%)
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10  KeyEvent :      1 |         8/        8 (  0.0%)
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10 FramebufferUpdate   :      3 |        30/       30 (  0.0%)
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10  SetEncodings :      1 |        24/       24 (  0.0%)
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10  SetPixelFormat :      1 |        20/       20 (  0.0%)
Nov 19 11:01:10 ttutil01 x11vnc[1596]: 19/11/2025 11:01:10  TOTALS :      8 |       139/      139 (  0.0%)

✔️ Expected Behavior

Working desktop

❌ Actual Behavior

segfault at 8 ip
00007f43386515ae sp 00007ffe6851f080 error 4 in
libcommon.so.0.0.0

Child 2667 terminated
unexpectedly with signal SIGSEGV

Anything else?

xrdp.ini

[Globals]
fork=True
port=3389
tcp_nodelay=True
tcp_keepalive=True
security_layer=negotiate
crypt_level=high
certificate=/etc/xrdp/cert.pem
key_file=/etc/xrdp/key.pem
ssl_protocols=TLSv1.2, TLSv1.3
tls_ciphers=HIGH
allow_channels=True
allow_multimon=True
bitmap_cache=True
bitmap_compression=True
bulk_compression=True
use_fastpath=both
enable_token_login=True
dark_grey=374248
blue=374248
ls_title=RONIN Remote Desktop Login
ls_top_window_bg_color=202020
ls_bg_color=E8E8E8
ls_logo_filename=/usr/local/share/xrdp/login-user.bmp
ls_logo_x_pos=111
ls_input_y_pos=200
[Logging]
LogFile=xrdp.log
LogLevel=0
EnableSyslog=True
SyslogLevel=WARN
[Channels]
cliprdr=True
[xrdp1]
name=Physical Desktop
lib=libvnc.so
ip=127.0.0.1
port=5900
username=na
password=ask

[matt335672]

Hi @shanock01

Thanks fo raising this.

I'm unable to provoke a crash here on v0.10.4 on my test rig using x11vnc and a resolution of 1680x1050. I've run up xrdp with a memory checker and it's clean. So I imagine we're not agreeing on settings.

Can you provide the following:

1. The command line you're using for x11vnc?
2. The stanza in xrdp.ini which you're using to connect to the session

Thanks very much.

[shanock01]

Hi, thanks for your attention.

My x11vnc command line is:

ExecStart=/usr/bin/x11vnc -rfbauth /etc/x11vnc.pass -auth /var/run/lightdm/root/:0 -forever -loop -noxdamage -repeat -rfbport 5900 -shared

The xrdp.ini I provided initially is the complete file. If I understand the question, the stanza I'm using is 'xrdp1'.

The xrdp.ini, and the x11vnc line above, are standardized between all the computers that are exhibiting this behavior.

[shanock01]

I have narrowed something down.

If I set:

[Channels]
drdynvc=True

Everything works fine, no segfault.

[matt335672]

Great detective work, @shanock01 ! I've managed to reproduce this now.

We added a 'resize queue' to support client-side dynamic resize requests for v0.10.x. These requests come in on the drdynvc channel. When drdynvc is not enabled, the resize queue is not created, as no resize requests can be received from the client. However, the server can still generate resize requests in some configurations, like yours. When the server resize request is added to the queue, the world ends.

The workaround is to enable drdynvc for now. I'll get patches into the active build streams for your configuration.