libgcrypt crashing when connecting to Mac hosts [Trace: basic_initialization, _gcry_rngfips_initialize]

[krf]

Heya,

this has been already discussed at the beginning of 2014:
http://sourceforge.net/p/libvncserver/mailman/message/32165820/

This is exactly the same issue, but I'm resending this in hope to get this
fixed. KRDC is unusable for me now.

I'm also getting a crash when using KRDC to connect to a Mac host (running OSX
10.7, if that helps). Backtrace is attached, see [1]

Error message is:

Fatal: failed to create the RNG lock: Invalid argument
FATAL: failed to acquire the FSM lock in libgrypt: Invalid argument

libgcrypt version is: 1.6.1-2ubuntu1

I also tried compiling libvncserver with SSL support and without gcrypt
support but compilation fails in this mode:

../libvncserver/.libs/libvncserver.so: error: undefined reference to 'gcry_md_open'
../libvncserver/.libs/libvncserver.so: error: undefined reference to 'gcry_md_write'
../libvncserver/.libs/libvncserver.so: error: undefined reference to 'gcry_md_ctl'
../libvncserver/.libs/libvncserver.so: error: undefined reference to 'gcry_md_get_algo_dlen'
../libvncserver/.libs/libvncserver.so: error: undefined reference to 'gcry_md_read' collect2: error: ld returned 1 exit status

Note: I have no real clue about VNC/libvncserver/libgcrypt altogether, but
this seems like a real issue people are having which has been around for more
than a year now. Let's get this fixed, happy to help.

Greets

[1] Backtrace:

(gdb) bt
#0  0x00007ffff3e41d27 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff3e43418 in __GI_abort () at abort.c:89
#2  0x00007fffe391e8f4 in lock_fsm () at fips.c:235
#3  0x00007fffe391ec01 in fips_new_state (new_state=STATE_FATALERROR) at fips.c:759
#4  0x00007fffe391f0b8 in _gcry_fips_signal_error (srcfile=0x7fffe39b49f0 "misc.c", srcline=140, srcfunc=0x7fffe39b4b00 <__FUNCTION__.7625> 
"_gcry_logv", is_fatal=1, description=0x7fffe39b4a3b "internal
 error (fatal or bug)") at fips.c:732
#5  0x00007fffe3917efa in _gcry_logv (level=level@entry=40, fmt=fmt@entry=0x7fffe39d24e8 "failed to create the RNG lock: %s\n", 
arg_ptr=arg_ptr@entry=0x7fffe0bd68f8) at misc.c:140
#6  0x00007fffe3918274 in _gcry_log_fatal (fmt=fmt@entry=0x7fffe39d24e8 "failed to create the RNG lock: %s\n") at misc.c:230
#7  0x00007fffe39a9665 in basic_initialization () at random-fips.c:201
#8  0x00007fffe39aa2e1 in _gcry_rngfips_initialize (full=1) at random-fips.c:746
#9  _gcry_rngfips_randomize (buffer=0x7fffd4003a90, length=16, level=GCRY_STRONG_RANDOM) at random-fips.c:835
#10 0x00007fffe39a7c80 in _gcry_random_bytes (nbytes=nbytes@entry=16, level=GCRY_STRONG_RANDOM) at random.c:324
#11 0x00007fffe39b23d7 in _gcry_mpi_randomize (w=0x7fffd4003a50, nbits=<optimized out>, level=<optimized out>) at mpiutil.c:586
#12 0x00007fffe3bff25c in InitialiseRFBConnection () from /usr/lib/x86_64-linux-gnu/libvncclient.so.0
#13 0x00007fffe3c01064 in rfbInitClient () from /usr/lib/x86_64-linux-gnu/libvncclient.so.0
#14 0x00007fffe3e17e84 in VncClientThread::clientCreate 
(this=this@entry=0xb25220, reinitialising=reinitialising@entry=false) at ../../vnc/vncclientthread.cpp:555
#15 0x00007fffe3e182b0 in VncClientThread::run (this=0xb25220) at ../../vnc/vncclientthread.cpp:463
#16 0x00007ffff5fe239f in ?? () from /usr/lib/x86_64-linux-gnu/libQtCore.so.4
#17 0x00007ffff23320a5 in start_thread (arg=0x7fffe0bd7700) at pthread_create.c:309
#18 0x00007ffff3f0577d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

[bk138]

Tbh, no idea what's going on there. I will provide a minimum Program that triggers the issue and that will hopefully help us to track down the problem.

[bk138]

Ok, can you please try with the provided sample clients in client_examples? I suppose it's got to do with multithreading...

[bk138]

Also, I guess that this might be a bug within krdc as it's not calling gcry_control() to setup gcrypt threading. A note in the libvncclient README would be nice though...

[krf]

I figured that I could use gtkvncviewer to try to connect. However: My target host requires a unix-like login, with username + password. I don't see a way to specify the username using that binary(?).

So, in fact, rfbCredential.userCredential.username needs to be filled for me... (I think)

[bk138]

You can try https://github.com/bk138/multivnc which has the GetCredential callback implemented.

[bk138]

Also, http://remmina.sourceforge.net/ 1.0 should also work by looking at its code...

[krf]

FYI: I got the same crash when trying to connect using multivnc:

Fatal: failed to create the RNG lock: Invalid argument
FATAL: failed to acquire the FSM lock in libgrypt: Invalid argument

Program received signal SIGABRT, Aborted.
0x00007ffff46d0d27 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007ffff46d0d27 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff46d2418 in __GI_abort () at abort.c:89
#2  0x00007ffff57c38f4 in lock_fsm () at fips.c:235
#3  0x00007ffff57c3c01 in fips_new_state (new_state=STATE_FATALERROR) at fips.c:759
#4  0x00007ffff57c40b8 in _gcry_fips_signal_error (srcfile=0x7ffff58599f0 "misc.c", srcline=140, srcfunc=0x7ffff5859b00 <__FUNCTION__.7625> "_gcry_logv", is_fatal=1, description=0x7ffff5859a3b "internal
 error (fatal or bug)") at fips.c:732
#5  0x00007ffff57bcefa in _gcry_logv (level=level@entry=40, fmt=fmt@entry=0x7ffff58774e8 "failed to create the RNG lock: %s\n", arg_ptr=arg_ptr@entry=0x7fffffffb9e8) at misc.c:140
#6  0x00007ffff57bd274 in _gcry_log_fatal (fmt=fmt@entry=0x7ffff58774e8 "failed to create the RNG lock: %s\n") at misc.c:230
#7  0x00007ffff584e665 in basic_initialization () at random-fips.c:201
#8  0x00007ffff584f2e1 in _gcry_rngfips_initialize (full=1) at random-fips.c:746
#9  _gcry_rngfips_randomize (buffer=0x82fa80, length=16, level=GCRY_STRONG_RANDOM) at random-fips.c:835
#10 0x00007ffff584cc80 in _gcry_random_bytes (nbytes=nbytes@entry=16, level=GCRY_STRONG_RANDOM) at random.c:324
#11 0x00007ffff58573d7 in _gcry_mpi_randomize (w=0x7078b0, nbits=<optimized out>, level=<optimized out>) at mpiutil.c:586
#12 0x000000000048a0f6 in InitialiseRFBConnection ()
#13 0x000000000048c5f4 in rfbInitClient ()
#14 0x0000000000425c33 in VNCConn::Init(wxString const&, wxString const&, int, int, bool, int, int) ()
#15 0x000000000043aa60 in MyFrameMain::spawn_conn(wxString, int) ()
#16 0x000000000043c1a4 in MyFrameMain::machine_connect(wxCommandEvent&) ()
#17 0x00007ffff5f628be in wxAppConsoleBase::CallEventHandler(wxEvtHandler*, wxEventFunctor&, wxEvent&) const () from /usr/lib/x86_64-linux-gnu/libwx_baseu-3.0.so.0
#18 0x00007ffff60f4368 in wxEvtHandler::ProcessEventIfMatchesId(wxEventTableEntryBase const&, wxEvtHandler*, wxEvent&) () from /usr/lib/x86_64-linux-gnu/libwx_baseu-3.0.so.0
#19 0x00007ffff60f446b in wxEventHashTable::HandleEvent(wxEvent&, wxEvtHandler*) () from /usr/lib/x86_64-linux-gnu/libwx_baseu-3.0.so.0
#20 0x00007ffff60f4818 in wxEvtHandler::TryHereOnly(wxEvent&) () from /usr/lib/x86_64-linux-gnu/libwx_baseu-3.0.so.0
#21 0x00007ffff60f48a3 in wxEvtHandler::ProcessEventLocally(wxEvent&) () from /usr/lib/x86_64-linux-gnu/libwx_baseu-3.0.so.0
#22 0x00007ffff60f4905 in wxEvtHandler::ProcessEvent(wxEvent&) () from /usr/lib/x86_64-linux-gnu/libwx_baseu-3.0.so.0
#23 0x00007ffff6843b48 in wxWindowBase::TryAfter(wxEvent&) () from /usr/lib/x86_64-linux-gnu/libwx_gtk2u_core-3.0.so.0
#24 0x00007ffff60f4677 in wxEvtHandler::SafelyProcessEvent(wxEvent&) () from /usr/lib/x86_64-linux-gnu/libwx_baseu-3.0.so.0
#25 0x00007ffff6818ec0 in wxToolBarBase::OnLeftClick(int, bool) () from /usr/lib/x86_64-linux-gnu/libwx_gtk2u_core-3.0.so.0
#26 0x00007ffff7295474 in _g_closure_invoke_va (closure=0x372e, closure@entry=0x6e3b40, return_value=0x372e, return_value@entry=0x0, instance=0x6, instance@entry=0x573b70, args=0xffffffffffffffff, args@
entry=0x7fffffffcea8, n_params=0, param_types=0x85a010) at /build/buildd/glib2.0-2.42.1/./gobject/gclosure.c:831
#27 0x00007ffff72af057 in g_signal_emit_valist (instance=0x573b70, signal_id=<optimized out>, detail=0, var_args=0x7fffffffcea8) at /build/buildd/glib2.0-2.42.1/./gobject/gsignal.c:3218
#28 0x00007ffff72afefa in g_signal_emit_by_name (instance=0x372e, detailed_signal=0x372e <error: Cannot access memory at address 0x372e>) at /build/buildd/glib2.0-2.42.1/./gobject/gsignal.c:3405
#29 0x00007ffff7295474 in _g_closure_invoke_va (closure=0x372e, closure@entry=0x6dab40, return_value=0x372e, return_value@entry=0x0, instance=0x6, instance@entry=0x6c9930, args=0xffffffffffffffff, args@
entry=0x7fffffffd1c0, n_params=0, param_types=0x85a010) at /build/buildd/glib2.0-2.42.1/./gobject/gclosure.c:831
#30 0x00007ffff72af057 in g_signal_emit_valist (instance=0x6c9930, signal_id=<optimized out>, detail=0, var_args=var_args@entry=0x7fffffffd1c0) at /build/buildd/glib2.0-2.42.1/./gobject/gsignal.c:3218
#31 0x00007ffff72af9af in g_signal_emit (instance=<optimized out>, signal_id=<optimized out>, detail=<optimized out>) at /build/buildd/glib2.0-2.42.1/./gobject/gsignal.c:3365
#32 0x00007ffff7819425 in ?? () from /usr/lib/x86_64-linux-gnu/libgtk-x11-2.0.so.0
#33 0x00007ffff7295474 in _g_closure_invoke_va (closure=0x372e, closure@entry=0x6d5f80, return_value=0x372e, return_value@entry=0x0, instance=0x6, instance@entry=0x6c9930, args=0xffffffffffffffff, args@
entry=0x7fffffffd490, n_params=0, param_types=0x85a010) at /build/buildd/glib2.0-2.42.1/./gobject/gclosure.c:831
#34 0x00007ffff72af057 in g_signal_emit_valist (instance=0x6c9930, signal_id=<optimized out>, detail=0, var_args=var_args@entry=0x7fffffffd490) at /build/buildd/glib2.0-2.42.1/./gobject/gsignal.c:3218
#35 0x00007ffff72af9af in g_signal_emit (instance=<optimized out>, signal_id=<optimized out>, detail=<optimized out>) at /build/buildd/glib2.0-2.42.1/./gobject/gsignal.c:3365

[bk138]

What about remmina?

[maxnet]

Also, I guess that this might be a bug within krdc as it's not calling gcry_control() to setup gcrypt threading.

Can't libvncclient just assume pthreads are used on unix platforms, and set it up for that?
It seems to be already making that same assumption when setting up other dependent libraries like OpenSSL (in tls_openssl.c)

[bk138]

Might be an option as per https://www.gnupg.org/documentation/manuals/gcrypt/Multi_002dThreading.html, yep.

[maxnet]

libgcrypt version is: 1.6.1-2ubuntu1

Might be an option as per https://www.gnupg.org/documentation/manuals/gcrypt/Multi_002dThreading.html,

Functions mentioned there are dummies in 1.6.x

gcrypt.h

/* NOTE: Since Libgcrypt 1.6 the thread callbacks are not anymore
   used.  However we keep it to allow for some source code
   compatibility if used in the standard way.  */

So problem must be something different.

[bk138]

Should be fixed by #51