lldb on FreeBSD picks the wrong address for functions found in both libc and libsys

[jgopensource]

In FreeBSD 15.0 and later when lldb is doing JIT it may pick the wrong address for functions that are available in both libc and libsys.

Steps to reproduce:

1. Put the following 3 lines into a file named empty.c:

int main(void) {
        return 0;
}

2. Compile the program: clang -O0 -g -o empty empty.c
3. Run lldb: lldb empty
4. Execute the following lldb commands:
   4.1. breakpoint set -n main
   4.2. run
   4.3. p (uint32_t)getuid()

The observed outcome will be one of these two:

A. The value 4294967295 is printed, which is incorrect. That value is -1 interpreted as an unsigned 32-bit integer.
In this case instead of finding an address for getuid() in libsys an address in libc is found and the wrong function gets executed. The symbol getuid is present in both libc.so.7 and libsys.so.7 which is also the case for other syscalls (or all of them?).

In the logs of lldb (when enabled) one can see these:

Process::CanJIT pid 2492 allocation test passed, CanJIT () is true
IRExecutionUnit::getSymbolAddress(Name="getuid") = 83944f90

The address value matches the address of getuid in /lib/libc.so.7.

B. The error message error: Can't evaluate the expression without a running target due to: Interpreter doesn't handle one of the expression's opcodes is displayed, which is deceptive.
The real problem is that when performing JIT lldb tried to allocate memory via mmap() in the process being debugged and failed.

In the logs of lldb (when enabled) one can see these:

Process::CanJIT pid 1677 allocation test failed, CanJIT () is false: unable to allocate 4096 bytes of memory with permissions rwx
ProcessGDBRemote::DoAllocateMemory no direct stub support for memory allocation, and InferiorCallMmap also failed - is stub missing register context save/restore capability?

The wrong address is picked for mmap() (in lldb_private::InferiorCallMmap()) and as a result the wrong function gets executed. The address is again from libc instead of from libsys.

The problem was observed with lldb version 19.1.7 on FreeBSD 15.0-RELEASE-p2 and FreeBSD 16.0-CURRENT on both amd64 (x86_64) and arm64.

[llvmbot]

@llvm/issue-subscribers-libc

Author: Jordan Gordeev (jgopensource)

Details

In FreeBSD 15.0 and later when lldb is doing JIT it may pick the wrong address for functions that are available in both `libc` and `libsys`.

Steps to reproduce:

1. Put the following 3 lines into a file named empty.c:

int main(void) {
        return 0;
}

2. Compile the program: clang -O0 -g -o empty empty.c
3. Run lldb: lldb empty
4. Execute the following lldb commands:
   4.1. breakpoint set -n main
   4.2. run
   4.3. p (uint32_t)getuid()

The observed outcome will be one of these two:

A. The value 4294967295 is printed, which is incorrect. That value is -1 interpreted as an unsigned 32-bit integer.
In this case instead of finding an address for getuid() in libsys an address in libc is found and the wrong function gets executed. The symbol getuid is present in both libc.so.7 and libsys.so.7 which is also the case for other syscalls (or all of them?).

In the logs of lldb (when enabled) one can see these:

Process::CanJIT pid 2492 allocation test passed, CanJIT () is true
IRExecutionUnit::getSymbolAddress(Name="getuid") = 83944f90

The address value matches the address of getuid in /lib/libc.so.7.

B. The error message error: Can't evaluate the expression without a running target due to: Interpreter doesn't handle one of the expression's opcodes is displayed, which is deceptive.
The real problem is that when performing JIT lldb tried to allocate memory via mmap() in the process being debugged and failed.

In the logs of lldb (when enabled) one can see these:

Process::CanJIT pid 1677 allocation test failed, CanJIT () is false: unable to allocate 4096 bytes of memory with permissions rwx
ProcessGDBRemote::DoAllocateMemory no direct stub support for memory allocation, and InferiorCallMmap also failed - is stub missing register context save/restore capability?

The wrong address is picked for mmap() (in lldb_private::InferiorCallMmap()) and as a result the wrong function gets executed. The address is again from libc instead of from libsys.

The problem was observed with lldb version 19.1.7 on FreeBSD 15.0-RELEASE-p2 and FreeBSD 16.0-CURRENT on both amd64 (x86_64) and arm64.

[llvmbot]

@llvm/issue-subscribers-lldb

Author: Jordan Gordeev (jgopensource)

Details

In FreeBSD 15.0 and later when lldb is doing JIT it may pick the wrong address for functions that are available in both `libc` and `libsys`.

Steps to reproduce:

1. Put the following 3 lines into a file named empty.c:

int main(void) {
        return 0;
}

2. Compile the program: clang -O0 -g -o empty empty.c
3. Run lldb: lldb empty
4. Execute the following lldb commands:
   4.1. breakpoint set -n main
   4.2. run
   4.3. p (uint32_t)getuid()

The observed outcome will be one of these two:

A. The value 4294967295 is printed, which is incorrect. That value is -1 interpreted as an unsigned 32-bit integer.
In this case instead of finding an address for getuid() in libsys an address in libc is found and the wrong function gets executed. The symbol getuid is present in both libc.so.7 and libsys.so.7 which is also the case for other syscalls (or all of them?).

In the logs of lldb (when enabled) one can see these:

Process::CanJIT pid 2492 allocation test passed, CanJIT () is true
IRExecutionUnit::getSymbolAddress(Name="getuid") = 83944f90

The address value matches the address of getuid in /lib/libc.so.7.

B. The error message error: Can't evaluate the expression without a running target due to: Interpreter doesn't handle one of the expression's opcodes is displayed, which is deceptive.
The real problem is that when performing JIT lldb tried to allocate memory via mmap() in the process being debugged and failed.

In the logs of lldb (when enabled) one can see these:

Process::CanJIT pid 1677 allocation test failed, CanJIT () is false: unable to allocate 4096 bytes of memory with permissions rwx
ProcessGDBRemote::DoAllocateMemory no direct stub support for memory allocation, and InferiorCallMmap also failed - is stub missing register context save/restore capability?

The wrong address is picked for mmap() (in lldb_private::InferiorCallMmap()) and as a result the wrong function gets executed. The address is again from libc instead of from libsys.

The problem was observed with lldb version 19.1.7 on FreeBSD 15.0-RELEASE-p2 and FreeBSD 16.0-CURRENT on both amd64 (x86_64) and arm64.

[emaste]

On FreeBSD libc is an auxiliary filter library for libsys

Dynamic section at offset 0x215b90 contains 31 entries:
  Tag                Type           Name/Value
  0x000000007ffffffd (AUXILIARY)    Auxiliary library: [libsys.so.7]
  0x0000000000000001 (NEEDED)       Shared library: [libsys.so.7]
  0x000000000000000e (SONAME)       Library soname: [libc.so.7]
...

libsys contains the actual syscall stubs. The symbols are present in libc, but are just forwarded to libsys. I wonder if ELF filter libraries are just not well supported (also in gdb perhaps)?

[emaste]

@mchoo7 and I did some experimentation, it appears that ELF filter libraries are just not handled well by LLDB, and this is not a FreeBSD-specific issue. [Edit: I claimed GDB also got this wrong, but I think I'm mistaken]

[jimingham]

MachO doesn't have the notion of filter libraries, but it does have "reexported" symbols where library A says it exports something from library B as some_other_name. lldb had to have special handling for those when building up the symbol table.

Maybe you could do something along the same lines when processing these libraries?

[emaste]

@jimingham thanks, this indeed sounds broadly similar to what we'll need to support for ELF.

[emaste]

@jgopensource this should probably be retitled lldb does not resolve symbols from ELF filter libraries correctly

[mchoo7]

@jgopensource this should probably be retitled lldb does not resolve symbols from ELF filter libraries correctly

I created a separate issue for that at #180860.