crashed in photon::prelocked_thread_interrupt

[ivanallen]

photon: 0.9.2

It has occurred twice so far, and the probability of occurrence is very low.

From the coredump analysis, the cvar queue in the thread holds a photon thread of 0xffffffff, which is the same value for both crashes.

#0  0x0000000001f584bc in std::__atomic_base<bool>::exchange (__m=std::memory_order_acquire, __i=true, this=0x40) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/atomic_base.h:503
503           exchange(__int_type __i,
[Current thread is 1 (Thread 0x7fae84998640 (LWP 87))]
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.34-168.el9_6.23.x86_64 hwloc-libs-2.4.1-5.el9.x86_64 libaio-0.3.111-13.el9.x86_64 libdwarf-0.3.4-1.el9.1.x86_64 libgcc-11.3.1-2.1.el9.x86_64 libibverbs-54.0-1.el9.x86_64 libnl3-3.11.0-1.el9.x86_64 librdmacm-54.0-1.el9.x86_64 libstdc++-11.3.1-2.1.el9.x86_64 liburing-2.5-1.el9.x86_64 libuuid-2.37.4-21.el9.x86_64 lz4-libs-1.9.3-5.el9.x86_64 numactl-libs-2.0.19-1.el9.x86_64 openssl-libs-3.2.2-6.el9_5.1.x86_64 re2-20211101-20.el9.x86_64 systemd-libs-250-12.el9_1.1.x86_64 zlib-1.2.11-35.el9_1.x86_64
(gdb) bt
#0  0x0000000001f584bc in std::__atomic_base<bool>::exchange (__m=std::memory_order_acquire, __i=true, this=0x40) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/atomic_base.h:503
#1  std::atomic<bool>::exchange (__m=std::memory_order_acquire, __i=true, this=0x40) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/atomic:120
#2  photon::spinlock::xchg (this=0x40) at /home/jenkins/workspace/photon/include/photon/thread/thread.h:240
#3  photon::spinlock::lock (this=0x40) at /home/jenkins/workspace/photon/include/photon/thread/thread.h:220
#4  photon::locker<photon::spinlock>::lock (must_lock=true, this=<synthetic pointer>)
    at /home/jenkins/workspace/photon/include/photon/thread/thread.h:340
#5  photon::locker<photon::spinlock>::locker (mutex=0x40, do_lock=2, this=<synthetic pointer>)
    at /home/jenkins/workspace/photon/include/photon/thread/thread.h:323
#6  photon::locker<photon::spinlock>::locker (do_lock=2, mutex=0x40, this=<synthetic pointer>)
    at /home/jenkins/workspace/photon/include/photon/thread/thread.h:320
#7  photon::locker<photon::spinlock>::locker (do_lock=2, mutex=..., this=<synthetic pointer>)
    at /home/jenkins/workspace/photon/include/photon/thread/thread.h:328
#8  photon::vcpu_t::_move_to_standbyq_atomic (th=0xffffffff, this=0x0) at /home/jenkins/workspace/photon/thread/thread.cpp:557
#9  photon::vcpu_t::move_to_standbyq_atomic<photon::thread*> (x=0xffffffff, this=0x0)
    at /home/jenkins/workspace/photon/thread/thread.cpp:538
#10 photon::prelocked_thread_interrupt (th=th@entry=0xffffffff, error_number=<optimized out>)
    at /home/jenkins/workspace/photon/thread/thread.cpp:1444
#11 0x0000000001f58f6d in photon::waitq::resume_one (this=this@entry=0x3d923f60, error_number=error_number@entry=-1)
    at /home/jenkins/workspace/photon/thread/thread.cpp:1677
#12 0x0000000001f5d053 in photon::condition_variable::notify_one (this=0x3d923f60)
    at /home/jenkins/workspace/photon/include/photon/thread/thread.h:427
#13 photon::ThreadPoolBase::thread_create_ex (this=this@entry=0x36062f00, start=start@entry=0x1f56c40 <photon::WorkPool::impl::delegate_helper(void*)>, arg=arg@entry=0x3d923e30, joinable=joinable@entry=false)
    at /home/jenkins/workspace/photon/thread/thread-pool.cpp:60
#14 0x0000000001f57692 in photon::ThreadPoolBase::thread_create (arg=0x3d923e30, start=0x1f56c40 <photon::WorkPool::impl::delegate_helper(void*)>, this=0x36062f00)
    at /home/jenkins/workspace/photon/include/photon/thread/thread-pool.h:43
#15 photon::WorkPool::impl::main_loop (this=this@entry=0x32a14000) at /home/jenkins/workspace/photon/thread/workerpool.cpp:134
#16 0x0000000001f56bd2 in photon::WorkPool::impl::join_current_vcpu_into_workpool (this=0x32a14000)
    at /home/jenkins/workspace/photon/thread/workerpool.cpp:165
#17 photon::WorkPool::join_current_vcpu_into_workpool (this=<optimized out>)
    at /home/jenkins/workspace/photon/thread/workerpool.cpp:221
#18 0x000000000103e56e in xbase::photon::WorkerPool::join_current_vcpu_into_workpool (id=4, this=0x2a96dc70)
    at /home/jenkins/workspace/r-build_XFinity_4.2.000.0.251225/xmake_globaldir/.xmake/packages/x/xbase/2.4.3/aeb6a6d6a0fc48e68ef63cfef8c8df3d/include/xbase/photon/worker_pool.h:32
#19 bs::shard::XdfsSdkModule::init()::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda()#1}::operator()() const (__closure=<optimized out>) at src/bdev/store/xdfs_sdk_adaptor/xdfs_sdk_module.cc:109
#20 photon::FunctorWrapper<bs::shard::XdfsSdkModule::init()::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda()#1}>::operator()() (this=<optimized out>)
    at /home/jenkins/workspace/include/photon/thread/thread11.h:54
#21 tuple_assistance::apply_impl<photon::FunctorWrapper<bs::shard::XdfsSdkModule::init()::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda()#1}>, std::tuple<>>(photon::FunctorWrapper<bs::shard::XdfsSdkModule::init()::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda()#1}>&&, std::tuple<>&&, std::integer_sequence<unsigned long>) (t=..., f=...)
    at /home/jenkins/workspace/include/photon/common/tuple-assistance.h:80
#22 tuple_assistance::apply<photon::FunctorWrapper<bs::shard::XdfsSdkModule::init()::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda()#1}>, std::tuple<> >(photon::FunctorWrapper<bs::shard::XdfsSdkModule::init()::{lambda(unsigned long)#1}::operator()(unsigned long) const::{lambda()#1}>&&, std::tuple<>&&) (t=..., f=...)
    at /home/jenkins/workspace/include/photon/common/tuple-assistance.h:87
#23 photon::__stub11<std::pair<photon::FunctorWrapper<bs::shard::XdfsSdkModule::init(bs::shard::XdfsSdkModule::init()::_ZN2bs5shard13XdfsSdkModule4initEv.Frame*)::<lambda(std::size_t)>::<lambda()> >, std::tuple<> > >(void *) ()
    at /home/jenkins/workspace/include/photon/thread/thread11.h:31
#24 0x0000000001f5caa7 in _photon_thread_stub ()
#25 0x0000000000000000 in ?? ()
(gdb) f 11
#11 0x0000000001f58f6d in photon::waitq::resume_one (this=this@entry=0x3d923f60, error_number=error_number@entry=-1)
    at /home/jenkins/workspace/photon/thread/thread.cpp:1677
1677    /home/jenkins/workspace/photon/thread/thread.cpp: No such file or directory.
(gdb) p *this
$1 = {q = {th = 0xffffffff, lock = {_lock = {_M_base = {static _S_alignment = 1, _M_i = false}}}}}
(gdb) f 12
#12 0x0000000001f5d053 in photon::condition_variable::notify_one (this=0x3d923f60)
    at /home/jenkins/workspace/photon/include/photon/thread/thread.h:427
427     /home/jenkins/workspace/photon/include/photon/thread/thread.h: No such file or directory.
(gdb) p *this
$2 = {<photon::waitq> = {q = {th = 0xffffffff, lock = {_lock = {_M_base = {static _S_alignment = 1, _M_i = false}}}}}, <No data fields>}
(gdb) f 13
#13 photon::ThreadPoolBase::thread_create_ex (this=this@entry=0x36062f00, start=start@entry=0x1f56c40 <photon::WorkPool::impl::delegate_helper(void*)>, arg=arg@entry=0x3d923e30, joinable=joinable@entry=false)
    at /home/jenkins/workspace/photon/thread/thread-pool.cpp:60
60      /home/jenkins/workspace/photon/thread/thread-pool.cpp: No such file or directory.
(gdb) p pCtrl
$3 = (photon::TPControl *) 0x3d923f40
(gdb) p pCtrl[0]
$4 = {th = 0x0, pool = 0x36062f00, start = 0x1f56c40 <photon::WorkPool::impl::delegate_helper(void*)>, arg = 0x3d923e30, cvar = {<photon::waitq> = {q = {th = 0xffffffff, lock = {_lock = {_M_base = {static _S_alignment = 1,
              _M_i = false}}}}}, <No data fields>}, joinable = false, joining = false, m_mtx = {_lock = {_M_base = {static _S_alignment = 1, _M_i = true}}}}
(gdb) p this
$5 = (photon::ThreadPoolBase * const) 0x36062f00
(gdb) p pCtrl[0].pool
$6 = (photon::ThreadPoolBase *) 0x36062f00
(gdb) p pCtrl[0].pool[0]
$7 = {<IdentityPoolBaseT<photon::TPControl>> = {<IdentityPoolBase> = {<intrusive_list_node<IdentityPoolBase>> = {<__intrusive_list_node> = {__prev_ptr = 0x36062f00, __next_ptr = 0x36062f00}, <No data fields>}, m_capacity = 64,
      m_size = 0, m_refcnt = 2820, min_size_in_interval = 0, m_mtx = {_lock = {_M_base = {static _S_alignment = 1, _M_i = false}}}, m_cvar = {<photon::waitq> = {q = {th = 0x0, lock = {_lock = {_M_base = {static _S_alignment = 1,
                  _M_i = false}}}}}, <No data fields>}, m_ctor = {<Delegate_Base> = {<No data fields>}, _obj = 0x36062f00, _func = 0x1f5cf40 <photon::ThreadPoolBase::ctor(photon::ThreadPoolBase*, photon::TPControl**)>},
      m_dtor = {<Delegate_Base> = {<No data fields>}, _obj = 0x36062f00, _func = 0x1f5d3b0 <photon::ThreadPoolBase::dtor(photon::ThreadPoolBase*, photon::TPControl*)>},
      m_reserved = 0x800000 <spdlog::details::elapsed_formatter<spdlog::details::null_scoped_padder, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >::format(spdlog::details::log_msg const&, tm const&, fmt::v9::basic_memory_buffer<char, 250ul, std::allocator<char> >&)+128>, autoscale = false, m_items = 0x36062f68}, <No data fields>}, <No data fields>}
(gdb)

[lihuiba]

Are you using thread_pool in a multi-vcpu mode?

[ivanallen]

Are you using thread_pool in a multi-vcpu mode?

In this case, we borrowed a WorkerPool, and each vcpu has a WorkerPool. It is possible that on one vcpu, the WorkerPool of another vcpu will be called.

But I have another case where a crash also occurred at the same location. I only use ThreadPool within the same thread and do not use it across vcpus.

[lihuiba]

Are you using WorkerPools together with thread_pool (thread_mod > 0) and multi-vcpus?

[ivanallen]

Are you using WorkerPools together with thread_pool (thread_mod > 0) and multi-vcpus?

In this issue:

vcpu_num = 0 and thread_mod = 64.

new ::photon::WorkPool(/*vcpu_num = */ 0, ::photon::INIT_EVENT_DEFAULT, ::photon::INIT_IO_NONE, /*thread_mod = */64)

// then join_current_vcpu_into_workpool 

Another issue, juse use thread pool in thread local:

thread_local photon::ThreadPoolBase* g_sdk_photon_tp = nullptr;
photon::ThreadPoolBase* get_photon_thread_pool() {
    if (!g_sdk_photon_tp) {
        g_sdk_photon_tp = photon::ThreadPoolBase::new_thread_pool(1024, 256*1024);
    }
    return g_sdk_photon_tp;
}

[lihuiba]

vcpu_num = 0 and thread_mod = 64.

How many vCPUs joined the WorkPool afterwards?

use thread pool in thread local

This seems fine.

[ivanallen]

vcpu_num = 0 and thread_mod = 64.

How many vCPUs joined the WorkPool afterwards?

use thread pool in thread local

This seems fine.

Only one vCPU joined.

[lihuiba]

Only one vCPU joined.

I suspect that another thread(s) has a bug that destroys related memory region, and makes it look like this.

[ivanallen]

Only one vCPU joined.

I suspect that another thread(s) has a bug that destroys related memory region, and makes it look like this.

For the second issue, we used thread_local ThreadPool. Theoretically, it's impossible for it to be used across threads, but it crashed for the same reason.

But we will continue to investigate whether the problem you mentioned exists.