bug: multi-thread race condition

[bentyeh]

Problem

Running the following command multiple times led to a crash each time but with different errors:

splitcode -c splitcode-config_oligo.tsv -t 10 --nFastqs=2 --assign --out-fasta --no-outb \
    --x-names --mod-names --no-x-out --mapping=/dev/null \
    --summary=summary.txt --select=0 --pipe \
    R1.fastq.gz R2.fastq.gz > out.fasta

Attempt 1 error message:

Segmentation fault (core dumped)

Attempt 2 error message:

free(): invalid pointer
Aborted (core dumped)

The same command has not crashed (at least on a few trial runs) when I only use 1 thread (-t 1).

---

System

System: Caltech HPC

• Operating System: Red Hat Enterprise Linux 9.3 (Plow)
• Kernel: Linux 5.14.0-362.24.1.el9_3.x86_64
• CPU: Intel(R) Xeon(R) Platinum 8352Y

Splitcode

• Version 0.31.3
• I tried compiling using the default CMakeLists.txt file or with the edits suggested in instance of 'std::overflow_error' - robin_hood::map overflow #37, and both crash on the above command.

---

Analysis

I think there may be some race condition. I re-compiled splitcode with ThreadSanitizer (TSan) enabled by modifying the root CMakeLists.txt file as follows:

...

if(${CMAKE_VERSION} VERSION_LESS 3.1)
    add_compile_options(-std=c++17)
else()
    set(CMAKE_CXX_STANDARD 17)
    set(CMAKE_CXX_STANDARD_REQUIRED ON)
    set(CMAKE_CXX_EXTENSIONS OFF)
endif()

option(ENABLE_SANITIZERS "Enable Address/Undefined sanitizer" OFF)

if(ENABLE_SANITIZERS)
  message(STATUS "Enabling Address/Undefined sanitizers")
  set(SANITIZE_FLAGS "-fsanitize=thread -g -O1")
  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${SANITIZE_FLAGS}")
  set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} ${SANITIZE_FLAGS}")
endif()

...

Then, I compiled with

cd build
cmake -DENABLE_SANITIZERS=ON -DCMAKE_BUILD_TYPE=Debug ..
make

Then, I ran the same command as above:

splitcode -c splitcode-config_oligo.tsv -t 10 --nFastqs=2 --assign --out-fasta --no-outb \
    --x-names --mod-names --no-x-out --mapping=/dev/null \
    --summary=summary.txt --select=0 --pipe \
    R1.fastq.gz R2.fastq.gz > tsan.txt

Here's the TSan output: tsan.txt

I hope this is helpful for you to help pin down the bug.

[Yenaled]

Made a new release; hopefully it fixes it (haven't tested though).

[bentyeh]

Noting here for future reference that commit 9efb531 (incorporated into release 0.31.4) prevents the segfault observed in this issue but does not fully address the underlying race condition. See comments on the commit itself.

[Yenaled]

@bentyeh the latest update should address the race condition with the increment operator (though not throughly tested).

[bentyeh]

Thanks. Adding a reference here to the commit 955771c