Issue with Core Dumps in Cloudberry Tests

[avamingli]

Apache Cloudberry version

Main

What happened

Hi Team,

I have a concern regarding the Cloudberry tests. Even when all tests pass, a core dump is generated on the system if it's enabled.

Typically, a core dump indicates a crash, suggesting that something has gone wrong. However, since all tests are passing, it appears that this might be an expected behavior intended for testing purposes.
I believe this issue may stem from Greenplum, as I've noticed similar problems reported there for quite some time(That issue was created by me, but no response and closed as an expected one).

Core dump files can be quite large and consume significant disk space. I believe we should avoid generating them if possible. It would be beneficial to identify the specific case that triggers the core dump and explore alternative methods for testing such scenarios.

What you think should happen instead

No response

How to reproduce

Run regression test.

Operating System

Ubuntu.

Anything else

No response

Are you willing to submit PR?

• Yes, I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct.

[yjhjstz]

which test case caused core ？

[avamingli]

which test case caused core ？

Unfortunately, I don't know which specific test case is causing the core dump at this point; identifying that is the key of addressing this issue.
The call stack indicates that the core dump is triggered by an auxiliary process, but it's unclear which test case activates that process

[edespino]

Thanks for raising this concern about core dumps being generated during test execution. This is definitely something we should address to improve our test infrastructure.

Proposed Solution

We could enhance the test framework by adding an optional core file existence check. Here's how it would work:

- test: ic-resgroup-v2
  make_target: installcheck-resgroup-v2
  make_directory: --directory=src/test/isolation2
  num_primary_mirror_pairs: 3
  perform_core_file_existence_check: true

When perform_core_file_existence_check is enabled, the framework would:

1. Monitor for core file generation during test execution
2. Automatically fail tests if unexpected core files are detected
3. Capture stack trace information for debugging purposes
4. Include clear failure messages indicating why the test failed (core file found)

This approach would help us:

• Identify potential issues early in the development cycle
• Manage disk space more effectively by avoiding unnecessary core dumps
• Maintain debugging capabilities through stack traces
• Distinguish between intentional and unintentional core dumps

Next Steps

Before proceeding, could you clarify if any of the current core file generations are intentional test cases? This would help us avoid flagging legitimate test scenarios as failures.

I'd be happy to collaborate on implementing this enhancement. Let me know if you'd like to move forward with this approach or if you have any other suggestions for handling this situation.

[avamingli]

Hi, @edespino

Truly appreciate your willingness to help with this issue in our CI/CD pipeline. Implementing a system to detect core dump files and fail the build if they exist is an excellent proactive measure. This will not only enhance our testing process but also ensure that we maintain a high standard of stability and reliability in our builds.

Before proceeding, could you clarify if any of the current core file generations are intentional test cases?

Only have this one after regression passed on my Ubuntu.
Do you mean which case triggers that ?
There are:

Unfortunately, I don't know which specific test case is causing the core dump at this point; identifying that is the key of addressing this issue.
The call stack indicates that the core dump is triggered by an auxiliary process, but it's unclear which test case activates that process

And if you’re asking about which core file names might be intentional?
It’s important to note that the naming of core dump files is determined by the Linux user settings using core_pattern.

/proc/sys/kernel/core_pattern

Essentially, the user configuration dictates how these files are generated and named.
What we currently have is the information from the call stack.

[edespino]

@avamingli I am definitely familiar with the kernel.core_pattern (name and location of core files). I have utilized something similar in a previous project were we controlled core file generation (including in containers).

Can you share which Ubuntu version you are using? Is there a specific branch which generates a core file? I'd like to be able to reproduce the crash in some form to test out the perform_core_file_existence_check mechanism.

[edespino]

@avamingli If you don't mind, I'll assign this issue to myself and hopefully be able to come up with a reasonable approach. I will be attempting to add this to the existing GitHub action workflow.

[avamingli]

Can you share which Ubuntu version you are using? Is there a specific branch which generates a core file? I'd like to be able to reproduce the crash in some form to test out the perform_core_file_existence_check mechanism.

Linux i-v3roa8c2 5.15.0-76-generic #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Apache Cloudberry version
Main

[avamingli]

@avamingli If you don't mind, I'll assign this issue to myself and hopefully be able to come up with a reasonable approach. I will be attempting to add this to the existing GitHub action workflow.

Yes, that sounds great! Your initiative is much appreciated.

[edespino]

Can you share which Ubuntu version you are using? Is there a specific branch which generates a core file? I'd like to be able to reproduce the crash in some form to test out the perform_core_file_existence_check mechanism.

Linux i-v3roa8c2 5.15.0-76-generic #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Apache Cloudberry version
Main

@avamingli Last set of questions ... I hope :) ... Can you share:

• The output of the following commands:

$GPHOME/bin/pg_config --configure
gcc --version

• How you ran the regression test (PGOPTIONS='-c optimizer=off' make --directory=~/cloudberry installcheck) in which a core file was generated?

Initially, I will attempt to reproduce the issue manually on Ubuntu 22.04 and go from there.

[avamingli]

$GPHOME/bin/pg_config --configure

$GPHOME/bin/pg_config --configure
'--prefix=/home/gpadmin/install/gp9' '--enable-debug' '--enable-cassert' '--with-perl' '--enable-tap-tests' 'CFLAGS=-O0 -g3' 'CXXFLAGS=-O0 -g3'

gcc --version

gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0

• How you ran the regression test (PGOPTIONS='-c optimizer=off' make --directory=~/cloudberry installcheck) in which a core file was generated?

I didn't turn ORCA off, just make a normal cluster, under regress dir and make installcheck.

[edespino]

@avamingli Again, thank you for raising this important concern about core dumps in our test suite. I've been investigating this issue and have made several improvements to help us better handle and track core dumps:

1. I've identified that the mirror_replay test suite intentionally generates a core file as part of its test scenarios. It is included in greenplum_schedule. I do not know if this is intentional behavior.

2. To better analyze core dumps, I've implemented a debug build mechanism:

   • Added debug-specific configure flags (--enable-debug, --enable-profiling, etc.)
   • Created a separate debug RPM build that preserves debug symbols
   • Debug builds are clearly marked with 'dbg' in the package name

3. During this investigation, I discovered that the resgroup/resgroup_views test exhibits a small memory difference when run with debug builds enabled.

4. I've enhanced our test infrastructure to:

   • Properly configure core dump generation
   • Ensure core dumps are detectable when they occur
   • Store core dumps in a consistent location

5. Added automated core dump detection:

   • Each test configuration now executes a new Check for Core Dumps Across All Configurations: ${{ matrix.test }} step
   • This step scans for core files and provides detailed analysis
   • Test runs will fail if unexpected core dumps are found

These changes will help us better distinguish between intentional and unexpected core dumps, while providing more detailed information for debugging when issues occur.

Can you please review the following test runs and let me know if this is they type of core dump support you were thinking of? In addition the the core file that is created in mirror_replay, you will see that this new mechanism also identified an additional core file. See if you can spot it.

• Apache Cloudberry Build (DEBUG)
• Apache Cloudberry Build