Fix NTFS SD card mount failure by adding sync and udevadm settle (BugFix) by Jefferyyen · Pull Request #1717 · canonical/checkbox

Jefferyyen · 2025-02-11T09:00:37Z

Fix an issue where Checkbox aborts during the SDHC storage test due to a race condition when mounting an NTFS-formatted SD card. The test fails with:

ntfs-3g-mount: mount failed: Device or resource busy

because mount_usb_storage() executes before Ubuntu's auto-mount process completes.

Add sync to flush pending writes and udevadm settle to wait for udev events, ensuring the device is ready before mounting.

Description

This PR enhances the storage mounting function in Checkbox by adding sync and udevadm settle. These additions ensure that the system has fully registered the device before attempting to mount it, preventing premature failures.

Resolved issues

When running the insertion + read/write + removal test, if the SD card is formatted as NTFS, Checkbox may attempt to mount the device before Ubuntu's auto-mount process is complete. This results in an error log:

ntfs-3g-mount: mount failed: Device or resource busy

which causes the test to abort prematurely.

By adding sync and udevadm settle, we ensure that the system completes the device registration before proceeding with the mounting process, preventing this issue.

Documentation

Tests

The fix was verified by running the following test procedure:

Test Steps:

Run the command:checkbox-cli run com.canonical.certification::mediacard/sdhc-storage-manual
When prompted, insert an NTFS-formatted SD card.
Observe the output logs:

--------- Testing insertion ---------  
INSERT NOW  
Timeout: 30 seconds

Checkbox detects the card and attempts to mount it. Before applying this fix, the following error occurs:

------- Insertion test passed -------  
--------- Testing read/write --------  
DEBUG: generating a random file  
Mounting the USB storage  
sda1  
DEBUG: try to mount USB storage for testing  
ntfs-3g-mount: mount failed: Device or resource busy    🔴this one
ERROR: mount /dev/sda1 on /tmp/tmpXXXXXX failed.

With the applied fix, sync and udevadm settle ensure that the mounting process does not fail due to a premature device state.

Fix an issue where Checkbox aborts during the SDHC storage test due to a race condition when mounting an NTFS-formatted SD card. The test fails with: ntfs-3g-mount: mount failed: Device or resource busy because `mount_usb_storage()` executes before Ubuntu's auto-mount process completes. Add `sync` to flush pending writes and `udevadm settle` to wait for udev events, ensuring the device is ready before mounting. Signed-off-by: Jeffery Yen <songpao226@gmail.com> Co-authored-by: Wayne Lin <q123456462@gmail.com>

codecov · 2025-02-11T09:03:22Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 49.14%. Comparing base (9a81f3b) to head (fc66d65).
Report is 176 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1717      +/-   ##
==========================================
+ Coverage   49.12%   49.14%   +0.02%     
==========================================
  Files         372      372              
  Lines       40338    40334       -4     
  Branches     6809     6810       +1     
==========================================
+ Hits        19816    19824       +8     
+ Misses      19799    19786      -13     
- Partials      723      724       +1

Flag	Coverage Δ
checkbox-support	`62.17% <100.00%> (+0.19%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

fernando79513

Thank you very much for this PR!
It's really well documented and explained. I've just added a couple of comments.
The main issue is that I think you have to validate your gpg keys. That will fix the issue with the deb validator and with the merge of the PR.

checkbox-support/checkbox_support/scripts/tests/test_mount_usb_storage.py

.github/workflows/deb_validator.yaml

checkbox-support/checkbox_support/scripts/tests/test_mount_usb_storage.py

…rage.py to test_usb_read_write.py

checkbox-support/checkbox_support/scripts/tests/test_usb_read_write.py

Co-authored-by: Fernando Bravo <39527354+fernando79513@users.noreply.github.com>

- Simplify test: mock subprocess and check log - Enforce comments under 80 characters - Merge import statements from the same module - Remove unused imports - Update copyright and add author info

Jefferyyen · 2025-03-20T04:03:30Z

Thank you for your suggestion. I've committed the changes to the PR again. Please don't hesitate to let me know if anything needs to change.

fernando79513 · 2025-03-20T09:15:28Z

As mentioned here, please try to remove the changes to the GitHub actions, these shouldn't be needed after the latest fixes, If they still don't work, try rebasing your branch with main again.
Also, since you have merged checkbox-support/checkbox_support/tests/test_usb_read_write.py with checkbox-support/checkbox_support/scripts/tests/test_usb_read_write.py, you can now remove checkbox-support/checkbox_support/tests/test_usb_read_write.py

Jefferyyen · 2025-03-23T07:30:02Z

Thanks for the feedback. I've removed the GitHub Actions changes and deleted the redundant test file. Please let me know if you have any further suggestions.

fernando79513

Thanks a lot for finding this bug and for this change!
LGTM +1!

…Fix) (#1717) * Fix NTFS SD card mount failure by adding sync and udevadm settle Fix an issue where Checkbox aborts during the SDHC storage test due to a race condition when mounting an NTFS-formatted SD card. The test fails with: ntfs-3g-mount: mount failed: Device or resource busy because `mount_usb_storage()` executes before Ubuntu's auto-mount process completes. Add `sync` to flush pending writes and `udevadm settle` to wait for udev events, ensuring the device is ready before mounting. Signed-off-by: Jeffery Yen <songpao226@gmail.com> Co-authored-by: Wayne Lin <q123456462@gmail.com> * Fix PPA GPG key retrieval issue in CI * Fix CI: Use gpg --dearmor for PPA key handling * Add test for mount_usb_storage functionality * Format test_mount_usb_storage.py using Black * Update test contents from existing file and rename test_mount_usb_storage.py to test_usb_read_write.py * Update test contents and merge tests into test_usb_read_write.py * Apply Black formatting to test_usb_read_write.py * Delet the wrong comment * Apply Black formatting to test_usb_read_write.py * Simplify test: mock subprocess and check log Co-authored-by: Fernando Bravo <39527354+fernando79513@users.noreply.github.com> * keep comments under 80 lines Co-authored-by: Fernando Bravo <39527354+fernando79513@users.noreply.github.com> * Try to keep comments under 80 lines Co-authored-by: Fernando Bravo <39527354+fernando79513@users.noreply.github.com> * Merge import statements from the same module Co-authored-by: Fernando Bravo <39527354+fernando79513@users.noreply.github.com> * Remove unused imports Co-authored-by: Fernando Bravo <39527354+fernando79513@users.noreply.github.com> * Update copyright and author Co-authored-by: Fernando Bravo <39527354+fernando79513@users.noreply.github.com> * Refactor and cleanup code - Simplify test: mock subprocess and check log - Enforce comments under 80 characters - Merge import statements from the same module - Remove unused imports - Update copyright and add author info * Remove old test_usb_read_write.py * Revert changes to GitHub Actions configuration in deb_validator.yaml --------- Signed-off-by: Jeffery Yen <songpao226@gmail.com> Co-authored-by: Wayne Lin <q123456462@gmail.com> Co-authored-by: Fernando Bravo <39527354+fernando79513@users.noreply.github.com>

Jefferyyen changed the title ~~Fix NTFS SD card mount failure by adding sync and udevadm settle~~ Fix NTFS SD card mount failure by adding sync and udevadm settle (BugFix) Feb 11, 2025

Jefferyyen added 4 commits February 11, 2025 19:36

Fix PPA GPG key retrieval issue in CI

3cfd0d3

Fix CI: Use gpg --dearmor for PPA key handling

8a85ecd

Add test for mount_usb_storage functionality

3b2a234

Format test_mount_usb_storage.py using Black

ee5de08

fernando79513 self-assigned this Feb 14, 2025

fernando79513 reviewed Feb 26, 2025

View reviewed changes

checkbox-support/checkbox_support/scripts/tests/test_mount_usb_storage.py Outdated Show resolved Hide resolved

.github/workflows/deb_validator.yaml Outdated Show resolved Hide resolved

checkbox-support/checkbox_support/scripts/tests/test_mount_usb_storage.py Outdated Show resolved Hide resolved

Jefferyyen added 5 commits February 27, 2025 16:25

Update test contents from existing file and rename test_mount_usb_sto…

8991e72

…rage.py to test_usb_read_write.py

Update test contents and merge tests into test_usb_read_write.py

3d8b662

Apply Black formatting to test_usb_read_write.py

efc30ee

Delet the wrong comment

5d86d65

Apply Black formatting to test_usb_read_write.py

c502265

fernando79513 reviewed Mar 18, 2025

View reviewed changes

fernando79513 added the waiting-for-changes The review has been completed but the PR is waiting for changes from the author label Mar 19, 2025

Jefferyyen and others added 7 commits March 20, 2025 10:07

Simplify test: mock subprocess and check log

28d0f08

Co-authored-by: Fernando Bravo <39527354+fernando79513@users.noreply.github.com>

keep comments under 80 lines

61556f7

Co-authored-by: Fernando Bravo <39527354+fernando79513@users.noreply.github.com>

Try to keep comments under 80 lines

0142347

Co-authored-by: Fernando Bravo <39527354+fernando79513@users.noreply.github.com>

Merge import statements from the same module

0f99aff

Co-authored-by: Fernando Bravo <39527354+fernando79513@users.noreply.github.com>

Remove unused imports

a4fdb5f

Co-authored-by: Fernando Bravo <39527354+fernando79513@users.noreply.github.com>

Update copyright and author

98c6c2c

Co-authored-by: Fernando Bravo <39527354+fernando79513@users.noreply.github.com>

Refactor and cleanup code

ed5d4fa

- Simplify test: mock subprocess and check log - Enforce comments under 80 characters - Merge import statements from the same module - Remove unused imports - Update copyright and add author info

Jefferyyen added 2 commits March 23, 2025 14:51

Remove old test_usb_read_write.py

1a02fe6

Revert changes to GitHub Actions configuration in deb_validator.yaml

fc66d65

fernando79513 approved these changes Mar 24, 2025

View reviewed changes

fernando79513 merged commit 04293a1 into canonical:main Mar 25, 2025
19 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix NTFS SD card mount failure by adding sync and udevadm settle (BugFix)#1717

Fix NTFS SD card mount failure by adding sync and udevadm settle (BugFix)#1717
fernando79513 merged 19 commits intocanonical:mainfrom
Jefferyyen:fix-ntfs-mount

Jefferyyen commented Feb 11, 2025

Uh oh!

codecov bot commented Feb 11, 2025 •

edited

Loading

Uh oh!

fernando79513 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jefferyyen commented Mar 20, 2025

Uh oh!

fernando79513 commented Mar 20, 2025 •

edited

Loading

Uh oh!

Jefferyyen commented Mar 23, 2025

Uh oh!

fernando79513 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Jefferyyen commented Feb 11, 2025

Description

Resolved issues

Documentation

Tests

Uh oh!

codecov bot commented Feb 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

fernando79513 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jefferyyen commented Mar 20, 2025

Uh oh!

fernando79513 commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jefferyyen commented Mar 23, 2025

Uh oh!

fernando79513 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Feb 11, 2025 •

edited

Loading

fernando79513 commented Mar 20, 2025 •

edited

Loading