[v25.2.x] [CORE-14558] `storage`: prevent race between `disk_log_appender` and `remove_prefix_full_segments()`

[vbotbuildovich]

Backport #28472 to branch v25.2.x. This issue was created as a placeholder because the original PR's commit(s) could not be automatically cherry-picked.
Here are the commands to execute:

git checkout v25.2.x
git checkout -b manual-backport-28472-v25.2.x-580
git cherry-pick -x f7ca2ec7da df9089acbf 485b24d8ad e31ae0bdca a0ae9625a2 799ff148b7

git push origin manual-backport-28472-v25.2.x-580
gh pr create \
  --title "[v25.2.x] [CORE-14558] `storage`: prevent race between `disk_log_appender` and `remove_prefix_full_segments()`" \
  --base "v25.2.x" \
  --label "kind/backport" \
  --head "manual-backport-28472-v25.2.x-580" \
  --draft \
  --repo "redpanda-data/redpanda" \
  --milestone "v25.2.x-next" \
  --body "Backport of PR https://github.com/redpanda-data/redpanda/pull/28472 "

[CORE-14558]: https://redpandadata.atlassian.net/browse/CORE-14558?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ