Always resize! solver (volume integral) after AMR

[DanielDoehring]

I noted that if we call a resize! function in the AMR routine, we might be able to condense the functions from #2764 quite significantly.

Furthermore, we already have similar functionality:

Trixi.jl/src/solvers/dg.jl

Lines 605 to 615 in e5ba942

	# Required to be able to run `SimpleSSPRK33` without `VolumeIntegralSubcellLimiting`
	Base.resize!(semi, volume_integral::AbstractVolumeIntegral, new_size) = nothing
	function Base.resize!(semi, volume_integral::VolumeIntegralSubcellLimiting, new_size)
	# Resize container antidiffusive_fluxes
	resize!(semi.cache.antidiffusive_fluxes, new_size)
	# Resize container subcell_limiter_coefficients
	@unpack limiter = volume_integral
	return resize!(limiter.cache.subcell_limiter_coefficients, new_size)
	end

[github-actions]

Review checklist

This checklist is meant to assist creators of PRs (to let them know what reviewers will typically look for) and reviewers (to guide them in a structured review process). Items do not need to be checked explicitly for a PR to be eligible for merging.

Purpose and scope

• The PR has a single goal that is clear from the PR title and/or description.
• All code changes represent a single set of modifications that logically belong together.
• No more than 500 lines of code are changed or there is no obvious way to split the PR into multiple PRs.

Code quality

• The code can be understood easily.
• Newly introduced names for variables etc. are self-descriptive and consistent with existing naming conventions.
• There are no redundancies that can be removed by simple modularization/refactoring.
• There are no leftover debug statements or commented code sections.
• The code adheres to our conventions and style guide, and to the Julia guidelines.

Documentation

• New functions and types are documented with a docstring or top-level comment.
• Relevant publications are referenced in docstrings (see example for formatting).
• Inline comments are used to document longer or unusual code sections.
• Comments describe intent ("why?") and not just functionality ("what?").
• If the PR introduces a significant change or new feature, it is documented in NEWS.md with its PR number.

Testing

• The PR passes all tests.
• New or modified lines of code are covered by tests.
• New or modified tests run in less then 10 seconds.

Performance

• There are no type instabilities or memory allocations in performance-critical parts.
• If the PR intent is to improve performance, before/after time measurements are posted in the PR.

Verification

• The correctness of the code was verified using appropriate tests.
• If new equations/methods are added, a convergence test has been run and the results
  are posted in the PR.

Created with ❤️ by the Trixi.jl community.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.12%. Comparing base (adbf238) to head (6e9a04d).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2818      +/-   ##
==========================================
+ Coverage   97.10%   97.12%   +0.01%     
==========================================
  Files         592      592              
  Lines       45621    45644      +23     
==========================================
+ Hits        44299    44328      +29     
+ Misses       1322     1316       -6     

Flag	Coverage Δ
unittests	97.12% <100.00%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jlchan]

Thanks for adding this - did you run any benchmarks for before/after this PR?

[ranocha]

Thanks, this sounds reasonable in general.

[ranocha]

Thanks for adding this - did you run any benchmarks for before/after this PR?

@DanielDoehring?

[DanielDoehring]

Thanks for adding this - did you run any benchmarks for before/after this PR?

Essentially this PR adds only a few wrappers on top of the existing functionality, which makes it hard to benchmark this.
In a real simulation, for instance, I get for https://github.com/trixi-framework/Trixi.jl/blob/main/examples/p4est_3d_dgsem/elixir_advection_amr_unstructured_curved.jl with amr_interval = 1

AMR                                       179    1.85s   45.5%  10.3ms   1.16GiB   97.0%  6.65MiB
  coarsen                                 179    942ms   23.2%  5.26ms    682MiB   55.6%  3.81MiB
    solver                                179    672ms   16.5%  3.75ms    676MiB   55.1%  3.78MiB
      reinitialize data structures         94    524ms   12.9%  5.57ms    264MiB   21.5%  2.81MiB
      ~solver~                            179    148ms    3.6%   826μs    412MiB   33.6%  2.30MiB
    mesh                                  179    270ms    6.7%  1.51ms   6.02MiB    0.5%  34.4KiB
      rebalance                           179    263ms    6.5%  1.47ms     0.00B    0.0%    0.00B
      ~mesh~                              179   5.35ms    0.1%  29.9μs   5.36MiB    0.4%  30.7KiB
      coarsen!                            179   1.59ms    0.0%  8.91μs    678KiB    0.1%  3.79KiB
    ~coarsen~                             179    124μs    0.0%   692ns   1.47KiB    0.0%    8.40B
  refine                                  179    827ms   20.4%  4.62ms    506MiB   41.3%  2.83MiB
    solver                                179    549ms   13.5%  3.07ms    505MiB   41.1%  2.82MiB
      reinitialize data structures         65    428ms   10.5%  6.58ms    200MiB   16.3%  3.07MiB
      ~solver~                            179    121ms    3.0%   678μs    305MiB   24.9%  1.70MiB
    mesh                                  179    278ms    6.9%  1.55ms   1.77MiB    0.1%  10.1KiB
      rebalance                           179    274ms    6.8%  1.53ms     0.00B    0.0%    0.00B
      ~mesh~                              179   2.05ms    0.1%  11.4μs   1.77MiB    0.1%  10.1KiB
      refine                              179   1.75ms    0.0%  9.79μs     0.00B    0.0%    0.00B
    ~refine~                              179   91.6μs    0.0%   512ns   1.47KiB    0.0%    8.40B
  indicator                               179   72.6ms    1.8%   405μs   0.99MiB    0.1%  5.66KiB
  ~AMR~                                   179   3.30ms    0.1%  18.4μs   1.15MiB    0.1%  6.59KiB
initial condition AMR                       1   36.4ms    0.9%  36.4ms   34.7MiB    2.8%  34.7MiB
  AMR                                       3   35.3ms    0.9%  11.8ms   34.7MiB    2.8%  11.6MiB
    refine                                  3   34.1ms    0.8%  11.4ms   33.0MiB    2.7%  11.0MiB
      solver                                3   28.8ms    0.7%  9.59ms   33.0MiB    2.7%  11.0MiB
        reinitialize data structures        2   27.1ms    0.7%  13.5ms   25.5MiB    2.1%  12.7MiB
        ~solver~                            3   1.72ms    0.0%   573μs   7.49MiB    0.6%  2.50MiB
      mesh                                  3   5.37ms    0.1%  1.79ms   29.9KiB    0.0%  10.0KiB
        rebalance                           3   5.27ms    0.1%  1.76ms     0.00B    0.0%    0.00B
        refine                              3   57.7μs    0.0%  19.2μs     0.00B    0.0%    0.00B
        ~mesh~                              3   35.6μs    0.0%  11.9μs   29.9KiB    0.0%  10.0KiB
      ~refine~                              3   2.24μs    0.0%   747ns   1.47KiB    0.0%     501B
    indicator                               3   1.04ms    0.0%   347μs   32.6KiB    0.0%  10.9KiB
    ~AMR~                                   3    107μs    0.0%  35.7μs   1.65MiB    0.1%   564KiB
    coarsen                                 3    370ns    0.0%   123ns      192B    0.0%    64.0B
  ~initial condition AMR~                   1   1.15ms    0.0%  1.15ms      752B    0.0%     752B

i.e., reinitialize_datastructures takes 979 ms

while the new change gives

AMR                                       179    1.77s   44.5%  9.90ms   1.16GiB   97.0%  6.65MiB
  coarsen                                 179    934ms   23.4%  5.22ms    682MiB   55.6%  3.81MiB
    solver                                179    664ms   16.7%  3.71ms    676MiB   55.1%  3.78MiB
      reinitialize data structures         94    542ms   13.6%  5.76ms    264MiB   21.5%  2.81MiB
      ~solver~                            179    122ms    3.1%   683μs    412MiB   33.6%  2.30MiB
    mesh                                  179    270ms    6.8%  1.51ms   6.02MiB    0.5%  34.4KiB
      rebalance                           179    263ms    6.6%  1.47ms     0.00B    0.0%    0.00B
      ~mesh~                              179   5.82ms    0.1%  32.5μs   5.36MiB    0.4%  30.7KiB
      coarsen!                            179   1.45ms    0.0%  8.09μs    678KiB    0.1%  3.79KiB
    ~coarsen~                             179    101μs    0.0%   563ns   1.47KiB    0.0%    8.40B
  refine                                  179    763ms   19.2%  4.27ms    506MiB   41.3%  2.83MiB
    solver                                179    488ms   12.3%  2.73ms    505MiB   41.1%  2.82MiB
      reinitialize data structures         65    406ms   10.2%  6.24ms    200MiB   16.3%  3.07MiB
      ~solver~                            179   82.7ms    2.1%   462μs    305MiB   24.9%  1.70MiB
    mesh                                  179    275ms    6.9%  1.54ms   1.77MiB    0.1%  10.1KiB
      rebalance                           179    269ms    6.7%  1.50ms     0.00B    0.0%    0.00B
      ~mesh~                              179   4.40ms    0.1%  24.6μs   1.77MiB    0.1%  10.1KiB
      refine                              179   1.56ms    0.0%  8.72μs     0.00B    0.0%    0.00B
    ~refine~                              179    100μs    0.0%   557ns   1.47KiB    0.0%    8.40B
  indicator                               179   71.8ms    1.8%   401μs   0.99MiB    0.1%  5.66KiB
  ~AMR~                                   179   3.16ms    0.1%  17.7μs   1.15MiB    0.1%  6.59KiB
initial condition AMR                       1   25.9ms    0.6%  25.9ms   34.7MiB    2.8%  34.7MiB
  AMR                                       3   24.7ms    0.6%  8.24ms   34.7MiB    2.8%  11.6MiB
    refine                                  3   23.6ms    0.6%  7.87ms   33.0MiB    2.7%  11.0MiB
      solver                                3   18.2ms    0.5%  6.07ms   33.0MiB    2.7%  11.0MiB
        reinitialize data structures        2   17.1ms    0.4%  8.54ms   25.5MiB    2.1%  12.7MiB
        ~solver~                            3   1.14ms    0.0%   381μs   7.49MiB    0.6%  2.50MiB
      mesh                                  3   5.38ms    0.1%  1.79ms   29.9KiB    0.0%  10.0KiB
        rebalance                           3   5.29ms    0.1%  1.76ms     0.00B    0.0%    0.00B
        refine                              3   48.6μs    0.0%  16.2μs     0.00B    0.0%    0.00B
        ~mesh~                              3   35.9μs    0.0%  12.0μs   29.9KiB    0.0%  10.0KiB
      ~refine~                              3   1.94μs    0.0%   647ns   1.47KiB    0.0%     501B
    indicator                               3   1.02ms    0.0%   341μs   32.6KiB    0.0%  10.9KiB
    ~AMR~                                   3   91.6μs    0.0%  30.5μs   1.65MiB    0.1%   564KiB
    coarsen                                 3    440ns    0.0%   147ns      192B    0.0%    64.0B
  ~initial condition AMR~                   1   1.14ms    0.0%  1.14ms      752B    0.0%     752B
─────────────────────────────────────────────────────────────────────────────────────────────────

i.e., reinitialize_datastructures takes 965 ms

overall I would say this is expected.

If we are really concerned about this we could think about inlining some functions I guess