[diffusion] benchmark: enable performance benchmark with warmup

[fsygd]

Motivation

Performance benchmark will be better with warmup to get more accurate performance, especially when something like DeepGemm is involved. (e.g. #15403)

without warmup on H200:

with warmup on H100:

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.
• Work with maintainers to merge your PR. See the PR Merge Process

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[mickqian]

why this😂

[fsygd]

Pre-commit did this, not me. Should I rollback?

[mickqian]

@BBuf would you take a look?

[fsygd]

Any blocker？ @mickqian

[BBuf]

If warmup is enabled by default here, it might mask some performance issues that would otherwise be discoverable, such as the problem found previously here: #15511. For offline inference, you must accept the warmup cost. For serving, the first request also has to accept the warmup cost. For subsequent requests, if the resolution changes or similar things happen, deepgemm still needs to recompile, which also has a cost. These should not be removed, especially for profiling.

[fsygd]

If warmup is enabled by default here, it might mask some performance issues that would otherwise be discoverable, such as the problem found previously here: #15511. For offline inference, you must accept the warmup cost. For serving, the first request also has to accept the warmup cost. For subsequent requests, if the resolution changes or similar things happen, deepgemm still needs to recompile, which also has a cost. These should not be removed, especially for profiling.

I see, warmup by default may hide something we concern. May i add warmup for option maybe sometimes we need it, or just close the pr?

[fsygd]

Warmup is an option now, PTAL @mickqian @BBuf @RubiaCx @FlamingoPg