Remove glm41v from CI to speed up CI

[byjiang1996]

Motivation

Follow up to #13222

glm41v test already exists in nightly test

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.