[Feature Request] Enable default throttling with thresholds based on task type for Cluster Manager in 3.0

[gargmanik13]

Is your feature request related to a problem? Please describe

Currently, by default Cluster Manager throttling is disabled for all tasks in ClusterManagerThrottle.

Describe the solution you'd like

We can enable throttling with some thresholds for each task type by default.

Related component

Cluster Manager

Describe alternatives you've considered

No response

Additional context

No response

[andrross]

Thanks @gargmanik13! Linking #17170 for reference.

[rajiv-kv]

[Triage - 1, 2, 3, 4]

Thanks @gargmanik13 for filing the issue. We discussed the following two options

1. Enable with a default value (say 50) for all the task-type
2. Enable with a default value based on specific task-type

Please evaluate these options.

We also discussed concern of throttling for critical recovery tasks such as node-join and node-left if we enable by default. However by design, the throttling is not enabled for default task type which means that only certain task-type executions are throttled. Please confirm the same as well.

[gargmanik13]

Thanks @rajiv-kv for the suggestions.

• Proposing to use default value based on specific task-type, since certain tasks like - put_mapping with high frequency use might throttle.
• Throttling will be enabled by default only for tasks which are registered in ClusterManagerTaskThrottler, critical recovery tasks are not registered