Limit the number of connections used by snapshot file downloads during recoveries

[fcofdez]

Today we limit the number of concurrent snapshot file downloads during recoveries to 5 per recovery:

elasticsearch/server/src/main/java/org/elasticsearch/indices/recovery/RecoverySettings.java

Lines 141 to 148 in 9958c3c

	public static final Setting<Integer> INDICES_RECOVERY_MAX_CONCURRENT_SNAPSHOT_FILE_DOWNLOADS =
	Setting.intSetting("indices.recovery.max_concurrent_snapshot_file_downloads",
	5,
	1,
	20,
	Property.Dynamic,
	Property.NodeScope
	);

This setting combined with the default node_concurrent_recoveries=2 works well in most scenarios.

elasticsearch/server/src/main/java/org/elasticsearch/cluster/routing/allocation/decider/ThrottlingAllocationDecider.java

Line 49 in 6e875d0

public static final int DEFAULT_CLUSTER_ROUTING_ALLOCATION_NODE_CONCURRENT_RECOVERIES = 2;

But when the number of concurrent recoveries is higher, it's possible to exhaust the repository connections
just with recoveries, possibly affecting other repository operations such as snapshot/restore.

We should limit the number of concurrent snapshot file downloads in the target node to 25 as that would leave
room for the rest of the operations to make progress if node_concurrent_recoveries is larger than the default.

Relates #73496

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)