Limit concurrent requests to 28000

[Bslabe123]

This PR fixes the ClientConnectorErrors seen at high qps due to port exhaustion by upper bounding the number of concurrent requests to 28000. This is a temporary fix until the approach for high qps benchmarking is decided on. 28000 is roughly the number of ephemeral ports available when containerized. Added active_connections_metric gauge.

LatencyProfileGenerator:time_to_first_token_created 1.7431152424978657e+09
# HELP LatencyProfileGenerator:active_requests How many requests actively being processed
# TYPE LatencyProfileGenerator:active_requests gauge
LatencyProfileGenerator:active_requests 29888.0
# HELP LatencyProfileGenerator:active_connections How many active connections
# TYPE LatencyProfileGenerator:active_connections gauge
LatencyProfileGenerator:active_connections 28000.0

[jjk-g]

I am less inclined to hard code 28k vs set to 0 (no limit) here and catch the appropriate error, log, and retry.

The added metric and logging will help observability. The retry is effectively the same outcome (qps slowdown).

[kaushikmitr]

yes if we can log failures due to ephemeral port exhaustion so that we know the experiment is not valid and the user needs to reduce the QPS or num_prompts

[Bslabe123]

+1 to logging when we exhaust ports, added and prevented including server metrics when we exhaust ports since the wait time invalidates these. Non-server metrics could still be valuable since the measured e2e latency includes the time waiting to send the request, if no requests are ever being queued on any model server and the bottleneck is this tool then yes the experiment data would for certain be invalid.