Add backpressure guard for chat text generations

[pandego]

Summary

Add backpressure protection for chat text generations in the master API.

Problem

When too many text generations are already in flight, new requests can overload the master process and degrade responsiveness.

Root cause

There was no explicit in-flight guard for text generation requests at the API entry point.

Fix

• Add a configurable backpressure guard for in-flight text generation requests.
• Return HTTP 429 when the in-flight count is at or above the configured limit.
• Add focused tests for both allow and reject behavior.

Validation

• uv run pytest src/exo/master/tests/test_text_generation_backpressure.py src/exo/master/tests/test_cancel_command.py

Closes #1664.

[Evanev7]

i like the direction - can we pass this through the construction of the API & a cli flag rather than an env var? i'm working towards implementing live configuration and it'd be good to start plumbing these things instead of using env.

[pandego]

Great suggestion - done in acef4c92.

I replaced the env-based limit with explicit API/CLI plumbing:

• added CLI flag --max-in-flight-text-generations (default 2)
• wired Args.max_in_flight_text_generations through Node.create into API(...)
• updated API to use self.max_in_flight_text_generations in _enforce_text_generation_backpressure
• updated backpressure tests to assert configured limit behavior without env patching

Validation run:

• uv run pytest src/exo/master/tests/test_text_generation_backpressure.py -q (2 passed)

[pandego]

Refreshed this branch onto current main and force-pushed the updated PR branch.

Validation blocker from this environment:

• uv run pytest src/exo/master/tests/test_text_generation_backpressure.py src/exo/master/tests/test_cancel_command.py -q
• blocked while building the mlx dependency because the local toolchain cannot find metal (xcrun: error: unable to find utility "metal")

The PR diff is still the intended focused scope against current main:

• src/exo/main.py
• src/exo/master/api.py
• src/exo/master/tests/test_text_generation_backpressure.py

[pandego]

I rebased this branch onto the current main so the dirty merge state is cleared.

Validation note: I re-ran the focused local check path, but this environment still hits the same mlx Metal Toolchain blocker before the target path can complete, so I could not produce a fresh green local run here.

If you want, I can keep digging on the mlx blocker next, but the branch itself is refreshed and ready for another look.

[pandego]

I rebased this branch onto the current main so the dirty merge state is cleared.

Validation note: I re-ran the focused local check path, but this environment still hits the same mlx Metal Toolchain blocker before the target path can complete, so I could not produce a fresh green local run here.

If you want, I can keep digging on the mlx blocker next, but the branch itself is refreshed and ready for another look.