[serve] Support 'root_path' parameter across uvicorn versions

[axreldable]

Why are these changes needed?

Since 0.26.0 uvicorn changed how it processes root_path. To support all uvicorn versions, injecting root_path to ASGI app instead of passing it to uvicorn.Config starting from version 0.26.0.

Before the change:

# uvicorn==0.22.0
pytest -s -v python/ray/serve/tests/test_standalone.py::test_http_root_path - pass
# uvicorn==0.40.0 - latest
pytest -s -v python/ray/serve/tests/test_standalone.py::test_http_root_path - failed
# FAILED python/ray/serve/tests/test_standalone.py::test_http_root_path - assert 404 == 200

After the change:

# uvicorn==0.22.0
pytest -s -v python/ray/serve/tests/test_standalone.py::test_http_root_path - pass
# uvicorn==0.40.0 - latest
pytest -s -v python/ray/serve/tests/test_standalone.py::test_http_root_path - pass

Related issue number

Closes #55776.

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run pre-commit jobs to lint the changes in this PR. (pre-commit setup)
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[axreldable]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a compatibility layer to handle the root_path parameter across different uvicorn versions. The change correctly identifies the breaking change in uvicorn==0.26.0 and uses an ASGI middleware to inject the root_path for newer versions, while maintaining the old behavior for older versions. The logic is sound and is accompanied by a new test file with good coverage for various root_path scenarios. I have one suggestion to improve the readability of the version check logic.

[axreldable]

Hi @harshit-anyscale , @abrarsheikh ! Could you please review it?

[github-actions]

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

[ok-scale]

Hi @harshit-anyscale , @abrarsheikh ! Could you please review it?

@axreldable Looking as well :)

[axreldable]

Hi @harshit-anyscale , @abrarsheikh ! Could you please review it?

@axreldable Looking as well :)

Thank you very much for looking into it, @ok-scale ! This is my attempt to support different versions of uvicorn, but I'm not 100% sure that it's the best way to do it. I’d really appreciate your feedback if there’s a better approach.

[harshit-anyscale]

lgtm
thanks @axreldable for contributing, nice work!

[harshit-anyscale]

cc: @abrarsheikh for review.

[abrarsheikh]

one question but looks good

[cursor]

Test case expects incorrect root_path value

Medium Severity

The test case ("/root_path", "/root_path") has an incorrect expected value. When both FastAPI's root_path and Serve's root_path are set, Starlette's __call__ method concatenates them: scope["root_path"] = self.root_path + scope.get("root_path", ""). This means the actual scope["root_path"] will be /root_path/root_path, but the test expects /root_path. The assertion scope_root_path = app_root_path or serve_root_path doesn't account for Starlette's concatenation behavior.

Additional Locations (1)

• python/ray/serve/tests/test_fastapi_2.py#L24-L28

 

[abrarsheikh]

can you verify if this is true?

[axreldable]

The short answer is that the test case ("/root_path", "/root_path") verifies that the scope["root_path"] is /root_path and not /root_path/root_path.

[axreldable]

Starlette behavior is irrelevant here as we're working with uvicorn, but the same breaking change was introduced in both uvicorn (version 0.26.0) and Starlette (version 0.35.0)

[axreldable]

To understand the change better I came up with the following unit test to check the behavior of uvicorn + FastAPI app:

import multiprocessing
import socket
import time
from dataclasses import dataclass
from pprint import pprint
from typing import Any, Dict, Tuple

import httpx
import pytest
import uvicorn
from fastapi import FastAPI, Request


def _pick_free_port() -> int:
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.bind(("127.0.0.1", 0))
        return s.getsockname()[1]


def _wait_until_listening(host: str, port: int, timeout_s: float = 3.0) -> None:
    deadline = time.time() + timeout_s
    last_err = None
    while time.time() < deadline:
        try:
            with socket.create_connection((host, port), timeout=0.2):
                return
        except OSError as e:
            last_err = e
            time.sleep(0.05)
    raise RuntimeError(f"Server did not start listening on {host}:{port}. Last error: {last_err}")


def _parse_version(v: str) -> Tuple[int, int, int]:
    parts = v.split(".")
    major = int(parts[0]) if len(parts) > 0 else 0
    minor = int(parts[1]) if len(parts) > 1 else 0
    patch = int(parts[2].split("+")[0].split("-")[0]) if len(parts) > 2 else 0
    return major, minor, patch


def _uvicorn_rewrites_path_and_raw_path() -> bool:
    # behavior change starts in 0.26.0
    return _parse_version(uvicorn.__version__) >= (0, 26, 0)


def create_app(app_root_path: str) -> FastAPI:
    app = FastAPI(root_path=app_root_path)

    @app.get("/hello")
    async def hello(request: Request) -> Dict[str, Any]:
        raw = request.scope.get("raw_path")
        return {
            "uvicorn_version": uvicorn.__version__,
            "scope.root_path": request.scope.get("root_path"),
            "scope.path": request.scope.get("path"),
            "scope.raw_path": raw.decode("latin-1", errors="replace") if raw else None,
            "request.url.path": request.url.path,
        }

    return app


def run_uvicorn_in_proc(app_root_path: str, uvicorn_root_path: str, port: int) -> None:
    app = create_app(app_root_path)
    uvicorn.run(
        app,
        host="127.0.0.1",
        port=port,
        root_path=uvicorn_root_path,
        log_level="error",
    )


@dataclass(frozen=True)
class Resp:
    status: int
    json: Dict[str, Any] | None


async def _get_json(url: str) -> Resp:
    async with httpx.AsyncClient(timeout=2.0) as client:
        r = await client.get(url)
        if r.headers.get("content-type", "").startswith("application/json"):
            return Resp(r.status_code, r.json())
        return Resp(r.status_code, None)


def _start_server(app_root_path: str, uvicorn_root_path: str) -> Tuple[multiprocessing.Process, int]:
    port = _pick_free_port()
    proc = multiprocessing.Process(
        target=run_uvicorn_in_proc,
        args=(app_root_path, uvicorn_root_path, port),
        daemon=True,
    )
    proc.start()
    _wait_until_listening("127.0.0.1", port)
    return proc, port


async def verify_urls(app_root_path, uvicorn_root_path, expected_url_1, expected_url_2):
    proc, port = _start_server(app_root_path=app_root_path, uvicorn_root_path=uvicorn_root_path)
    try:
        base = f"http://127.0.0.1:{port}"

        test_urls = [
            f"{base}/hello",
            f"{base}{app_root_path}/hello",
            f"{base}{uvicorn_root_path}/hello",
            f"{base}{app_root_path}{uvicorn_root_path}/hello",
            f"{base}{uvicorn_root_path}{app_root_path}/hello",
            f"{base}{app_root_path}{app_root_path}/hello",
            f"{base}{uvicorn_root_path}{uvicorn_root_path}/hello",
        ]

        print(f"\nTesting with app_root_path='{app_root_path}', uvicorn_root_path='{uvicorn_root_path}'")
        print("Available URLs:")

        tested = set()
        working_urls_body_pairs = []
        async with httpx.AsyncClient() as client:
            for test_url in test_urls:
                if test_url not in tested:
                    try:
                        response = await client.get(test_url)
                        resp = await _get_json(f"{base}/hello")
                        if response.status_code == 200:
                            body = resp.json
                            working_urls_body_pairs.append((test_url, body))
                    except Exception as e:
                        print(f"  ✗ {test_url} - Error: {str(e)}")
                    tested.add(test_url)
            pprint(working_urls_body_pairs)
            if expected_url_1:
                assert working_urls_body_pairs[0][0].endswith(expected_url_1)
                if expected_url_2:
                    assert working_urls_body_pairs[1][0].endswith(expected_url_2)
            else:
                print(f"No available urls from {test_urls}")
                assert len(working_urls_body_pairs) == 0
    finally:
        proc.terminate()
        proc.join(timeout=2)


@pytest.mark.asyncio
@pytest.mark.parametrize("app_root_path,uvicorn_root_path,expected_path_1,expected_path_2", [
    ("", "", "/hello", ""),
    ("/app_root_path", "", "/hello", "/app_root_path/hello"),
    ("", "/uvicorn_root_path", "/hello", ""),
    ("/app_root_path", "/uvicorn_root_path", "/hello", "/app_root_path/hello"),
    ("/root_path", "/root_path", "/hello", "/root_path/hello"),
])
async def test_http_root_path_0_25(app_root_path, uvicorn_root_path, expected_path_1, expected_path_2):
    # yes | pip uninstall uvicorn && pip install uvicorn==0.25.0
    await verify_urls(app_root_path, uvicorn_root_path, expected_path_1, expected_path_2)


@pytest.mark.asyncio
@pytest.mark.parametrize("app_root_path,uvicorn_root_path,expected_url_1,expected_url_2", [
    ("", "", "/hello", ""),
    ("/app_root_path", "", "/hello", "/app_root_path/hello"),
    ("", "/uvicorn_root_path", "/hello", ""),
    ("/app_root_path", "/uvicorn_root_path", "", ""),  # no available urls
    ("/root_path", "/root_path", "/hello", ""),
])
async def test_http_root_path_0_35(app_root_path, uvicorn_root_path, expected_url_1, expected_url_2):
    # yes | pip uninstall uvicorn && pip install uvicorn==0.35.0
    await verify_urls(app_root_path, uvicorn_root_path, expected_url_1, expected_url_2)

[axreldable]

Versions 0.25.0 and 0.35.0 are used as they are smaller and greater than the breaking version 0.26.0
This is the summary for the unit test above:

Uvicorn 0.25.0 (< 0.26.0)

app_root_path	uvicorn_root_path	Working URLs	scope.root_path	scope.path	scope.raw_path	request.url.path	Notes
""	""	/hello	""	/hello	/hello	/hello	Baseline
/app_root_path	""	/hello, /app_root_path/hello	/app_root_path	/hello	/hello	/hello	App root is informational
""	/uvicorn_root_path	/hello, /uvicorn_root_path/hello	/uvicorn_root_path	/hello	/hello	/hello	Uvicorn root is informational
/app_root_path	/uvicorn_root_path	/hello, /app_root_path/hello	/app_root_path	/hello	/hello	/hello	Both roots coexist
/root_path	/root_path	/hello, /root_path/hello	/root_path	/hello	/hello	/hello	No double-prefixing

Uvicorn 0.35.0 (> 0.26.0)

app_root_path	uvicorn_root_path	Working URLs	scope.root_path	scope.path	scope.raw_path	request.url.path	Notes
""	""	/hello	""	/hello	/hello	/hello	Baseline
/app_root_path	""	/hello, /app_root_path/hello	/app_root_path	/hello	/hello	/hello	App root still informational
""	/uvicorn_root_path	/hello	/uvicorn_root_path	/uvicorn_root_path/hello	/uvicorn_root_path/hello	/uvicorn_root_path/hello	Path rewritten
/app_root_path	/uvicorn_root_path	❌ none	—	—	—	—	All URLs fail (404)
/root_path	/root_path	/hello	/root_path	/root_path/hello	/root_path/hello	/root_path/hello	Implicit prefixing

[axreldable]

I updated the unit test for this change to be more descriptive:

@pytest.mark.parametrize(
    "app_root_path,serve_root_path,expected_params_1,expected_params_2",
    [
        ("", "", ["/hello", "", "/hello"], []),
        ("/app_root_path", "", ["/hello", "/app_root_path", "/hello"], ["/app_root_path/hello", "/app_root_path", "/app_root_path/hello"]),
        ("", "/serve_root_path", ["/hello", "/serve_root_path", "/serve_root_path/hello"], []),
        ("/app_root_path", "/serve_root_path", [], []),
        ("/root_path", "/root_path", ["/hello", "/root_path", "/root_path/hello"], []),
    ],
)
def test_root_path(ray_shutdown, app_root_path, serve_root_path,expected_params_1,expected_params_2):
    """
    | Case | `app_root_path`  | `serve_root_path`  | Expected Working URL #1 (suffix) | `root_path` (req.scope) | `path` (req.scope)       | Expected Working URL #2 (suffix) | `root_path` #2   | `path` #2              |
    | ---: | ---------------- | ------------------ | -------------------------------- | ----------------------- | ------------------------ | -------------------------------- | ---------------- | ---------------------- |
    |    1 | `""`             | `""`               | `/hello`                         | `""`                    | `/hello`                 | —                                | —                | —                      |
    |    2 | `/app_root_path` | `""`               | `/hello`                         | `/app_root_path`        | `/hello`                 | `/app_root_path/hello`           | `/app_root_path` | `/app_root_path/hello` |
    |    3 | `""`             | `/serve_root_path` | `/hello`                         | `/serve_root_path`      | `/serve_root_path/hello` | —                                | —                | —                      |
    |    4 | `/app_root_path` | `/serve_root_path` | *(none)*                         | —                       | —                        | —                                | —                | —                      |
    |    5 | `/root_path`     | `/root_path`       | `/hello`                         | `/root_path`            | `/root_path/hello`       | —                                | —                | —                      |
    """
    # Works for both versions:
    # yes | pip uninstall uvicorn && pip install uvicorn==0.25.0
    # yes | pip uninstall uvicorn && pip install uvicorn==0.35.0
    app = FastAPI(root_path=app_root_path)

    @app.get("/hello")
    def func(request: Request):
        return {
            "root_path": request.scope.get("root_path"),
            "path": request.scope.get("path"),
        }

    @serve.deployment
    @serve.ingress(app)
    class App:
        pass

    serve.start(http_options={"root_path": serve_root_path})
    serve.run(App.bind())

    base = get_application_url("HTTP")
    test_urls = [
        f"{base}/hello",
        f"{base}{app_root_path}/hello",
        f"{base}{serve_root_path}/hello",
        f"{base}{app_root_path}{serve_root_path}/hello",
        f"{base}{serve_root_path}{app_root_path}/hello",
        f"{base}{app_root_path}{app_root_path}/hello",
        f"{base}{serve_root_path}{serve_root_path}/hello",
    ]

    tested = set()
    working_url_params = []
    for test_url in test_urls:
        if test_url not in tested:
            try:
                response = httpx.get(test_url)
                if response.status_code == 200:
                    body = response.json()
                    print(body)
                    params = {}
                    params["url"] = test_url
                    params["root_path"] = body["root_path"]
                    params["path"] = body["path"]
                    working_url_params.append(params)
            except Exception as e:
                print(f"  ✗ {test_url} - Error: {str(e)}")
            tested.add(test_url)
    print(working_url_params)
    if expected_params_1:
        assert working_url_params[0]["url"].endswith(expected_params_1[0])
        assert working_url_params[0]["root_path"] == expected_params_1[1]
        assert working_url_params[0]["path"] == expected_params_1[2]
        if expected_params_2:
            assert working_url_params[1]["url"].endswith(expected_params_2[0])
            assert working_url_params[1]["root_path"] == expected_params_2[1]
            assert working_url_params[1]["path"] == expected_params_2[2]
    else:
        print(f"No available urls from {test_urls}")
        assert len(working_url_params) == 0

It works across uvicorn versions.
From the url/root_path/path perspective, the behavior is the same as for the regular FastAPI+uvicorn for version 0.35.0 except the test case 2 (different path for the url /app_root_path/hello: /hello -> /app_root_path/hello)

[axreldable]

The updated test works before and after the change across uvicorn versions (tested with 0.25.0 and 0.35.0). Will verify again and open a separate PR to update the test. This version is more descriptive.

[axreldable]

Created #60270