feat: Add SageMaker support

[andjsmi]

Motivation

SageMaker Endpoints support /ping for healthchecks and /invocations for invocation payloads however sglang currently doesn't support this invocation pattern to make the package usable on SageMaker Endpoints.

Modifications

This pull request adds two endpoints for /ping/ and /invocations in http_server.py.

/ping provides the same functionality as /health. At present /invocations acts the same as /v1/chat/completions however it may be worth expanding this to invoke as based on the request content.

I've included test cases as well and have been able to test on a SageMaker endpoint.

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[zhaochenyang20]

@andjsmi This should be a nice PR. But do you find someone to test it on sagemaker? We do not have the access.

[andjsmi]

Hey @zhaochenyang20.

Thanks! Yes, I've tested it on SageMaker myself and have included a screenshot below.

The main requirements are responding empty 200 OK from the /ping endpoint and then accepting POST requests on /invocations, with the ability to stream chunked encoding back. I have tested all these usecases on a SageMaker endpoint.

If there's a particular way you'd like me to test further, please let me know.

[zhyncs]

Can we use the lmsysorg/sglang:latest as the base image?

[zhyncs]

This change overall looks good, I can merge it first, minor changes can be submitted in a follow-up, thank you very much for AWS's support!