Add object store support to llm-d storage offloading

[effi-ofer]

This PR adds object store backend to the llm-d storage offloading. It does this by introducing a "backend" configuration parameter which can be set as follows, with POSIX as the default.

  "POSIX"           - POSIX engine, GDS disabled
  "POSIX_GDS_READ"  - GDS for reads only
  "POSIX_GDS_WRITE" - GDS for writes only
  "POSIX_GDS"       - GDS for reads and writes
  "POSIX_BB_READ"   - bounce-buffer GDS for reads only
  "POSIX_BB_WRITE"  - bounce-buffer GDS for writes only
  "POSIX_BB"        - bounce-buffer GDS for reads and writes
  "OBJ"             - NIXL S3 object store backend

The object store access is implemented using NIXL (for PUT/GET) and boto3 (for lookup).

Documentation:
The full configuration of the object store is described in docs/object_store.md.

Testing:

• all the tests in the tests directories
• added a new test_obj_backend test case

[github-actions]

Unsigned commits detected! Please sign your commits.

For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation.

[kfirtoledo]

Can you rename the files to manager and worker like the fs

[effi-ofer]

It's not really a manager and worker. Rather it's encapsulation that was used to support the various nixl backends (posix, obj, gds). I can flatten it all down since we support only obj in the PR. But keeping the code in this form doesn't increase the complexity or decrease the readability of the code and enable us to very quickly add other nixl backends and run tests on them. Let me know what you think.

[kfirtoledo]

I think the structure should be similar like storage backend and CPU backend, I don't see a reason this backend will be different

[kfirtoledo]

I don't think that anything that related to object store should be in the file system

[kfirtoledo]

You should open an issue about that

[kfirtoledo]

check_xfer_state is called unwrapped here, but get_finished wraps the same call in try/except nixlBackendError. Inconsistent — if it can raise in one place, it can raise in the other.

[kfirtoledo]

while True: time.sleep(0.1) with no upper bound — if NIXL's state machine wedges, this hangs forever. Consider a timeout: float | None argument so shutdown paths can bail out.

[kfirtoledo]

Tuple swap makes the intent obvious ("take what was there, replace with empty"):

results, self._pending_results = self._pending_results, []

[kfirtoledo]

_complete_read blocks the polling loop. Line 110 does self._h2d_stream.synchronize() — called from get_finished() → _complete_transfer() → _complete_read(). If many blocks finish at once, every get_finished() call waits for all H2D copies on all just-completed jobs. Under load, this serializes completion. For an async engine, surprising.

[kfirtoledo]

make test runs pytest ./tests/ -v -sq across everything, so this test is picked up even when no S3 is available. The docstring says "Skips the entire session if required credentials are not provided," but assert raises AssertionError — pytest reports it as a failure, not a skip. Please use pytest.skip(...) instead:

if not endpoint:
    pytest.skip("OBJ endpoint not set — skipping S3 integration tests")
if not bucket:
    pytest.skip("OBJ bucket not set")
# ...

That way make test passes cleanly on dev machines without S3 and still exercises the path on CI where credentials are configured.

[kfirtoledo]

Objects written to S3 are never cleaned up. Every test run leaves orphan objects under root_dir="kv-test", so the bucket grows without bound. Either add a fixture teardown that deletes objects by prefix, or use a unique per-run prefix (UUID/timestamp) so leftovers are at least separable.

[kfirtoledo]

The module docstring says "Assumes the object store bucket already exists and is reachable," but nothing checks this. If the bucket doesn't exist the test fails cryptically deep in NIXL. A one-line head_bucket probe inside obj_config that calls pytest.skip(...) on failure would surface the problem clearly (and also covers the case where the endpoint is configured but unreachable).

[kfirtoledo]

Naming inconsistency: other options use hyphens (--obj-access-key, --obj-secret-key) but this one mixes a hyphen then an underscore (--obj-ca_bundle). Suggest --obj-ca-bundle for consistency.