Skip to content

ai/live: Remote signer discovery#3866

Merged
j0sh merged 21 commits intomasterfrom
ja/remote-discovery
Feb 25, 2026
Merged

ai/live: Remote signer discovery#3866
j0sh merged 21 commits intomasterfrom
ja/remote-discovery

Conversation

@j0sh
Copy link
Collaborator

@j0sh j0sh commented Feb 4, 2026

This adds an optional remote discovery feature to the remote signer node. Client using the remote signer can now discover orchestrators and their supported capabilities on the network. The remote discovery endpoint is compatible with the existing orchestrator webhook, making it usable by go-livepeer as well as local gateway SDKs.

Usage

  • Remote signer node gets a -remoteDiscovery flag. This is only usable in remote signer mode for now and is opt-in. All the other gateway-side orchestrator configuration flags are supported here: -orchWebhookUrl, -orchAddr , -orchBlocklist, -extraNodes etc. Additionally, operators can tune the orchestrator refresh interval with the existing -liveAICapReportInterval flag.

  • If the gateway is using a remote signer (via -remoteSignerUrl) and has no other orchestrator sources (eg, webhook, orchestrator list) then it will fall back to using the remote signer's discovery endpoint.

  • The discovery endpoint is GET /discover-orchestrators. It returns a JSON list of [{"address":...,"capabilities":[...]}] where the address is the URI the orchestrator can be reached at, and the capabilities is a list of capabilities the orchestrator supports, eg live-video-to-video/streamdiffusion. There is also an optional caps query parameter to return only the subset of orchestrators matching the capability (exact string match). Multiple caps parameters can be included (OR).

Remote Discovery Implementation

  • Remote discovery builds on the existing DB discovery pool's orchestrator polling. The DB discovery pool periodically (via -liveAICapReportInterval) fetches orchestrator info from the network and updates the node's capability cache.
  • The remote discovery pool lazily reads from this cache and builds a capability-indexed view for quick lookups.
  • Price filtering happens during refresh, so it doesn't return orchestrators that will be later rejected by the remote signer due to price

Supporting Changes

  • DB Discovery pool and Webhook Discovery pool are updated to use a "builder pattern" of config structs + initializer, matching the regular orchestrator pool. The existing constructor interfaces are left untouched, but the plumbing underneath uses the new builder pattern. This is done to support the IgnoreCapacityCheck flag (see below)

  • To avoid returning capacity errors during the DB Discovery refresh call (it is capabilities we want), we need to thread through the IgnoreCapacityCheck flag a couple places from the webhook, DB discovery, etc. Note that this flag is only in enabled when the node is in remote signer mode, so this change should not affect other modes such as the gateway.

  • DB Discovery cache refreshes now incorporate the ExtraNodes field in order to fully traverse the orchestrators available on the network.

  • To support price filtering during remote discovery, the PriceInfo field is added to OrchNetworkCapabilities.

  • The LiveAICapReportInterval field was added to the LivepeerNode; this is just storing an existing flag so the remote signer node can use it.

  • Tests, docs, etc.

@github-actions github-actions bot added go Pull requests that update Go code docs labels Feb 4, 2026
@j0sh j0sh force-pushed the ja/remote-discovery branch from d63d90f to 8b60101 Compare February 11, 2026 04:11
j0sh added 4 commits February 11, 2026 15:54
This requires threading through an additional field for each of the
discovery types to pass into the OrchestratorInfo request.

The behavior for non-discovery gateways should remain the same.
This reverts commit 555ab44.
@codecov
Copy link

codecov bot commented Feb 14, 2026

Codecov Report

❌ Patch coverage is 68.75000% with 100 lines in your changes missing coverage. Please review.
✅ Project coverage is 32.79594%. Comparing base (8de5e30) to head (58a3374).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
cmd/livepeer/starter/starter.go 3.63636% 53 Missing ⚠️
server/remote_discovery.go 81.95489% 16 Missing and 8 partials ⚠️
server/remote_signer.go 65.71429% 11 Missing and 1 partial ⚠️
discovery/db_discovery.go 84.12698% 5 Missing and 5 partials ⚠️
cmd/livepeer/starter/flags.go 0.00000% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@                 Coverage Diff                 @@
##              master       #3866         +/-   ##
===================================================
+ Coverage   32.56835%   32.79594%   +0.22759%     
===================================================
  Files            170         171          +1     
  Lines          41700       41993        +293     
===================================================
+ Hits           13581       13772        +191     
- Misses         27094       27189         +95     
- Partials        1025        1032          +7     
Files with missing lines Coverage Δ
common/types.go 0.00000% <ø> (ø)
core/livepeernode.go 75.16779% <ø> (ø)
discovery/discovery.go 84.12698% <100.00000%> (+0.34320%) ⬆️
discovery/wh_discovery.go 67.67677% <100.00000%> (+4.88607%) ⬆️
server/broadcast.go 80.17928% <100.00000%> (+0.19920%) ⬆️
cmd/livepeer/starter/flags.go 0.00000% <0.00000%> (ø)
discovery/db_discovery.go 68.85714% <84.12698%> (+3.75647%) ⬆️
server/remote_signer.go 63.12500% <65.71429%> (+0.31798%) ⬆️
server/remote_discovery.go 81.95489% <81.95489%> (ø)
cmd/livepeer/starter/starter.go 21.79487% <3.63636%> (-0.69987%) ⬇️

... and 2 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8de5e30...58a3374. Read the comment docs.

Files with missing lines Coverage Δ
common/types.go 0.00000% <ø> (ø)
core/livepeernode.go 75.16779% <ø> (ø)
discovery/discovery.go 84.12698% <100.00000%> (+0.34320%) ⬆️
discovery/wh_discovery.go 67.67677% <100.00000%> (+4.88607%) ⬆️
server/broadcast.go 80.17928% <100.00000%> (+0.19920%) ⬆️
cmd/livepeer/starter/flags.go 0.00000% <0.00000%> (ø)
discovery/db_discovery.go 68.85714% <84.12698%> (+3.75647%) ⬆️
server/remote_signer.go 63.12500% <65.71429%> (+0.31798%) ⬆️
server/remote_discovery.go 81.95489% <81.95489%> (ø)
cmd/livepeer/starter/starter.go 21.79487% <3.63636%> (-0.69987%) ⬇️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@j0sh j0sh marked this pull request as ready for review February 18, 2026 22:00
@j0sh
Copy link
Collaborator Author

j0sh commented Feb 18, 2026

Tagging @mjh1 / @leszko since this touches some of the existing discovery code. For context, see #3791 and #3822

Copy link
Contributor

@leszko leszko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skimmed through the PR and it looks good. Saying that, I won't be able to review it in detail.

@eliteprox eliteprox self-assigned this Feb 23, 2026
Copy link
Collaborator

@eliteprox eliteprox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes LGTM, tested remote signer capability discovery with single orch. Approved with one small nit.

// Register the remote signer endpoints
ls.HTTPMux.Handle("POST /sign-orchestrator-info", http.HandlerFunc(ls.SignOrchestratorInfo))
ls.HTTPMux.Handle("POST /generate-live-payment", http.HandlerFunc(ls.GenerateLivePayment))
if ls.LivepeerNode.OrchestratorPool != nil {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we gate the handler with ls.LivepeerNode.RemoteDiscovery?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good spot, yes that would have been a bug. Fixed in 7d31811

j0sh added 6 commits February 24, 2026 22:00
Broadcaster capability max-price state is a global, and unit test cleanup which
calls SetCapabilityMaxPrice(..., nil) can leave a "default" entry with a nil
value. The new remote discovery tests read that shared capability pricing and
previously dereferenced the nil default, causing intermittent panics and
cross-test coupling. Fix by adding a nil guard in getCapabilityMaxPrice()

Switch SetCapabilityMaxPrice() to a write lock to correctly synchronize global
state updates.
@j0sh
Copy link
Collaborator Author

j0sh commented Feb 25, 2026

Pushed a number of small cleanups to remote discovery and one bona fide fix.

Fix in 58a3374 :

  • The broadcaster price config here is pretty old code
  • Makes heavy use of globals
  • Unit tests exercising this area don't always leave things the way they found it
  • Use of globals + messy cleanup led to cross-test contamination with the new remote discovery tests (specifically, a nil panic crash)
  • Locks were also not being used correctly in the price config setter; it was taking a read lock for a write operation

@j0sh j0sh enabled auto-merge (squash) February 25, 2026 08:14
@j0sh j0sh merged commit 48c5d86 into master Feb 25, 2026
16 checks passed
@j0sh j0sh deleted the ja/remote-discovery branch February 25, 2026 08:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs go Pull requests that update Go code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants