Skip to content

Bug: --maxTotalEV flag ignored, always uses default value (20 trillion wei) #3880

@chrishobcroft

Description

@chrishobcroft

Environment

  • Version: v0.8.10 (latest as of 2026-03-13)
  • Platform: ARM64 Linux
  • Node Type: Gateway (--gateway)
  • Network: arbitrum-one-mainnet

Summary

The --maxTotalEV command-line flag is defined and parsed but never applied at runtime. The gateway always enforces the hardcoded default of 20000000000000 (20 trillion wei) regardless of the flag value provided.

This prevents using AI models like image-to-video which require higher payment limits (~30 trillion wei per request).

Steps to Reproduce

  1. Start gateway with explicit maxTotalEV value:
livepeer \
  --network arbitrum-one-mainnet \
  --gateway \
  --maxTotalEV 500000000000000 \
  --maxPricePerCapability /path/to/maxPrices.json \
  --v 99
  1. Send an image-to-video request that exceeds the default 20T limit:
curl -X POST "http://localhost:9935/image-to-video" \
  -F "model_id=stabilityai/stable-video-diffusion-img2vid-xt-1-1" \
  -F "image=@test.jpg" \
  -F "height=512" \
  -F "width=512"
  1. Observe the error in logs

Expected Behavior

Payment validation should use the flag value (500000000000000 wei):

total ticket EV 29999999999990.00000 < max total ticket EV 500000000000000.00000  ✅

Actual Behavior

Payment validation uses the hardcoded default (20000000000000 wei):

E0313 10:51:01 ai_process.go:1657 Could not create payment err="total ticket EV 29999999999990.00000 for 10 tickets > max total ticket EV 20000000000000.00000"

Note: Error shows 20 trillion despite flag set to 500 trillion.

Code Analysis

The flag appears to be correctly defined and passed through the code:

1. Flag Definition (cmd/livepeer/starter/flags.go:101)

cfg.MaxTotalEV = fs.String("maxTotalEV", *cfg.MaxTotalEV, "The maximum acceptable expected value for one PM payment")

2. Default Value (cmd/livepeer/starter/starter.go:266)

defaultMaxTotalEV := "20000000000000"

3. Parsing (cmd/livepeer/starter/starter.go:1106-1109)

maxTotalEV, _ := new(big.Rat).SetString(*cfg.MaxTotalEV)
if maxTotalEV.Cmp(big.NewRat(0, 1)) < 0 {
    panic(fmt.Errorf("-maxTotalEV must not be negative, but %v provided. Restart the node with a valid value for -maxTotalEV", *cfg.MaxTotalEV))
}

⚠️ Issue: The boolean return value from SetString is ignored. Compare with maxTicketEV parsing (lines 1099-1105) which properly validates:

maxEV, _ := new(big.Rat).SetString(*cfg.MaxTicketEV)
if maxEV == nil {  // ← This check is missing for maxTotalEV
    panic(fmt.Errorf("-maxTicketEV must be a valid rational number..."))
}

4. Sender Creation (cmd/livepeer/starter/starter.go:1124)

n.Sender = pm.NewSender(n.Eth, timeWatcher, senderWatcher, maxEV, maxTotalEV, *cfg.DepositMultiplier)

5. Validation (pm/sender.go:220-222)

totalEV := ev.Mul(ev, new(big.Rat).SetInt64(int64(numTickets)))
if totalEV.Cmp(s.maxTotalEV) > 0 {
    return fmt.Errorf("total ticket EV %v for %v tickets > max total ticket EV %v", ...)
}

Hypothesis

Despite the flag being defined and passed through, the actual value of *cfg.MaxTotalEV at line 1106 appears to still be the default "20000000000000" instead of the user-provided value.

Possible causes:

  1. Flag parsing system not updating the config pointer
  2. Config being overridden by a file or environment variable
  3. Sender being recreated later with default values

Logs

Command line verified via ps aux:

--maxTotalEV 500000000000000

Runtime error from journalctl:

Mar 13 10:51:01 livepeer[790611]: E0313 10:51:01.973430 ai_process.go:1657 
  Could not create payment err="total ticket EV 29999999999990.00000 for 10 tickets > max total ticket EV 20000000000000.00000"

No startup logs show the maxTotalEV value being applied (unlike other config values which appear in the startup table).

Suggested Fix

Add proper validation and logging similar to maxTicketEV:

maxTotalEV, ok := new(big.Rat).SetString(*cfg.MaxTotalEV)
if !ok || maxTotalEV == nil {
    panic(fmt.Errorf("-maxTotalEV must be a valid rational number, but %v provided. Restart the node with a valid value for -maxTotalEV", *cfg.MaxTotalEV))
}
if maxTotalEV.Cmp(big.NewRat(0, 1)) < 0 {
    panic(fmt.Errorf("-maxTotalEV must not be negative, but %v provided. Restart the node with a valid value for -maxTotalEV", *cfg.MaxTotalEV))
}
glog.Infof("Using maxTotalEV: %v wei", maxTotalEV.FloatString(0))  // ← Add logging

Additionally, include maxTotalEV in the startup configuration table that's logged (like MaxPricePerCapability at line 1152).

Impact

This bug prevents gateways from using AI models that require payments exceeding 20 trillion wei per request, including:

  • stabilityai/stable-video-diffusion-img2vid-xt-1-1 (image-to-video)
  • Potentially other high-cost AI pipelines

Workaround

The only current workaround is to artificially lower the maxPricePerCapability values to keep total payments under 20 trillion wei, which excludes higher-priced orchestrators from the selection pool.

Additional Context

Tested with quoted and unquoted flag values:

  • --maxTotalEV 500000000000000
  • --maxTotalEV "500000000000000"

Both produce the same result (default value used at runtime).


Related Code References:

  • Flag definition: cmd/livepeer/starter/flags.go:101
  • Default value: cmd/livepeer/starter/starter.go:266
  • Parsing: cmd/livepeer/starter/starter.go:1106-1124
  • Sender creation: pm/sender.go:62-69
  • Validation: pm/sender.go:220-222

Metadata

Metadata

Assignees

No one assigned

    Labels

    status: triagethis issue has not been evaluated yet

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions