Client Cache breaks Resumable guarantee

[serathius]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

Cache is a experimental library https://github.com/etcd-io/etcd/tree/main/cache developed as part of GSoC #19371

Run robustness test on latest implementation of cache #20354

If found the issue

    logger.go:146: 2025-08-14T11:30:16.618+0200 ERROR   Broke watch guarantee   
{
  "guarantee": "resumable",
  "client": 3,
  "request": {
    "Key": "",
    "Revision": 989,
    "WithPrefix": true,
    "WithProgressNotify": false,
    "WithPrevKV": false
  },
  "got-event": {
    "Type": "put-operation",
    "Key": "key42",
    "Value": {
      "Value": "1556"
    },
    "Revision": 1540
  },
  "want-event": {
    "Type": "put-operation",
    "Key": "key13",
    "Value": {
      "Value": "1004"
    },
    "Revision": 989
  }

What did you expect to happen?

Cache should provide the same guarantees as etcd.

How can we reproduce it (as minimally and precisely as possible)?

Pull the #20354

And run:

make gofail-enable
make 
make gofail-disable
GO_TEST_FLAGS='--count=1 -v --run TestRobustnessExploratory  --failfast' make test-robustness

/cc @MadhavJivrajani @apullo777

/assign @apullo777

Anything else we need to know?

No response

Etcd version (please run commands below)

Details

$ etcd --version
# paste output here

$ etcdctl version
# paste output here

Etcd configuration (command line flags or environment variables)

Details

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

Details

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

[k8s-ci-robot]

@serathius: GitHub didn't allow me to assign the following users: apullo777.

Note that only etcd-io members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

Details

In response to this:

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via security@etcd.io.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

Cache is a experimental library https://github.com/etcd-io/etcd/tree/main/cache developed as part of GSoC #19371

Run robustness test on latest implementation of cache #20354

If found an issue

   logger.go:146: 2025-08-14T11:30:16.618+0200 ERROR   Broke watch guarantee   {"guarantee": "resumable", "client": 3, "request": {"Key":"","Revision":989,"WithPrefix":true,"WithProgressNotify":false,"WithPrevKV":false}, "got-event": {"Type":"put-operation","Key":"key42","Value":{"Value":"1556"},"Revision":1540}, "want-event": {"Type":"put-operation","Key":"key13","Value":{"Value":"1004"},"Revision":989}}
   logger.go:146: 2025-08-14T11:30:16.618+0200 ERROR   Watch validation failed {"duration": "15.339682ms", "error": "broke Resumable - A broken watch can be resumed by establishing a new watch starting after the last revision received in a watch event before the break, so long as the revision is in the history window"

What did you expect to happen?

Cache should provide the same guarantees as etcd.

How can we reproduce it (as minimally and precisely as possible)?

Pull the #20354

And run:

make gofail-enable
make 
make gofail-disable
GO_TEST_FLAGS='--count=1 -v --run TestRobustnessExploratory  --failfast' make test-robustness

/cc @MadhavJivrajani @apullo777

/assign @apullo777

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --version
# paste output here

$ etcdctl version
# paste output here

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[serathius]

We have more reproductions in https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/etcd-io_etcd/20354/pull-etcd-robustness-amd64/1955924393838252032

[apullo777]

Thanks! After some initial investigation, here is my understanding of what might contribute to this issue:

• The cache allows watchers to register while history is still empty, as ready is set before ringBuffer history exists (on the first upstream watch response). So PeekOldest() returns 0
• The demux mis-classifies the watcher as active when history is empty instead of treating it as lagging.
• We Broadcast event without checking the gap (batch's earliest revision is newer than the watcher's expected revision)

I will try to figure out how to fix this and open a PR. Some options in mind:

1. In cache.Watch we need to decide compaction correctly when history is empty
2. In demux.Register make sure to register non-zero startRev as lagging when history is empty
3. detect gaps in demux.Broadcast and move watcher to lagging