Skip to content

fix: reqresp flake & add logging#21334

Merged
deffrian merged 3 commits intomerge-train/spartanfrom
nikita/reqresp-flakes
Mar 11, 2026
Merged

fix: reqresp flake & add logging#21334
deffrian merged 3 commits intomerge-train/spartanfrom
nikita/reqresp-flakes

Conversation

@deffrian
Copy link
Copy Markdown
Collaborator

@deffrian deffrian commented Mar 10, 2026

Ref: A-634

The test is set up this way:

  • Block proposers are connected via gossip between each other.
  • All other nodes drop all gossip messages.
  • Each proposer gets a part of the entire tx set. In a normal scenario they exchange these txs via gossip, so every proposer has every tx.

What happened:

  • P2 sends Tx2 to P1 — this send succeeds.
  • P1 sends Tx1 to P2 — this send fails.
  • Tx2 gets mined by P1.
  • P2 proposes an empty block because the only tx it had was mined by P1.
  • This is where the test ends and we start waiting for all txs to get mined (including Tx1).

Changes

I didn't find any evidence for why the gossip failure happened. This PR adds more logs to the gossip pipeline to try to find why this failure occurred. The flake could not be reproduced after hundreds of test runs.

Also found an unrelated flake: after transactions are mined, we query checkpoints, but the checkpoint might not yet be published to L1 and synced by the archiver, causing the assertion to fail.

@deffrian deffrian added the ci-full Run all master checks. label Mar 10, 2026
@deffrian deffrian changed the title fix: reqresp flakes fix: reqresp flake & add logging Mar 11, 2026
@AztecBot
Copy link
Copy Markdown
Collaborator

Flakey Tests

🤖 says: This CI run detected 1 tests that failed, but were tolerated due to a .test_patterns.yml entry.

\033FLAKED\033 (8;;http://ci.aztec-labs.com/bc65a0e5021a387e�bc65a0e5021a387e8;;�):  yarn-project/end-to-end/scripts/run_test.sh simple src/e2e_p2p/duplicate_attestation_slash.test.ts (299s) (code: 0) group:e2e-p2p-epoch-flakes

@deffrian deffrian merged commit 193dfb5 into merge-train/spartan Mar 11, 2026
10 checks passed
@deffrian deffrian deleted the nikita/reqresp-flakes branch March 11, 2026 20:22
github-merge-queue Bot pushed a commit that referenced this pull request Mar 11, 2026
BEGIN_COMMIT_OVERRIDE
fix: (A-623) increase committee timeout in scenario smoke test (#21193)
feat: orchestrator enqueues via serial queue (#21247)
feat: rollup mana limit gas validation (#21219)
fix: make e2e HA test more deterministic (#21199)
chore: fix chonk_browser lint warning (#21265)
chore: deploy SPONSORED_FPC in test networks (#21254)
fix: (A-635) e2e bot flake on nonce mismatch (#21288)
chore: deflake duplicate attestations and proposals slash tests (#21294)
fix(sequencer): fix log when not enough txs (#21297)
chore: send env var to pods (#21307)
fix: Simulate gas in n tps test. Set min txs per block to 1 (#21312)
fix: update dependabot dependencies (#21238)
test: run nightly bench of block capacity (#20726)
fix: update block_capacity test to use new send() result types (#21345)
fix(node): fix index misalignment in findLeavesIndexes (#21327)
fix(log): do not log validation error if unregistered handler (#21111)
fix: limit parallel blocks in prover to max AVM parallel simulations
(#21320)
fix: use native sha256 to speed up proving job id generation (#21292)
chore: remove v4-devnet-1 (#21044)
fix(validator): wait for l1 sync before processing block proposals
(#21336)
fix(txpool): cap priority fee with max fees when computing priority
(#21279)
chore: Properly compute finalized block (#21156)
fix: remove extra argument in KVArchiverDataStore constructor call
(#21361)
chore: revert l2 slot time 72 -> 36 on scenario network (#21291)
fix(archiver): do not error if proposed block matches checkpointed
(#21367)
fix(claude): rule to not append echo exit (#21368)
chore: reduce severity of errors due to HA node not acquiring signature
(#21311)
fix: make reqresp batch retry test deterministic (#21322)
fix: (A-643) add buffer to maxFeePerBlobGas for gas estimation and fix
bump loop truncation (#21323)
fix(e2e): use L2 priority fee in deploy_method same-block test (#21373)
fix: reqresp flake & add logging (#21334)
END_COMMIT_OVERRIDE
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-full Run all master checks.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants