Skip to content

Conversation

@ChrisPenner
Copy link
Member

@ChrisPenner ChrisPenner commented Jan 9, 2026

Overview

A recently submitted contribution included references to unknown builtins, this caused the contribution diff to crash, but the queue didn't have any error handling, so it would keep retrying without backoff and ended up gumming up the diff queue and also using up all our Postgres credits :'(

This adds error handling which should catch invalid diffs and mark them as errors so they're not retried.

  • Run the migration on staging and prod

Implementation notes

  • Catch exceptions in diff transactions
  • Mark these entries with an error
  • Don't pull erroring diffs from the queue

Test coverage

I'll test one known invalid diff on production to ensure it's all working correctly.

Loose ends

I'd prefer we had a proper queueing solution; ideally one with auto-backoff, a GUI interface, and maybe even some form of alerting. Maybe we can look at adding one.

@ChrisPenner ChrisPenner force-pushed the cp/causal-diff-deadletter branch from 844fb8e to eb2f610 Compare January 9, 2026 20:22
@ChrisPenner ChrisPenner marked this pull request as ready for review January 9, 2026 20:23
@ChrisPenner ChrisPenner merged commit f4b9869 into main Jan 9, 2026
5 checks passed
@ChrisPenner ChrisPenner deleted the cp/causal-diff-deadletter branch January 9, 2026 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants