[v25.3.x] CORE-14479 schema_registry: fix avro external reference collection#29627
Merged
pgellert merged 5 commits intoredpanda-data:v25.3.xfrom Feb 26, 2026
Merged
Conversation
(cherry picked from commit 8c5b546) Manual backport changes: - update bazel lockfile for v25.3
For consistency across the codebase, use relative references for bazel dependencies that refer to targets that are in the same module. (cherry picked from commit 38af379)
Various tests require a minimal store fixture for testing, so extract a shared implementation that can be reused in existing tests and in the one I am about to introduce. (cherry picked from commit 7b9bf85)
Previously, avro external references were not handled correctly during schema processing. The old implementation attempted to build schemas by concatenating them, which resulted in incorrect parsing - for example, a schema A that references B would be parsed as schema B (the first schema in the concatenated result). This commit fixes the reference collection logic to properly traverse and collect all external schema references, and uses compileJsonSchemaWithNamedReferences to parse schemas with named references correctly. As a result: - Avro schemas with external references are now compiled correctly. This affects compatibility checks and server-side schema ID validation. - The schema definition returned by the schema registry API endpoints is unchanged. - The way schemas are serialized onto the schemas topic is unchanged. Manual backport change: - The fix is gated by the schema_registry_avro_use_named_references config which defaults to false, preserving existing behavior. When enabled, schemas with external references are compiled using compileJsonSchemaWithNamedReferences instead of schema concatenation, which fixes compatibility checks and schema validation issues. - Adds a test for the above gated behaviour. (cherry picked from commit f579884)
Collaborator
nguyen-andrew
approved these changes
Feb 18, 2026
Comment on lines
+384
to
+386
| // Explicitly ensure the legacy behavior is active (default is false) | ||
| config::shard_local_cfg() | ||
| .schema_registry_avro_use_named_references.reset(); |
Member
There was a problem hiding this comment.
nit: I think it would be nice to explicitly set it to false, or reset it + assert/require that the default is false
| // schema in the concatenation (Address), not from Person. | ||
| auto compiled = register_schema( | ||
| subject("PersonSubject"), referencing_schema, schema_version{1}); | ||
| EXPECT_EQ(compiled.name(), "com.example.Address"); |
Member
There was a problem hiding this comment.
nit: can add const auto ref_name = "com.example.Address"; replace this and line 405 with ref_name.
Collaborator
Retry command for Build#80655please wait until all jobs are finished before running the slash command |
7 tasks
Collaborator
Retry command for Build#80772please wait until all jobs are finished before running the slash command |
Collaborator
Retry command for Build#81128please wait until all jobs are finished before running the slash command |
Contributor
Author
|
/ci-repeat 1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Manual backport of #28780
See the commit messages for details on what is changing compared to the original PR. The main difference is that now the new (potentially breaking-change) behaviour is hidden behind a default-disable cluster config
schema_registry_avro_use_named_referencesto enable the safe backporting of this bug fix.Output of
git range-diff f579884~4..f579884 HEAD~5..HEADto show the diff in this commit range: https://gist.github.com/pgellert/84c33796b3c02fbeea1a11fbe06f4e48Relates to #29611
Fixes CORE-15528
Backports Required
Release Notes
Bug Fixes
schema_registry_avro_use_named_referencesis enabled, while starting v26.1.1 it the new behaviour is always enabled.