[DMS-591] Add common extension override and array uniqueness constraint merging

[simpat-jesus]

• Update API schema packages from 1.0.322 to 1.0.324 (DataStandard52, TPDM, Sample, Homograph)
• Add commonExtensionOverrides support — extensions can now inject _ext schema fragments at specified JSONPath locations in the core jsonSchemaForInsert (e.g., extending Address within Contact)
• Fix duplicate arrayUniquenessConstraints — when extensions share the same top-level paths as core constraints, nested constraints are merged instead of creating duplicates
• Tighten JSON schema validation for commonExtensionOverrides in JsonSchemaForApiSchema.json
• Add log sanitization to all external data logged in EffectiveApiSchemaProvider

[bradbanister]

AI Review

2026-02-19T17:13:02Z by Showboat 0.6.0

This branch adds support for common extension overrides when building the effective API schema at startup.

Why this exists

Extensions can add fields to common/complex types (for example, an extension that adds fields to EdFi.Address). Those common types are embedded inside core resources (for example, Contact.addresses[*]).

Previously, schema merging only added missing keys into the core jsonSchemaForInsert and skipped keys that already exist in core (like addresses). That meant extension _ext fields for common types were not inserted into the core insert schema, even though the extension schema contained them.

What changed

Extension resource schemas can now include commonExtensionOverrides entries that explicitly tell DMS where to insert/merge _ext fragments into the core jsonSchemaForInsert.

git rev-parse --abbrev-ref HEAD && git log -2 --oneline

DMS-591-v2
46f9348e Add common extension override support
9d886f27 Update schema packages

git show -1 --name-only --oneline

46f9348e Add common extension override support
src/dms/core/EdFi.DataManagementService.Core/ApiSchema/EffectiveApiSchemaProvider.cs
src/dms/core/EdFi.DataManagementService.Core/ApiSchema/JsonSchemaForApiSchema.json

rg -n "commonExtensionOverrides|ApplyCommonExtensionOverrides|NavigateJsonPath" src/dms/core/EdFi.DataManagementService.Core/ApiSchema/EffectiveApiSchemaProvider.cs src/dms/core/EdFi.DataManagementService.Core/ApiSchema/JsonSchemaForApiSchema.json

src/dms/core/EdFi.DataManagementService.Core/ApiSchema/JsonSchemaForApiSchema.json:235:                        "commonExtensionOverrides": {
src/dms/core/EdFi.DataManagementService.Core/ApiSchema/EffectiveApiSchemaProvider.cs:116:                ApplyCommonExtensionOverrides(extensionResources, coreResources);
src/dms/core/EdFi.DataManagementService.Core/ApiSchema/EffectiveApiSchemaProvider.cs:243:            // entries handled by ApplyCommonExtensionOverrides — skip them here.
src/dms/core/EdFi.DataManagementService.Core/ApiSchema/EffectiveApiSchemaProvider.cs:308:    /// used in Contact), the extension schema includes commonExtensionOverrides that specify:
src/dms/core/EdFi.DataManagementService.Core/ApiSchema/EffectiveApiSchemaProvider.cs:313:    private void ApplyCommonExtensionOverrides(
src/dms/core/EdFi.DataManagementService.Core/ApiSchema/EffectiveApiSchemaProvider.cs:331:            var overrides = extensionResource["commonExtensionOverrides"]?.AsArray();
src/dms/core/EdFi.DataManagementService.Core/ApiSchema/EffectiveApiSchemaProvider.cs:374:                    var targetNode = NavigateJsonPath(coreJsonSchemaForInsert, jsonPath);
src/dms/core/EdFi.DataManagementService.Core/ApiSchema/EffectiveApiSchemaProvider.cs:425:    private static JsonNode? NavigateJsonPath(JsonNode root, string jsonPath)

Schema package update

This branch also bumps the Ed-Fi ApiSchema NuGet packages so the downloaded schema inputs can include the new commonExtensionOverrides structure.

rg -n "EdFi\.(DataStandard52|Homograph|Sample|TPDM)\.ApiSchema" src/Directory.Packages.props

17:        <PackageVersion Include="EdFi.DataStandard52.ApiSchema" Version="1.0.324" />
18:        <PackageVersion Include="EdFi.Homograph.ApiSchema" Version="1.0.324" />
19:        <PackageVersion Include="EdFi.Sample.ApiSchema" Version="1.0.324" />
20:        <PackageVersion Include="EdFi.TPDM.ApiSchema" Version="1.0.324" />

ApiSchema contract

commonExtensionOverrides lives on a resource extension schema and is an array of entries with:

• insertionLocations: one or more JSONPath strings (dot-notation) that are evaluated starting at the core resource's jsonSchemaForInsert node.
• schemaFragment: the schema object that becomes (or merges into) the _ext property at each insertion location.

Example (simplified):

{
  "resourceName": "Contact",
  "isResourceExtension": true,
  "commonExtensionOverrides": [
    {
      "insertionLocations": ["$.properties.addresses.items"],
      "schemaFragment": {
        "type": "object",
        "properties": {
          "sample": { "type": "string" }
        }
      }
    }
  ]
}

Note: the implementation currently supports only dot-notation property access (no wildcards/filters/array indices).

import json, pathlib
path = pathlib.Path('src/dms/core/EdFi.DataManagementService.Core/ApiSchema/JsonSchemaForApiSchema.json')
schema = json.loads(path.read_text(encoding='utf-8'))
props = schema['definitions']['ResourceSchemas']['patternProperties']['^[A-Za-z0-9]+$']['properties']
common = props['commonExtensionOverrides']
item_props = common['items']['properties']
print('commonExtensionOverrides' in props)
print(sorted(item_props.keys()))
print(common['items']['required'])

True
['insertionLocations', 'schemaFragment']
['insertionLocations', 'schemaFragment']

Behavior (simplified example)

The override mechanism navigates to each insertionLocation, then adds or merges an _ext entry under that schema node’s properties.

import json

core_insert = {
    'properties': {
        'addresses': {
            'items': {
                'properties': {
                    'streetNumberName': {'type': 'string'}
                }
            }
        }
    }
}

override = {
    'insertionLocations': ['$.properties.addresses.items'],
    'schemaFragment': {
        'type': 'object',
        'properties': {
            'sample': {'type': 'string'}
        }
    }
}

def navigate(root, json_path):
    path = json_path[2:] if json_path.startswith('$.') else json_path
    current = root
    for segment in path.split('.'):
        current = current.get(segment)
        if current is None:
            return None
    return current

for loc in override['insertionLocations']:
    target = navigate(core_insert, loc)
    target_props = target.get('properties', {})
    if '_ext' in target_props:
        target_props['_ext'].setdefault('properties', {}).update(
            override['schemaFragment'].get('properties', {})
        )
    else:
        target_props['_ext'] = override['schemaFragment']
    target['properties'] = target_props

print(json.dumps(core_insert['properties']['addresses']['items']['properties'], indent=2, sort_keys=True))

{
  "_ext": {
    "properties": {
      "sample": {
        "type": "string"
      }
    },
    "type": "object"
  },
  "streetNumberName": {
    "type": "string"
  }
}

[bradbanister]

Inline notes focus on correctness, validation, and simplifying duplication.

[bradbanister]

NavigateJsonPath isn’t really JSONPath (dot-property only) and will silently fail for valid JSONPath syntax (arrays, wildcards, etc.). Since we already have JsonHelpers.SelectNodeFromPath (Json.Path), consider using that here and decide whether an override that can’t be applied should warn/fail. Also prefer StartsWith("$.", StringComparison.Ordinal) for non-culture behavior.

[bradbanister]

This now silently skips any non-_ext duplicate keys (previously logged). That can hide unexpected schema collisions beyond the common-extension case. Consider logging (Debug/Warning) when a key is skipped, or restricting the skip to scenarios where commonExtensionOverrides covers it.

[bradbanister]

Building coreResourceByName with assignment avoids ToDictionary throwing, but “last write wins” can mask duplicate resourceName entries. Suggest detecting duplicates and logging/throwing with context rather than silently overwriting.

[bradbanister]

ApplyCommonExtensionOverrides currently continues on many malformed/unsatisfied conditions (missing properties, unexpected fragment shape, etc.), so overrides may silently not apply. Recommend failing-fast when overrides are present but can’t be applied.

[bradbanister]

When _ext exists, this merges only _ext.properties and ignores other schema keywords (required, additionalProperties, allOf, etc.), so behavior differs from adding the full fragment. Should we be merging the full object? Also duplicates _ext merging logic with MergeExtensionObjectIntoCore—consider extracting a helper.

[bradbanister]

coreResourceByName is constructed here and in CopyResourceExtensionNodeToCore. Consider building it once in Initialize() and passing it to helpers to reduce duplication and repeated scans.