[DMS-981] Core Emits Concrete JSON Locations for Document References

[JBrenesSimpat]

Summary

Implements the only required DMS Core change called out in reference/design/backend-redesign/design-docs/overview.md and reference/design/backend-redesign/design-docs/flattening-reconstitution.md:

• Document reference extraction now includes a concrete JSON location with numeric indices (e.g., $.addresses[2].periods[0].calendarReference) instead of wildcard paths ([*]).
• This enables efficient write-time FK population for references inside nested collections without per-row JSONPath evaluation.

Changes

• DocumentReference — Added JsonPath Path field for concrete reference-object paths
• SuperclassIdentity — Passes Path through to base DocumentReference (always $ for root-level superclass)
• ReferenceExtractor — Refactored from SelectNodesFromArrayPathCoerceToStrings() (values only) to SelectNodesAndLocationFromArrayPathCoerceToStrings() (values + paths), deriving concrete reference-object paths by stripping the final segment
• IdentityExtractor — Updated SuperclassIdentity construction to include Path
• IntermediateReferenceElement — Deleted (dead code after refactor)
• DatabaseTest.cs — Updated test helper to pass Path
• ReferenceExtractorTests.cs — Added path assertions to existing fixtures + new nested collection test fixture

Acceptance Criteria

• Each extracted document reference instance includes a concrete JSON path with indices (no [*] wildcards)
• Paths are stable/canonical (same input JSON -> same path strings)
• Existing descriptor reference extraction behavior remains correct (descriptor references already include paths)
• Core unit tests cover:
  • A reference at root
  • A reference inside a collection
  • A reference inside nested collections

Test Plan

• All 1,152 Core unit tests pass
• All Backend, DDL, and Frontend unit tests pass (no regressions)
• New nested collection fixture validates paths: $.addresses[0].periods[0].calendarReference, $.addresses[1].periods[0].calendarReference, $.addresses[1].periods[1].calendarReference

[JBrenesSimpat]

Code Review

Reviewed by Claude Code (claude-opus-4-6)

Jira Context

DMS-981: Core Emits Concrete JSON Locations for Document References — document reference extraction must include concrete JSON paths with numeric indices (no [*] wildcards) to enable efficient write-time FK population.

Summary

1 issue found requiring changes.

All four Jira acceptance criteria are addressed:

1. Concrete JSON paths with indices (no wildcards) ✅
2. Paths are stable/canonical ✅
3. Descriptor extraction unchanged ✅
4. Unit tests for root, collection, and nested collection references ✅

Additional Notes

[Warning] Dead code: IntermediateReferenceElement.cs should be deleted

src/dms/core/EdFi.DataManagementService.Core/Extraction/IntermediateReferenceElement.cs is no longer referenced anywhere in the codebase. The ReferenceExtractor refactor replaced its usage with anonymous types. Change requested: Delete IntermediateReferenceElement.cs to avoid leaving orphaned code.

[JBrenesSimpat]

Before DMS-981, ReferenceExtractor.cs was the sole consumer of IntermediateReferenceElement. The refactor in this PR replaced:

• SelectNodesFromArrayPathCoerceToStrings() (values only) → SelectNodesAndLocationFromArrayPathCoerceToStrings() (values + paths)
• IntermediateReferenceElement(IdentityJsonPath, string[] ValueSlice) → anonymous type { IdentityJsonPath, JsonPathAndValue[] PathValues }

That swap removed the only usage of IntermediateReferenceElement, making it dead code as a direct consequence of our changes. So it's our responsibility to clean it up in this same
PR.

[Copilot]

Pull request overview

This PR implements a backend redesign requirement to emit concrete JSON locations for document references. Previously, document references included only the identity and referential ID; now they include a concrete JSON path with numeric indices (e.g., $.addresses[0].periods[1].calendarReference) instead of wildcard paths. This enables efficient write-time FK population for references inside nested collections without per-row JSONPath evaluation, bringing DocumentReference into alignment with DescriptorReference which already used concrete paths.

Changes:

• Added JsonPath Path field to DocumentReference and SuperclassIdentity records for concrete reference-object paths
• Refactored ReferenceExtractor to use SelectNodesAndLocationFromArrayPathCoerceToStrings() which returns both values and concrete paths, deriving reference-object paths by stripping the final path segment
• Removed IntermediateReferenceElement record (replaced by anonymous type in refactored code)
• Added comprehensive test coverage for root-level, collection, and nested collection reference scenarios

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/dms/core/EdFi.DataManagementService.Core.External/Model/DocumentReference.cs	Added required Path parameter to record for concrete JSON location
src/dms/core/EdFi.DataManagementService.Core.External/Model/SuperclassIdentity.cs	Added required Path parameter (always "$" for root-level) and updated base record call
src/dms/core/EdFi.DataManagementService.Core/Extraction/ReferenceExtractor.cs	Refactored to extract concrete paths alongside values, derive reference-object paths, and pass to DocumentReference constructor
src/dms/core/EdFi.DataManagementService.Core/Extraction/IntermediateReferenceElement.cs	Deleted dead code replaced by anonymous type
src/dms/core/EdFi.DataManagementService.Core/Extraction/IdentityExtractor.cs	Updated SuperclassIdentity instantiation to include Path parameter
src/dms/backend/EdFi.DataManagementService.Old.Postgresql.Tests.Integration/DatabaseTest.cs	Updated test helper methods to provide Path parameter
src/dms/core/EdFi.DataManagementService.Core.Tests.Unit/Extraction/ReferenceExtractorTests.cs	Added path assertions to existing tests and comprehensive nested collection test fixture

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.