-
Notifications
You must be signed in to change notification settings - Fork 21
feat(shacl): SHACL SDNA Migration - Prolog to SHACL links parser #654
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
102 commits
Select commit
Hold shift + click to select a range
391ea28
feat: add SHACL core data structures
data-coasys 7d56e4c
feat: add SHACL generation to ModelOptions decorator
data-coasys 3f06eed
docs: add SHACL migration progress tracker
data-coasys 5003f1a
feat: add SHACL storage methods to PerspectiveProxy
data-coasys f003bfe
feat: integrate SHACL into ensureSDNASubjectClass workflow
data-coasys 2c8dfee
docs: update progress - core implementation complete
data-coasys dd2529a
feat(shacl): Add named property shapes for queryable SHACL
data-coasys 1c6b894
feat(model): Populate 'name' field in SHACL property shapes
data-coasys f0d34dd
fix(perspectives): Use correct Literal API in SHACL methods
data-coasys fbc6646
test(shacl): Add test for named property shapes generation
data-coasys a5766dc
feat(shacl): Extract property name in fromLinks() for named shapes
data-coasys ef0e14c
chore: Remove Deno test file from TypeScript build
data-coasys f0f3056
docs: Add comprehensive SHACL architecture analysis
data-coasys a84f2f1
feat(shacl): Add Rust SHACL parser and integrate with add_sdna()
data-coasys 9d2941b
feat(shacl): Wire TypeScript SHACL generation to Rust backend
data-coasys a5bb3b9
docs: Add SHACL implementation completion summary
data-coasys 919bbc6
refactor(shacl): Remove Prolog generation, use SHACL exclusively
data-coasys 8a8c1cc
fix(shacl): Fix Rust borrow error and unused variable warning
data-coasys 5c64679
fix(shacl): Fix extract_namespace() logic
data-coasys 82b62b5
refactor(shacl): Restore Prolog generation alongside SHACL
data-coasys 2836b24
docs: Final SHACL architecture explanation
data-coasys 2b56387
wip(shacl): Start Prolog removal, force test failures to find depende…
data-coasys 7b1a72c
docs: Clear status and path forward for Prolog removal
data-coasys f1c3ff2
docs(shacl): Complete link structure overview with W3C conformance
data-coasys b99eb7b
docs(shacl): Unified separate links approach for Classes + Flows
data-coasys d5c928b
feat(shacl): Add action infrastructure for constructor/destructor/pro…
data-coasys c2c62b2
feat(shacl): Implement SHACL-first action execution with Prolog fallback
data-coasys 40a197c
fix(shacl): Fix tuple destructuring and link normalization before sig…
data-coasys 895d007
feat(shacl): Remove Prolog fallbacks - SDNA is now SHACL-only
data-coasys d6cadd0
fix(shacl): Remove duplicate ad4m://has_subject_class link from parse…
data-coasys 6a8c810
feat(shacl): Add Prolog SDNA to SHACL links parser for backward compa…
data-coasys c3aed79
style: cargo fmt
data-coasys 5e7dbfc
test: add comprehensive SHACLShape tests
data-coasys 9985989
docs: add deprecation guidance for sdnaCode parameter
data-coasys 9d67a3b
docs: recommend addShacl() with SHACLShape type over JSON strings
data-coasys a1fa6ef
feat(shacl): Add SHACLFlow type for SHACL-based state machines
data-coasys 67df25f
fix: Address CodeRabbit review comments
data-coasys ce9fda9
style: cargo fmt
data-coasys 248cae2
fix: Address additional CodeRabbit review comments
data-coasys 3f7e7c1
fix: TypeScript build errors
data-coasys 9ecd607
fix: Rust borrow checker error in collection name matching
data-coasys 94c641f
fix: SHACLShape constructor auto-derives shape URI + remove unused sh…
data-coasys c68ea30
fix: Address CodeRabbit review comments - security and data handling
data-coasys 8d80531
test: Add comprehensive toJSON/fromJSON tests for SHACLShape
data-coasys 6aebe1f
refactor: DRY - import AD4MAction from SHACLShape instead of duplicating
data-coasys f9a56e1
feat: Remove Prolog code storage - SHACL is now source of truth
data-coasys cbf31e7
fix: Generate minimal Prolog facts from SHACL links for backward comp…
data-coasys 886ac3a
Merge branch 'dev' into feat/shacl-sdna-migration
lucksus 98a7b6b
fix: Generate comprehensive Prolog facts from SHACL links for backwar…
data-coasys 32ab946
Merge remote-tracking branch 'origin/dev' into feat/shacl-sdna-migration
data-coasys e30c7e9
fix(sdk): Pass SHACL JSON to backend in addSdna calls
data-coasys ab3cbbf
fix: Disable Prolog mode, implement SHACL-only subject class lookup
data-coasys 5e4d2b3
fix: resolve compilation errors in SHACL-only mode
data-coasys b2af23b
fix(tests): Add shaclJson argument to mock PerspectiveResolver
data-coasys a75d90c
feat: SHACL-based subject class lookup (Prolog-free)
data-coasys f843ab7
debug: Add logging to trace SHACL link storage and lookup
data-coasys 9e4ef8d
Remove Prolog fallbacks from Ad4mModel system - make fully SHACL-native
data-coasys b12b0f5
Fix SurrealDB queries in getSubjectClassMetadataFromSDNA
data-coasys 14ae3c0
Fix SHACL class query - use string::ends_with instead of CONTAINS
data-coasys 8c1b1ba
Rewrite getSubjectClassMetadataFromSDNA to use link API
data-coasys edd2690
Restore ad4m://sdna link storage for backward compatibility
data-coasys 81cdd03
test: add test for parse_prolog_sdna_to_shacl_links
data-coasys dbefc57
chore: add debug logging for SHACL link flow tracing
data-coasys e520841
chore: use warn level logging for SHACL debugging to ensure visibilit…
data-coasys 1cb0841
chore: add UUID to add_sdna logging for correlation
data-coasys 529e0f7
chore: add database-level logging for SHACL link storage and retrieval
data-coasys 6a83126
Merge remote-tracking branch 'origin/dev' into feat/shacl-sdna-migration
data-coasys 2199f51
fix: align find_subject_class_from_shacl_by_query with SHACL link pat…
data-coasys 1e58e69
chore: retrigger CI
data-coasys e1c8437
Merge branch 'dev' into feat/shacl-sdna-migration
jhweir 8757a26
getSubjectClassMetadataFromSDNA made public
jhweir d31d107
feat: Complete SHACL migration with Prolog-disabled support
jhweir d400594
Cargo fmt
jhweir 4178ded
refactor: Apply CodeRabbit improvements for SHACL query optimization …
jhweir 15f3a22
fix: collection name double-pluralization in queries and legacy SDNA
jhweir b865d25
feat(shacl): fix subject instance creation and collection filtering
jhweir 9edc0fa
Merge branch 'dev' into feat/shacl-sdna-migration
jhweir 82309f6
Lock file updates
jhweir f9adedf
Cargo fmt
jhweir ae8d0bc
fix(shacl_parser): unwrap Result in test_extract_namespace assertions
jhweir dcd44df
fix: prevent invalid URIs from empty literals and bare SHACL node kinds
jhweir e810916
refactor: use batching for SHACL/Flow link writes, single surreal que…
HexaField 844d08b
Merge pull request #685 from coasys/fix/shacl-batching-and-imports
lucksus 61ad475
Merge branch 'dev' into feat/shacl-sdna-migration
lucksus 3ccfb38
refactor: make sdnaCode optional in addSdna API
data-bot-coasys f2898f7
fix: restore Prolog disabled warnings while returning empty matches
data-bot-coasys 56d81cc
refactor: extract SHACL→Prolog compat code to separate module
data-bot-coasys dd0ee70
test: remove skipped Prolog-only tests
data-bot-coasys 9e9b564
fix: make sdnaCode nullable in GraphQL schema
data-bot-coasys 49f411c
Merge branch 'dev' into feat/shacl-sdna-migration
lucksus 6e91e01
refactor: address PR review comments for SHACL SDNA migration
data-bot-coasys beb212f
fix: remove non-existent 'collection' property from SHACLPropertyShap…
data-bot-coasys 5c48831
Convert ends_with to SurrealDB queries, migrate flow methods to SHACL…
data-bot-coasys 708cc28
fix: migrate test addSdna calls to ensureSDNASubjectClass (Prolog→SHA…
data-bot-coasys bc405d1
test: add SHACL-based flow test for TODO workflow
data-bot-coasys 80b75d8
refactor: remove subjectClassesFromSHACL GraphQL endpoint, use link q…
data-bot-coasys cc88ffe
fix: replace SQL LIKE with SurrealQL string::starts::with in getFlow()
data-bot-coasys 5e0ae4d
fix: correct SurrealQL function name string::starts_with (not starts:…
data-bot-coasys e68adc7
Update core/src/perspectives/PerspectiveClient.ts
lucksus dbb1660
refactor: replace Prolog query-based SubjectClassOption with client-s…
data-bot-coasys 2a96113
refactor: replace findClassByProperties with single SurrealDB query
data-bot-coasys 4e480bd
fix: subjectClassesByTemplate falls back to findClassByProperties
data-bot-coasys File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -65,3 +65,5 @@ rust-executor/CUSTOM_DENO_SNAPSHOT.bin | |
| rust-executor/test_data | ||
|
|
||
| .npmrc | ||
| docs-src/ | ||
| .worktrees/ | ||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,255 @@ | ||
| # SHACL SDNA Architecture | ||
|
|
||
| **Replacing Prolog-based Subject DNA with W3C SHACL + AD4M Extensions** | ||
|
|
||
| ## Overview | ||
|
|
||
| AD4M uses Subject DNA (SDNA) to define data schemas and their operational behavior. This document describes the migration from Prolog-based SDNA to a SHACL-based approach that: | ||
|
|
||
| 1. **W3C Conformant** - Schema definitions use standard SHACL predicates | ||
| 2. **Extensible** - AD4M-specific actions use a separate `ad4m://` namespace | ||
| 3. **Queryable** - All metadata stored as links, queryable via SurrealQL or simple link queries | ||
| 4. **Consistent** - Same pattern for Classes and Flows | ||
|
|
||
| --- | ||
|
|
||
| ## Architecture: Separate Links Approach | ||
|
|
||
| ### Design Principles | ||
|
|
||
| 1. **One predicate = one action type** - Clear semantics | ||
| 2. **SHACL for schema, AD4M for behavior** - Clean separation | ||
| 3. **Named URIs** - Property shapes have queryable URIs (not blank nodes) | ||
| 4. **JSON in literals** - Actions stored as JSON arrays in `literal://string:` format | ||
|
|
||
| ### Link Structure | ||
|
|
||
| For a class named `Recipe` with namespace `recipe://`: | ||
|
|
||
| | Link Type | Source | Predicate | Target | | ||
| | ------------- | ----------------------- | -------------------- | ------------------------ | | ||
| | Shape Type | `recipe://RecipeShape` | `rdf://type` | `sh://NodeShape` | | ||
| | Target Class | `recipe://RecipeShape` | `sh://targetClass` | `recipe://Recipe` | | ||
| | Has Property | `recipe://RecipeShape` | `sh://property` | `recipe://Recipe.name` | | ||
| | Property Type | `recipe://Recipe.name` | `rdf://type` | `sh://PropertyShape` | | ||
| | Property Path | `recipe://Recipe.name` | `sh://path` | `recipe://name` | | ||
| | Datatype | `recipe://Recipe.name` | `sh://datatype` | `xsd://string` | | ||
| | Constructor | `recipe://RecipeShape` | `ad4m://constructor` | `literal://string:[...]` | | ||
| | Destructor | `recipe://RecipeShape` | `ad4m://destructor` | `literal://string:[...]` | | ||
| | Setter | `recipe://Recipe.name` | `ad4m://setter` | `literal://string:[...]` | | ||
| | Adder | `recipe://Recipe.items` | `ad4m://adder` | `literal://string:[...]` | | ||
| | Remover | `recipe://Recipe.items` | `ad4m://remover` | `literal://string:[...]` | | ||
|
|
||
| --- | ||
|
|
||
| ## Example: Complete Recipe Class | ||
|
|
||
| ### TypeScript Definition | ||
|
|
||
| ```typescript | ||
| @ModelOptions({ | ||
| name: "Recipe", | ||
| namespace: "recipe://", | ||
| }) | ||
| class Recipe { | ||
| @SubjectProperty({ through: "recipe://name", writable: true }) | ||
| name: string = ""; | ||
|
|
||
| @SubjectProperty({ through: "recipe://rating", writable: true }) | ||
| rating: number = 0; | ||
|
|
||
| @SubjectCollection({ through: "recipe://has_ingredient" }) | ||
| ingredients: string[] = []; | ||
| } | ||
| ``` | ||
|
|
||
| ### W3C SHACL Links (Schema) | ||
|
|
||
| ```turtle | ||
| # Shape definition | ||
| recipe://RecipeShape rdf:type sh:NodeShape . | ||
| recipe://RecipeShape sh:targetClass recipe://Recipe . | ||
|
|
||
| # Property: name | ||
| recipe://Recipe.name rdf:type sh:PropertyShape . | ||
| recipe://Recipe.name sh:path recipe://name . | ||
| recipe://Recipe.name sh:datatype xsd:string . | ||
| recipe://Recipe.name sh:maxCount 1 . | ||
| recipe://RecipeShape sh:property recipe://Recipe.name . | ||
|
|
||
| # Property: rating | ||
| recipe://Recipe.rating rdf:type sh:PropertyShape . | ||
| recipe://Recipe.rating sh:path recipe://rating . | ||
| recipe://Recipe.rating sh:datatype xsd:integer . | ||
| recipe://Recipe.rating sh:maxCount 1 . | ||
| recipe://RecipeShape sh:property recipe://Recipe.rating . | ||
|
|
||
| # Collection: ingredients | ||
| recipe://Recipe.ingredients rdf:type ad4m:CollectionShape . | ||
| recipe://Recipe.ingredients sh:path recipe://has_ingredient . | ||
| recipe://Recipe.ingredients sh:nodeKind sh:IRI . | ||
| recipe://RecipeShape sh:property recipe://Recipe.ingredients . | ||
| ``` | ||
|
|
||
| ### AD4M Action Links (Behavior) | ||
|
|
||
| ```turtle | ||
| # Constructor - creates instance with default values | ||
| recipe://RecipeShape ad4m://constructor """literal://string:[ | ||
| {"action": "addLink", "source": "this", "predicate": "recipe://name", "target": ""}, | ||
| {"action": "addLink", "source": "this", "predicate": "recipe://rating", "target": "0"} | ||
| ]""" . | ||
|
|
||
| # Destructor - removes instance links | ||
| recipe://RecipeShape ad4m://destructor """literal://string:[ | ||
| {"action": "removeLink", "source": "this", "predicate": "recipe://name"}, | ||
| {"action": "removeLink", "source": "this", "predicate": "recipe://rating"} | ||
| ]""" . | ||
|
|
||
| # Property setters | ||
| recipe://Recipe.name ad4m://setter """literal://string:[ | ||
| {"action": "setSingleTarget", "source": "this", "predicate": "recipe://name", "target": "value"} | ||
| ]""" . | ||
|
|
||
| recipe://Recipe.rating ad4m://setter """literal://string:[ | ||
| {"action": "setSingleTarget", "source": "this", "predicate": "recipe://rating", "target": "value"} | ||
| ]""" . | ||
|
|
||
| # Collection operations | ||
| recipe://Recipe.ingredients ad4m://adder """literal://string:[ | ||
| {"action": "addLink", "source": "this", "predicate": "recipe://has_ingredient", "target": "value"} | ||
| ]""" . | ||
|
|
||
| recipe://Recipe.ingredients ad4m://remover """literal://string:[ | ||
| {"action": "removeLink", "source": "this", "predicate": "recipe://has_ingredient", "target": "value"} | ||
| ]""" . | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Action Types | ||
|
|
||
| ### Shape-Level Actions | ||
|
|
||
| | Predicate | Purpose | Bound To | | ||
| | -------------------- | ----------------------------- | ----------------------------- | | ||
| | `ad4m://constructor` | Create instance with defaults | `{namespace}{ClassName}Shape` | | ||
| | `ad4m://destructor` | Remove instance and links | `{namespace}{ClassName}Shape` | | ||
|
|
||
| ### Property-Level Actions | ||
|
|
||
| | Predicate | Purpose | Bound To | | ||
| | ---------------- | -------------------------- | ----------------------------------------- | | ||
| | `ad4m://setter` | Set single-valued property | `{namespace}{ClassName}.{propertyName}` | | ||
| | `ad4m://adder` | Add to collection | `{namespace}{ClassName}.{collectionName}` | | ||
| | `ad4m://remover` | Remove from collection | `{namespace}{ClassName}.{collectionName}` | | ||
|
|
||
| ### Action JSON Format | ||
|
|
||
| ```json | ||
| [ | ||
| { | ||
| "action": "addLink|removeLink|setSingleTarget|collectionSetter", | ||
| "source": "this|uuid|literal", | ||
| "predicate": "namespace://predicate", | ||
| "target": "value|*|specific_value", | ||
| "local": true // optional | ||
| } | ||
| ] | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Querying SHACL Links | ||
|
|
||
| ### Get Constructor Actions | ||
|
|
||
| ```rust | ||
| // Find shape with constructor | ||
| let links = self.get_links(&LinkQuery { | ||
| predicate: Some("ad4m://constructor".to_string()), | ||
| ..Default::default() | ||
| }).await?; | ||
|
|
||
| // Find one matching class name | ||
| for link in links { | ||
| if link.data.source.ends_with(&format!("{}Shape", class_name)) { | ||
| // Parse JSON from literal://string:{json} | ||
| let actions = parse_literal_json(&link.data.target)?; | ||
| return Ok(actions); | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ### Get Property Setter | ||
|
|
||
| ```rust | ||
| // Find property shape with setter | ||
| let prop_suffix = format!("{}.{}", class_name, property_name); | ||
| let links = self.get_links(&LinkQuery { | ||
| predicate: Some("ad4m://setter".to_string()), | ||
| ..Default::default() | ||
| }).await?; | ||
|
|
||
| for link in links { | ||
| if link.data.source.ends_with(&prop_suffix) { | ||
| let actions = parse_literal_json(&link.data.target)?; | ||
| return Ok(actions); | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Implementation Status | ||
|
|
||
| ### Phase 1: SHACL Infrastructure (Complete) | ||
|
|
||
| - [x] TypeScript: Generate SHACL JSON with actions in `generateSDNA()` | ||
| - [x] TypeScript: Serialize SHACL to links in `PerspectiveProxy.ensureSdnaLinks()` | ||
| - [x] Rust: Parse SHACL JSON in `shacl_parser.rs` | ||
| - [x] Rust: Store action links with separate predicates | ||
|
|
||
| ### Phase 2: Replace Prolog Queries (In Progress) | ||
|
|
||
| - [x] `get_constructor_actions()` - Try SHACL first, fallback to Prolog | ||
| - [x] `get_destructor_actions()` - Try SHACL first, fallback to Prolog | ||
| - [x] `get_property_setter_actions()` - Try SHACL first, fallback to Prolog | ||
| - [x] `get_collection_adder_actions()` - Try SHACL first, fallback to Prolog | ||
| - [x] `get_collection_remover_actions()` - Try SHACL first, fallback to Prolog | ||
| - [x] `resolve_property_value()` - Try SHACL for resolve language | ||
| - [x] TypeScript `removeSubject()` - Try SHACL for destructor actions | ||
|
|
||
| ### Phase 3: Remove Prolog Fallbacks ✅ (Completed in this PR) | ||
|
|
||
| > **Note:** Prolog engines (scryer-prolog) are kept available for complex queries and | ||
| > future advanced features. Only the _fallback pattern_ is removed - SHACL is the | ||
| > single source of truth for all SDNA actions. | ||
|
|
||
| - [x] Remove Prolog fallbacks for action retrieval (SHACL-first is now SHACL-only) | ||
| - [x] Migrate Flows to same SHACL link pattern (`SHACLFlow` class with `toLinks()`/`fromLinks()`) | ||
| - [x] Keep scryer-prolog dependency (for complex Prolog queries later) | ||
| - [x] Refactor TypeScript to use `SHACLShape.fromLinks()` / `toJSON()` throughout | ||
| - [x] Use batched link operations (`addLinks()`) and single SurrealDB queries | ||
|
|
||
| --- | ||
|
|
||
| ## Benefits | ||
|
|
||
| 1. **W3C Standard** - Interoperable with SHACL ecosystem | ||
| 2. **Cleaner Runtime** - SHACL as single source for SDNA actions (Prolog still available for complex queries) | ||
| 3. **Queryable** - All metadata as links in SurrealDB | ||
| 4. **Debuggable** - Inspect schema as regular links | ||
| 5. **Extensible** - Add new action types without schema changes | ||
|
|
||
| --- | ||
|
|
||
| ## Files | ||
|
|
||
| | File | Purpose | | ||
| | -------------------------------------------------------- | ------------------------------------------------- | | ||
| | `core/src/model/decorators.ts` | `generateSHACL()` - Creates SHACL from decorators | | ||
| | `core/src/shacl/SHACLShape.ts` | SHACL shape class with `toLinks()` | | ||
| | `core/src/perspectives/PerspectiveProxy.ts` | `ensureSdnaLinks()` - Stores SHACL as links | | ||
| | `rust-executor/src/perspectives/shacl_parser.rs` | Parses SHACL JSON, generates links | | ||
| | `rust-executor/src/perspectives/perspective_instance.rs` | Action retrieval with SHACL-first | | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.