Skip to content

Comments

feat: MIF v2 vendor-neutral schema with format adapters#62

Merged
varun29ankuS merged 2 commits intomainfrom
feat/mif-v2-schema
Feb 22, 2026
Merged

feat: MIF v2 vendor-neutral schema with format adapters#62
varun29ankuS merged 2 commits intomainfrom
feat/mif-v2-schema

Conversation

@varun29ankuS
Copy link
Owner

Summary

Redesigns the Memory Interchange Format (MIF) as a vendor-neutral standard with pluggable adapters, fixing all known issues with the v1 implementation.

What was broken (v1)

  • Entity types exported as "UNKNOWN" — graph lookup was missing
  • Import destroyed UUIDs — generated new IDs, breaking all references
  • O(n*k) dedup — recalled all memories per import candidate
  • Merge strategies were no-ops — silently ignored
  • Encryption was dead code — unused stubs bloating the module
  • Export bulk-loaded all memories — OOM risk on large stores
  • handlers/mif.rs was 1688 lines with search handlers mixed in

What's fixed (v2)

New module: src/mif/

  • schema.rs — Vendor-neutral types: MifDocument, MifMemory, MifKnowledgeGraph, MifTodo, MifProject, MifReminder with vendor extensions
  • export.rs — Streaming export with entity type resolution from GraphMemory, per-memory vendor extensions, PII redaction
  • import.rs — UUID-preserving import via remember_with_id(), O(1) SHA256 content-hash dedup, graph reconstruction
  • pii.rs — PII detection/redaction (email, phone, SSN, API key, credit card)

Adapter pattern: src/mif/adapters/

  • MifAdapter trait with detect(), to_mif(), from_mif()
  • shodh.rs — MIF v2 JSON (lossless round-trip) + v1 backward compat
  • mem0.rs — mem0 format: [{memory, metadata, ...}]
  • generic.rs — Generic JSON array: [{content, timestamp, tags, ...}]
  • markdown.rs — Markdown with YAML frontmatter (Letta/Obsidian style)
  • AdapterRegistry with auto-detection (most specific → generic fallback)

Handler cleanup

  • handlers/mif.rs rewritten: 1688 → ~300 lines (thin HTTP layer)
  • Multimodal/robotics search moved to handlers/search.rs
  • Dead encryption code removed entirely

New API endpoint: GET /api/mif/adapters — lists available formats

Files changed

  • 10 new files in src/mif/
  • 5 modified files (handlers, lib.rs, memory/mod.rs)
  • +2958 / -1413 lines net

Test plan

  • cargo check — clean ✅
  • cargo clippy — no warnings in new files ✅
  • Round-trip: export MIF v2 → import → verify UUIDs, entity types preserved
  • Import mem0 JSON → memories created with correct content/timestamps
  • Import markdown → memory created with frontmatter metadata
  • Existing multimodal/robotics search routes work after file split
  • PII redaction works on export with redact_pii=true

Redesign Memory Interchange Format as a vendor-neutral standard:

- MIF v2 schema: raw UUIDs, typed entities, vendor extensions for
  shodh-specific metadata (importance, tier, Hebbian state, LTP)
- Adapter pattern: MifAdapter trait with detect/to_mif/from_mif for
  extensible format support (shodh, mem0, generic JSON, markdown)
- AdapterRegistry: auto-detection via ordered detect() calls
- Streaming export: entity types resolved from GraphMemory (no more
  "UNKNOWN"), per-memory vendor extensions, PII redaction
- UUID-preserving import: new remember_with_id() for explicit MemoryId,
  O(1) content-hash dedup via SHA256 (replaces O(n*k) recall queries)
- File split: multimodal/robotics search moved to handlers/search.rs
- Dead code removal: encryption stubs, old v1 types, duplicate PII code
- handlers/mif.rs rewritten from 1688 to ~300 lines (thin HTTP layer)
entity_refs on memories are empty (entities come through
experience.entities path instead), so UUID-based graph lookup
always missed. Added name-based fallback for both code paths:
entity_refs and experience.entities now resolve types from the
graph entity name map. Resolves 97.3% of entity types (was 0%).
@varun29ankuS varun29ankuS merged commit 3e11d75 into main Feb 22, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant