feat: structured collections — typed CRUD tools for agent workspaces

[standardtoaster]

Collections: Typed Structured Storage for Agent Workspaces

Summary

"Add milk to the grocery list." Most deployments can't do this out of the box. The agent either creates a new document every time (fragmenting the list) or tries to edit an existing markdown file and corrupts it. Across 28 test scenarios and 2 models, modifying structured data stored in memory documents fails every time.

This PR adds schema-defined collections with auto-generated typed CRUD tools. Register a collection with a schema, get a tool that handles inserts, queries, updates, deletes, and aggregations with validation.

Model	Collections	Memory docs	Delta
Qwen 3.5-35B (local)	76%	37%	+39
Claude Haiku 4.5	70%	26%	+44

35 files changed, ~12,600 lines. Dual-backend (PostgreSQL + libSQL), per-user tool isolation. Also provides the storage layer for #1474.

The problem

A user says "add milk to the grocery list." Both options fail:

1. Write a new document. Creates fragments. "What's on the list?" requires deduplication across scattered files.

2. Edit an existing document. Read, parse, modify, write back. Models can't do this — they lose items, duplicate entries, or corrupt formatting. Fails on every model tested.

The problem is structural: append-only documents don't support mutation.

What a user sees

POST /api/collections
{
  "collection": "grocery_items",
  "description": "Shopping list",
  "fields": {
    "name":     { "type": "text", "required": true },
    "quantity": { "type": "number" },
    "category": { "type": { "enum": ["produce", "dairy", "meat", "bakery", "other"] } },
    "store":    { "type": "text" },
    "done":     { "type": "bool", "default": false }
  }
}

This generates a tool called grocery_items. The model calls:

grocery_items(operation: "add", data: { "name": "milk", "category": "dairy" })
grocery_items(operation: "query")
grocery_items(operation: "summary", agg_operation: "count")

The schema validates inputs and coerces LLM quirks: "12" becomes 12, "true" becomes true, "tomorrow" becomes an ISO date.

Evidence

Collections vs memory docs (28 scenarios, same data, same models)

Category	Qwen+Coll	Qwen+Mem	Haiku+Coll	Haiku+Mem
Grocery (7)	76%	29%	76%	28%
Nanny hours (7)	56%	21%	69%	24%
Todo (6)	72%	52%	72%	35%
Transactions (8)	59%	48%	65%	21%

Tool design matters

Approach	Score	Why
1 unified tool/collection + skills	76%	Low tool count + guided operations
5 per-operation tools + skills	65%	Self-documenting but 20 tools adds noise
1 unified tool/collection, no skills	68%	Model handles the operation enum fine
Flat files (memory docs)	37%	Writes broken
Generic CRUD (5 tools for all)	41%	Model forgets collection names

One tool per collection with an operation parameter is the best design. Auto-generated skills (+8%) teach the model which operation to use for natural language intents.

How it works

Storage

StructuredStore trait, fully implemented for both PostgreSQL (JSONB) and libSQL (json_extract). Records in a shared structured_records table, discriminated by (user_id, collection) with composite index + GIN index. 7 field types: Text, Number, Date, Time, DateTime, Bool, Enum.

Schema alteration supports adding/removing fields and enum values. Existing records are preserved — queries, filters, and aggregations handle missing fields correctly (SUM/AVG skip records where the field doesn't exist).

Tools

Each collection gets one tool named {user}_{collection} with parameters:

• operation (required enum: query, add, update, delete, summary)
• data (object — validated against the collection's field schema)
• record_id (string — for update/delete)
• filters (object — field → {op, value}, supports eq/neq/gt/gte/lt/lte/is_null/is_not_null)
• field, agg_operation, group_by (for summary aggregations)

Tool names include the owner prefix; the dispatcher filters per-user via tool_definitions_for_user().

When a collection is registered, a SKILL.md is auto-generated with activation keywords from the schema. Keyword/regex matching injects the skill into the system prompt when relevant. On restart, existing schemas are loaded and tools registered before the first conversation.

REST API

6 endpoints for external integration (webhooks, cron, scripts). Inherits gateway auth.

• GET/POST /api/collections — List / register schemas
• GET/POST /api/collections/{name}/records — Query / insert records
• PUT/DELETE /api/collections/{name}/records/{id} — Update / delete records

Multi-tenant scoping

Collections are scoped by user_id. Every query includes WHERE user_id = $1. Tool definitions are filtered per-user in the dispatcher, job workers, and routine engine.

For cross-user read access, source_scope allows a tool to query another user's data. source_scope is stripped in CollectionRegisterTool::execute() and collections_register_handler — only server-side seeding can set it.

Tool scaling

Each collection adds 1 tool. 20 collections = 20 tools, ~300 extra tokens with compressed descriptions. Per-user filtering and on-demand skill injection keep prompts lean.

Drawbacks

• Adds a new storage abstraction alongside memory documents. Zero impact if unused.
• Fields must be defined upfront. Adding a required field doesn't backfill old records.
• Delete by description succeeds ~40%. Record IDs in query results make delete-by-ID reliable.

Alternatives considered

• Improve document-based search: writes remain 0%.
• External database/service: breaks single-deployment model.
• Generic CRUD tools: 41%. Model forgets collection names as parameters.
• 5 per-operation tools per collection: 65%. Scales poorly (50 tools at 10 collections).

Compatibility

Additive only. No existing tables modified, no existing tool behavior changed, no changes to memory documents. V16 migration adds two new tables (structured_schemas, structured_records). owner_user_id() added to Tool trait with default None — existing tools unaffected. Both backends implemented and tested at parity. Set COLLECTION_TOOL_MODE=unified to enable (recommended).

[gemini-code-assist]

Code Review

This pull request introduces "Collections," a typed structured storage system for agent workspaces that provides schema-validated CRUD tools with support for PostgreSQL and libSQL backends. The code review identified several critical issues, including the omission of workspace read scopes in API handlers which breaks cross-lens access, and missing event broadcasts in the unified collection tool that would prevent routine triggers from firing. Additionally, feedback highlighted a correctness issue in libSQL aggregations where app-layer processing combined with record limits could yield wrong results, and pointed out that ORDER BY clauses fail to handle top-level database columns correctly. Other improvements were suggested regarding transaction locking in libSQL, robust group key extraction, and the use of UTC for consistent date resolution.

[gemini-code-assist]

The call to resolve_collection_owner passes an empty vector for scopes, effectively disabling cross-lens access for queries. To support the multi-tenant scoping described in the RFC, the user's workspace_read_scopes should be passed here. Similar issues exist in the insert, update, and delete handlers (lines 281, 384, and 478).

Suggested change

	let owner = match resolve_collection_owner(
	db.as_ref(),
	&user.user_id,
	&Vec::<String>::new(),
	&name,
	)
	let owner = match resolve_collection_owner(
	db.as_ref(),
	&user.user_id,
	&user.workspace_read_scopes,
	&name,
	)

[gemini-code-assist]

The aggregate_in_rust function performs aggregations in the application layer after fetching records. However, query_records enforces a limit of 1000, leading to incorrect results for larger collections. Following repository rules, use targeted database queries (e.g., SQL SUM/AVG) to perform aggregations at the database level to ensure correctness and prevent performance/DoS vulnerabilities.

References

1. Use targeted database queries to fetch specific records instead of loading all records and filtering in the application, especially for public endpoints, to prevent performance bottlenecks and DoS vulnerabilities.

[gemini-code-assist]

The execute_update method is missing the broadcast of a CollectionWriteEvent. This will prevent routine triggers based on collection updates from firing when this tool is used. An event should be sent to collection_write_tx similar to the implementation in execute_add.

[gemini-code-assist]

The execute_delete method is missing the broadcast of a CollectionWriteEvent. This will prevent routine triggers based on collection deletions from firing when this tool is used.

[gemini-code-assist]

The collections_list_handler is missing the logic to include the user's workspace read scopes, as indicated by the redundant extension with an empty vector. This prevents users from seeing collections they have cross-lens access to, which contradicts the PR description regarding cross-lens support.

Suggested change

	let mut user_ids = vec![user.user_id.clone()];
	user_ids.extend(Vec::<String>::new().iter().cloned());
	let mut user_ids = vec![user.user_id.clone()];
	user_ids.extend(user.workspace_read_scopes.iter().cloned());

[gemini-code-assist]

A deferred transaction (BEGIN) is used here for a read-modify-write cycle. To prevent race conditions and potential deadlocks in libSQL/SQLite, use BEGIN IMMEDIATE to acquire the write lock at the start of the transaction, as per repository guidelines.

Suggested change

	conn.execute("BEGIN", ())
	conn.execute("BEGIN IMMEDIATE", ())

References

1. To prevent race conditions in get-or-create operations in SQLite/libSQL, use BEGIN IMMEDIATE transactions to serialize concurrent writers.

[gemini-code-assist]

The ORDER BY clause interpolates the field name directly into json_extract. If a user attempts to order by a top-level column like created_at or updated_at, the query will attempt to extract these from the JSON data field (where they don't exist), resulting in an incorrect sort order. Consider handling these special fields similarly to resolve_filter_field.

[gemini-code-assist]

Using row.get::<String>(0) will fail if the group key is a numeric or boolean type in the JSON data. Using the get_text helper is safer as it handles type conversion and NULL values more robustly.

Suggested change

	.get::<String>(0)
	.unwrap_or_else(|_| "null".to_string());
	let value = extract_agg_value(&row, 1, &aggregation.operation)?;
	let key = get_text(&row, 0);

[gemini-code-assist]

Similar to the libSQL implementation, ordering by top-level columns like created_at or updated_at will fail because the query attempts to access them via the data->> JSONB operator. These fields should be handled as direct column references.

[gemini-code-assist]

Using chrono::Local::now() for natural language date resolution can lead to inconsistent results depending on the server's local timezone configuration. It is generally safer to use Utc::now() to ensure consistent behavior across different deployment environments.

Suggested change

	let today = chrono::Local::now().date_naive();
	let today = chrono::Utc::now().date_naive();

[ilblackdragon]

What about modelling this as a SQL database with tables, and let agent just run SQL queries on it?
This would then work also if user connects to external databases too. And allows to leverage all the same SQL knowledge that it has. And just can maintain metadata of tables that agent can fetch.

For multi tenant case would be great to have: per-user, shared across everyone and also group-based. This would allow to also share these tables between groups of users or the whole organization.

[standardtoaster]

Thanks for the feedback. I think SQL is interesting as a backing store, but I'd push back on exposing raw SQL to the agent:

Safety surface. SQL gives the model room to accidentally DELETE, DROP TABLE, or ALTER schema. Multi-tenant safety would need sharded databases and row-level ACLs — a lot of infrastructure to protect against something typed tools prevent by design.

Model interface. Small models likely know SQL syntax well — it's heavily represented in training data. But SQL shifts the scaling problem: instead of knowing which tool to call, the model needs to know every table name, column name, and type. That's the same context pressure as tool count, but less constrained. Typed tools give the model a bounded action space with self-documenting parameters. Happy to benchmark SQL vs typed tools if you feel strongly — I have the test harness for it.

Tool scaling is the real concern. With cross-scope access, tool count can balloon (107 tools in my deployment, 40 from collection duplicates). I think the answer is a few things working together:

1. Deferred tool schemas — compressed descriptions (~15 tokens/tool) instead of full JSON schemas. I have a branch where 87 tools = ~1,300 tokens vs ~18K for full schemas.
2. Unified search pipeline with tool activation — embed tool descriptions into the same search index as memory. When the agent does a memory_search, it gets back relevant memories and relevant tool schemas for actions it might want to take — similar to how skills are already injected based on context. As the system grows in capabilities, the search pipeline becomes the discovery mechanism rather than the tool list.
3. REST endpoints / routines for bulk operations — imports, syncs, and batch updates don't need to go through the agent's tool list. A webhook-triggered routine or dedicated endpoint handles bulk ingestion directly, keeping the agent's tool surface focused on interactive CRUD.

v2's available_actions(leases) has the right hook for this — just needs filtering wired up alongside the existing execution-time gating.

On multi-tenant sharing: I just pushed new code for cross-scope auto-discovery in this PR but haven't had a chance to fully test it yet. The approach uses workspace scoping (workspace_read_scopes) to make collections from other scopes available without duplicating tools.

[ilblackdragon]

Here is another suggest I'm actually want to try: memory files with schema validation.

We already have metadata for files in workspace - we can add schema validation.
Whenever agent reads folder or file - it gets schema and when it writes - it gets validated automatically.

Would be great to test this approach. This uses all the same infra we already have.

You can still create alias tools like grocery_list which effectively does memory_tree("collections/groceries", ).

[standardtoaster]

I prototyped this to get some signal. Quick hack: same 30 benchmark scenarios from my earlier comment on #1736, but seeded as JSON files in workspace folders instead of collection records or markdown. That comment has the full breakdown across all variants including the workspace skills from #1736.

Setup: Each entity type gets a folder (collections/commitments/, collections/signals/, etc.). Each record is a {id}-{slug}.json file. Each folder has a _schema.json describing fields and types. AGENTS.md tells the model to use memory_tree to list folders, memory_read to fetch individual records, and memory_search to find items by keyword. No custom tools — just the standard memory tools with structured data and explicit guidance. ~50 lines of seeding code added to bench-commitments.py (--variant schema-workspace-bare).

Results (Qwen 3.5 4B, 30 scenarios):

Approach	Score
collections-bare (typed tools, #1937)	0.65
workspace + skills (#1736)	0.47
workspace-bare (markdown)	0.45
schema-workspace (JSON + _schema.json)	0.37

Per-category:

Category	schema-workspace	collections-bare
Commitment query	0.24	0.72
Commitment CRUD	0.30	0.58
Signal	0.50	0.46
Decision	0.53	0.56
Aggregation	0.35	0.44
Cross-entity	0.46	0.48

Schema-workspace lands below plain markdown workspace. The model has to memory_tree → memory_read each file → parse JSON → reason across all results. For "how many commitments are critical?" that's 10+ tool calls vs 1 with a typed collection tool. Filtering by field value (urgency=critical, status=open) is where workspace approaches fundamentally struggle — memory_search finds relevant files semantically but can't do structured comparisons.

Where it does fine: signal detection (0.50) and decisions (0.53) — these are mostly "find the thing and read it" queries where semantic search works. The gap opens on filtering, counting, and cross-record aggregation.

I think both ideas have a place: collections for data you filter/aggregate (transactions, time tracking, tasks), schema-validated workspace files for semi-structured content where search is more useful than structured queries.

[standardtoaster]

This has been bouncing around in my head since your first comment about SQL. I've been looking at #2025 too, the glob/grep/file_undo tools and coding skills, and I keep coming back to a tension between two philosophies of how memory could work: everything is a file (schema validation, _schema.json, the workspace as universal substrate) versus structured data behind typed query tools. They're not necessarily in conflict, but they pull in different directions when you're deciding where to put new features.

Text files clearly work well for agent memory. But sometimes what you want isn't a blank notebook, it's a preprinted one. A grocery pad with lines for item, quantity, aisle. A timesheet with columns for date, hours, name. The structure isn't constraining the agent, it's telling it exactly what to write down and giving it a way to look things up later without reading every page. That's what my testing is showing across the variants in #1736 and above: schema-workspace scored 0.37 not because the data format was wrong, but because memory_search can't flip to "all entries where urgency=critical" the way a typed tool can. I tried throwing larger models at it (Sonnet) and it didn't seem to close the gap.

For the use cases I'm building toward (financial transactions, hour tracking, to-do management with "what's overdue?" and nagging when things go stale), the query interface is load-bearing. Even modest use accumulates: a year of nanny shifts, a few months of to-dos, eventually bank statements feeding in. That's the preprinted notebook. The schema tells the agent what fields matter, and the typed tool lets it flip straight to the right page instead of reading every entry.

I think both belong in the system. Workspace for the blank notebook (prose, context, meeting notes, identity), collections for the preprinted pads (transactions, hours, tasks). Schema validation on workspace files could bridge the gap for semi-structured content. Curious whether you see a path to making the file approach work for these query patterns, or whether you're thinking of them as separate use cases too.

[standardtoaster]

I ran these scenarios through the nearai/benchmarks harness — 52 structured data scenarios across 8 entity types (commitments, signals, decisions, parked ideas, grocery, todos, nanny hours, transactions). Same model (Qwen 3.5), same harness, same scenarios — only the storage and tool surface changes.

Workspace files (0.356) vs Collections (0.850):

Category	Workspace	Collections	Delta
commitments	0.516	0.937	+0.421
cross-entity	0.000	1.000	+1.000
grocery	0.000	0.833	+0.833
todos	0.400	0.800	+0.400
transactions	0.000	0.857	+0.857
overall	0.356	0.850	+0.494

The gap is largest for high-item-count entities — transactions (60 items: 0→0.857), grocery (12 items: 0→0.833) — and cross-entity queries (0→1.0) where workspace requires reading every file but a collection tool handles it in one call.

Benchmark PR: nearai/benchmarks#14. Harness features PR: nearai/benchmarks#13. The generator supports --variant workspace and --variant collections so anyone can reproduce both runs.