fix(libsql): support flexible embedding dimensions

[zmanian]

Summary

Fixes #494.

• Adds incremental migration infrastructure to the libSQL backend (_migrations table tracking + run_incremental() runner)
• V9 migration rebuilds memory_chunks table with BLOB column (was F32_BLOB(1536)) so any embedding dimension works
• Drops the libsql_vector_idx vector index (requires fixed-dimension F32_BLOB(N)); vector search gracefully falls back to FTS-only, matching PostgreSQL behavior after its V9 migration
• Updates base schema for fresh installs (no vector index, BLOB column)
• Removes the now-incorrect "dimension is not 1536" warnings from app.rs and main.rs

Upgrade path: Existing users get the V9 migration automatically on next startup. All data (documents, chunks, embeddings) is preserved. Users only need to re-embed if they change their embedding model/dimension, which backfill_embeddings() handles.

Test plan

• cargo test --lib -- 1853 passed, 0 failed
• cargo clippy --all --all-features -- zero warnings
• cargo check --no-default-features --features libsql -- compiles
• Manual: verify existing libSQL database upgrades cleanly on startup
• Manual: verify 768-dim embeddings (e.g. nomic-embed-text via Ollama) store and search correctly

Generated with Claude Code

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the libSQL database backend by introducing support for flexible embedding dimensions. It achieves this by updating the memory_chunks table schema to use a generic BLOB type for embeddings and implementing a robust incremental migration system to manage these schema changes. The update also adapts the vector search mechanism to function without a fixed-dimension index, ensuring compatibility and reducing previous configuration constraints for embedding storage and retrieval.

Highlights

• Flexible Embedding Dimensions: The libSQL backend now supports flexible embedding dimensions by changing the memory_chunks table to store embeddings as BLOB instead of F32_BLOB(1536).
• Incremental Migration Infrastructure: A new incremental migration system has been added for libSQL, tracking applied migrations in a _migrations table and running them sequentially.
• Vector Index Removal and Fallback: The libsql_vector_idx vector index has been dropped, as it requires fixed-dimension embeddings. Vector search in libSQL will now gracefully fall back to FTS-only, matching PostgreSQL behavior.
• Removed Dimension Warnings: Warnings in app.rs and main.rs related to non-1536 embedding dimensions have been removed, as the system now supports arbitrary dimensions.

Changelog

• src/app.rs
  • Removed the warning message that appeared when libSQL was used with embedding dimensions other than 1536.
• src/db/libsql/mod.rs
  • Added a call to libsql_migrations::run_incremental() to apply new incremental migrations on database initialization.
• src/db/libsql/workspace.rs
  • Modified the vector search logic to handle potential failures when querying the libsql_vector_idx and to fall back to full-text search (FTS) if the index is absent.
• src/db/libsql_migrations.rs
  • Updated comments to reflect the new incremental migration system.
  • Changed the VECTOR(1536) mapping to BLOB for flexible embedding dimensions in the base schema description.
  • Modified the memory_chunks table definition in the base schema to use BLOB for the embedding column and removed the libsql_vector_idx index creation.
  • Introduced INCREMENTAL_MIGRATIONS constant, defining a V9 migration that rebuilds the memory_chunks table, drops the old vector index, and recreates FTS triggers to support flexible embedding dimensions.
  • Added the run_incremental function to manage the application and tracking of incremental migrations.
• src/main.rs
  • Removed the warning message that appeared when libSQL was used with embedding dimensions other than 1536.

Activity

• The author provided a detailed summary of the changes, including an upgrade path and test plan.
• The pull request was generated using Claude Code.
• The author reported passing cargo test --lib and cargo clippy --all --all-features checks.
• The author reported successful compilation with cargo check --no-default-features --features libsql.
• Manual verification steps for database upgrade and 768-dim embeddings were outlined.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces support for flexible embedding dimensions in the libSQL backend, which is a great improvement for users who want to use different embedding models. The changes include a new incremental migration system, a V9 migration to alter the memory_chunks table, and updates to gracefully handle the absence of a vector index. The implementation is well-structured. I've found one potential issue regarding the atomicity of migrations which could lead to an inconsistent database state if a migration is interrupted.

[gemini-code-assist]

The current implementation of run_incremental executes the migration SQL and records its completion as two separate operations. If the application were to be interrupted after the migration SQL runs but before it's recorded in the _migrations table, the migration would be partially or fully applied but not marked as complete. This could lead to failures on subsequent startups when the application tries to re-run the same migration on an altered schema.

To ensure atomicity, each migration should be wrapped in a database transaction. This guarantees that either both the migration and its tracking record are committed, or neither is, preventing an inconsistent state. The libsql crate provides a convenient transaction() API that can be used for this purpose. The transaction will automatically roll back if not explicitly committed.

        let tx = conn.transaction().await.map_err(|e| {
            DatabaseError::Migration(format!(
                "Failed to start transaction for V{version} ({name}): {e}"
            ))
        })?;

        tx.execute_batch(sql).await.map_err(|e| {
            DatabaseError::Migration(format!(
                "libSQL migration V{version} ({name}) failed: {e}"
            ))
        })?;

        // Record as applied
        tx.execute(
            "INSERT INTO _migrations (version, name) VALUES (?1, ?2)",
            libsql::params![version, name],
        )
        .await
        .map_err(|e| {
            DatabaseError::Migration(format!(
                "Failed to record migration V{version} ({name}): {e}"
            ))
        })?;

        tx.commit().await.map_err(|e| {
            DatabaseError::Migration(format!(
                "Failed to commit transaction for V{version} ({name}): {e}"
            ))
        })?;

[zmanian]

Addressed in d3cf7a4. The migration execution and recording are now wrapped in a conn.transaction() with explicit tx.commit(), so they succeed or fail atomically. If the process crashes mid-migration, the transaction rolls back and the migration retries on next startup.