Hemant/rebuild updated by hemant-endee · Pull Request #172 · endee-io/endee

hemant-endee · 2026-04-09T05:24:27Z

No description provided.

github-actions · 2026-04-09T05:25:24Z

VectorDB Benchmark — Dense — Passed

Triggered by @omnish-endee · Commit 4a3982f

Step	Status
Provision Servers	Up
Deploy Endee Server	Done
Run Benchmark	Passed
Results	See below
Teardown	Done ✓

No result file found on benchmark server.

hemant-endee · 2026-04-09T08:51:42Z

/correctness_benchmarking dense

omnish-endee · 2026-04-10T07:21:43Z

/correctness_benchmarking dense

omnish-endee · 2026-04-10T08:41:21Z

/correctness_benchmarking dense

omnish-endee · 2026-04-10T09:47:43Z

/correctness_benchmarking dense

github-actions · 2026-04-21T09:59:03Z

VectorDB Benchmark - Ready To Run

CI Passed ([lint + unit tests] (https://github.com/endee-io/endee/actions/runs/24928018536)) - benchmark options unlocked.

Post one of the command below. Only members with write access can trigger runs.

Available Modes

Mode	Command	What runs
Dense	`/correctness_benchmarking dense`	HNSW insert throughput · query P50/P95/P99 · recall@10 · concurrent QPS
Hybrid	`/correctness_benchmarking hybrid`	Dense + sparse BM25 fusion · same suite + fusion latency overhead

Infrastructure

Server	Role	Instance
Endee Server	Endee VectorDB — code from this branch	`t2.large`
Benchmark Server	Benchmark runner	`t3a.large`

Both servers start on demand and are always terminated after the run — pass or fail.

How Correctness Benchmarking Works

1. Post /correctness_benchmarking <mode>
2. Endee Server Create  →  this branch's code deployed  →  Endee starts in chosen mode
3. Benchmark Server Create  →  benchmark suite transferred
4. Benchmark Server runs correctness benchmarking against Endee Server
5. Results posted back here  →  pass/fail + full metrics table
6. Both servers terminated   →  always, even on failure

After a new push, CI must pass again before this menu reappears.

hemant-endee · 2026-04-21T10:18:36Z

/correctness_benchmarking dense

github-actions · 2026-04-21T10:18:55Z

VectorDB Benchmark — Dense — Failed

Triggered by @hemant-endee · Commit 117a81f

Step	Status
Provision Servers	Up
Deploy Endee Server	Done
Run Benchmark	Failed
Results	See reason below
Teardown	Done ✓

Benchmark job status: skipped

shaleenji · 2026-04-23T06:19:26Z

+struct RebuildResult {
+    bool success;
+    int http_code;
+    std::string message;
+};



use unsigned char and std::string. declare a generic return struct which will be used everywhere in the codebase eventually.

for each function returning this struct, at the comments above the function definition will define what each return code means. This can be later conglomerated as ENUMS.

shaleenji · 2026-04-23T06:20:21Z

-                        int code = (message.find("already in progress") != std::string::npos) ? 409 : 400;
-                        return json_error(code, message);
+                    if (!result.success) {
+                        return json_error(result.http_code, result.message);


do not couple http codes to the core source. decide the http_code to be returned based on the return code of the function,.

shaleenji · 2026-04-23T06:36:23Z

dont use atomics in status variables

shaleenji · 2026-04-23T06:37:06Z

remove jthread from here

shaleenji · 2026-04-23T06:38:26Z

dont use string comparisons for this.

shaleenji · 2026-04-23T06:40:40Z

-        } catch (const std::exception&) {
-            // Silently ignore cleanup errors on startup
+        } catch (const std::exception& e) {
+            LOG_WARN(2053, "rebuild", "Failed to cleanup temp files on startup: " << e.what());


use 1800 series for rebuild logs. add it to docs/logs.md

* Rebuild index with new config * fix 1 * index name in get stattu api correction * docs changes * Rebuild Status Persistence

…me as addVectors)

…y + IndexManager to executeJob instead of unpacking fields

shaleenji · 2026-04-24T12:42:16Z

+void Rebuild::cleanupTempFiles(const std::string& data_dir) {
+    if (!std::filesystem::exists(data_dir)) {
+        return;
+    }
+    try {
+        std::string temp_filename = std::string(settings::DEFAULT_SUBINDEX) + ".idx.temp";
+        std::string ts_prefix     = std::string(settings::DEFAULT_SUBINDEX) + ".idx.";
+        for (const auto& entry : std::filesystem::recursive_directory_iterator(data_dir)) {
+            if (!entry.is_regular_file()) continue;
+            const std::string fname = entry.path().filename().string();
+            bool is_temp = (fname == temp_filename);
+            bool is_ts   = fname.size() > ts_prefix.size()
+                           && fname.substr(0, ts_prefix.size()) == ts_prefix
+                           && std::all_of(fname.begin() + ts_prefix.size(), fname.end(), ::isdigit);
+            if (is_temp || is_ts) {
+                std::filesystem::remove(entry.path());
+            }
+        }
+    } catch (const std::exception& e) {
+        LOG_WARN(1803, "rebuild", "Failed to cleanup temp files on startup: " << e.what());
+    }
+}


should not throw error if there are not cleanup files left in the directory.

shaleenji · 2026-04-26T12:34:02Z

+        std::filesystem::copy_file(p.timestamped_path, p.index_path,
+            std::filesystem::copy_options::overwrite_existing);


why is renaming the index file not enough here ?

shaleenji · 2026-04-26T12:37:15Z

+        std::filesystem::copy_file(p.timestamped_path, p.index_path,
+            std::filesystem::copy_options::overwrite_existing);
+
+        // Cannot call reloadIndex() here — we hold operation_mutex and reloadIndex acquires


st.stop_requested() is not checked in phase 3. makes the shutdown time unbounded for large indexes.

shaleenji · 2026-04-26T12:40:19Z

+        new_alg->saveIndex(p.timestamped_path);
+        std::filesystem::copy_file(p.timestamped_path, p.index_path,
+            std::filesystem::copy_options::overwrite_existing);
+
+        // Cannot call reloadIndex() here — we hold operation_mutex and reloadIndex acquires
+        // indices_mutex_, while deleteIndex holds indices_mutex_ then acquires operation_mutex.
+        // Calling reloadIndex here would deadlock with a concurrent delete on the same index.
+        auto fresh_alg = std::make_unique<hnswlib::HierarchicalNSW<float>>(p.index_path, 0);
+        IndexManager::wireVectorFetchers(fresh_alg.get(), entry->vector_storage);


new_alg already has all the data and was just persisted.

can use entry->alg = std::move(new_alg) directly.

doesnt seem useful to re load again

shaleenji · 2026-04-26T12:42:42Z

@@ -0,0 +1,124 @@
+# Index Rebuild


Things that are missing in the doc. make it more comprehensive

Disk space: rebuild needs 2× the index size on disk (timestamped + canonical co-exist briefly).

Memory: 2× the graph in RAM during Phase 2 (old + new).

Expected duration: rough order of magnitude per million vectors helps capacity planning.

What "manually restart" means after a crash — operationally how does an operator know a rebuild was incomplete? (There's no persisted state.)

shaleenji · 2026-04-26T12:43:29Z

+std::string Rebuild::formatTime(std::chrono::system_clock::time_point tp) {
+    return timeToISO8601(tp);
+}


no one is using this function

shaleenji · 2026-04-26T12:44:50Z

+    rebuild_.setActiveRebuild(username, index_id, current_count);
+
+    // Spawn thread — lambda calls rebuild_.executeJob directly (execution lives in Rebuild)
+    std::jthread t([this, params = std::move(params)](std::stop_token st) mutable {


here mutable is not required. executeJob takes const RebuildJobParams&, so the lambda doesn't actually mutate

shaleenji · 2026-04-26T12:46:25Z

+        case RebuildStatus::IN_PROGRESS: return "in_progress";
+        case RebuildStatus::COMPLETED:   return "completed";
+        case RebuildStatus::FAILED:      return "failed";
+        default:                         return "unknown";


can never reach this line since RebuildStatus has only three defined values

shaleenji requested changes Apr 21, 2026

View reviewed changes

shaleenji reviewed Apr 23, 2026

View reviewed changes

Comment thread src/core/rebuild.hpp Outdated

Comment on lines 29 to 30

Copy link
Copy Markdown

Collaborator

shaleenji Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dont use atomics in status variables

shaleenji requested changes Apr 23, 2026

View reviewed changes

hemant-endee added 8 commits April 23, 2026 16:37

Rebuild index with new config (#136)

85aded6

* Rebuild index with new config * fix 1 * index name in get stattu api correction * docs changes * Rebuild Status Persistence

using jthread with stop token

bb2f3d6

Shared parallel addPoint utility function — static chunk partition(sa…

cbaa744

…me as addVectors)

comments resolved

cb9f73d

rebuild also handles execute_rebuild_job

fd2c034

correction on cleantempfiles and error code

3b32a2e

rebuild status enum and logs code change

13fe6ce

rebuild.cpp

6f4083e

hemant-endee force-pushed the hemant/rebuild_updated branch from 117a81f to 6f4083e Compare April 23, 2026 16:09

Using Rebuild as friend class on Indexmanager helps to pass CacheEntr…

2202923

…y + IndexManager to executeJob instead of unpacking fields

shaleenji requested changes Apr 26, 2026

View reviewed changes

		std::filesystem::copy_file(p.timestamped_path, p.index_path,
		std::filesystem::copy_options::overwrite_existing);

Conversation

hemant-endee commented Apr 9, 2026

Uh oh!

github-actions Bot commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

VectorDB Benchmark — Dense — Passed

Uh oh!

hemant-endee commented Apr 9, 2026

Uh oh!

omnish-endee commented Apr 10, 2026

Uh oh!

omnish-endee commented Apr 10, 2026

Uh oh!

omnish-endee commented Apr 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

VectorDB Benchmark - Ready To Run

Available Modes

Infrastructure

How Correctness Benchmarking Works

Uh oh!

hemant-endee commented Apr 21, 2026

Uh oh!

github-actions Bot commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

VectorDB Benchmark — Dense — Failed

Uh oh!

shaleenji Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Apr 9, 2026 •

edited

Loading

github-actions Bot commented Apr 21, 2026 •

edited

Loading

github-actions Bot commented Apr 21, 2026 •

edited

Loading

shaleenji Apr 23, 2026 •

edited

Loading