Skip to content

Commit c828277

Browse files
author
MPCoreDeveloper
committed
current status progression blite zvec compare
1 parent 2332af0 commit c828277

26 files changed

+4909
-37
lines changed

‎.github/copilot-instructions.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
- Test programs should be located in the `tests` folder and not in the repository root.
55
- All project documentation must be written in English. This includes docs, README files, technical specs, implementation plans, and code comments.
66
- Provide periodic progress updates while work is ongoing to ensure the assistant is not stuck.
7+
- When benchmarking competitor databases (like BLite), document the developer experience (DX) honestly. If a library's API is hard to use, poorly documented, or has mismatches between docs and actual API, note that as a real finding in the benchmark report. User-friendliness and ease of integration matter as much as raw performance numbers.
78

89
## Code Style
910
- Use specific formatting rules.

‎docs/benchmarks/SHARPCOREDB_COMPARATIVE_BENCHMARKS.md‎

Lines changed: 269 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,269 @@
1+
# SharpCoreDB — Comparative Benchmark Report
2+
3+
**Date:** 2026-03-06
4+
**SharpCoreDB Version:** 1.5.0 (.NET 10, C# 14)
5+
**Test Machine:** Intel i7-10850H (6C/12T), 32 GB RAM, Windows 10, NVMe SSD
6+
7+
---
8+
9+
## Executive Summary
10+
11+
SharpCoreDB was benchmarked against four competing embedded databases across two categories:
12+
13+
| Category | Competitors | SharpCoreDB Result |
14+
|---|---|---|
15+
| **Document CRUD** | SQLite, LiteDB, BLite | 🥇 Fastest INSERT, competitive UPDATE/DELETE |
16+
| **Vector Search** | Zvec (Alibaba, C++) | Competitive for managed .NET; Zvec faster on raw throughput |
17+
18+
**Key finding:** SharpCoreDB delivers the fastest batch INSERT performance of all tested databases (202K ops/sec), while SQLite remains king for point reads and updates. BLite 2.0.2 could not be benchmarked due to severe developer experience issues.
19+
20+
---
21+
22+
## Part 1: Document CRUD Benchmark
23+
24+
### Test Configuration
25+
26+
| Parameter | Value |
27+
|---|---|
28+
| **Inserts** | 100,000 documents (10K batches) |
29+
| **Reads** | 10,000 point queries by name field |
30+
| **Updates** | 10,000 row updates (batched) |
31+
| **Deletes** | 10,000 row deletes (batched) |
32+
| **Schema** | 5 columns: name (TEXT), email (TEXT), age (INTEGER), score (REAL), data (TEXT) |
33+
| **Mode** | All databases: WAL mode, optimal batch settings |
34+
35+
### Results
36+
37+
| Database | Version | INSERT ops/sec | READ ops/sec | UPDATE ops/sec | DELETE ops/sec |
38+
|---|---|---|---|---|---|
39+
| **SharpCoreDB** | 1.5.0 | **202,222** 🥇 | 6,102 | 8,411 | 7,203 |
40+
| **SQLite** | 10.0.3 | 167,363 | **96,724** 🥇 | **252,482** 🥇 | **378,961** 🥇 |
41+
| **LiteDB** | 5.0.21 | 91,845 | 13,317 | 9,218 | 13,907 |
42+
| **BLite** | 2.0.2 | ❌ DNF | ❌ DNF | ❌ DNF | ❌ DNF |
43+
44+
> DNF = Did Not Finish — could not successfully run benchmark (see [BLite Analysis](#blite-202--developer-experience-analysis) below)
45+
46+
### Time Breakdown (seconds)
47+
48+
| Database | INSERT (100K) | READ (10K) | UPDATE (10K) | DELETE (10K) |
49+
|---|---|---|---|---|
50+
| **SharpCoreDB** | 0.49s | 1.64s | 1.19s | 1.39s |
51+
| **SQLite** | 0.60s | 0.10s | 0.04s | 0.03s |
52+
| **LiteDB** | 1.09s | 0.75s | 1.08s | 0.72s |
53+
54+
### Analysis
55+
56+
#### SharpCoreDB — Fastest Batch Inserts
57+
58+
SharpCoreDB's `InsertBatch()` API achieves **202K inserts/sec**, beating SQLite by 21% and LiteDB by 120%. This is powered by:
59+
- SQL-free direct insert path (bypasses SQL parsing entirely)
60+
- Prepared statement caching for repeated schemas
61+
- Batched WAL flushing (single fsync per batch instead of per-row)
62+
- StreamingRowEncoder for zero-allocation bulk inserts
63+
64+
**Trade-off:** Read/Update/Delete performance is behind SQLite. SharpCoreDB's current read path (10K individual `ExecuteSQL` SELECT queries) is bottlenecked by SQL parsing overhead per query. SQLite has 20+ years of B-tree optimization and compiled C native code.
65+
66+
#### SQLite — The Gold Standard for Point Operations
67+
68+
SQLite dominates READ (97K ops/sec), UPDATE (252K ops/sec), and DELETE (379K ops/sec). This is expected — SQLite is a 20+ year C library with:
69+
- Native compiled code (no managed runtime overhead)
70+
- Highly optimized B-tree with page-level caching
71+
- Prepared statement API (parameterized queries avoid reparsing)
72+
- WAL mode with shared memory for concurrent readers
73+
74+
#### LiteDB — Solid .NET Document Database
75+
76+
LiteDB shows balanced performance across all operations. As a pure .NET BSON document database, it's a fair peer comparison to SharpCoreDB:
77+
- INSERT: SharpCoreDB is **2.2x faster** (202K vs 92K)
78+
- READ: LiteDB is **2.2x faster** (13K vs 6K) — LiteDB's FindById uses direct B-tree lookup
79+
- UPDATE: Comparable (9.2K vs 8.4K)
80+
- DELETE: LiteDB is **1.9x faster** (14K vs 7.2K)
81+
82+
**Overall vs LiteDB:** SharpCoreDB wins bulk inserts decisively; LiteDB wins point reads. Both are competitive on updates.
83+
84+
---
85+
86+
### BLite 2.0.2 — Developer Experience Analysis
87+
88+
BLite was intended as a fourth benchmark competitor. It is positioned as a "zero-allocation embedded document database" for .NET with BSON, B-tree indexing, HNSW vector search, and LINQ support. On paper, it's an impressive project.
89+
90+
**However, we were unable to successfully run any BLite benchmarks.** This section documents the issues encountered, because developer experience is just as important as raw performance.
91+
92+
#### What Went Wrong
93+
94+
| # | Issue | Details |
95+
|---|---|---|
96+
| 1 | **Source generator failure** | BLite's `DocumentDbContext` requires a compile-time source generator (`BLite.SourceGenerators`) to emit `InitializeCollections()`. The generator silently produced no output, leaving the `Docs` property `null` at runtime → `NullReferenceException`. |
97+
| 2 | **API documentation mismatch** | The README documents `b.Set("field", value)` for `BsonDocumentBuilder`, but this method does not exist in 2.0.2. Neither do `b["field"]`, `b.Add()`, `b.Write()`, or any other setter we tried. |
98+
| 3 | **`BsonDocument` has no parameterless constructor** | Cannot create a `new BsonDocument()` — the only path is `col.CreateDocument(fields, builder)`, but the builder API is undiscoverable (see #2). |
99+
| 4 | **`BsonDocument` has no indexer** | The README shows `doc.GetString("field")`, `doc.GetInt32("field")`, `doc.Id` — none of these compiled against BLite.Bson 2.0.2. |
100+
| 5 | **No IntelliSense discoverability** | With no parameterless constructors, no working extension methods, and no XML doc comments visible in the NuGet package, the API is essentially undiscoverable without source code access. |
101+
| 6 | **Package structure confusion** | BLite is a meta-package referencing `BLite.Core`, `BLite.Bson`, and `BLite.SourceGenerators`. Adding `BLite.Core` separately alongside `BLite` caused subtle resolution issues. |
102+
103+
#### What We Tried
104+
105+
```
106+
Attempt 1: DocumentDbContext + DocumentCollection<ObjectId, T> → Source generator silent failure
107+
Attempt 2: BLiteEngine + DynamicCollection + CreateDocument(fields, b => b.Set(...)) → "Set" does not exist
108+
Attempt 3: BLiteEngine + DynamicCollection + CreateDocument(fields, b => { b["key"] = val; }) → No indexer
109+
Attempt 4: BLiteEngine + DynamicCollection + CreateDocument(fields, b => b.Write(...)) → "Write" does not exist
110+
Attempt 5: BLiteEngine + DynamicCollection + new BsonDocument() → No parameterless constructor
111+
```
112+
113+
#### Verdict
114+
115+
> **BLite may be a capable database engine, but in its 2.0.2 NuGet release, the public API for the schema-less path (`BLiteEngine` + `DynamicCollection`) is not usable without access to the source code.** The documented API does not match the shipped binary. The strongly-typed path (`DocumentDbContext`) requires a source generator that silently fails to produce output.
116+
>
117+
> This is a critical developer experience issue. If a senior .NET developer with AI assistance cannot write a basic Insert → Read → Update → Delete flow after 5+ attempts, the library is not ready for production adoption.
118+
119+
#### Fair Disclosure
120+
121+
- BLite is an active open-source project under development. Version 2.0.2 may have been published ahead of documentation sync.
122+
- The source generator may work correctly in a standalone project (our test was inside a larger solution).
123+
- Future versions may resolve these issues. We recommend re-evaluating when 3.0 ships.
124+
- BLite's feature set (BSON, HNSW, LINQ, zero-allocation) is ambitious and technically interesting.
125+
126+
---
127+
128+
## Part 2: Vector Search Benchmark
129+
130+
### SharpCoreDB HNSW vs Zvec (Alibaba)
131+
132+
Full details: [SHARPCOREDB_VS_ZVEC_COMPARISON.md](SHARPCOREDB_VS_ZVEC_COMPARISON.md)
133+
134+
#### Index Build (128-dimensional vectors)
135+
136+
| Dataset | SharpCoreDB | Zvec (estimated) |
137+
|---|---|---|
138+
| 1K vectors | 2,934 vec/sec | — |
139+
| 10K vectors | 1,676 vec/sec | — |
140+
| 100K vectors | 573 vec/sec | ~10,000+ vec/sec |
141+
142+
#### Search Latency (100K index, K=10)
143+
144+
| Metric | SharpCoreDB | Zvec |
145+
|---|---|---|
146+
| p50 | 0.530ms | <0.5ms (est.) |
147+
| p95 | 0.758ms | — |
148+
| p99 | 0.964ms | — |
149+
150+
#### Search Throughput (100K index, K=10, 10s)
151+
152+
| Threads | SharpCoreDB QPS | Zvec QPS |
153+
|---|---|---|
154+
| 1 thread | 1,815 | — |
155+
| 8 threads | 9,231 | 15,000+ |
156+
157+
#### Key Differentiator: Adaptive SIMD
158+
159+
SharpCoreDB automatically uses the best SIMD instruction set available on the host CPU:
160+
161+
| SIMD Tier | Width | SharpCoreDB | Zvec |
162+
|---|---|---|---|
163+
| AVX-512 | 512-bit (16 floats) | ✅ Auto-detected | ✅ Hard requirement |
164+
| AVX2+FMA | 256-bit (8 floats) | ✅ Auto-detected | ❌ Crashes (`Illegal instruction`) |
165+
| SSE2 | 128-bit (4 floats) | ✅ Auto-detected | ❌ |
166+
| ARM NEON | 128-bit (4 floats) | ✅ Auto-detected | ❌ |
167+
| Scalar | 32-bit | ✅ Fallback | ❌ |
168+
169+
> **Zvec crashed with `Illegal instruction (core dumped)` on our test CPU** (i7-10850H, AVX2) because it hard-requires AVX-512. SharpCoreDB's single binary runs on any CPU and adapts automatically.
170+
171+
---
172+
173+
## Part 3: Overall Comparison Matrix
174+
175+
### CRUD Databases
176+
177+
| Feature | SharpCoreDB | SQLite | LiteDB | BLite |
178+
|---|---|---|---|---|
179+
| **Type** | SQL + Document + Vector | SQL | Document (BSON) | Document (BSON) |
180+
| **Language** | C# (.NET 10) | C (native) | C# (.NET Standard) | C# (.NET 10) |
181+
| **Best at** | Batch INSERT | Point R/U/D | Balanced CRUD | Unknown (untestable) |
182+
| **INSERT 100K** | **202K ops/sec** 🥇 | 167K ops/sec | 92K ops/sec | ❌ |
183+
| **READ 10K** | 6K ops/sec | **97K ops/sec** 🥇 | 13K ops/sec | ❌ |
184+
| **SQL Support** | ✅ Full | ✅ Full | ❌ | ❌ |
185+
| **Vector Search** | ✅ HNSW | ❌ | ❌ | ✅ HNSW |
186+
| **Encryption** | ✅ Built-in | ❌ (extension) | ❌ | ❌ |
187+
| **API Usability** | ✅ Simple | ✅ Standard | ✅ Good | ❌ Undiscoverable |
188+
| **NuGet Package** | ✅ | ✅ | ✅ | ⚠️ Issues |
189+
| **Documentation** | ✅ | ✅ Extensive | ✅ Good | ⚠️ Mismatches |
190+
191+
### Vector Databases
192+
193+
| Feature | SharpCoreDB | Zvec (Alibaba) |
194+
|---|---|---|
195+
| **Type** | Full database + vector | Vector-only |
196+
| **Language** | C# (.NET 10) | C++ |
197+
| **Platform** | Windows + Linux + macOS + ARM | Linux only (AVX-512 only) |
198+
| **QPS (8 threads)** | 9,231 | 15,000+ |
199+
| **Sub-ms latency** | ✅ (p50 = 0.53ms) | ✅ |
200+
| **SQL + Vector** | ✅ Hybrid queries | ❌ |
201+
| **Adaptive SIMD** | ✅ (any CPU) | ❌ (crashes without AVX-512) |
202+
| **Single binary** | ✅ | ❌ |
203+
| **.NET native** | ✅ | ❌ (Python/C++ bridge) |
204+
205+
---
206+
207+
## Conclusions
208+
209+
### When to Use SharpCoreDB
210+
211+
- **Bulk data ingestion** — fastest INSERT of all tested databases (202K ops/sec)
212+
- **Hybrid workloads** — SQL + vector search + encryption in a single embedded database
213+
- **Cross-platform .NET** — runs on Windows, Linux, macOS, ARM with adaptive SIMD
214+
- **Simplicity** — one NuGet package, no native dependencies, no external processes
215+
216+
### When to Use SQLite
217+
218+
- **Read-heavy workloads** — 16x faster point reads than SharpCoreDB
219+
- **Update/Delete-heavy** — 30-50x faster individual row operations
220+
- **Mature ecosystem** — 20+ years of production use, tooling, and documentation
221+
222+
### When to Use LiteDB
223+
224+
- **Balanced .NET document database** — good all-around performance
225+
- **Schema-less BSON** — when you need MongoDB-like document storage embedded in .NET
226+
- **Simple API** — well-documented, easy to pick up
227+
228+
### When NOT to Use BLite (as of 2.0.2)
229+
230+
- **Production use** — public API is not usable from NuGet without source code access
231+
- **Quick prototyping** — documented API does not match shipped binary
232+
- **Any project that values time-to-first-query** — basic CRUD required 5+ failed attempts
233+
234+
---
235+
236+
## Methodology & Reproducibility
237+
238+
### Test Harness
239+
240+
All benchmarks used the same test structure:
241+
1. Create database in temp directory
242+
2. INSERT N documents with identical schema across all databases
243+
3. READ M documents by field match
244+
4. UPDATE M documents
245+
5. DELETE M documents
246+
6. Measure wall-clock time per operation
247+
7. Clean up temp files
248+
249+
### Fairness
250+
251+
| Aspect | Approach |
252+
|---|---|
253+
| **Same schema** | All databases used the same 5-column document structure |
254+
| **Same data** | Identical test data generated for all databases |
255+
| **Optimal API** | Each database used its recommended batch/bulk API |
256+
| **WAL mode** | SQLite: `PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL` |
257+
| **Warm start** | All measurements exclude database creation time |
258+
| **No indexing** | No secondary indexes — tests raw storage engine performance |
259+
260+
### Source Code
261+
262+
- Comparative benchmark: `tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/`
263+
- Zvec benchmark: `tests/benchmarks/zvec_python/` and `tests/benchmarks/QuickZvecTest/`
264+
- Raw results: `tests/benchmarks/SharpCoreDB.Benchmarks.Comparative/results/`
265+
266+
---
267+
268+
*Report generated 2026-03-06 by SharpCoreDB Benchmark Suite v1.5.0*
269+
*All tests run on: .NET 10.0.3, Intel i7-10850H, 32GB RAM, NVMe SSD, Windows 10*

0 commit comments

Comments
 (0)