v0.4.0

What's Changed

• docs: complete collections API documentation for remaining metrics by @sanjeed5 in #2420
• feat: support GPT-5 and o-series models with automatic temperature and top_p constraint handling by @anistark in #2418
• update: add llm options as tabs to quickstart by @anistark in #2421
• feat: migrate to instructor.from_provider for universal provider support by @anistark in #2424
• docs: fix typos in some files by @Edge-Seven in #2429
• feat: implement prompt class for context precision by @anistark in #2433
• fix: docs for quickstart by @anistark in #2434
• Fix 2432: Update import statements for langchain modules by @rvernica in #2436
• Fix 2440 Adjust LLM parameters in evals.py by @rvernica in #2441
• feat: migrate context recall, answer relevancy, and context entity recall metrics to modular BasePrompt architecture by @anistark in #2435
• feat: migrate 6 metrics (ContextRelevance, Response Groundedness, AnswerAccuracy, Faithfulness, AnswerCorrectness, SummaryScore) to BasePrompt by @anistark in #2443
• feat: migrate final metrics (FactualCorrectness, NoiseSensitivity) to modular BasePrompt architecture and update docs by @anistark in #2444
• chore: add COC by @anistark in #2437
• docs: clarify MLflow is required, not optional in RAG evaluation guide by @anistark in #2447
• chore: cleanup old patterns and update links by @anistark in #2449
• chore: rebranding efforts by @jjmachan in #2445
• feat: dual adapter support (Instructor + LiteLLM) by @anistark in #2446
• fix: resolve InstructorLLM detection bug and add EvaluationDataset backend support for experiments by @anistark in #2451
• fix: retrieved_contexts string filtering in LangChain integration by @dhyaneesh in #2452
• fix: correct MultiTurnSample user_input validation logic by @harshil-sanghvi in #2426
• fix: automatic embedding provider matching for LLMs by @anistark in #2454
• fix: make GitPython an optional dependency by @anistark in #2453
• docs: Update customizations how-to guides to use collections API and LLM factory by @sanjeed5 in #2425
• fix: detect async clients in closures for instructor-wrapped litellm routers by @anistark in #2458
• fix: quickstart by @anistark in #2463
• chore: update calendar email by @anistark in #2462
• fix: make GoogleEmbeddings handle GenerativeModel clients by auto-extracting genai module by @anistark in #2466
• docs: add migration guide for v0.4 by @anistark in #2461

New Contributors

• @Edge-Seven made their first contribution in #2429
• @rvernica made their first contribution in #2436
• @dhyaneesh made their first contribution in #2452

Full Changelog: v0.3.9...v0.4.0

v0.3.9

What's Changed

• fix(docs): add missing line break so the step title and description a… by @nkch1k in #2391
• Migrate SummaryScore by @rhlbhatnagar in #2376
• feat: add metadata fields for synthetic data traceability by @dev-jonathan in #2389
• Migrate noise sensitivity by @rhlbhatnagar in #2379
• docs: quickstart guide with interactive LLM and project structure by @anistark in #2380
• Migrate Faithfullness by @rhlbhatnagar in #2384
• fix: docs for discrete, numeric and ranking using instructor by @anistark in #2397
• Migrate Answer Accuracy + Context Relevance by @rhlbhatnagar in #2390
• refactor: remove aspect critic and simple criteria metrics with discrete metric examples by @anistark in #2399
• Migrate Context Pricision with + without ref by @rhlbhatnagar in #2398
• docs: fix recall formula label in SQL metrics by @tysoncung in #2405
• chore: remove deprecated ground_truths by @anistark in #2402
• docs: Add documentation for metrics.collections API by @sanjeed5 in #2407
• Migrate factual correctness by @rhlbhatnagar in #2401
• refactor: remove redundant AnswerSimilarity from collections API in favor of SemanticSimilarity by @anistark in #2410
• docs: Update documentation structure to reflect experiments-first paradigm by @sanjeed5 in #2394
• Response Groundedness by @rhlbhatnagar in #2403
• fix: office hours link update by @anistark in #2415
• Refactor/removing deprecated by @anistark in #2412
• fix: handle max_completeion_tokens error for newer openai models by @anistark in #2413
• refactor: make embeddings optional in AnswerCorrectness when using pure factuality mode by @anistark in #2414
• Feat/migrate context recall by @jjmachan in #2372
• chore: update quickstart llm config by @anistark in #2417

New Contributors

• @nkch1k made their first contribution in #2391
• @dev-jonathan made their first contribution in #2389
• @tysoncung made their first contribution in #2405

Full Changelog: v0.3.8...v0.3.9

v0.3.8

What's Changed

• feat: semantic similarity migrated to collections by @anistark in #2361
• feat: Add reusable testing infrastructure for metrics migration by @jjmachan in #2370
• add: console scripts for ragas_examples by @anistark in #2367
• feat: add quickstart cmd with templates to run by @anistark in #2374
• fix: detect uvloop and skip nest_asyncio to prevent patching errors by @anistark in #2369
• Remove import not used by @ChenyangLi4288 in #2364
• Migrate answer_correctness by @rhlbhatnagar in #2365
• Migrate context_entity_recall by @rhlbhatnagar in #2366
• feat: aspect critic metric for coherence, harmfulness, maliciousness, correctness by @anistark in #2375
• Fixed: NameError during evalutation of llamaindex query engine by @Prigoistic in #2331
• Remove error suppressor in async_utils.py and engine.py by @ChenyangLi4288 in #2362
• docs: clarify Context Relevance implementation differs from paper design by @anistark in #2378
• fix: add missing metrics (ToolCallF1, ChrfScore) to sidebar and document deprecated ContextUtilization by @anistark in #2381
• refactor: instructor_llm_factory merge with llm_factory by @anistark in #2382
• fix: handle tuple-formatted entities in SingleHopSpecificQuerySynthesizer by @anistark in #2377
• feat: simple criteria migrated to collections by @anistark in #2386
• chore: remove deprecation warnings for LangchainLLMWrapper, LlamaIndexLLMWrapper, and embedding wrappers by @anistark in #2387

New Contributors

• @ChenyangLi4288 made their first contribution in #2364

Full Changelog: v0.3.7...v0.3.8

v0.3.7

What's Changed

• refactor: improve metrics code quality by @anistark in #2337
• chore: remove old analtyics by @jjmachan in #2338
• Fix/query distribution robustness by @yatoyun in #2340
• Simplify earlier how to guides in docs by @sanjeed5 in #2319
• docs: reorganize prompt evaluation guides in navigation by @sanjeed5 in #2346
• Metrics migration, migrate rouge + answer relevance by @rhlbhatnagar in #2335
• fix: streamline theme extraction from overlaps in MultiHopSpecificQue… by @kenzoyan in #2347
• Test/metric new compare by @anistark in #2349
• feat: bleu score migrated to collections by @anistark in #2352
• fix: Add List[List[str]] formats for overlapped items in theme extration (Continuation in #2347) by @kenzoyan in #2355
• feat: string metrics migrated to collections by @anistark in #2356
• feat: answer similarity migrated to collections by @anistark in #2358
• fix: add missing props token_usage_parser for test generation methods #2359 by @bhkj9999 in #2360
• feat: add bypass_n option to LangchainLLMWrapper for n-completion control by @SimFG in #2354
• docs: Add how-to guide for aligning LLM-as-Judge by @sanjeed5 in #2348

New Contributors

• @yatoyun made their first contribution in #2340
• @kenzoyan made their first contribution in #2347
• @bhkj9999 made their first contribution in #2360
• @SimFG made their first contribution in #2354

Full Changelog: v0.3.6...v0.3.7

v0.3.6

What's Changed

• Feature/chrf score by @kauabh in #2221
• Fix/asyncio by @anistark in #2294
• Fix: update simple RAG init to use embed_text(s) (docs) by @s3pi in #2292
• Update _bleu_score.py by @kauabh in #2297
• Refactor/update gemini to genai sdk by @sahusiddharth in #2240
• Feature/metrics input flexibility by @anistark in #2298
• Ensure old_temperature is set correctly. Fixes #1937 and #2110 by @claudepi in #2295
• Enhance EmbeddingExtractor to support both async and sync methods for… by @telesoho in #2286
• Tokens counting by @anistark in #2299
• Fix/tool call accuracy by @anistark in #2300
• fix: coroutine warning for bleu by @anistark in #2301
• Add base_url parameter to embedding_factor by @anistark in #2303
• fix: add disallowed_special on tiktoken encode by @anistark in #2304
• Feat/tool call f1 1893 by @anistark in #2305
• Feature/azure token usage extraction by @anistark in #2306
• fix: improve metric decorators with better validation and error handling by @jjmachan in #2302
• Metric/parallel tool call by @anistark in #2307
• Fix: avoid ambiguous truth value for empty numpy array in HuggingfaceEmbeddings (fixes #2080) by @Rahul2512Chauhan in #2308
• Devpod cn/main by @anistark in #2309
• Feat/quoted spans metric by @anistark in #2311
• Fix noise sensitivity compute by @anistark in #2312
• Corrected numerous typos in Markdown files. by @ker2xu in #1994
• Deprecation warnings for LLMs and Prompts by @rhlbhatnagar in #2253
• Docs/eval_rag_agent - how to evaluate and improve rag app by @sanjeed5 in #2293
• Add llamaindex agentic evals gemini by @anistark in #2317
• fix: type str in tests by @anistark in #2318
• Fix generate_multiple caching issue (#1980) by @Rahul2512Chauhan in #2314
• fix: metric inheritance patterns: separate factory-created metrics from class-instantiated metrics by @jjmachan in #2316
• fix: concurrent ResponseRelevancy by @anistark in #2328
• fix: answer_relevancy scoring logic to prevent false zero by @anistark in #2327
• feat: Add OCI Gen AI Integration for Direct LLM Support by @harshil-sanghvi in #2321
• feat: Add save/load functionality and improved repr for LLM-based metrics by @jjmachan in #2320
• Fix: Fixed the Numpy 3.13 issue by @Prigoistic in #2282
• refactor: docs and warnings for metric base new structure by @anistark in #2333
• fix: typing by @anistark in #2334

New Contributors

• @kauabh made their first contribution in #2221
• @s3pi made their first contribution in #2292
• @claudepi made their first contribution in #2295
• @telesoho made their first contribution in #2286
• @ker2xu made their first contribution in #1994
• @harshil-sanghvi made their first contribution in #2321
• @Prigoistic made their first contribution in #2282

Full Changelog: v0.3.5...v0.3.6

v0.3.5

What's Changed

• Docs/howto-texttosqlagent by @sanjeed5 in #2264
• fix: preview logo was too small. by @anistark in #2277
• modified the documentation to be in sync with current output format by @kotalaraghava in #2281
• removed some meta properties to test by @jjmachan in #2278
• feature: improve async / executor functionality by @ahgraber in #2070
• modification of the translate instruction by @anistark in #2284
• Remove experimental from docs and fix examples in docs by @sanjeed5 in #2270
• fix: resolve TypeError in TopicAdherenceScore bitwise operations by @anistark in #2258
• Knowledge graph/optimize for large corpus by @anistark in #2267
• Update _nv_metrics.py by @titericz in #2053
• Add telemetry by @rhlbhatnagar in #2260
• OpenAI model cost by @anistark in #2287
• docs: agent metrics code examples improvement by @yesidc in #1983
• Prompt Optimization Tutorial by @sahusiddharth in #1993
• Feature/metric type checking by @anistark in #2288
• improved the release script for ragas-examples by @jjmachan in #2289
• fix: removed the need for regex patterns by @jjmachan in #2290

New Contributors

• @kotalaraghava made their first contribution in #2281
• @yesidc made their first contribution in #1983

Full Changelog: v0.3.4...v0.3.5

v0.3.5rc2

fix: removed the need for regex patterns (#2290)

v0.3.5rc1

What's Changed

• Docs/howto-texttosqlagent by @sanjeed5 in #2264
• fix: preview logo was too small. by @anistark in #2277
• modified the documentation to be in sync with current output format by @kotalaraghava in #2281
• removed some meta properties to test by @jjmachan in #2278
• feature: improve async / executor functionality by @ahgraber in #2070
• modification of the translate instruction by @anistark in #2284
• Remove experimental from docs and fix examples in docs by @sanjeed5 in #2270
• fix: resolve TypeError in TopicAdherenceScore bitwise operations by @anistark in #2258
• Knowledge graph/optimize for large corpus by @anistark in #2267
• Update _nv_metrics.py by @titericz in #2053
• Add telemetry by @rhlbhatnagar in #2260
• OpenAI model cost by @anistark in #2287
• docs: agent metrics code examples improvement by @yesidc in #1983
• Prompt Optimization Tutorial by @sahusiddharth in #1993
• Feature/metric type checking by @anistark in #2288
• improved the release script for ragas-examples by @jjmachan in #2289

New Contributors

• @kotalaraghava made their first contribution in #2281
• @yesidc made their first contribution in #1983

Full Changelog: v0.3.4...v0.3.5rc1

v0.3.4

What's Changed

• Update context_precision.md by @anupamck in #2262
• Reduce find_indirect_clusters() runtime through neighborhood detection and sampling by @ahgraber in #2144
• fix: handle langchain multiple batching by @anistark in #2257
• chore: fix the dead space in the header for docsite by @jjmachan in #2265
• total bin coverage for default_transform() in Knowledge Graph transformations by @tolgaerdonmez in #1950
• Improved context precision documentation by @anupamck in #2266
• Fix docs: correct inheritance class for Non-LLM metrics by @AlanPonnachan in #2272
• added rb2b analytics by @jjmachan in #2273

New Contributors

• @anupamck made their first contribution in #2262
• @tolgaerdonmez made their first contribution in #1950
• @AlanPonnachan made their first contribution in #2272

Full Changelog: v0.3.3...v0.3.4

v0.3.3

What's Changed

• feat: refactor Ragas embeddings to maintain backward compatibility by @NirantK in #2184
• feat: move tracing from experimental to ragas/integrations by @NirantK in #2188
• feat: Refactor LLM structure and add InstructorLLM by @NirantK in #2191
• fix: setup and local ci by @anistark in #2199
• docs: correct small typos in Testset Generation page and related md components by @deibid in #2123
• Docs/how-to-benchmark-new-llm-guide by @sanjeed5 in #2158
• fix: access model fields via class by @emmanuel-ferdman in #2146
• add: PR template for better review and overview of changes by @anistark in #2202
• Move prompts to ragas by @NirantK in #2197
• Fix: make the answer be the same as the split sentences by @wlbksy in #2128
• Merge workspace.toml into pyproject.toml by @NirantK in #2189
• Cleanup/move-benchmark-llm-to-examples by @sanjeed5 in #2203
• fix: shields badge for downloads replace with pepy. by @anistark in #2204
• refactor: removing some section to streamline pr template by @anistark in #2206
• Move dataset module to main package and update imports by @NirantK in #2205
• fix: type check for pyright by @anistark in #2211
• refactor: move ragas out to project root. by @anistark in #2209
• chore: fix the OIDC error in claude code by @jjmachan in #2212
• fix: _version.py is to be ignored in git by @anistark in #2213
• chore: only review for /claude-review command by @jjmachan in #2214
• Fix a couple of typos in docs/getstarted/evals.md by @sadra-barikbin in #2081
• Fix repeating template bug in _nv_metrics.py by @dev-nid in #2216
• fix: ragas dir structure path fixes by @anistark in #2215
• fix: minor typo in context_precision.md by @AbhishekRP2002 in #2050
• fix: update docstrings for single-hop use case by @terry07 in #1947
• Fix issue with cache hits by @Nicba1010 in #1977
• Use generate_multiple instead of generate in a loop in ResponseRelevancy by @aabanic in #1978
• fix: update required columns in RefusalRate (docs) by @DmitrySirakov in #2008
• Add support for make install-minimal to speed up CI + dev by @rhlbhatnagar in #2218
• Move experimental metrics to main package by @NirantK in #2220
• Fix e2e test failures with datasets>=4.0.0 and missing dependencies by @anistark in #2222
• Move Utils / Exceptions module to main package and update imports by @rhlbhatnagar in #2219
• update compare_embeddings doc links by @resitaydin in #2037
• chore: add py.typed marker file to support type checking by @sisp in #2029
• Retire ragas.experimental namespace by @NirantK in #2223
• chore: cleanup docs for ragas directory by @anistark in #2224
• fix: update make commands to use uv for docs by @anistark in #2217
• Remove experimental folder by @NirantK in #2225
• Docs/how-to-iterate-prompt by @sanjeed5 in #2226
• security: stop supporting v0.2 security releases by @anistark in #2230
• Optimize factual correctness metric runtime by 50% by @vignesh14052002 in #2153
• Fixed the mermaid which was breaking on navigation by @ganeshrvel in #2121
• Fix typos and clean style by @kwinkunks in #2042
• Feature/examples-as-installable-package by @sanjeed5 in #2210
• chore: added reo.js analytics by @jjmachan in #2232
• Move tutorials by @rhlbhatnagar in #2235
• Move howtos by @rhlbhatnagar in #2236
• Move references to docs/ by @rhlbhatnagar in #2234
• Fix BadRequestError while using OpenAI O1 series models | Add option to bypass temperature by @vignesh14052002 in #2151
• feature!: knowledge graph flexibility by @ahgraber in #2030
• return False when pred_sequence length and ref_requence length is dif… by @licux in #1894
• Feature: improve relationship builders for better async and reduced memory utilization by @ahgraber in #2077
• Mirgate all leftover docks from experimental by @rhlbhatnagar in #2243
• Add deprecation warnings for embeddings by @rhlbhatnagar in #2244
• set temperature to 0.01 for better llm compatibility by @0Falli0 in #939
• docs: fix broken link in FAQ (metrics tracing) by @Rahul2512Chauhan in #2238
• fix: temperature change to 0.01 bugs by @anistark in #2247
• Scrubbed the references to app.ragas.io by @ganeshrvel in #2207
• feat: cancellable task by @anistark in #2242
• update: add og images and metadata to docs by @anistark in #2248
• fix: add support for both langchain llm and ragas llm by @anistark in #2229
• chore: analytics support for commonroom and scarf by @jjmachan in #2250

New Contributors

• @deibid made their first contribution in #2123
• @wlbksy made their first contribution in #2128
• @sadra-barikbin made their first contribution in #2081
• @dev-nid made their first contribution in #2216
• @AbhishekRP2002 made their first contribution in #2050
• @terry07 made their first contribution in #1947
• @Nicba1010 made their first contribution in #1977
• @aabanic made their first contribution in #1978
• @rhlbhatnagar made their first contribution in #2218
• @resitaydin made their first contribution in #2037
• @sisp made their first contribution in #2029
• @vignesh14052002 made their first contribution in #2153
• @kwinkunks made their first contribution in #2042
• @0Falli0 made their first contribution in #939
• @Rahul2512Chauhan made their first contribution in #2238

Full Changelog: v0.3.2...v0.3.3