A modular RAG evaluation harness. Every part of the pipeline (chunker, embedder, retriever, reranker, LLM) is swappable from a config file, so you can change one piece and measure what it does to retrieval and answer quality.
Components are Strategies behind small interfaces and are built from plain config dicts by a Factory.
20 documents, 20 labeled queries (with near-miss distractors), top_k = 3. Real runs, not illustrative numbers.
semantic_hybrid beat baseline on every quality metric here. The cost is a one-time ~36s indexing step to embed the corpus. Generation and judging used a small local model (Qwen2.5-0.5B-Instruct), so faithfulness numbers reflect that model's limits.
| Metric | Baseline | Semantic hybrid |
|---|---|---|
| Recall@3 | 1.000 | 1.000 |
| Precision@3 | 0.483 | 0.750 |
| MRR | 0.975 | 1.000 |
| nDCG@3 | 0.974 | 0.980 |
Implemented from scratch, with no external metrics library. Ground truth is tracked at document level.
pip install -r requirements.txt python run_experiment.py --config experiments/configs/baseline.yaml # interactive pipeline builder uvicorn webapp.backend.main:app --reload
The interactive UI needs a Python backend, so this page is a static showcase and not the live app.