[Atlas] Hybrid lexical+semantic search with quest-relevance + recency reranker

Goal

/api/search (api.py:16833) does FTS5/PostgreSQL ts_rank only. scidex/core/vector_search.py (543 LoC) exists but isn't fused into
the main endpoint. Result: searching for "synaptic pruning microglia"
ranks a 2014 wiki page above a fresh, high-quest-priority hypothesis
because BM25 doesn't know what the user cares about. Build a hybrid
rerank: take top-50 lexical + top-50 vector results, normalise scores,
then rerank with a learned blend of (lexical_score, vector_score, quest_relevance, recency, epistemic_tier).

Acceptance Criteria

☐ scidex/atlas/search_rerank.py::hybrid(q, types, limit) -> list[Result].

☐ Lexical pull: existing _fts_search path; vector pull: existing VectorSearchEngine.search.

☐ Reciprocal-rank-fusion (RRF) baseline → produce top 100 candidates → rerank by 0.4RRF + 0.2quest_priority_norm + 0.2recency_decay + 0.1tier_bonus + 0.1*author_calibration.

☐ quest_priority_norm: max-quest-priority of any quest the artifact is linked to; recency_decay = exp(-age_days/180); tier_bonus: T0=1.0, T1=0.7, T5=0.0; author_calibration: from agent_calibration.brier.

☐ /api/search?rerank=hybrid activates the new path; default stays lexical for one release; metrics on search_queries track CTR per mode.

☐ CTR reported on /api/search/analytics with rerank_mode breakdown.

☐ Test: dual-fixture query where the right answer is a high-quest-priority recent hypothesis; lexical alone returns it at rank 8; hybrid returns it at rank 1.

Approach

Reuse _fts_search builder closures from api.py — extract them into a callable list so search_rerank can iterate without copy/paste.

Score normalisation: min-max within each list before fusion, then RRF.

Cache rerank weights as a row in search_rerank_weights(version, weight_json, set_at) so future LTR (learning-to-rank) just inserts a new row — no code change needed.

Latency budget: 250 ms p95; fall back to lexical if vector engine times out at 100 ms.

Dependencies

q-er-calibration-tracker (agent_calibration.brier).
epistemic_tiers (existing).

Work Log

2026-04-27 11:15 PT — Slot 0 (minimax:78)

Implemented hybrid lexical+semantic search with quest-relevance + recency reranker
Created scidex/atlas/search_rerank.py with:

- hybrid(q, types, limit) -> list[Result] function
- RRF fusion of top-50 lexical + top-50 vector results
- Reranking by: 0.4RRF + 0.2quest_priority_norm + 0.2recency_decay + 0.1tier_bonus + 0.1*author_calibration
- log_search_query() for analytics tracking

Modified api.py:

- Added rerank parameter to /api/search endpoint
- Added rerank_mode column to search_queries table
- Added search_rerank_weights table for LTR weight caching
- Updated /api/search/analytics to include rerank_mode_breakdown

Notes:

- quest_priority uses hypothesis_missions + missions.index_score (no quests table exists)
- author_calibration falls back to 0.5 when agent_calibration table unavailable
- Some FTS tables/columns don't exist (papers_fts, papers.published_date) - fallback handles gracefully

Tested: hybrid search returns 5 results with proper scoring (final_score, rrf_score, recency_decay, tier_bonus)
Committed and pushed

Tasks using this spec (1)

[Atlas] Hybrid lexical+semantic search with quest-relevance

Search done P91

File: q-srch-hybrid-rerank_spec.md

Modified: 2026-05-01 20:13

Size: 3.4 KB