SciDEX — Task: [Senate] Add convergence monitoring dashboard sect

Add convergence monitoring to /senate: (1) Detect duplicate/near-duplicate hypotheses by title similarity, (2) Track score drift over time, (3) Show global progress metrics (new hypotheses/day, KG growth rate, debate throughput). Surface as a new dashboard section on the senate page. ## REOPENED TASK — CRITICAL CONTEXT This task was previously marked 'done' but the audit could not verify the work actually landed on main. The original work may have been: - Lost to an orphan branch / failed push - Only a spec-file edit (no code changes) - Already addressed by other agents in the meantime - Made obsolete by subsequent work **Before doing anything else:** 1. **Re-evaluate the task in light of CURRENT main state.** Read the spec and the relevant files on origin/main NOW. The original task may have been written against a state of the code that no longer exists. 2. **Verify the task still advances SciDEX's aims.** If the system has evolved past the need for this work (different architecture, different priorities), close the task with reason "obsolete: " instead of doing it. 3. **Check if it's already done.** Run `git log --grep=''` and read the related commits. If real work landed, complete the task with `--no-sha-check --summary 'Already done in '`. 4. **Make sure your changes don't regress recent functionality.** Many agents have been working on this codebase. Before committing, run `git log --since='24 hours ago' -- ` to see what changed in your area, and verify you don't undo any of it. 5. **Stay scoped.** Only do what this specific task asks for. Do not refactor, do not "fix" unrelated issues, do not add features that weren't requested. Scope creep at this point is regression risk. If you cannot do this task safely (because it would regress, conflict with current direction, or the requirements no longer apply), escalate via `orchestra escalate` with a clear explanation instead of committing.

Completion Notes

Auto-completed by supervisor after successful deploy to main

Git Commits (3)

Squash merge: orchestra/task/ee6448a7-add-convergence-monitoring-dashboard-sec (1 commits)2026-04-19

[Senate] Add convergence monitoring to /senate — duplicate detection, score distribution, throughput metrics [task:ee6448a7-03a6-43a1-b2e2-6044955ab7be]2026-04-02

Spec File

[Senate] Add convergence monitoring dashboard section

Goal

Add a convergence monitoring section to the /senate page that tracks hypothesis duplication, score distribution, and system throughput metrics. Also enhance the /api/convergence endpoint to return live-computed data instead of reading from a potentially stale JSON file.

Acceptance Criteria

☑ Convergence section visible on /senate page

☑ Duplicate hypothesis detection using Jaccard title similarity (>=60% threshold)

☑ Score distribution bar chart showing hypothesis quality spread

☑ 24h throughput metrics with delta comparison to previous period

☑ Overall convergence health status (HEALTHY/ATTENTION/DIVERGING)

☑ /api/convergence returns live computed data (duplicates, scores, throughput)

☑ Python syntax validates clean

Approach

Add convergence section builder in senate_page() after quality trend section

Compute Jaccard word-set similarity for all hypothesis title pairs

Build score distribution bins and throughput delta metrics

Replace file-based /api/convergence with live DB queries

Add "Convergence" bullet to Senate governance role description

Work Log

2026-04-02 23:50 PT — Slot 23

Reviewed existing quality gates (9 checks already live)
Added convergence monitoring section to senate_page() with 3 sub-sections:

- Duplicate detection (Jaccard similarity on title words)
- Score distribution (5-bin bar chart with mean/median)
- 24h throughput with period-over-period deltas

Enhanced /api/convergence to compute live data instead of reading stale JSON
Verified syntax clean
Committed and pushed

Verification — 2026-04-19 11:10 UTC

Checked origin/main (commit a179deebd) for implementation evidence:

/api/convergence endpoint — git show origin/main:api.py | grep -n "def api_convergence" → line 14649

- Returns: status, score_distribution, avg_score, median_score, duplicate_pairs, duplicate_count, throughput
- Uses live DB queries, no JSON file read

_get_duplicate_hypothesis_pairs — line 14535

- Jaccard word-set similarity with 0.6 threshold

senate_page convergence_section — line 41706+

- Duplicate detection, score distribution bar chart, 24h throughput with deltas

Python syntax — python3 -m py_compile api.py → Syntax OK

All 7 acceptance criteria satisfied on origin/main.

Payload JSON

{
  "_reset_note": "This task was reset after a database incident on 2026-04-17.\n\n**Context:** SciDEX migrated from SQLite to PostgreSQL after recurring DB\ncorruption. Some work done during Apr 16-17 may have been lost.\n\n**Before starting work:**\n1. Check if the task's goal is ALREADY satisfied (run the relevant checks)\n2. Check `git log --all --grep=task:YOUR_TASK_ID` for prior commits\n3. If complete, verify and mark done. If partial, continue. If not done, proceed.\n\n**DB change:** SciDEX now uses PostgreSQL. `get_db()` auto-detects via\nSCIDEX_DB_BACKEND=postgres env var.",
  "_reset_at": "2026-04-18T06:29:22.046013+00:00",
  "_reset_from_status": "done"
}