[Agora] Periodic Falsifier-of-Truth on top-Elo hypotheses

Goal

Even hypotheses that survived an adversarial debate
(q-rt-adversarial-debate-runner) at time T may have been falsified
by post-T literature. PubMed grows by ~3500 articles/day; a hypothesis
proposed in March can have its mechanism contradicted in April without
SciDEX noticing. This task ships a recurring Falsifier-of-Truth
runner that re-probes the top-Elo hypotheses against new literature
since their last falsification check, and downgrades any whose
support has decayed. It is the temporal complement of the adversarial
debate runner: that one tests reasoning quality, this one tests
literature freshness.

Effort: deep

Acceptance Criteria

- select_targets(top_n=50, min_elo=1600,
min_days_since_last_check=14) -> list[hypothesis_id].
- run(hypothesis_id) -> FalsifierReport orchestrates:
1. Pull new PubMed citations published since
last_falsifier_check_at (default: hypothesis
created_at) using the existing paper-corpus-search
skill or scidex/agora/pubmed_utils.py.
2. Score each new paper for relevance via the existing
Skeptic+Falsifier persona path (already wired in
scidex/senate/falsifier-related code; grep
Falsifier in scidex/agents/).
3. Aggregate: count contradicts, weakens,
supports, unrelated from the trailing window.
4. Compute a falsification_score = (contradicts + 0.5*weakens) /
total_relevant.
5. If falsification_score > 0.30 AND ≥3 contradicts: flag
hypothesis as lifecycle='under_review', dock
composite_score by 15%, post a structured comment to the
hypothesis with the new contradicting PMIDs.

ALTER TABLE hypotheses ADD COLUMN IF NOT EXISTS
        last_falsifier_check_at TIMESTAMPTZ;
      ALTER TABLE hypotheses ADD COLUMN IF NOT EXISTS
        falsification_score DOUBLE PRECISION;
      CREATE TABLE falsifier_of_truth_run (
        id BIGSERIAL PRIMARY KEY,
        hypothesis_id TEXT NOT NULL,
        ran_at        TIMESTAMPTZ NOT NULL DEFAULT NOW(),
        new_pmids     TEXT[] NOT NULL,
        contradicts   INT NOT NULL DEFAULT 0,
        weakens       INT NOT NULL DEFAULT 0,
        supports      INT NOT NULL DEFAULT 0,
        unrelated     INT NOT NULL DEFAULT 0,
        falsification_score DOUBLE PRECISION,
        verdict       TEXT NOT NULL CHECK (verdict IN
                      ('survives','weakened','falsified')),
        comment_id    TEXT
      );
      CREATE INDEX idx_fot_h ON falsifier_of_truth_run(hypothesis_id);

(LLM cost cap), prioritised by
Elo desc, last_falsifier_check_at asc nulls first. Lands as
a recurring Orchestra task in the Agora queue. reuse scidex/agora/crosslink_emitter.py style: a structured
comment of comment_type_labels=['refutation'] with the new
PMIDs as cited evidence so it can re-trigger
q-perc-refutation-debate-emitter (closing the loop). twice across runs; tracked via a small
falsifier_seen_pmid(hypothesis_id, pmid) row written each
run. lands, falsifier-of-truth respects the honeypot quarantine
(does not surface honeypot PMIDs as real falsification
evidence). counts of survives/weakened/falsified and a
"Most-falsified hypotheses" list. verdict bucketing arithmetic, dedup across runs, honeypot
exclusion, comment-write integration.

Approach

Stand up the new-PMID-since-T helper around the existing

pubmed-utils path; smoke against a 30d-old hypothesis.

Migration; module; tests.

Recurring registration; first manual run on top 10.

Verify the refutation-emitter loop actually re-triggers from

falsifier comments (or confirm it falls in the existing
comment_type_labels=['refutation'] path which already wires
spawned_debate_id).

Tile + smoke.

Dependencies

• scidex/agora/pubmed_utils.py — PubMed query helper.
• q-rt-citation-honeypot — honeypot quarantine respected.
• q-perc-refutation-debate-emitter — receives the falsifier

comments and may spawn a fresh debate.

Dependents

• q-trust-provenance-integrity-scanner — uses falsification deltas

as one signal of trust drift.

Work Log

2026-04-27 — Implementation complete [task:7c31cf87-3a13-457e-8ddd-241ff75cfe91]

Delivered:

• migrations/20260428_falsifier_of_truth.sql — Adds lifecycle, last_falsifier_check_at, falsification_score to hypotheses; creates falsifier_of_truth_run and falsifier_seen_pmid tables with indexes. Migration applied to the live DB.

• scidex/agora/falsifier_of_truth.py — Full implementation:

- select_targets(top_n=50, min_elo=1600, min_days_since_last_check=14) — queries elo_ratings.leaderboard, filters deprecated/no-score/recently-checked hypotheses.
- run(hypothesis_id) -> FalsifierReport — end-to-end pipeline: PubMed search with date window (search_pubmed with mindate), honeypot guard, seen-PMID dedup, LLM scoring per paper (contradicts/weakens/supports/unrelated), falsification_score = (contradicts + 0.5*weakens) / total_relevant, verdict decision (survives/weakened/falsified). Falsified → lifecycle='under_review', composite_score docked 15%, Elo penalty via elo_ratings.record_match, refutation comment posted to artifact_comments with comment_type_labels=['refutation'].
- run_nightly(max_hypotheses=10) — nightly batch driver with error isolation per hypothesis.
- get_dashboard_stats(days=30) — Senate tile aggregation (survives/weakened/falsified counts + most-falsified list).
- Honeypot guard: _is_honeypot_pmid queries citation_honeypot table (no-op until that migration lands).

• tests/agora/test_falsifier_of_truth.py — 28 passing tests covering: score arithmetic edge cases, verdict threshold logic, search query building, select_targets filtering, run() happy/falsified/weakened/error paths, nightly batch cap, dedup exclusion, honeypot exclusion.

Design notes:

• lifecycle column added to hypotheses (also needed by adversarial_debate.py which already references it).
• Comment artifact_id format: hypothesis-{hypothesis_id} (matches existing comments in artifact_comments).
• Recurring Orchestra task registration deferred — the nightly run_nightly() is callable directly; Orchestra cron setup is a separate operational step.

Tasks using this spec (1)

[Agora] Periodic Falsifier-of-Truth on top-Elo hypotheses

File: q-rt-falsifier-of-truth-cron_spec.md

Modified: 2026-05-01 20:13

Size: 7.0 KB