Effort: thorough
scidex/forge/forge_tools.py:67 log_tool_call records duration_ms and
success per invocation, and Orchestra's cost_ledger tracks token spend,
but no view ties the two together to answer "for $1 of compute spent on
SciDEX scientific work, which of the 58+ tools in tools.py produced the
highest-impact output?". We're flying blind on tool ROI: agents over-use
the cheap-but-shallow pubmed_search and under-use expensive but
high-information gtex_tissue_expression not because of evidence but
because of habit. Build a tool ROI dashboard that joins
tool_invocations × cost_ledger × downstream impact (citations of the
result, hypothesis price moves it triggered, debate outcomes it shifted)
and ranks tools by (impact / dollar).
scidex/forge/tool_roi.py::compute_tool_roi(tool_id, window_days=30) -> dict returning {invocations, total_cost_usd, impact_score, roi_per_dollar, percentile_rank}.artifact_links link_type='derived_from' count), 0.4 downstream_hypothesis_price_delta (sum of |delta| from price_history within 24h of invocation), 0.3 * debate_outcome_shift (1.0 if a debate cited the invocation and outcome flipped).cost_ledger model_cost row for the invoking agent's model) + per-tool API cost (lookup tool_api_costs(tool_name PK, cost_per_call_usd) table; default 0).tool_roi_daily(day, tool_id, invocations, total_cost_usd, impact_score, roi_per_dollar, PRIMARY KEY (day, tool_id)).scidex/senate/scheduled_tasks.py populates tool_roi_daily./forge/tools/roi page renders a sortable table + bubble chart (x=cost, y=impact, size=invocations).GET /api/forge/tools/roi?days=30 returns the leaderboard JSON.roi_per_dollar < 10th_percentile for a tool over 4 consecutive weeks, auto-emit a tool_review Senate proposal asking whether to deprecate or boost.scidex/forge/forge_tools.py:22 init_tool_invocations_table and :67 log_tool_call for the source schema.artifact_links WHERE source = invocation_id; downstream_hypothesis_price_delta = JOIN tool_invocations.outputs JSON for hypothesis_id, then market_dynamics.get_price_history.orchestra.cost.compute_cost_for_window(model, tokens) rather than re-implementing pricing.market_dynamics.generate_market_overview_svg (line 1428) — no JS chart library.tool_roi_archive).scidex/forge/forge_tools.py:22,67 — invocations source.scidex/exchange/market_dynamics.py:747 get_price_history — price impact.cost_ledger (orchestra/cost.py:41).q-tools-deprecated-detector (consumes ROI signals).q-tools-skill-marketplace (uses ROI as a price signal for scarce tools).init_tool_roi_tables() — creates tool_api_costs, tool_roi_daily, tool_roi_archive + indexescompute_cost(skill_id, duration_ms, invocations) — estimates cost from duration_ms + API fees_citation_impact(invocation_ids) — 0.3 × COUNT(artifact_links WHERE link_type='derived_from')_hypothesis_price_impact(invocation_ids) — 0.4 × sum of |price deltas| within 24h_debate_outcome_shift(invocation_ids) — 0.3 if debate cited invocation and outcome flippedcompute_tool_roi(tool_id, window_days=30) — main ROI functioncompute_all_tool_roi(window_days=30) — batch compute all tools with O(1) percentile ranks (no recursive calls)_compute_tool_roi_raw() — raw ROI without percentile (called by compute_all_tool_roi)populate_daily_roi(target_date=None) — upserts ROI into tool_roi_daily + archives old rowsdetect_chronic_low_roi_tools() — finds tools below 10th pctile for 4+ consecutive weeksemit_tool_review_proposals() — inserts Senate tool_review proposals for chronic low-ROI toolsToolRoiDailyTask scheduled task ("tool-roi-daily", daily, 1440 min)GET /api/forge/tools/roi — JSON leaderboardGET /forge/tools/roi — HTML page with sortable table + SVG bubble chart— in f-string, already in origin/main before this branch). Not caused by this work.3e0f452e8 on branch orchestra/task/8e3829d0-tool-call-cost-benchmark-which-tool-maxiFixed two _PgRow tuple-unpacking bugs that caused crashes in production:
detect_chronic_low_roi_tools: for (tool_id, day, roi) in rows: → for row in rows: tool_id, day, roi = row[0], row[1], row[2]_PgRow doesn't support tuple unpacking despite behaving like a sequence.
compute_tool_roi: O(n²) percentile computation replaced by O(n) batch approach in compute_all_tool_roi; also fixed same tuple-unpacking bug.8c3e9f12d — [Forge] Fix _PgRow tuple-unpacking bugs in tool_roi [task:8e3829d0-...]