Goal
Improve notebook coverage checks so demo-critical analyses are not falsely counted as covered when notebook references exist but files are missing. Add deterministic, CI-friendly diagnostics that separate true missing coverage from stale notebook pointers. Keep existing stub generation behavior intact while making failures actionable.
Acceptance Criteria
☑ ci_notebook_coverage.py reports stale/missing notebook artifact references distinctly from coverage counts.
☑ Coverage output includes strict file-backed coverage metrics in addition to reference-only metrics.
☑ Script still runs successfully with and without --register.
☑ Work log records start and completion with concrete verification commands/results.
Approach
Inspect current coverage SQL and identify false-positive coverage conditions.
Add file-existence-aware diagnostics for notebook paths and notebook table links.
Print deterministic summary blocks suitable for recurring CI logs.
Run bounded verification and record results.Dependencies
Dependents
[Artifacts] CI: Verify notebook coverage and generate summaries for recent analyses (73ee83a5-c702-45de-8f3d-b41171c061a9)
Work Log
2026-04-04 09:07 PDT — Slot 4
- Started one-shot task
fc1bf0a8-a072-4a6f-a5bf-aa2e17b5450e.
- Read
AGENTS.md, /home/ubuntu/Orchestra/AGENTS.md, and QUESTS.md.
- Verified baseline system state with
scidex status.
- Began implementation on
ci_notebook_coverage.py to add strict diagnostics for stale notebook artifact references.
2026-04-04 09:09 PDT — Slot 4
- Implemented coverage hardening in
ci_notebook_coverage.py:
- Added strict file-existence checks for
analyses.notebook_path and
notebooks.(file_path, rendered_html_path).
- Split metrics into reference coverage vs strict file-backed coverage.
- Added deterministic diagnostic counters and sample analysis IDs for missing refs, stale paths, and stale notebook links.
-
timeout 300 python3 -c "import py_compile; py_compile.compile('ci_notebook_coverage.py', doraise=True)" (pass)
-
timeout 300 python3 ci_notebook_coverage.py --limit 0 (pass)
-
timeout 300 python3 ci_notebook_coverage.py --limit 0 --register (pass)
-
timeout 300 curl -s -o /dev/null -w '%{http_code}' http://localhost:8000/api/status =>
200 -
timeout 300 curl -s -o /dev/null -w '%{http_code}' http://localhost:8000/notebooks =>
200 -
scidex status checked after change
- Result: notebook coverage CI now exposes false-positive coverage and stale artifact pointers for demo triage.