[Forge] Immune repertoire pipeline — TCR/BCR FASTQ → MiXCR clones → epitope-link artifact

Effort: thorough

Goal

Build an immune-receptor repertoire pipeline for the immunology
vertical: ingest TCR/BCR sequencing FASTQ (or a precomputed AIRR
TSV), call clonotypes with MiXCR, compute repertoire diversity
(Shannon, Gini, Hill), match clonotypes against IEDB epitopes via the
new iedb_epitopes tool, and emit a clonotype-to-epitope linkage
artifact a debate can cite. Closes the immunology-vertical's biggest
data gap: SciDEX has no way to argue from actual repertoire data today.

Why this matters

Immunology hypotheses ("expanded autoreactive TCRs drive RA flares",
"hospital-acquired SARS-CoV-2 strains evade convalescent BCR
responses") need real repertoire evidence to be debate-grade. Without
a pipeline, the immunology Theorist has nothing to ground claims on
beyond text reviews. This pipeline absorbs publicly available AIRR
datasets (10X Genomics, ImmuneSpace, AIRR-DB) and turns them into
SciDEX artifacts the persona pack can argue from.

Acceptance Criteria

- ingest(source) — accepts FASTQ paths, an AIRR-format TSV,
or a 10X cellranger-vdj output dir.
- call_clonotypes_mixcr(fastqs) — invokes MiXCR via subprocess;
returns AIRR-format clones table.
- diversity_metrics(clones) — computes Shannon entropy, Gini,
Hill numbers (q=1, q=2), Chao1 estimator.
- link_to_epitopes(clones) — calls
tools.iedb_epitopes per clonotype CDR3; returns matches
with sequence-similarity score (Levenshtein ≤ 2 = match,
≤ 4 = candidate).
- pipeline(source, chain='TRB') — composes; commits artifact
under data/scidex-artifacts/immune_repertoire/<run_id>/
with the clones table, diversity JSON, and linkage CSV. source_spec_json, chain TEXT CHECK IN ('TRA','TRB','IGH','IGK','IGL'),
n_clonotypes, shannon, gini, n_epitope_matches, mixcr_version,
pipeline_version, started_at, finished_at, artifact_id). chain) with @log_tool_call. (Pareto), a diversity-metric panel, and a clonotype-to-epitope
table linking out to the matched IEDB record. (q-vert-vertical-personas-pack) consumes a
repertoire_block when a debate's hypothesis names a disease
with a recent run. end-to-end in <20 min, produces ≥1 epitope match, artifact
registered. → diversity metrics in expected ranges, mock IEDB linkage
returns expected matches.

Approach

MiXCR has a free academic license; ship install instructions in

docs/setup/mixcr.md. Subprocess wrapper handles installed +
missing cases gracefully.

Diversity formulas implemented once in pure NumPy.

Levenshtein matching uses python-Levenshtein (lightweight).

Cache IEDB lookups by CDR3 hash to avoid repeat calls.

Persona injection mirrors the prior pattern.

Dependencies

• MiXCR (subprocess); iedb_epitopes from

q-vert-vertical-evidence-providers.

• q-vert-vertical-personas-pack — immunology-expert consumer.
• data/scidex-artifacts/ submodule.

Work Log

2026-04-27 — Implementation

• Created scidex/forge/immune_repertoire.py (~480 LoC) with:

- ingest(source) — accepts FASTQ paths, AIRR TSV, or 10X cellranger dir
- call_clonotypes_mixcr(fastqs, chain) — subprocess MiXCR wrapper; AIRR output parsed to list of dicts
- load_airr_tsv(path) — parses AIRR-format TSV clonotypes file
- diversity_metrics(clones) — Shannon entropy, Gini coefficient, Hill numbers (q=1, q=2), Chao1 estimator in pure NumPy
- link_to_epitopes(clones, chain) — Levenshtein matching (≤2=match, ≤4=candidate); short-circuits on empty clones to avoid circular import
- pipeline(source, chain) — full pipeline: ingest → MiXCR → diversity → epitope linkage → artifact commit → DB persist; writes clones.tsv, diversity.json, epitope_linkage.csv, metadata.json under data/scidex-artifacts/immune_repertoire/<run_id>/
- _levenshtein_distance_py() — pure-Python fallback (no C extension required)
- _persist_run() — writes repertoire_run row with ON CONFLICT upsert

• Created migrations/021_add_repertoire_run_table.py — repertoire_run table with PK, chain CHECK, indexes
• Added immune_repertoire_pipeline() to scidex/forge/tools.py with @log_tool_call decorator and TOOL_NAME_MAPPING entry
• Created tests/test_immune_repertoire.py — 23 passing tests covering ingest, diversity metrics (6 cases), AIRR TSV loading, Levenshtein matching logic (5 cases), fallback Levenshtein (6 cases)
• All 23 tests pass

Note on circular import: iedb_epitopes is imported locally inside link_to_epitopes() and after the empty-clones short-circuit to avoid triggering forge_tools.py's instrumentation block at import time. This is a pre-existing latent issue in tools.py (the pathway_flux_pipeline instrumentation references a function defined after it); the short-circuit fix is the right solution.

Tasks using this spec (1)

[Forge] Immune repertoire pipeline - TCR/BCR FASTQ to MiXCR

File: q-tool-immune-repertoire-pipeline_spec.md

Modified: 2026-05-01 20:13

Size: 5.4 KB