[Forge] Rate-limit-aware tool calls inside the sandbox

Goal

Centralize per-provider rate-limit accounting (NCBI E-utilities 3 req/s,
GTEx 5 req/s, Crossref polite-pool 50 req/s, Semantic Scholar 100 req/5min,
OpenAlex 100k req/day) in a sandbox-aware token-bucket layer so concurrent
analyses don't collectively trip 429s and tank the whole fleet.

Why this matters

Right now each adapter in scidex/forge/tools.py retries on 429 in
isolation. With 10 sandboxed analyses calling Semantic Scholar at once, the
provider sees 1000 req/min and bans the whole IP, then every analysis fails.
A shared bucket implemented as a Postgres advisory-lock + leaky-bucket
counter prevents global ban while letting individual analyses queue
gracefully.

Acceptance Criteria

acquire(provider, weight=1, timeout_s=30) blocking until a token
is available; refill rates configured per provider. tokens_remaining, max_tokens, refill_per_sec) table with row-level
locking. hitting external APIs. "retry_after_s": N} instead of raising — the analysis stores it as
partial evidence and re-queues. 24-h request volumes. 429 responses observed.

Approach

Token bucket maintained in PG so it survives multi-process and multi-

host sandbox runs.

Each acquire() is idempotent on retry (use a UUID nonce).

Defaults loaded from scidex/forge/rate_limits.yaml; overridable via

env var SCIDEX_RATE_LIMIT_<PROVIDER>=<rps>.

Dependencies

• scidex/forge/tools.py adapters.

Work Log

2026-04-27 09:30 PT — Slot minimax:79

• Created scidex/forge/rate_limiter.py: Postgres token-bucket with pg_advisory_xact_lock,

acquire(provider, weight=1, timeout_s=30) blocking until token available, returns
{"status": "throttled", "retry_after_s": N} on timeout. Fail-open on errors.
Includes @rate_limited(provider) decorator for tools.py wrappers.

• Created scidex/forge/rate_limits.yaml: Default rates for 15 providers (NCBI 3/s,

GTEx 5/s, Crossref 50/s, SemanticScholar 0.33/s, OpenAlex 1.16/s, etc.), overridable
via SCIDEX_RATE_LIMIT_<PROVIDER> env vars.

• Created migrations/021_rate_buckets.py: Creates provider_rate_buckets table.
• Modified scidex/forge/tools.py: All @log_tool_call wrappers now call acquire()

before hitting external APIs. Replaced all manual time.sleep(1) rate-limit hacks
with proper token-bucket acquire calls (NCBI, GTEx, SemanticScholar, Crossref, STRING,
DisGeNET, GWAS, ChEMBL, OpenTargets, Ensembl, AlphaFold, EuropePMC).

• Added /forge/rate-limits HTML page and /api/forge/rate-limits JSON endpoint showing

live bucket levels per provider with visual progress bars.

• Fixed nested-try bug in retraction_check (inner try: with acquire("NCBI") had

mismatched indentation causing syntax error; fixed by moving acquire into outer try block).

• Committed to main as 223a3b165 and pushed successfully.

Verification

Evidence of completion:

• Commit 223a3b165 on main contains 5 files changed, +89835 insertions:

- New: scidex/forge/rate_limiter.py (token-bucket module)
- New: scidex/forge/rate_limits.yaml (15 provider configs)
- New: migrations/021_rate_buckets.py (DB schema)
- Modified: scidex/forge/tools.py (+acquire() calls, -sleep hacks)
- Modified: api.py (+rate-limits pages/endpoints)

Already Resolved — 2026-04-27T22:30:00Z

Verification performed by minimax:79 slot

• provider_rate_buckets table exists in PostgreSQL (verified via direct query: SELECT 1 FROM provider_rate_buckets LIMIT 1 → "table exists")
• scidex/forge/tools.py imports rate_limiter and has 25 occurrences of rate_limiter or acquire( calls
• /forge/rate-limits HTML page returns 200 with rate limit UI (title: "SciDEX — Rate Limits")
• /api/forge/rate-limits JSON endpoint returns provider list with tokens_remaining, max_tokens, refill_per_sec fields
• Confirmed NCBI (3/s), GTEx (5/s), Crossref (50/s), SemanticScholar (0.33/s), OpenAlex (1.16/s) all configured
• Commit 223a3b165 on main verified via git log --oneline origin/main | grep 223a3b165

Tasks using this spec (1)

[Forge] Rate-limit-aware tool calls inside the sandbox

File: q-sand-rate-limit-aware-tools_spec.md

Modified: 2026-05-01 20:13

Size: 4.5 KB