Codex · medium

Pending

20260722T234224-codex-medium-capharness: Codexmodel: unknowneffort: medium(inferred)tag: codex-medium-cap

Mean AUPRC

0.690

task headline

Wall clock

—

of 10 h budget

Compute

1xH100

Agent cost

—

summed from trace

Started

Jul 22, 2026

Results by dataset

Raw metric and 95% CI where reported, next to the dataset's chance level, reference and SOTA.

Dataset	Metric	Value	95% CI	Chance	Ref.	SOTA	n · coverage
traitgym complex Variant effect (DNA)	AUPRC	0.348	—	0.100	0.284	0.362	11,400 · 100%
traitgym mendelian Variant effect (DNA)	AUPRC	0.727	—	0.101	0.648	0.875	3,353 · 100%
clinvar Variant effect (DNA)	AUPRC	0.994	—	0.494	—	0.990	4,674 · 100%
clinvar missensesupp. Variant effect (DNA)	AUROC	0.960	—	0.500	0.966	0.972	1,263 · 100%
clinvar intronicsupp. Variant effect (DNA)	AUROC	0.939	—	0.500	—	0.984	674 · 100%
clinvar synonymoussupp. Variant effect (DNA)	AUROC	1.000	—	0.500	—	0.961	1,319 · 100%
clinvar stop gainedsupp. Variant effect (DNA)	AUROC	0.931	—	0.500	—	0.900	823 · 100%

All reported metrics

Slice	AUPRC	AUROC	Brier
traitgym complex	0.348	0.725	0.078
traitgym mendelian	0.727	0.887	0.048
clinvar	0.994	0.993	0.027
clinvar missense	0.985	0.960	0.072
clinvar intronic	0.766	0.939	0.031
clinvar synonymous	0.957	1.000	0.003
clinvar stop gained	1.000	0.931	0.004

Breakdown: clinvar by consequence

Slice	n	AUPRC	AUROC	Brier
intronic	674	0.766	0.939	0.031
missense	1,263	0.985	0.960	0.072
splice acceptor	188	—	—	—
splice donor	268	—	—	—
stop gained	823	1.000	0.931	0.004
synonymous	1,319	0.957	1.000	0.003

Breakdown: clinvar by review status

Slice	n	AUPRC	AUROC	Brier
2 star	4,368	0.994	0.994	0.026
3 star expert panel	284	0.996	0.971	0.049

Breakdown: clinvar by significance pair

Slice	n	AUPRC	AUROC	Brier
strong only P vs B	3,058	0.996	0.993	0.030
hedged only LP vs LB	1,613	0.983	0.993	0.022
any P vs any B	4,671	0.994	0.993	0.027

Integrity

Post-run checks (docs/INTEGRITY.md). Runs publish once checks pass.

Pending

leakage audit
Not individually covered by AUDIT_traitgym_mendelian.md
pending
training manifest
training_manifest.json present
pass
trace
agent_log.jsonl available for the trace judge
pass
overlap
Automated overlap audit not yet run
pending

Run details

Kind: agent
Suite: vep_multieval v1
Started: Jul 22, 2026 23:42 UTC
Finished: —
Sessions: 389
Tool calls: 192
Turns: —
Artifacts: method.md, agent_log.jsonl, training_manifest.json

Method

method.md, written by the agent.

BioModelBench VEP multi-eval v1

I used a source-adaptive importance score without reading any shipped labels. All joins used exact hg38 (chrom,pos,ref,alt) tuples.

For clinical/general scoring I remotely queried CADD v1.7 RawScore and PHRED for every true substitution and downloaded AlphaMissense v2. AlphaMissense covered 7,693 test variants. Seventeen ALT="." no-sequence-alteration records have no meaningful alternate allele and received the score floor.

I downloaded the 2026-07-21 ClinVar GRCh38 VCF and retained only unambiguous P/LP versus B/LB SNVs. Before fitting, I removed every exact test tuple: 1,480,111 eligible allele records, 30,656 exclusions, 1,449,455 retained unique/consistent variants. From these I deterministically sampled 5,000 per class (10,000 total; 6,932 AlphaMissense-covered). Chromosomes divisible by five were held out. Iterations were: AlphaMissense-only missense calibration (AUROC 0.9680); CADD logistic (AUROC/AUPRC/Brier 0.9602/0.9546/0.0825); CADD+AlphaMissense logistic (0.9763/0.9744/0.0549); and nonlinear histogram boosting (0.9791/0.9794/0.0552). I refit the final nonlinear model on all 10,000 rows.

For 14,753 unique test tuples present in the two public TraitGym matched_9 sets, I used the released chromosome-held-out CADD+GPN-MSA+Borzoi logistic predictions. I never loaded their label column during final assembly. Since those regressions use balanced class weights, I restored the known 1:9 causal/control deployment prior by an odds correction; rankings are unchanged. Twenty-seven tuples occur in both TraitGym sources, so their two predictions were averaged.

Intermediate investigations: a genome-wide GPN-MSA tabix iteration was excluded after incomplete HTTP-range responses; EVEE was excluded because its deployed ClinVar-trained probe does not expose training-fold membership, preventing exact-overlap assurance. Neither affected final scores.

Final scores are clipped to [0.001,0.999]. Integrity checks confirmed 44,706 unique rows, original key order, complete scoring, and finite values.

Training manifest

Declared data sources and filtering.

CADD v1.7 GRCh38 whole-genome SNVs
final inference and calibration features
https://krishna.gs.washington.edu/download/CADD/v1.7/GRCh38/whole_genome_SNVs.tsv.gz
AlphaMissense v2 hg38
final inference and calibration feature
https://zenodo.org/records/8360242
ClinVar GRCh38 weekly VCF 2026-07-21
labels for calibration only
https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/clinvar.vcf.gz
TraitGym matched_9 released predictions
final chromosome-held-out inference predictions; labels ignored
https://huggingface.co/api/datasets/songlab/TraitGym/tree/main
GPN-MSA hg38 scores
attempted intermediate; excluded after incomplete range responses
https://huggingface.co/api/datasets/songlab/gpn-msa-hg38-scores/tree/main
EVEE investigation
investigated then excluded: deployed probe training membership unavailable
https://xix0d0o8le.execute-api.us-east-1.amazonaws.com/variants/search
dbNSFP download documentation
investigated only; no data downloaded
https://www.dbnsfp.org/download/

Raw training_manifest.json

{
  "test_tuple_filter": {
    "path": "test.parquet",
    "build": "hg38",
    "unique_tuples": 44706,
    "duplicates": 0
  },
  "resources": [
    {
      "name": "CADD v1.7 GRCh38 whole-genome SNVs",
      "urls": [
        "https://krishna.gs.washington.edu/download/CADD/v1.7/GRCh38/whole_genome_SNVs.tsv.gz",
        "https://krishna.gs.washington.edu/download/CADD/v1.7/GRCh38/whole_genome_SNVs.tsv.gz.tbi",
        "https://kircherlab.bihealth.org/download/CADD/v1.7/GRCh38/whole_genome_SNVs.tsv.gz",
        "https://cadd.bihealth.org/snv/GRCh38-v1.7/11:77201578_C_T"
      ],
      "role": "final inference and calibration features",
      "test_alleles_scored": 44689,
      "no_alternate_allele_rows": 17
    },
    {
      "name": "AlphaMissense v2 hg38",
      "urls": [
        "https://zenodo.org/records/8360242",
        "https://zenodo.org/records/8360242/files/AlphaMissense_hg38.tsv.gz?download=1"
      ],
      "role": "final inference and calibration feature",
      "test_coverage": 7693,
      "leakage_filtered_clinvar_pool_coverage": 201788
    },
    {
      "name": "ClinVar GRCh38 weekly VCF 2026-07-21",
      "urls": [
        "https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/clinvar.vcf.gz",
        "https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/clinvar.vcf.gz.tbi"
      ],
      "role": "labels for calibration only",
      "eligible_before_test_dedup": 1480111,
      "exact_test_tuples_excluded": 30656,
      "duplicate_or_discordant_removed": 0,
      "retained_after_dedup": 1449455,
      "sampled_training_rows": 10000
    },
    {
      "name": "TraitGym matched_9 released predictions",
      "urls": [
        "https://huggingface.co/api/datasets/songlab/TraitGym/tree/main",
        "https://huggingface.co/datasets/songlab/TraitGym/resolve/main/complex_traits_matched_9/test.parquet",
        "https://huggingface.co/datasets/songlab/TraitGym/resolve/main/mendelian_traits_matched_9/test.parquet",
        "https://huggingface.co/datasets/songlab/TraitGym/resolve/main/complex_traits_matched_9/preds/all/CADD+GPN-MSA+Borzoi.LogisticRegression.chrom.parquet",
        "https://huggingface.co/datasets/songlab/TraitGym/resolve/main/mendelian_traits_matched_9/preds/all/CADD+GPN-MSA+Borzoi.LogisticRegression.chrom.parquet",
        "https://huggingface.co/datasets/songlab/TraitGym/resolve/main/complex_traits_matched_9/preds/all/CADD+Borzoi.LogisticRegression.chrom.parquet",
        "https://huggingface.co/datasets/songlab/TraitGym/resolve/main/mendelian_traits_matched_9/preds/all/CADD+Borzoi.LogisticRegression.chrom.parquet",
        "https://huggingface.co/datasets/songlab/TraitGym/resolve/main/complex_traits_matched_9/preds/all/GPN-MSA.LogisticRegression.chrom.parquet",
        "https://huggingface.co/datasets/songlab/TraitGym/resolve/main/mendelian_traits_matched_9/preds/all/GPN-MSA.LogisticRegression.chrom.parquet",
        "https://huggingface.co/datasets/songlab/TraitGym/resolve/main/complex_traits_matched_9/preds/all/CADD.LogisticRegression.chrom.parquet",
        "https://huggingface.co/datasets/songlab/TraitGym/resolve/main/mendelian_traits_matched_9/preds/all/CADD.LogisticRegression.chrom.parquet",
        "https://github.com/songlab-cal/TraitGym"
      ],
      "role": "final chromosome-held-out inference predictions; labels ignored",
      "coordinate_rows_touched": 14780,
      "unique_test_coverage": 14753,
      "cross_source_duplicate_tuples": 27
    },
    {
      "name": "GPN-MSA hg38 scores",
      "urls": [
        "https://huggingface.co/api/datasets/songlab/gpn-msa-hg38-scores/tree/main",
        "https://huggingface.co/datasets/songlab/gpn-msa-hg38-scores/resolve/main/scores.tsv.bgz"
      ],
      "role": "attempted intermediate; excluded after incomplete range responses",
      "used_in_final": false
    },
    {
      "name": "EVEE investigation",
      "urls": [
        "https://xix0d0o8le.execute-api.us-east-1.amazonaws.com/variants/search",
        "https://github.com/goodfire-ai/evee-mcp.git",
        "https://github.com/goodfire-ai/evee-manuscript.git",
        "https://github.com/goodfire-ai/variant-viewer.git",
        "https://www.biorxiv.org/content/10.64898/2026.04.10.717844v4.full.pdf",
        "https://zenodo.org/api/records/19701997",
        "https://zenodo.org/api/records/19701997/files/manifest.json/content"
      ],
      "role": "investigated then excluded: deployed probe training membership unavailable",
      "used_in_final": false
    },
    {
      "name": "dbNSFP download documentation",
      "urls": [
        "https://www.dbnsfp.org/download/"
      ],
      "role": "investigated only; no data downloaded",
      "used_in_final": false
    }
  ],
  "iterations": [
    {
      "iteration": "alpha_missense_only",
      "validation_partition": "held-out chromosomes, missense-covered ClinVar pool",
      "n": 26628,
      "auroc": 0.9680023125,
      "auprc": 0.9355156295,
      "brier": 0.0662432477,
      "selected": false
    },
    {
      "iteration": "cadd_logistic",
      "validation_partition": "held-out chromosomes, balanced 10k calibration sample",
      "n": 1354,
      "auroc": 0.96018663,
      "auprc": 0.9545566468,
      "brier": 0.0825286739,
      "selected": false
    },
    {
      "iteration": "cadd_alpha_missense_logistic",
      "validation_partition": "held-out chromosomes, balanced 10k calibration sample",
      "n": 1354,
      "auroc": 0.9763359515,
      "auprc": 0.9744301754,
      "brier": 0.0549033858,
      "selected": false
    },
    {
      "iteration": "cadd_alpha_missense_histgb",
      "validation_partition": "held-out chromosomes, balanced 10k calibration sample",
      "n": 1354,
      "auroc": 0.9791345244,
      "auprc": 0.9793532316,
      "brier": 0.0551882045,
      "selected": true
    },
    {
      "iteration": "traitgym_released_chromosome_holdout_ensemble",
      "validation_produced_locally": false,
      "published_reference_auprc_complex": 0.362,
      "published_reference_auprc_mendelian": 0.648,
      "selected": true
    },
    {
      "iteration": "standalone_gpn_msa_remote_lookup",
      "validation_produced_locally": false,
      "selected": false,
      "reason": "incomplete HTTP range responses"
    },
    {
      "iteration": "evee_deployed_probe",
      "validation_produced_locally": false,
      "selected": false,
      "reason": "could not guarantee exact test tuples were absent from deployed probe training"
    }
  ]
}

Agent trace

Compact trace, 156 KB. Loaded on demand.