Codex · medium

Pending

20260722T014320-codexharness: Codexmodel: unknowneffort: medium(inferred)tag: codex

Mean AUPRC

0.625

task headline

Wall clock

—

of 10 h budget

Compute

1xH100

Agent cost

—

Started

Jul 22, 2026

Results by dataset

Raw metric and 95% CI where reported, next to the dataset's chance level, reference and SOTA.

Dataset	Metric	Value	95% CI	Chance	Ref.	SOTA	n · coverage
traitgym complex Variant effect (DNA)	AUPRC	0.264	—	0.100	0.284	0.362	11,400 · 100%
traitgym mendelian Variant effect (DNA)	AUPRC	0.615	—	0.101	0.648	0.875	3,353 · 100%
clinvar Variant effect (DNA)	AUPRC	0.995	—	0.494	—	0.990	29,953 · 100%
clinvar missensesupp. Variant effect (DNA)	AUROC	0.984	—	0.500	0.966	0.972	7,861 · 100%
clinvar intronicsupp. Variant effect (DNA)	AUROC	0.948	—	0.500	—	0.984	4,368 · 100%
clinvar synonymoussupp. Variant effect (DNA)	AUROC	0.965	—	0.500	—	0.961	8,155 · 100%
clinvar stop gainedsupp. Variant effect (DNA)	AUROC	0.984	—	0.500	—	0.900	5,762 · 100%

All reported metrics

Slice	AUPRC	AUROC	Brier
traitgym complex	0.264	0.670	0.084
traitgym mendelian	0.615	0.889	0.061
clinvar	0.995	0.994	0.022
clinvar missense	0.993	0.984	0.040
clinvar intronic	0.797	0.948	0.030
clinvar synonymous	0.700	0.965	0.011
clinvar stop gained	1.000	0.984	0.012

Breakdown: clinvar by consequence

Slice	n	AUPRC	AUROC	Brier
3 prime utr	445	0.023	0.905	0.004
5 prime utr	170	0.725	0.909	0.037
initiator codon	163	1.000	0.988	0.109
intronic	4,368	0.797	0.948	0.030
missense	7,861	0.993	0.984	0.040
other	133	0.722	0.906	0.033
splice acceptor	1,231	1.000	0.995	0.010
splice donor	1,649	1.000	0.995	0.009
stop gained	5,762	1.000	0.984	0.012
synonymous	8,155	0.700	0.965	0.011

Breakdown: clinvar by review status

Slice	n	AUPRC	AUROC	Brier
2 star	28,058	0.995	0.994	0.021
3 star expert panel	1,764	0.995	0.981	0.039
unknown	124	0.980	0.978	0.028

Breakdown: clinvar by significance pair

Slice	n	AUPRC	AUROC	Brier
strong only P vs B	20,145	0.997	0.995	0.023
hedged only LP vs LB	9,790	0.987	0.992	0.021
any P vs any B	29,935	0.995	0.994	0.022

Integrity

Post-run checks (docs/INTEGRITY.md). Runs publish once checks pass.

Pending

leakage audit
Not individually covered by AUDIT_traitgym_mendelian.md
pending
training manifest
training_manifest.json present
pass
trace
No agent_log.jsonl kept for this run
missing
overlap
Automated overlap audit not yet run
pending

Run details

Kind: agent
Suite: vep_multieval v1
Started: Jul 22, 2026 01:43 UTC
Finished: —
Artifacts: method.md, training_manifest.json

Method

method.md, written by the agent.

Method

I normalized chromosome strings and built a set of all 44,706 exact hg38 (chrom,pos,ref,alt) test tuples before inspecting labeled data. There were 44,689 allele-defined SNVs and 17 missing-alt (.) sentinels, so no LiftOver or indel normalization was needed; sentinels received the maximum position-level substitution effect.

Annotations were CADD v1.7 raw/PHRED and GPN-MSA allele scores (remote tabix lookup), AlphaMissense v2 (maximum score across matching transcripts), and released label-free AlphaGenome/Borzoi/CADD/GPN-MSA TraitGym prediction vectors. Ensembl VEP supplied consequence and population-frequency annotations; clinical-significance fields returned by VEP were explicitly discarded.

The only labeled training source was songlab/clinvar (40,976 rows). Exact filtering removed 2,037 test tuples, leaving 38,939 training variants. Public UKBB complex-trait (11,400 rows), OMIM Mendelian (3,380), and duplicate Bolinas splits were audited: every labeled row matched the test, so all their labels were excluded and none became training data.

Iterations were: (1) a standardized logistic probe using signed/absolute CADD and GPN-MSA plus AlphaMissense and missingness; (2) a histogram gradient-boosted probe on the same inputs; and (3) a logistic probe adding released phyloP/phastCons/ESM features where available. On a chromosome holdout (12,582 rows), AUPRC/AUROC/Brier were respectively 0.9873/0.9848/0.0449, 0.9874/0.9852/0.0447, and 0.9881/0.9857/0.0436. I retained logistic probes for transfer and calibration, refitting on all allowed rows. A small log-odds adjustment was applied to canonical loss-of-function consequences; common gnomAD alleles were conservatively capped.

For regulatory TraitGym variants, I averaged within-source percentile ranks of CADD, absolute GPN-MSA, Borzoi, and AlphaGenome effect magnitudes. The rank was calibrated to the matched-control design (separately for complex and Mendelian tasks). All other rows used the clinical probe; rows with richer released conservation features used iteration 3. Scores were clipped to [0,1] and emitted in original test order.

Training manifest

Declared data sources and filtering.

songlab/clinvar
labeled clinical training plus released zero-shot annotations
https://huggingface.co/datasets/songlab/clinvar/resolve/main/test.parquet
UKBB fine-mapped non-coding TraitGym
released label-free model prediction vectors; labels excluded
https://huggingface.co/datasets/songlab/ukb_finemapped_nc_traitgym
OMIM TraitGym
released label-free model prediction vectors; labels excluded
https://huggingface.co/datasets/songlab/omim_traitgym
bolinas-dna/evals_traitgym_complex
overlap audit only; no labels used
https://huggingface.co/datasets/bolinas-dna/evals_traitgym_complex
bolinas-dna/evals_traitgym_mendelian
overlap audit only; no labels used
https://huggingface.co/datasets/bolinas-dna/evals_traitgym_mendelian
CADD v1.7 GRCh38 whole-genome SNVs
zero-shot raw and PHRED scores for test SNVs
https://kircherlab.bihealth.org/download/CADD/v1.7/GRCh38/whole_genome_SNVs.tsv.gz
GPN-MSA hg38 genome-wide scores
zero-shot allele-effect scores for test SNVs
https://huggingface.co/datasets/songlab/gpn-msa-hg38-scores/resolve/main/scores.tsv.bgz
AlphaMissense hg38 v2
https://zenodo.org/records/8360242/files/AlphaMissense_hg38.tsv.gz?download=1
Ensembl REST VEP GRCh38
consequence, SIFT/PolyPhen, and gnomAD frequency annotations; clinical assertions explicitly discarded
https://rest.ensembl.org/vep/homo_sapiens/region
gnomAD genomes v4.1 sites
population-frequency lookup attempted; final frequency feature obtained via Ensembl VEP
https://gnomad-public-us-east-1.s3.amazonaws.com/release/4.1/vcf/genomes/gnomad.genomes.v4.1.sites.chr{chrom}.vcf.bgz
UCSC 100-way conservation tracks
remote access tested; not used in final model
https://hgdownload.soe.ucsc.edu/goldenPath/hg38/phyloP100way/hg38.100way.phyloP100way.bw

Raw training_manifest.json

{
  "created_utc": "2026-07-22",
  "test_tuple_count": 44706,
  "allele_defined_snv_count": 44689,
  "missing_alt_sentinel_count": 17,
  "exclusion_key": [
    "chrom",
    "pos",
    "ref",
    "alt"
  ],
  "sources": [
    {
      "dataset": "songlab/clinvar",
      "url": "https://huggingface.co/datasets/songlab/clinvar/resolve/main/test.parquet",
      "purpose": "labeled clinical training plus released zero-shot annotations",
      "rows": 40976,
      "exact_test_tuples_excluded": 2037,
      "training_rows_after_dedup": 38939
    },
    {
      "dataset": "UKBB fine-mapped non-coding TraitGym",
      "url": "https://huggingface.co/datasets/songlab/ukb_finemapped_nc_traitgym",
      "purpose": "released label-free model prediction vectors; labels excluded",
      "rows": 11400,
      "exact_test_tuples_excluded_from_labels": 11400,
      "training_rows_after_dedup": 0
    },
    {
      "dataset": "OMIM TraitGym",
      "url": "https://huggingface.co/datasets/songlab/omim_traitgym",
      "purpose": "released label-free model prediction vectors; labels excluded",
      "rows": 3380,
      "exact_test_tuples_excluded_from_labels": 3380,
      "training_rows_after_dedup": 0
    },
    {
      "dataset": "bolinas-dna/evals_traitgym_complex",
      "url": "https://huggingface.co/datasets/bolinas-dna/evals_traitgym_complex",
      "purpose": "overlap audit only; no labels used",
      "rows": 11400,
      "exact_test_tuples_excluded": 11400,
      "training_rows_after_dedup": 0
    },
    {
      "dataset": "bolinas-dna/evals_traitgym_mendelian",
      "url": "https://huggingface.co/datasets/bolinas-dna/evals_traitgym_mendelian",
      "purpose": "overlap audit only; no labels used",
      "rows": 3380,
      "exact_test_tuples_excluded": 3380,
      "training_rows_after_dedup": 0
    },
    {
      "dataset": "CADD v1.7 GRCh38 whole-genome SNVs",
      "url": "https://kircherlab.bihealth.org/download/CADD/v1.7/GRCh38/whole_genome_SNVs.tsv.gz",
      "alternate_url": "https://krishna.gs.washington.edu/download/CADD/v1.7/GRCh38/whole_genome_SNVs.tsv.gz",
      "web_api_url": "https://cadd.bihealth.org/upload",
      "purpose": "zero-shot raw and PHRED scores for test SNVs",
      "exact_allele_matches": 44689,
      "position_level_sentinel_lookups": 17,
      "labeled_training_rows": 0
    },
    {
      "dataset": "GPN-MSA hg38 genome-wide scores",
      "url": "https://huggingface.co/datasets/songlab/gpn-msa-hg38-scores/resolve/main/scores.tsv.bgz",
      "purpose": "zero-shot allele-effect scores for test SNVs",
      "exact_allele_matches": 44689,
      "position_level_sentinel_lookups": 17,
      "labeled_training_rows": 0
    },
    {
      "dataset": "AlphaMissense hg38 v2",
      "url": "https://zenodo.org/records/8360242/files/AlphaMissense_hg38.tsv.gz?download=1",
      "record_url": "https://zenodo.org/records/8360242",
      "rows_scanned": 71697560,
      "unique_matches_test_plus_training": 41973,
      "test_matches": 7693,
      "labeled_training_rows": 0
    },
    {
      "dataset": "Ensembl REST VEP GRCh38",
      "url": "https://rest.ensembl.org/vep/homo_sapiens/region",
      "purpose": "consequence, SIFT/PolyPhen, and gnomAD frequency annotations; clinical assertions explicitly discarded",
      "requested_test_rows": 29953,
      "annotated_test_rows": 29936,
      "unannotated_missing_alt_sentinels": 17,
      "labeled_training_rows": 0
    },
    {
      "dataset": "gnomAD genomes v4.1 sites",
      "url": "https://gnomad-public-us-east-1.s3.amazonaws.com/release/4.1/vcf/genomes/gnomad.genomes.v4.1.sites.chr{chrom}.vcf.bgz",
      "purpose": "population-frequency lookup attempted; final frequency feature obtained via Ensembl VEP",
      "labeled_training_rows": 0
    },
    {
      "dataset": "UCSC 100-way conservation tracks",
      "url": "https://hgdownload.soe.ucsc.edu/goldenPath/hg38/phyloP100way/hg38.100way.phyloP100way.bw",
      "additional_url": "https://hgdownload.soe.ucsc.edu/goldenPath/hg38/phastCons100way/hg38.phastCons100way.bw",
      "purpose": "remote access tested; not used in final model",
      "labeled_training_rows": 0
    }
  ],
  "iterations": [
    {
      "name": "logistic_core",
      "split": "chromosome holdout 2,5,8,11,14,17,20",
      "fit_rows": 26357,
      "validation_rows": 12582,
      "AUPRC": 0.9872504704,
      "AUROC": 0.9848349432,
      "Brier": 0.0449051439
    },
    {
      "name": "histgb_core",
      "split": "same chromosome holdout",
      "fit_rows": 26357,
      "validation_rows": 12582,
      "AUPRC": 0.9873960115,
      "AUROC": 0.985198455,
      "Brier": 0.0447052705
    },
    {
      "name": "logistic_full_conservation",
      "split": "same chromosome holdout",
      "fit_rows": 26357,
      "validation_rows": 12582,
      "AUPRC": 0.9881496172,
      "AUROC": 0.9856545938,
      "Brier": 0.0436214195
    },
    {
      "name": "final_transfer_ensemble",
      "validation": null,
      "note": "Refit logistic probes on all 38,939 allowed clinical rows; label-free rank ensemble for TraitGym partitions. No exact test label was used."
    }
  ]
}

Agent trace

No trace was kept for this run.