Build a genomic foundation model, scored on genomic tasks across variant effect prediction, QTLs, epigenetic track prediction, DMS and embeddings
DNA, RNA & protein sequences24 h budgetheadline: Mean metric
About this task
The agent trains or post-trains sequence model(s) in 24 hours on 8×H100. Its final model is re-run by us, with no network, on ~25 public held-out datasets spanning DNA variant effect, chromatin/expression QTLs, protein, RNA and nucleotide deep mutational scans, genomic track prediction and sequence embeddings. Every dataset is named in the prompt; labels are never shipped.
Leaderboard
Ranked by Mean metric. Dataset cells show raw metric values, shaded by rank among the runs shown (darker = better); hover for CIs and the dataset's SOTA.
| # | Run | Integrity | |||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Refbaseline-zeroshot-nt-esm | 0.284 | 0.690 | 0.678 | 0.126 | 0.093 | 0.190 | 0.125 | 0.102 | 0.085 | 0.018 | 0.031 | 0.462 | 0.513 | 0.015 | 0.619 | 0.402 | 0.164 | 0.129 | 0.122 | · | · | · | 0.738 | 0.704 | 0.735 | 0.703 | 0.276 | — | — | ||
| 1 | Claude Code · highpilot-claude-40m-v2 · claude-sonnet-5 | 0.203 | 0.487 | 0.441 | 0.035 | 0.095 | 0.112 | 0.158 | 0.083 | 0.088 | 0.020 | 0.034 | 0.481 | 0.508 | -0.066 | 0.629 | 0.402 | -0.018 | 0.015 | -0.018 | 0.000 | -0.001 | -0.000 | 0.500 | 0.503 | 0.501 | 0.499 | 0.266 | Pending | — | — |
| Refbaseline-io-smoke | 0.197 | 0.515 | 0.412 | 0.040 | 0.109 | 0.123 | 0.151 | 0.082 | 0.077 | 0.021 | 0.028 | 0.510 | 0.497 | -0.009 | 0.642 | 0.009 | 0.002 | 0.006 | 0.013 | -0.000 | -0.000 | -0.001 | 0.507 | 0.504 | 0.571 | 0.556 | 0.257 | — | — |
≈ approximate reference value. “inf.” = agent / effort inferred from the run id or trace (no run_meta.json).
Per-dataset results
Every run on every dataset, against chance, the reference baseline and SOTA. Filters above apply.
agent runbest runbaselineSOTAreferencechance95% CI privateRaw metric values; each row has its own axis.
Suite datasets
From suite.yaml: each dataset's metric, chance level, reference and SOTA (raw values).
| Dataset | Visibility | Metric | Chance | Reference | SOTA | Source |
|---|---|---|---|---|---|---|
| Variant effect (DNA, clinical + GWAS) | ||||||
| clinvar gene holdout 2k | public | AUPRC | 0.500 | 0.851 Evo 2 7B | 0.919 CADD v1.7 | NCBI ClinVar GRCh38 VCF clinvar_20240301.vcf.gz (nearest weekly to the Evo2-paper 2024-02-28 release), >=2-star, P/LP vs B/LB, ref/alt <=64 bp; reference scores joined from HF goodarzilab/evo2-clinvar (Brixi et al. 2025) |
| songlab clinvar vs benign | public | AUPRC | 0.434 | 0.793 Evo2_7B | 0.906 GPN-Star-V100 | HF songlab/clinvar_vs_benign test.parquet (missense ClinVar P vs common benign; Ye, Benegas et al. 2025, GPN-Star) |
| songlab cosmic | public | AUPRC | 0.037 | 0.193 Evo2_7B | 0.581 GPN-Star-V100 | HF songlab/cosmic test.parquet (frequent COSMIC somatic missense vs gnomAD common missense) |
| songlab ukb finemapped coding | public | AUPRC | 0.101 | 0.168 Evo2_7B | 0.306 GPN-Star-M447 | HF songlab/ukb_finemapped_coding (UKB fine-mapped missense, PIP>0.9 vs PIP<0.01) |
| traitgym mendelian | public | chrom weighted auprc | 0.122 | 0.712 CADD (raw) | 0.779 GPN-Star-M447 | HF songlab/TraitGym mendelian_traits_matched_9 test (rev 1fde1955; Benegas, Eraslan, Song 2025, bioRxiv 10.1101/2025.02.11.637758) |
| traitgym complex | public | chrom weighted auprc | 0.140 | 0.137 Evo 2 7B (LLR) | 0.274 CADD (raw) | HF songlab/TraitGym complex_traits_matched_9 test (rev 1fde1955) |
| Molecular QTLs (chromatin, binding, expression, polyadenylation) | ||||||
| caqtl african | public | AUPRC | 0.085 | — | 0.576 AlphaGenome | AlphaGenome eval file storage.googleapis.com/alphagenome/evals/caqtl_african_variant_causality_human_predictions.feather (LCL caQTL, DeGorter 2023 via ChromBPNet benchmark; AG test chromosomes 3,6,9,12,16,18,19,21) |
| caqtl european | public | AUPRC | 0.083 | — | 0.342 AlphaGenome | AlphaGenome eval caqtl_european_variant_causality_human_predictions.feather (original eQTL Catalogue QTD100018, Zenodo 13848268) |
| dsqtl yoruba | public | AUPRC | 0.020 | — | 0.631 AlphaGenome | AlphaGenome eval dsqtl_yoruba_variant_causality_human_predictions.feather (Degner 2012 GEO GSE31388; hg19 IDs) |
| bqtl spi1 | public | AUPRC | 0.028 | — | 0.600 AlphaGenome | AlphaGenome eval bqtl_spi1_variant_causality_human_predictions.feather (Tehranchi 2016 PU.1 bQTL via ChromBPNet benchmark) |
| eqtl causality | public | auroc per tissue mean | 0.500 | 0.670 Borzoi | 0.717 AlphaGenome | AlphaGenome eval eqtl_variant_catalogue_causality_gene_balanced_human_predictions.feather (GTEx v8 via eQTL Catalogue SuSiE, distance-matched) |
| eqtl sign | public | auroc per tissue mean | 0.500 | 0.750 Borzoi | 0.834 AlphaGenome | AlphaGenome eval eqtl_variant_borzoi_sign_human_predictions.feather (GTEx v8 SuSiE PIP>0.9) |
| eqtl coefficient | public | spearman per tissue mean | 0.000 | 0.390 Borzoi | 0.511 AlphaGenome | AlphaGenome eval eqtl_variant_borzoi_coefficient_human_predictions.feather |
| paqtl | public | matched auprc | 0.628 | 0.621 Borzoi | 0.653 AlphaGenome | AlphaGenome eval paqtl_variant_causality_human_predictions.feather + PolyA_DB v3 (storage.googleapis.com/seqnn-share/helper/polyadb_human_v3.csv.gz) |
| Protein fitness (DMS, protein input) | ||||||
| proteingym substitutions | public | spearman per assay mean | 0.000 | 0.396 ESM2 650M | 0.514 VenusREM | ProteinGym v1.3 DMS_substitutions (HF OATML-Markslab/ProteinGym_v1; marks.hms.harvard.edu/proteingym/ProteinGym_v1.3); Notin et al. 2023 |
| Non-coding RNA fitness (DMS) | ||||||
| rnagym ncrna | public | spearman per assay mean | 0.000 | 0.155 RNA-FM | 0.240 RNAErnie | RNAGym fitness_prediction (HF Marks-lab/RNAgym; Arora et al. 2025) — Ribozyme + tRNA + Aptamer assays |
| Coding-sequence fitness (DMS, nucleotide input) | ||||||
| rnagym mrna coding | public | spearman per assay mean | 0.000 | 0.175 Evo 1 (131k) | 0.251 Evo 2 7B | RNAGym mRNA-coding assays (nucleotide CDS + codon-level mutants) — public substitute for the private Evo2-paper nucleotide DMS panels |
| evee sge dms | public | spearman per assay mean | 0.000 | 0.355 Evo 2 7B zero-shot loss | 0.492 AlphaMissense | MaveDB (CC0): BRCA1 urn:mavedb:00000097-a-2 (Findlay 2018), BRCA2 00001225-a-1 (HAP1 SGE, doi 10.1038/s41586-024-08388-8), LDLR 00001269-a-1 (Tabet 2025), TP53 00001213-a-1 (Funk 2025); EVEE construction (Pearce et al. 2026) |
| Functional-genomics track prediction | ||||||
| enformer tracks 128 | public | track pearson | 0.000 | — | — | Basenji2/Enformer human test set gs://basenji_barnyard/data/human (requester-pays), Avsec et al. 2021 |
| chrombpnet imr90 atac | public | profile pearson | 0.000 | 0.750 ChromBPNet (ENCODE predicted bigWig) | 0.860 AlphaGenome | ENCODE ChromBPNet annotation ENCSR978WIX (IMR-90 ATAC, experiment ENCSR200OML); fold-0 test peaks (chr1,3,6) from ENCFF802KJC; observed bigWig ENCFF650PHA |
| encode accessibility tiles | public | track pearson | 0.000 | 0.817 ChromBPNet (ENCODE predicted bigWigs) | — | ENCODE ChromBPNet annotations, observed 1 bp bigWigs: GM12878, K562, HepG2, A549, HCT116 x {ATAC, DNase} (e.g. DNase ENCSR003WJE/ENCSR296UHQ/ENCSR006CUK/ENCSR422EZD/ENCSR038BXQ; ATAC ENCSR389HIH/ENCSR467RSV/ENCSR380YGX/ENCSR311VBG/ENCSR799MEK) — public accessibility tracks |
| Sequence embeddings (Gene-MTEB, probe fitted by the evaluator) | ||||||
| gene mteb hv1 | public | gene mteb 8shot | 0.500 | 0.721 NT-2.5b-1000g | 0.828 METAGENE-1 | HF metagene-ai/HumanVirusInfecting config class-1 (sha 70876feb; Liu et al. 2025, METAGENE-1, arXiv 2501.02045) |
| gene mteb hv2 | public | gene mteb 8shot | 0.500 | 0.652 NT-2.5b-Multi | 0.742 METAGENE-1 | HF metagene-ai/HumanVirusInfecting config class-2 |
| gene mteb hv3 | public | gene mteb 8shot | 0.500 | 0.758 NT-2.5b-Multi | 0.835 METAGENE-1 | HF metagene-ai/HumanVirusInfecting config class-3 |
| gene mteb hv4 | public | gene mteb 8shot | 0.500 | 0.697 METAGENE-1 | 0.755 NT-2.5b-1000g | HF metagene-ai/HumanVirusInfecting config class-4 |
| diagnostics | ||||||
| t7 gp17 recovery | public | masked nt accuracy | 0.250 | — | — | NCBI NC_001604.1 (T7 phage, 39,937 bp), gp17 tail fiber; 4 masked ~201 bp windows (design-recovery diagnostic) |
Prompt
Exactly what the agent receives.
Show the full prompt
BioModelBench: Build a genomic foundation model, scored on genomic tasks across variant effect prediction, QTLs, epigenetic track prediction, DMS and embeddings
Goal
Build the best genomic sequence model you can. In 24 hours on H100×8 you may pretrain, continue pretraining, fine-tune or post-train any model(s) you like, starting from public weights or from scratch. Your final model is scored on every dataset below. It is not one task: it's a test of how general and how good your model is across biology.
After your run ends, we re-run your model ourselves, in a fresh container with no
network access, on our own copy of the test inputs, and compute every metric against
labels you never see. The headline score is the plain mean of the raw metric values over
every dataset in the table below except paqtl (reported, not averaged). A missing dataset scores 0, so cover everything.
The evaluation suite
| Group | Dataset | Test rows | Metric | Chance | SOTA / reference | Output |
|---|---|---|---|---|---|---|
| Variant effect (DNA, clinical + GWAS) | clinvar_gene_holdout_2k | 2,000 | auprc | 0.5 | 0.9192 (CADD v1.7) | scalar |
| Variant effect (DNA, clinical + GWAS) | songlab_clinvar_vs_benign | 5,000 | auprc | 0.4342 | 0.9061 (GPN-Star-V100) | scalar |
| Variant effect (DNA, clinical + GWAS) | songlab_cosmic | 5,000 | auprc | 0.0366 | 0.5811 (GPN-Star-V100) | scalar |
| Variant effect (DNA, clinical + GWAS) | songlab_ukb_finemapped_coding | 2,221 | auprc | 0.1009 | 0.306 (GPN-Star-M447) | scalar |
| Variant effect (DNA, clinical + GWAS) | traitgym_mendelian | 3,380 | chrom_weighted_auprc | 0.1225 | 0.7789 (GPN-Star-M447) | scalar |
| Variant effect (DNA, clinical + GWAS) | traitgym_complex | 2,000 | chrom_weighted_auprc | 0.1396 | 0.2738 (CADD (raw)) | scalar |
| Molecular QTLs (chromatin, binding, expression, polyadenylation) | caqtl_african | 5,000 | auprc | 0.0852 | 0.576 (AlphaGenome) | scalar |
| Molecular QTLs (chromatin, binding, expression, polyadenylation) | caqtl_european | 5,000 | auprc | 0.0828 | 0.3418 (AlphaGenome) | scalar |
| Molecular QTLs (chromatin, binding, expression, polyadenylation) | dsqtl_yoruba | 8,226 | auprc | 0.0201 | 0.6311 (AlphaGenome) | scalar |
| Molecular QTLs (chromatin, binding, expression, polyadenylation) | bqtl_spi1 | 5,000 | auprc | 0.0276 | 0.5998 (AlphaGenome) | scalar |
| Molecular QTLs (chromatin, binding, expression, polyadenylation) | eqtl_causality | 2,000 | auroc_per_tissue_mean | 0.5 | 0.7174 (AlphaGenome) | scalar |
| Molecular QTLs (chromatin, binding, expression, polyadenylation) | eqtl_sign | 2,000 | auroc_per_tissue_mean | 0.5 | 0.8343 (AlphaGenome) | signed |
| Molecular QTLs (chromatin, binding, expression, polyadenylation) | eqtl_coefficient | 2,000 | spearman_per_tissue_mean | 0.0 | 0.5108 (AlphaGenome) | signed |
| Molecular QTLs (chromatin, binding, expression, polyadenylation) | paqtl | 1,343 | matched_auprc | 0.6284 | 0.6531 (AlphaGenome) | scalar |
| Protein fitness (DMS, protein input) | proteingym_substitutions | 16,963 | spearman_per_assay_mean | 0.0 | 0.5136 (VenusREM) | scalar |
| Non-coding RNA fitness (DMS) | rnagym_ncrna | 23,113 | spearman_per_assay_mean | 0.0 | 0.2405 (RNAErnie) | scalar |
| Coding-sequence fitness (DMS, nucleotide input) | rnagym_mrna_coding | 14,000 | spearman_per_assay_mean | 0.0 | 0.2511 (Evo 2 7B) | scalar |
| Coding-sequence fitness (DMS, nucleotide input) | evee_sge_dms | 6,308 | spearman_per_assay_mean | 0.0 | 0.4919 (AlphaMissense) | scalar |
| Functional-genomics track prediction | enformer_tracks_128 | 1,000 | track_pearson | 0.0 | — | array |
| Functional-genomics track prediction | chrombpnet_imr90_atac | 12,920 | profile_pearson | 0.0 | 0.86 (AlphaGenome) | array |
| Functional-genomics track prediction | encode_accessibility_tiles | 1,024 | track_pearson | 0.0 | 0.8165 (ChromBPNet (ENCODE predicted bigWigs)) | array |
| Sequence embeddings (Gene-MTEB, probe fitted by the evaluator) | gene_mteb_hv1 | 15,404 | gene_mteb_8shot | 0.5 | 0.828 (METAGENE-1) | embedding |
| Sequence embeddings (Gene-MTEB, probe fitted by the evaluator) | gene_mteb_hv2 | 14,956 | gene_mteb_8shot | 0.5 | 0.742 (METAGENE-1) | embedding |
| Sequence embeddings (Gene-MTEB, probe fitted by the evaluator) | gene_mteb_hv3 | 15,744 | gene_mteb_8shot | 0.5 | 0.835 (METAGENE-1) | embedding |
| Sequence embeddings (Gene-MTEB, probe fitted by the evaluator) | gene_mteb_hv4 | 15,633 | gene_mteb_8shot | 0.5 | 0.755 (NT-2.5b-1000g) | embedding |
Exact input columns and output formats for every dataset are in IO_SPEC.md. Test inputs
are in datasets/<id>/test_inputs.parquet. Where a validation split exists, it's in
datasets/<id>/val.parquet with labels: use it freely for model selection, it's
disjoint from the test set. The reference genome is genome/hg38.analysisSet.fa (hg38, 1-based
coordinates, chr-prefixed).
What you must deliver
final_model/ (in your working directory) containing:
predict.py, run aspython predict.py --inputs <datasets dir> --genome <hg38.fa> --out <dir>. It must write one prediction file per dataset into--out, in the format ofIO_SPEC.md(<id>.parquetwithrow_id, score, or<id>.npy).- All weights and code it needs. It runs with no network in a container with
Python 3.11, torch 2.5.1 (CUDA), transformers 4.46, accelerate, safetensors, peft,
einops, numpy, pandas, pyarrow, scipy, scikit-learn, pyfaidx, pysam, pyBigWig and
biopython, on 8×H100 for at most 2 hours (16 H100-hours). Put any other packages as
wheels in
final_model/wheels/. It may contain several models (for example one per modality); route insidepredict.py. method.md: what you built and why, your training data, and your validation results.training_manifest.json: every data source (URL or dataset id plus version), what you filtered and why, and hashes of your final training files.
Test your predict.py end to end (--inputs datasets --out /tmp/preds) well before the
deadline. Keep checkpoints as you go.
Rules
- Don't train, tune or select on any listed test dataset: no inputs, no labels, nothing derived from them.
- Don't build data around specific test items. That includes analogues enumerated from test molecules or scaffolds, mutants of test proteins beyond what their own training sources contain, and windows chosen to cover test variants. Matching a dataset's general domain, modality or difficulty is fine.
- Exclude every identifier in
bundle/exclusions/from training data, and apply the provided checker before training. - Write
training_manifest.json: every data source (URL or dataset id plus version), what was filtered and why, and the final training files' hashes. - Don't look up this benchmark's repository, dashboard, past runs or traces.
- Use only the model and evaluation endpoints you're given. No hosted LLM APIs from your training or inference code.
- Don't modify the evaluation harness. Your model is re-scored outside your container regardless.
Exclusion lists for every test set are in exclusions/. Check every training file with:
python integrity/check_overlap.py --exclusions exclusions --train <your files or folders>
The same checks, plus an audit of your full trace, run after your run. Runs that break the rules are disqualified.
Resources
- H100×8, 64 CPUs, 1024 GB RAM, 24 hours of wall clock. Plan to use the whole budget.
- The internet, for public data, papers and public model weights (Hugging Face etc.).
- No hosted LLM APIs from your code.