Bacterial Gram-staining from genome, family-holdout blind eval

Bacterial genomes10 h budgetheadline: AUPRC
About this task

Predict Gram-staining status (positive vs. negative) for held-out bacterial families from genome assembly alone. Agent gets an NCBI assembly accession + phylum for each strain and — for the training strains — a Gram label. Nothing more granular than phylum is shipped (no genus, no species, no strain name), so the agent must extract features from the actual genome sequence rather than looking the answer up by name. Test strains come from families not represented in the training set.

Feature choice is on the agent: gene predictions, protein-family counts, KEGG/COG abundances, genome FM embeddings (GenomeOcean, Evo 2, Nucleotide Transformer), lipopolysaccharide biosynthesis-pathway hits, tRNA/rRNA counts, whole-genome k-mer profiles — anything extractable from the assembly.

Leaderboard

Ranked by AUPRC. Dataset cells show raw metric values, shaded by rank among the runs shown (darker = better); hover for CIs and the dataset's SOTA.

#RunIntegrity
1Claude Code · highinf.20260719T113801 · claude-opus-4-8[1m]0.9950.996Pending$3.10Jul 19, 2026
Refclass_prior (always Gram-)—0.500——
Refcodon-usage_+_tetranucleotide—0.850——
SOTAprotein-family-profile_classifier—0.950——

≈ approximate reference value. “inf.” = agent / effort inferred from the run id or trace (no run_meta.json).

Per-dataset results

Every run on every dataset, against chance, the reference baseline and SOTA. Filters above apply.

agent runbest runbaselineSOTAreferencechance95% CI privateRaw metric values; each row has its own axis.
gram family holdout AUROC0.471.03

Suite datasets

From suite.yaml: each dataset's metric, chance level, reference and SOTA (raw values).

DatasetVisibilityMetricChanceReferenceSOTASource
gram family holdoutpublicAUROC0.5000.850 codon-usage_+_tetranucleotide0.950 protein-family-profile_classifierMadin et al. 2020 (BacDive-derived), family-disjoint test

Prompt

Exactly what the agent receives.

Show the full prompt

BioModelBench — Bacterial Gram-staining (bacdive_gram)

Task

For each row in test.parquet, predict gram_prob ∈ [0, 1]: the probability the strain is Gram-positive. The label is binary (Gram-positive = 1, Gram-negative = 0).

Test strains are drawn from bacterial families not present in the training set. You have the assembly accession + phylum for each strain — that's it. No genus, no species, no strain name.

What's shipped

train.parquet

Columns:

  • assembly_accession (string, e.g. GCF_000005845.2 or GCA_…) — NCBI assembly accession.
  • phylum (string) — coarse taxonomy.
  • label (int, 0/1) — 1 = Gram-positive, 0 = Gram-negative.

test.parquet

Columns: assembly_accession, phylum. No label.

Compute

Modal container: A10G GPU (24 GB VRAM), 8 CPUs, 96 GB RAM. Python 3.11 with torch / transformers / peft / xgboost / lightgbm / scikit-learn / pandas / pyarrow / numpy / pyBigWig / pysam / biopython / requests pre-installed. pip install <x> works.

Budget: ~10 hours of GPU compute. Use it. Even after you produce a valid answer.parquet, keep iterating until the budget is exhausted. The runner will relaunch your session with a continuation prompt if you exit early — don't finalize prematurely.

Deliverables

  • answer.parquet — columns assembly_accession, gram_prob, one row per test strain, gram_prob ∈ [0, 1].
  • method.md (< 500 words).
  • training_manifest.json — every URL / API / dataset touched.

What the grader does

Reads your answer.parquet, joins on assembly_accession against the hidden Gram-staining labels, computes AUPRC / AUROC / Brier / coverage.

Reference bar

Test-set base rate: ~0.34 Gram-positive. Published Gram predictors on family-holdout land around AUPRC 0.7–0.9. Exact reference-baseline bar will be baked into grade.py before this task is treated as production.