In this post:
Summary
Sequence-to-expression (S2E) models predict sequencing assays like RNA-seq that measure some part of the regulatory processing of DNA directly from a DNA sequence. We don’t have a standardised way to compare S2E-models for yeast which is the gap we fill with https://github.com/Tom-Ellis-Lab/yeast-seq2expression-benchmark.
In the first release, we benchmark two recent models, Shorkie and Yorzoi, across a set of 10 benchmark tasks measuring everything from single-nucleotide variants to bacterial chromosomes in yeast. One result from this benchmark is that models seem to capture promoter grammar in training conditions (growth on glucose) well but completely ignore the effect of inserting a fixed expression cassette at different locations of the genome or put differently assign too little importance to the flanking sequence context.
We hope you find this collection of tasks useful to build and benchmark your own models on and are very happy to help with any kind of questions or hands-on support with setting it up for you!
In the coming months, we will publish our main findings as a paper - stay tuned!
Why this benchmark exists
The primary motivation for this benchmark is that yeast is an industrially relevant organism, for which sequence-to-function models offer a route to understanding how changes in DNA sequence alter transcription. It remains unclear, however, how well current models perform and how far they generalise across different aspects of this problem. In the medium term, better S2E models would accelerate DNA design by serving as in-silico oracles for ML-generated sequences.
Yeast is also substantially easier to engineer than mammalian cell lines. This tractability has produced resources with no mammalian counterpart, such as cells carrying exogenous chromosomes or scrambled genomes. Probing S2E models on tasks that are simply unavailable in mammalian systems may therefore reveal general strengths and weaknesses of this model class.
What this benchmark measures
In the first release, we benchmark two recent models, Shorkie and Yorzoi, across a set of 10 benchmark tasks measuring a range of regulatory mechanisms.
Overview
| Benchmark | Description | Task | Primary metric |
|---|---|---|---|
| Caudal eQTL | single-nucleotide cis-eQTLs vs matched controls | binary classification | AUROC / AUPRC (mean ± SEM over 4 negative sets) |
| Kita eQTL | single-nucleotide cis-eQTLs (independent panel) | binary classification | AUROC / AUPRC |
| Rafi / deBoer MPRA (promoter) | ~71k random 80 bp promoters in a dual-reporter plasmid GPRA | regression (marginalized logSED) | Pearson / Spearman |
| Shalem MPRA (terminator) | designed 3′-end / terminator variants (cleavage & termination) | regression (marginalized logSED) | Pearson r (Spearman alongside) |
| Chen synonymous MPRA | synonymous-codon effect on mRNA abundance | regression, 3 libraries | Pearson r + Spearman ρ on log2(R/D) |
| Wu RFP insertions | fixed RFP cassette across ORF-deletion loci (position effect) | regression | Pearson r + Spearman ρ (+ tail AUROC) |
| Hong IGR insertions | fixed reporter across intergenic loci (position effect) | regression | Spearman ρ on IntProp |
| Brooks SCRaMBLE | SCRaMBLE rearrangement (altered neighbour / downstream context) | coverage-track LFC | direction balanced accuracy, then Spearman / Pearson |
| Cuperus 5′-UTR | random 50 bp 5′-UTRs (Kozak, uORFs, structure) | regression (HIS3 reporter) | Spearman/Pearson + partial-corr over Kozak features |
| Meneu foreign DNA | whole bacterial chromosomes integrated in yeast (far-OOD sequence) | zero-shot coverage-track prediction | per-window Pearson + JS divergence (+ fold-change error) |
Feel free to reach out at [email protected]