The Bitter Lesson for Biology

TL;DR: We’ve demonstrated the first general scaling laws for biomolecular interactions, across a wide variety of assays, and achieved SOTA performance on binding affinity. This was made possible because of a relentless focus on data, and re-learning the bitter lesson after deviating from it.

Two years, in one meme

How we ended up back at “just use an LLM”

The bitter lesson

The bitter lesson states that general methods that leverage computation are ultimately the most effective, and by a large margin. In the modern compute regime, this is often used in the context of scaling laws - which state how scaling compute often leads to predictable gains.

Why do we believe in the bitter lesson? First, let us concede that the bitter lesson is an empirical rule.

Now, the short answer is that the bitter lesson keeps showing up across very different fields predictably. Time and time again, general methods that scale with computation, such as search and learning, have beaten methods built on human expertise about the problem. This may very well be because most human knowledge is brittle, especially in biology and chemistry.

The bitter lesson is bitter because it forces us to abide by a level of intellectual humility that we don’t want to accept. Progress in most fields has come from accepting that methods that use computation, when persistently scaled, outperforms our best attempts to hard-code intelligence, or model the physical world.

Why biology looked like the exception

Why should biology be any different? One intuition behind why biology might be different is that people think that it’s inherently harder in some relative sense compared to math, or coding, which are seen as more “verifiable domains”. While the specifics about the difficulty of a field are up for debate, this claim fails because human intuition about what’s hard to automate is a bad guide. Chess is objectively very difficult for most humans to play, however, with search and computation, DeepBlue was able to defeat Garry Kasparov.^[1]

Another reason why biology may very well be different is that historically, scaling laws for biology have just not worked! ESM-2’s scaling laws famously reversed when tested on Protein-Gym, Evo 2 was trained on about 30x more tokens than GPT-3, yet barely beats legacy computational tools at predicting assays. Researchers at GHDDI and Tsinghua found that compact, task-specific models usually match or beat large molecular models.^[2]

However, we have come to realize that the reason scaling laws haven’t worked very well is not because of some deep underlying ontology that biology possesses, but because of the kind of data most people collect and how they train on it. Public binding data is noisy and stripped of the assay context behind each measurement, and most models are built to ignore that context anyway.

We have spent the last two years trying to scale these models for general molecular discovery. This blog details the bitter lessons learnt after two years of modeling wet-lab assays, both accurately and inaccurately.

Protein–ligand binding affinity

Bindwell began by predicting protein-ligand binding affinity.

Protein-ligand binding affinity prediction (PLBA) is a problem at the core of drug-discovery. It deals with estimating how strongly a small-molecule (ligand) interacts with a target protein. PLBA is extremely important in drug-discovery, since it allows medicinal chemists to quickly identify initial hits. However, finding hits in a library is often a very time and cost intensive process, requiring expensive calibration of biochemical assays in a wet-lab environment. Thus, scientists often want to try and estimate the binding affinity of a small-molecule/ligand through computational methods, either via FEP (Free Energy Perturbations) and MD simulations, through docking (AutoDock VINA), or via ML methods.

Our early work in protein-ligand binding affinity included using embeddings from pre-trained protein and ligand transformers, attaching an MLP, and training a model on publicly available datasets such as BindingDB, CheMBL, Pubchem, PDBBind. We aggregated these datasets, normalized across different assays such as IC50, Ki, Kd, EC50, and curated a dataset that totaled to around ~2 million rows. Our initial results were not bad! We were one of the first people to realize that pre-trained transformers could defeat the human inductive biases that people baked into their models, such as SE3-equivariant GNNs or structure-based prediction models. Using our small pretrained transformers, we got some pretty encouraging in-silico results, but it was time to validate the model in the real world, in a wet lab.

Wet lab

We started by making a wet-lab in Santa Clara, at BioCurious, a community biolab. We had access to a couple of small benches, and we hired a full time lab technician who conducted assays. Using our small pre-trained transformer based model, we were actually able to get in-vitro hits for our target insect, Spodoptera Frugiperda.

It was while we were building the lab out that we realized that biology is fucking hard. The result of a binding assay depends on much more than the protein and the molecule. The temperature, the pH, the readout chemical, the number of replicates, the model used to fit the curve, the software used to fit, the method of extraction of the protein from the insect. Repeating the same experiment also gave different answers, because small differences in pipetting or incubation time were enough to make IC50 fits noisy.

We quickly realized that we needed to account for the variance that comes with biology, and that current datasets were insufficient.

An open secret

The open secret about binding affinity is that publicly available data that is used to train ML models is terrible. As stated before, most people use a variation of BindingDB, ChEMBL, Pubchem and PDBBind. The measurements in these datasets come from thousands of labs, each running its own assay with its own protein construct, substrate concentration, incubation time and readout. However, the sad reality is that none of this is readily documented in the datasets. As a result, one can observe very different results for the same protein-ligand pair, in the same dataset. The noisy nature of these datasets is well documented in the literature. When Kramer et al. searched ChEMBL for independent repeat measurements, 90% of the apparent repeat pairs turned out to be one measurement cited again. Leash Bio estimates that BindingDB has roughly 1.3M interactions, and that cleaning them up leaves only about 400K. But my personal favorite paper is by Landrum and Riniker, with the best figure I’ve ever seen in a journal.

Figure 4

Same compound, same target, two assays: R² ≈ 0.31

Each point is one compound measured against the same target in two ChEMBL assays, as pChEMBL on both axes. From Landrum & Riniker, “Combining IC50 or Ki Values from Different Sources Is a Source of Significant Noise” (J. Chem. Inf. Model., 2024).

Enlarge

Roughly 70% of the variance comes from how and where the assay was run, not from the compound or the target. That is the information assay metadata has to capture.

So we paid for better data and licensed an expert-curated dataset from a CRO (a contract research organization). We come back to that data later. First, the model.

Iterating on the models

It was at this time that I realized that the final form factor of our model should probably not include “just” protein and chemical embeddings, and should instead rely on more ‘nuanced’ features that codified human understanding.

Building in what chemists know

Binding affinity is an energy. When a molecule settles into a protein, the system gives up free energy, and the dissociation constant you read in a paper is energy on a log scale (ΔG = RT ln K_d). So in order to answer “what affects binding affinity?”, we really need to answer “what adds or removes energy when these two meet?” The textbook answer comes down to four things.

1. Shape. A molecule that fills the protein’s pocket snugly makes more close contacts, which is why size alone predicts affinity fairly well.
2. Specific contacts. Hydrogen bonds, salt bridges and stacked aromatic rings add directional pull; one well-placed hydrogen bond can tighten binding several-fold.
3. Flexibility. Every rotatable bond frozen on binding costs entropy, so a rigid molecule already shaped like the pocket beats a floppy one.
4. Charge. Protonation and tautomer state, both set by pH, change every interaction above.

We started with RDKit, the standard open-source chemistry toolkit. Our first feature set was a 128-bit Morgan fingerprint plus 18 descriptors that map almost one-to-one onto the list above:

• molecular weight and heavy-atom count for size;
• logP for the hydrophobic effect;
• polar surface area and hydrogen-bond donor and acceptor counts for polar contacts and desolvation;
• rotatable bonds for flexibility;
• formal charge for electrostatics.

Then we went bigger: a 2,048-bit fingerprint and all 217 of RDKit’s 2D descriptors. Then bigger again: we generated a 3D conformation of every molecule and added 912 descriptors of its shape and how its charge is spread through space. That came to 3,178 numbers per molecule, squeezed into one token the model could attend to. We also added multi-view encoders with chemical features, since SELFIES and SMILES represented different “views” of the same underlying molecular modality.

The frankenstein model

At its peak, our model had multiple towers: ESM-2 for the protein, ChemBERTa and SELFIES-TED for the ligand, Qwen3-Embedding-8B for the assay text, a projection of the RDKit features, a projection of structural features from Boltz-2 and ESMFold-2, and then docking features from SAIR, such as AEV-PLIG. This was truly a frankenstein model, one that I personally worked very hard on!

This was a severe violation of the bitter lesson, but we just continued adding these features that we thought correlated with affinity, because the model kept on getting better at predicting binding affinity on our held out set.

Running perturbation probes

We were curious as to what the most important parts of our frankenstein model were; so I ended up running inference probes. For each tower, we took a trained checkpoint, and perturbed inputs of that modality at inference time.

We started off with the protein sequence, and moved our way to the ligand, the RDKit features, the assay text, and checked if the model’s performance was recoverable. The hypothesis here was that if the tower mattered, predictions should have fallen apart, but rather surprisingly, for most of the encoders, they did not.

Scrambling the protein left the predictions almost unchanged. Scrambling the ligand moved them a little. Scrambling the RDKit features, on the best of these models, moved them not at all. But scrambling the assay text wiped the prediction out.^[3]

Fang et al. later found the same thing in biological reasoning models. They ran the same test as our probes, shuffling the DNA or protein input each model is given, and BioReason and BioReason-Pro scored no worse with it shuffled.

Text alone wins

It was… a bitter revelation, but towards the end, the best architecture was a single large pretrained transformer with a linear layer. We started with Gemma 4. The protein sequence, the ligand, and the assay metadata all went in as one piece of text, and the separate ESM-2, ESMFold, ChemBERTa, RDKit, and structure towers came off. On our 10k validation set that model reached a mean squared error of 0.629, ahead of the same Gemma with the encoder towers still attached (0.652) and ahead of the earlier multi-tower models (0.757 on that set). A LoRA of it, trained on 500,000 rows, reached 0.551.

Licensing CRO data

Once the model read everything as text, the text itself became the bottleneck. So we took a hard look at the CRO data we had licensed. Despite the data being expert curated, we felt there was still a lot of value left on the table that we could get from the $200,000 a year we were spending on the license.

The reasons for this were twofold. First, a lot of the data was just quite bad. The dataset was riddled with missing data and unit slips such as converting nanomolars to micromolars.

Figure 9

Unit errors, checked against the paper

We checked the CRO’s values against the papers they were curated from. Select a row to plot it; the shaded band is more concentrated than pure water (55.5 M).

Off by a millionTopo I inhibitors Borrowed the wrong column’s unitAkt inhibitors Every header says nMPPARγ/δ agonists Even the abstract says nMCCR5 antagonists Units switch mid-paperABBV-744

Naphthazarin derivatives (IV): inhibition of DNA topoisomerase I and cytotoxicity of acyl-dimethoxy-naphthoquinones

European Journal of Medicinal Chemistry, 2000

49 / 49CRO values in the wrong unit

Table I reports both columns in µM. The CRO data stored all 49 values in molar, a million times too high. Five compounds come out at “> 200 M”.

Compound · measurement	Paper	CRO data
Octanoyl, 2-isomerTopo I IC50	189 µM	189 M1,000,000×
CamptothecinTopo I IC50	6 µM	6 M1,000,000×
Decanoyl, 2-isomerTopo I IC50	> 200 µM	> 200 M1,000,000×
Acetyl, 2-isomerL1210 ED50	0.42 µM	0.42 M1,000,000×
Heptanoyl, 6-isomerL1210 ED50	0.11 µM	0.11 M1,000,000×

Table I as printed: both columns are labelled µM.

Design and synthesis of pyridine–pyrazolopyridine-based inhibitors of protein kinase B/Akt

Bioorganic & Medicinal Chemistry, 2007

11 / 12CRO values in the wrong unit

The enzyme columns are in nM; the cell-viability column beside them is in µM. The CRO data applied nM to 11 of the 12 cell values, so the compounds look 1,000× more potent in cells than reported. Two of them also have the wrong digits.

Compound · measurement	Paper	CRO data
9gMiaPaCa-2 MTT EC50	0.041 µM	0.041 nM1,000×
1MiaPaCa-2 MTT EC50	0.82 µM	0.82 µMmatches
2MiaPaCa-2 MTT EC50	0.38 µM	0.38 nM1,000×
25bMiaPaCa-2 MTT EC50	1.2 µM	1.4 nMunit + digits
25dMiaPaCa-2 MTT EC50	0.584 µM	0.548 nMunit + digits

Table 1: Akt1 and PKA in nM, the MTT column in µM.

Table 2: same layout. 25b is 1.2 and 25d is 0.584; the CRO data has 1.4 and 0.548.

Design and synthesis of a novel class of dual PPARγ/δ agonists

Bioorganic & Medicinal Chemistry Letters, 2007

90 / 90CRO values in the wrong unit

All six columns of Table 1 are in nM. The CRO data stored all 90 values in µM, which turns a 6 nM PPARδ agonist into a 6 µM one.

Compound · measurement	Paper	CRO data
21PPARδ transactivation EC50	6 nM	6 µM1,000×
1PPARγ binding IC50	5 nM	5 µM1,000×
1PPARα binding IC50	4,257 nM	4,257 µM1,000×
17PPARα binding IC50	10,400 nM	10,400 µM1,000×
RosiglitazonePPARγ transactivation EC50	308 nM	308 µM1,000×

Table 1: all six columns are labelled nM.

Orally active CCR5 antagonists as anti-HIV-1 agents 2: anilide derivatives containing a pyridine N-oxide moiety

Chemical & Pharmaceutical Bulletin, 2004

26 / 30CRO values in the wrong unit

Tables 2 and 3 are in nM, and the abstract quotes the lead compound at 7.2 nM. The CRO data stored those 26 values in µM. Table 1 really is in µM, and the CRO data got those 4 right.

Compound · measurement	Paper	CRO data
(S)-5sCCR5 binding IC50	7.2 nM	7.2 µM1,000×
5aCCR5 binding IC50	0.4 µM	0.4 µMmatches
1HIV-1 fusion IC50	1.4 nM	1.4 µM1,000×
(S)-5hCCR5 binding IC50	15 nM	15 µM1,000×
(R)-5fCCR5 binding IC50	3,600 nM	3,600 µM1,000×

The abstract: “IC50 = 7.2 nM”.

Table 3: both columns are labelled nM.

Discovery of ABBV-744, a BET bromodomain inhibitor with selectivity for the second bromodomain

Journal of Medicinal Chemistry, 2020

96 / 104CRO values in the wrong unit

Table 2 reports the cellular IC50s in µM; from Table 3 on, the same columns switch to nM. The CRO data kept µM throughout, so 96 of its 104 cellular IC50s for this paper are 1,000× too weak.

Compound · measurement	Paper	CRO data
33NCI-H1299 IC50 (Table 3)	730 nM	730 µM1,000×
17SKM-1 IC50 (Table 2)	0.01 µM	0.01 µMmatches
18NCI-H1299 IC50 (Table 2)	0.37 µM	0.37 µMmatches
21SKM-1 IC50 (Table 3)	3.3 nM	3.3 µM1,000×
28NCI-H1299 IC50 (Table 3)	3,200 nM	3,200 µM1,000×

Table 2: the cellular IC50 columns are in µM.

Table 3: the same columns switch to nM.

Next, the metadata for each assay was severely lacking. There was a lot of experimental context in the sources that the CRO simply failed to include in its metadata.

Figure 10

Same number, missing context

A measurement from the XY101 paper (J. Med. Chem., 2019). The CRO record has the right value; highlights mark what the paper reports that the record leaves out.

Gal4-RORα reporter, IC50 of the reference ligand T0901317. The CRO record and the paper agree: 2.6 µM. The paper prints “2.6 (55)”: the curve tops out at 55% inhibition.

CRO record

Target

Gal4-RORalpha

Cells

HEK 293T

Assay description

Inhibitory concentration of compound against GAL4-RORalpha LBD (25 ng) transfected in HEK 293T cells upon incubation for 24 hrs by LUCIFERASE ASSAY

Incubation

24 Hr

What the paper reports

The IC50 is 2.6 µM, with a maximum inhibition of 55%, so it is a partial inhibitor and the IC50 isn’t comparable to a full inhibitor’s.

A Gal4–RORα LBD dual-luciferase reporter in 293T cells, normalized to Renilla, with pG5-luc and Renilla plasmids at 60 ng and 2 ng per well. Run in triplicate.

Table 3: compound 1 at RORα reads “2.6 (55)”, i.e. 55% maximal inhibition.

Datawell

By now we had learned three things: text alone was likely the way to go, our so-called “expert labeled” CRO data fell short of what its sources actually contained, and its 10 million rows were not going to be enough. Spread across every target, compound and assay type, ten million rows is thin, and to find out whether a text model scales we needed an order of magnitude more data, with far richer descriptions of each assay than the CRO recorded. So we decided to extract our own assay data from public sources. That meant defining exactly what assay context and metadata we wanted from each source, finding the most cost-effective way to extract it, and running that extraction at scale on everything we could scrape from the Internet.

What data should look like

We wanted to teach the models to take everything about the assay into account, not just the protein and the ligand. We wanted our model to reason in assay space, not necessarily just protein or chemical space. An assay with less information about its conditions should, in theory, cause greater uncertainty in the model and thus lead to higher variance predictions. Unfortunately, this type of data does not nicely exist in extracted form at a large enough quantity to be useful for us.

So we decided to generate it ourselves with the goal of harvesting as detailed of descriptions that we could get for all assays that were relevant to us. The data was there, sitting in papers and patents, it was just a matter of getting it all extracted. And we didn’t think the task was hard enough to warrant the need for human experts to curate it for us. This was doubly useful as LLMs are both more scalable and cheaper than human data extractors. We were also able to fix the unit issues and data misses with better prompting leading to our data being more descriptive and less wrong than the human labeled CRO data.

With an infinite budget, we would ordinarily throw the best contemporary model at the task and call it a day. But given the fact that we were targeting around 200M data points, it was infeasible to use the best and most expensive models there were at the time. We needed something that could produce data at a much higher fidelity while being orders of magnitude cheaper. This meant finding the Pareto frontier of cost vs data quality and choosing the best model that would fit in our budget, which meant we needed an eval.

The eval

We framed extraction as a single-turn task: one model call per source, returning the assay data in a fixed structure.^[4] Our eval then had to answer one simple question: how well could a model extract the relevant assay data from a source?

We selected a collection of diverse and relevant sources and created rubrics for good extractions. Those rubrics would then be fed to an LLM to judge extractions from other models.

We observed strong inference time scaling results for models on this task and saw a clear relationship between cost and data extraction quality.

Datawell vs CRO data

All of this only matters if the model trained on the data gets better, and it did. We extracted data from the same sources as the CRO dataset and trained the same model on each. On the CRO data, validation loss bottoms out at 160k examples and then climbs as the model overfits. On Datawell it keeps falling, ending 28% lower, and for the first time we didn’t see severe overfitting.

Picking the validation set is the hard part of this comparison. If we scored both models against Datawell’s labels, we’d be grading our own homework; against the CRO’s, we’d be grading theirs. So we kept only measurements both sources agree on. We started from papers that both had extracted, and matched individual measurements across the two by paper, exact protein sequence, ligand and assay type (IC50, Ki, Kd…). That gave 10,972 measurements that each source had extracted independently. We kept the 8,566 where both reported exactly the same value. When two independent extractions of the same table land on the same number, that number is almost certainly what the paper says.

Scaling laws

So, after all of that, does it scale?

Yes! With clean data and a fixed recipe, our general model, Atomwell, gets better with size on every assay we can measure well enough to tell.

We fine-tuned four Qwen3.5 Base models, at 0.8B, 2B, 4B and 9B parameters (with 27B running right now as we speak), on 1 epoch of 76 million Datawell measurements. Our data mixture spans 13 kinds of physical quantities: potencies in molar, temperatures, half-lives, particle sizes, percentages and more.

We scored every model on our strict, out-of-distribution validation set. It holds 539,709 measurements from papers and patents we held out entirely, with no exact overlap with training in sources, molecules, targets, assay text or authors.^[5]

We report loss as the fraction of variance unexplained (FVU): the model’s mean squared error divided by the variance of the measured values, or 1 − R². Because our data mixes 13 kinds of physical quantity, it puts every assay on the same scale. A value of 0 is a perfect prediction, and 1 is no better than always predicting the mean. At 0.272, our 9B model explains about 73% of the variance in measurements from papers and patents it has never seen.

Comparison to Boltz-2

Scaling laws only matter if the model they describe is good, so we compared Atomwell against two public binding-affinity models on the same held-out data: Boltz-2, which predicts affinity from a co-folded 3D complex, and Nesso-1. We scored all three on the 1,885 IC50 measurements in our strict validation set that have exact values. Figure 2, repeated below, shows the result.

Atomwell wins on every metric. Its predictions correlate with the measured pIC50 at r = 0.681, against 0.554 for Nesso-1 and 0.429 for Boltz-2. It explains 43% of the variance (R² = 0.434), where Boltz-2 explains about 3%. Its mean squared error is 0.885 pIC50², 41% lower than Boltz-2’s 1.512 and 26% lower than Nesso-1’s 1.194. None of the three models was trained on these papers or patents, and Atomwell works from text alone, with no 3D structure.

References

1. Sutton, R. The Bitter Lesson (2019).
2. IBM. Deep Blue.
3. Moravec, H. Mind Children (Harvard University Press, 1988); see Moravec’s paradox.
4. Hou et al. Nature Computational Science (2026): protein language model performance on fitness prediction declines beyond a certain size.
5. Brixi et al. Evo 2. Nature (2026).
6. Brown et al. Language Models are Few-Shot Learners (GPT-3, 2020).
7. Guo & Ding. Do Larger Models Really Win in Drug Discovery? (2026).
8. Kramer, Kalliokoski, Gedeck & Vulpetti. The experimental uncertainty of heterogeneous public Ki data. J. Med. Chem. (2012).
9. Leash Bio. We need more, and it needs to be better (2024).
10. Landrum & Riniker. Combining IC50 or Ki Values from Different Sources Is a Source of Significant Noise. J. Chem. Inf. Model. (2024).
11. Fang et al. When Do Biological Reasoning Models Use Their Biological Inputs? (2026).
12. Datasets: BindingDB, ChEMBL, PubChem, PDBbind.
13. Trott & Olson. AutoDock Vina. J. Comput. Chem. (2010).
14. RDKit; Rogers & Hahn. Extended-Connectivity Fingerprints. J. Chem. Inf. Model. (2010).
15. Krenn et al. SELFIES. Mach. Learn.: Sci. Technol. (2020).
16. Lin et al. ESM-2 and ESMFold. Science (2023).
17. Ahmad et al. ChemBERTa-2 (2022); IBM Research, SELFIES-TED.
18. Zhang et al. Qwen3 Embedding (2025); Qwen3.5 (2026).
19. Passaro et al. Boltz-2 (2025).
20. Lemos et al. SAIR: Structurally Augmented IC50 Repository (2025).
21. Valsson et al. AEV-PLIG. Communications Chemistry (2025).
22. Google DeepMind. Gemma 4.
23. Hu et al. LoRA (2021).
24. Recursion / Valence Labs. Nesso-1 (2026).