Machine-Learned Peptide Property Prediction: What an AUC Licenses

A binary classifier trained on a public antimicrobial peptide set will report a high area under the ROC curve on its own benchmark, and it will return a score for any string of amino acids you hand it, synthesised or not. Those two facts are less connected than they look. The benchmark number describes how one particular collection of sequences was ordered under one particular way of choosing the negatives; the score on your designed sequence is an extrapolation whose error that benchmark never measured. Most of the work here is keeping the two apart.

The cleanest demonstration of the gap comes from varying the negatives and nothing else. Generating 660 predictive models from 12 machine learning architectures, a single positive dataset and 11 negative data sampling methods — architectures and methods defined on the basis of published AMP prediction software — the authors found that a similar training and benchmark dataset, produced by the same or a similar negative data sampling method, positively affects model performance, and concluded that all the benchmark analyses performed for AMP prediction models are significantly biased and that we do not know which model is the most accurate ^[1]. A reported accuracy is, in part, a measure of how somebody assembled the sequences the model was told were inactive.

What a representation can and cannot carry

One-hot encoding is the null option: an L×20 binary matrix, one column per position. It assumes nothing, which is both the point and the cost. Every pair of amino acids sits at the same distance, so a Lys→Arg substitution and a Lys→Asp substitution are the same size of change until the model learns otherwise — and it learns that only from your labels, however many of them you have. A 12-residue peptide becomes 240 binary features, and the chemistry of the alphabet has to be rediscovered from the data in front of you.

Physicochemical descriptors run the other way and impose the chemistry up front. Net charge at a stated pH, mean hydrophobicity, aliphatic index and hydrophobic moment collapse a sequence into a handful of numbers whose meaning is fixed before any fitting happens. The hydrophobic moment is the instructive case. Plotted against mean hydrophobicity, its magnitude measures the amphiphilicity of a helix and hence that helix's tendency to seek a surface between hydrophobic and hydrophilic phases, while the hydrophobicity axis measures affinity for the membrane interior; on that plot, helices from surface-seeking proteins such as melittin fall in the region of still higher hydrophobic moment than paired or bundled transmembrane helices, which in turn sit above monomeric transmembrane anchors ^[2]. Descriptor calculation of this kind, with retrieval of sequences from public or local sequence databases and instant access to precompiled datasets for machine learning, is what a peptide-specific package such as modlAMP supplies ^[3].

The cost of a global descriptor is that it is global. Net charge and mean hydrophobicity are invariant to permutation, so two sequences of identical composition and different residue order take identical values. The hydrophobic moment is not permutation-invariant, but it is defined on an assumed helical periodicity; computed for a sequence that never forms a helix, it is a number about a conformation the peptide does not adopt.

Learned embeddings sit between the two. Unsupervised training of a deep contextual language model on 86 billion amino acids across 250 million protein sequences produced a representation space with a multiscale organisation running from the biochemical properties of individual amino acids up to remote homology between proteins, with information about secondary and tertiary structure encoded such that linear projections can recover it ^[4]. A parallel effort trained autoregressive and autoencoder architectures on UniRef and BFD data containing up to 393 billion amino acids; the raw embeddings from unlabelled data captured some biophysical features of protein sequences, and used as the exclusive input they supported per-protein predictions of subcellular location at ten-state accuracy 81% and of membrane versus water-soluble at two-state accuracy 91%, and per-residue secondary structure at three-state accuracy 81–87% ^[5]. Scaled to 15 billion parameters, an atomic-resolution picture of protein structure emerges in the learned representations ^[6].

Every one of those figures was measured on proteins. The corpora are protein sequence databases and the validation tasks are protein tasks — secondary structure, subcellular localisation, remote homology, mutational effect. A 12- to 30-residue synthetic peptide, perhaps C-terminally amidated, perhaps carrying a D-residue or a non-proteinogenic side chain, is not drawn from that distribution, and the tokenizer has no symbol for most of what you might build into it. An embedding still comes back; its nearest neighbours in that space are short fragments of larger proteins, which is not the same object as a free peptide in buffer.

The data problem is most of the problem

Start with the label. A minimum inhibitory concentration is the lowest concentration of the assayed antimicrobial agent that, under defined test conditions, inhibits the visible growth of the bacterium being investigated, with growth assessed after incubation for a defined period of 16–20 h; the standard agar and broth dilution protocols apply only to aerobic bacteria ^[7]. Under defined test conditions is carrying the weight. A public activity table aggregates values generated across laboratories, media, inocula, strains and plate materials, and a regression is then asked to fit a single number against sequence as though the assay were a constant. Whatever share of the spread is assay provenance rather than sequence, the model has no way to book it anywhere except sequence.

Then the negatives. Positive entries exist because someone published an activity; negative entries are constructed, typically by sampling sequences that were never assayed at all — from proteomes, at matched length, or by shuffling. That construction is exactly the variable the 660-model study manipulated, and it moved performance ^[1]. It also means the boundary a classifier learns may be separating peptide taken from an activity database from random proteome fragment, a distinction with a composition-level shortcut and no necessary connection to membrane activity.

Then the class ratio. Curated benchmarks are usually balanced by construction; the library you will actually score is not. On strongly imbalanced datasets the visual interpretability of ROC plots can be deceptive with respect to conclusions about the reliability of classification performance, owing to an intuitive but wrong interpretation of specificity, whereas precision-recall plots evaluate the fraction of true positives among positive predictions ^[8]. That is a general result about binary classifier evaluation rather than a peptide finding, and it bites when a model trained near 1:1 is pointed at a design space where actives are rare: what you care about is the fraction of your shortlist worth synthesising, which is the precision-recall axis.

Random splits leak, and the leak is sequence similarity

Leakage is not a peptide pathology. A systematic survey of ML-based science found 17 fields where leakage has been found, collectively affecting 294 papers and, in some cases, leading to wildly overoptimistic conclusions, and organised the failure into a taxonomy of eight types; in the authors' own reproducibility study of civil war prediction, once the errors were corrected the complex ML models did not perform substantively better than decades-old logistic regression ^[9].

For sequence data the mechanism is homology. When splitting biological sequence data for the development and testing of predictive models it is necessary to avoid too-closely related pairs of sequences ending up in different partitions, and if this is ignored, performance of prediction methods will tend to be overestimated ^[10]. Activity databases are full of families — one parent scaffold with single substitutions, truncations, chain-length variants and species orthologues — so a random split will routinely put a peptide in training and its own alanine-scan variant in test. The model then reports how well it can look up an answer it has already been given. The fix is to cluster at a declared identity threshold and split at the level of clusters rather than sequences; a partitioning algorithm that keeps closely related sequences in the same partition can retain a larger number of sequences per dataset than homology reduction while providing homology separation on a par with reduction approaches, as evaluated on protein, DNA and RNA datasets ^[10].

Similarity among the actives is only half of it. AVE, a measure of training-validation redundancy for ligand-based classification that accounts for the similarity among inactive molecules as well as active ones, was found across seven widely used benchmarks to correlate strongly with the performance of ligand-based predictive methods, irrespective of the predicted property, chemical fingerprint, similarity measure, or previously applied unbiasing techniques; the authors put it that it may be the case that the previously reported performance of most ligand-based methods can be explained by overfitting to benchmarks rather than good prospective accuracy ^[11]. That work was done on small-molecule ligand sets, not peptides, so no number carries across. The structure of the argument does, and it matches the negative-sampling result: a benchmark can be easy because of where its inactives sit relative to its actives, and cross-validation inside that benchmark will not reveal it.

What an AUC on a curated benchmark licenses

An AUC is a ranking statistic. It estimates the probability that a randomly drawn positive scores above a randomly drawn negative, inside the test set you built. It is threshold-free, so it does not tell you where to cut. It is computed at a class ratio you chose. And given the two results above, part of what it measures is your split and your negative set. It licenses a comparison between two models evaluated on that identical benchmark under that identical split. It does not license a calibrated probability that one specific novel sequence has the property, and it says nothing at all about how far the model can be pushed from its training data before the ordering degrades.

That last question is measurable, and it is not settled. A panel of deep learning uncertainty quantification methods benchmarked on regression tasks from the FLIP protein fitness benchmark, compared across different degrees of distributional shift on metrics for accuracy, calibration, coverage, width and rank correlation, and run with both one-hot encoding and pretrained language model representations, gave no single best UQ method across all datasets, splits and metrics, and found that uncertainty-based sampling is often unable to outperform greedy sampling in Bayesian optimisation ^[12]. Those are protein fitness datasets rather than peptide activity sets. The lesson to import is the shape of the finding: even where uncertainty is modelled deliberately, the estimate you get depends on the dataset, the split and the metric you grade it with.

Applicability domain: ship the distance with the score

QSAR practice formalised this decades ago. One of the OECD principles for model validation requires defining the applicability domain, because reliable predictions are generally limited to query chemicals structurally similar to the training compounds used to build the model; descriptor-based approaches characterise that interpolation space in several different ways, and the thresholds adopted contribute significantly to what counts as an extrapolation ^[13]. The descriptors differ for a sequence model, but the obligation does not.

In practice this means reporting two numbers rather than one: the score, and a distance from the candidate to the training set measured in the space the model actually consumes. Maximum pairwise identity to any training sequence is the coarse version, and it is the same quantity your split threshold was set on. Better is a nearest-neighbour distance in the model's own feature space — in the descriptor vector, or in the pooled embedding — together with a per-descriptor range check. If a candidate carries net charge +9 and the training sequences span −2 to +6, the model has seen no evidence at that charge and is projecting a trend rather than interpolating between measurements. A score without a distance cannot be read, and a de novo designed sequence is by construction the case where that distance is largest.

Ranking for the bench is a different claim from predicting

The defensible use of these models is ordering a synthesis list, and two published campaigns bracket what that buys. Machine learning applied across 63,410 metagenomes and 87,920 prokaryotic genomes produced a catalogue of 863,498 non-redundant peptides, few of which match existing databases; of 100 candidates synthesised and tested, 79 were active ^[14]. Separately, recurrent neural networks trained on data from an antimicrobial activity and structure database to design short non-haemolytic peptides yielded 28 generated sequences, each at least 5 mutations away from the training data, of which synthesis and testing identified eight as new non-haemolytic antimicrobial peptides ^[15]. Those two fractions are not a controlled comparison — different organisms, different criteria for a hit, different provenance — but they mark opposite ends of the axis this article is about: the first campaign scored natural sequences from the kind of distribution the model was built on, the second stepped a fixed minimum distance away from it.

The second study also names the multi-property trap directly: models have successfully identified membrane-disruptive amphiphilic peptides, however mostly without addressing the associated toxicity to human red blood cells ^[15]. A single-property classifier tuned for a high AUC will push the most cationic, most amphiphilic sequences to the top of the list, which is the same direction as the liability. Running two models does not fix this by itself — you then need two scores and two applicability domains, and the intersection of two extrapolations is no more trustworthy than either one.

Framed as a search accelerator rather than an oracle, sequence representations earn their place. Simple models built on top of an unsupervised sequence representation were reported to generalise to unseen regions of sequence space, to predict the stability of natural and de novo designed proteins and the quantitative function of molecularly diverse mutants competitively with the then state-of-the-art methods, and to enable a two-orders-of-magnitude efficiency improvement in a protein engineering task ^[16]. Those results were obtained on proteins, and the honest version of the claim for a peptide model is the same in form: not that it knows the answer, but that it can reduce how many wells you have to open before you find one.

What to report so the result can be checked

• The split protocol: clustering tool, identity threshold, and whether partitions were formed on clusters or on sequences.
• How the negative set was constructed, and a second evaluation under a different negative sampling scheme — the 660-model result makes a single scheme uninterpretable on its own ^[1].
• The class ratio in the test set, a precision-recall curve alongside any ROC curve, and the confusion matrix at the operating threshold rather than the AUC alone ^[8].
• An applicability-domain distance for every prediction released, in the model's own feature space, with the training-set range for each descriptor.
• The assay conditions attached to each training label, or an explicit statement that they were heterogeneous and were not modelled ^[7].
• A prospective test: sequences selected, synthesised and assayed after the model was frozen, reported with their hit fraction.

Where this sits beside the other predictors

Two neighbouring tools in this library take the same input and return different objects. A sequence-based aggregation predictor returns a per-residue propensity profile with no axis for the buffer you will use; a structure predictor returns coordinates with a confidence array, and reading that array is most of the job. A property classifier of the kind discussed here returns a single scalar and no confidence array at all unless you build one, which is why the applicability-domain distance has to be constructed by hand. And an embedding from a language model whose representations encode structural information ^{[4, 6]} is not a structure: it is a vector that correlates with one.