Estimation & Prediction

GBLUP as an Attention Mechanism

---

Abstract

The 'transformer' architecture has been a breakthrough in Artificial Intelligence, allowing the dramatic improvement of Large Language Models and the emergence of revolutionary tools such as ChatGPT. The core of a transformer is the use of an 'Attention` mechanism which uses similarity between inputs of the training data and the inputs of the prediction set to derive predictions. In genomic models, the similarity between genomes of the training and prediction set also plays a central role in prediction. Indeed, the Genomic Best linear Unbiased Predictor (GBLUP) - one of the most popular and most effective genomic prediction models - can be seen as a simplified Transformer. An attention mechanism is built using n pairs of Keys x(i) the genotype of i-th subject and Values y(i), its phenotype. When the attention mechanism is queried with a Query x(i'), the genotype of a selection candidate, the mechanism returns an output which, in its simplest form, is a weighted average of the values. The weights are based on a similarity score between pairs of keys and queries. Commonly, the weights are computed using a score function (a common scoring function is the dot-product between centered keys and query). The scores are then mapped to non-negative weights using functions such as softmax. Thus, in its simplest form, the output function of an Attention mechanism is a weighted average of the values with higher weight assigned to values associated with keys that are similar (in input space) to the query - we can think of the attention mechanism as a local smoother. Self-attention is used to refer to the case where each key is also a query. In this case, we don't make a distinction between keys and queries and simply denote the data as the pairs of inputs and values. Interestingly, Reproducing Kernel Hilbert Spaces regressions and GBLUP (a special case of RKHS regression) have a structure which is similar to that of the self-attention mechanism above-described: Similarity between inputs is measured using the dot-product (S = k XX' in GBLUP) and the weights are derived from BLUP equations instead of the softmax operator. In our study, we provide an overview of Transformers and present GBLUP and RKHS as a transformer. Subsequently, we use real data to benchmark a simple transformer using a dot-product scoring and softmax weight, as well as a 'performer' - a modified version of a transformer that has important computational advantages - against GBLUP and RKHS in terms of computational cost

Keywords: 2026

How to Cite:

de los Campos, G. & Perez-Enciso, M., (2026) “GBLUP as an Attention Mechanism”, World Congress on Genetics Applied to Livestock Production Digital Archive 2026(1): 2291326. doi: https://doi.org/10.31274/wcgalp.24304

Rights: 1