Estimation & Prediction

Image embeddings for digital phenotyping and the construction of digital similarity matrices for genetic evaluations

---

Abstract

In recent years, incorporating additional sources of information in the mixed model equations, such as microbiome, metabolomic, proteomic, or transcriptomic similarity matrices as random effects, has been shown to improve the estimation of genetic merit within the genomic evaluation framework. In this study, we propose a novel approach for image-based digital phenotyping using latent representations extracted from Vision Transformer (ViT) and β-Variational Autoencoder (β-VAE) models to capture biological variation beyond existing fixed and random effects. Our approach processes segmented images to generate latent variables with dimensions determined by architecture: 768 to 1024 for the ViT variants and 128 for the β-VAE model. To evaluate how latent representations capture population structure, we computed attribute-specific correlations with environmental factors and assessed feature disentanglement with growth traits. We also estimated the heritability of each latent feature to assess the genetic potential encoded within these variables. Finally, we constructed digital similarity matrices from the latent features using three kernels: Pearson correlation, cosine similarity, and Gaussian RBF. Our results revealed that ViT-derived embeddings captured higher-resolution signals for environmental factors and phenotypic covariates than the β-VAE. Image-derived latent variables exhibited moderate, heterogeneous heritability, ranging from 0.06 to 0.36 for ViT variants and 0.11 to 0.49 for the β-VAE. Furthermore, ViT-based similarity matrices showed higher correlations with pedigree and genomic benchmarks than that from the β-VAE, with the ViT-B/32 model using a cosine similarity kernel achieving the highest performance, yielding r = 0.31 for pedigree and r = 0.27 for genomic benchmarks. In future work, we aim to investigate the feasibility of incorporating these digital similarity matrices into mixed model equations for routine genetic evaluation. Overall, our findings demonstrate that deep learning models can extract heritable latent features directly from images, thereby extending the scope and potential applications of digital phenotyping within animal breeding programs.

Keywords: 2026

How to Cite:

Billah, M., Bermann, M., Bussiman, F., Chen, C., Psota, E., Valente, B., Holl, J., Misztal, I. & Lourenco, D., (2026) “Image embeddings for digital phenotyping and the construction of digital similarity matrices for genetic evaluations”, World Congress on Genetics Applied to Livestock Production Digital Archive 2026(1): 2285607. doi: https://doi.org/10.31274/wcgalp.23750

Rights: 1