How reliable are genomic evaluations?

The author is an associate professor of dairy cattle genetics at Penn State University.

Recently, I have had breeders express to me a level of concern over the number of bulls whose proofs declined substantially once they had daughters. This naturally leads to questions regarding the reliability of genomic evaluations and whether the data can be trusted.

To analyze the accuracy of bull proofs, we are going to look at genomic predicted transmitting ability (gPTA) for bulls that had no daughters in 2014 and at least 500 daughters in 2024. I am picking 2014 because for longevity traits like productive life, it takes time for the daughters to mature and die before we can determine how long they lived.

There were 2,307 Holstein bulls that met this criteria. We will be comparing their gPTA changes to a second group of 2,451 Holstein bulls that had at least 100 daughters in 2014 and added at least 500 daughters since that time.

Realized reliability changed

I have plotted 2024 gPTA for fat yield against 2014 gPTA for fat yield in Figure 1. There was a genetic base change since 2014 that caused the average gPTA to drop, but that is not what we are interested in. We care about how stable the rankings were among our bulls.

As you can see, the general trend is positive and the gPTA for fat yield for bulls with no daughters mostly performed as expected. You will notice the R2 highlighted in the box on the chart. Without getting into too much detail, this number represents what I will call the “realized reliability.” This assumes that the 2024 proof represents the bull’s genetic merit with 100% accuracy, which is not entirely true, but it is a pretty good approximation.

The realized reliability in Figure 1 is 0.72 or 72%, which is close to the average reliabilities for those bulls at the time of 76%. In other words, the gPTA for these young bulls had the level of accuracy we expected.

So, does this mean that the concerns of breeders about bull proof stability are unfounded? Not exactly. Figure 2 zooms in on the top 10% of bulls in 2014. In this graph, you can see that the realized reliability is 24%. The relationship is still positive, but marginally so. For example, the third highest bull in 2014 is now ranked 790th in 2024.

This tells us that the reliability values we publish are only relevant when considering the entire population. If you are a seedstock breeder generating elite genetics for marketing purposes, you will only be selecting among bulls near the very top of the various rankings. The reality is that accurately picking the best bull among a group of elite sires is not possible.

True for all traits

I have summarized analyses for other traits including milk, fat percentage, productive life, daughter pregnancy rate, and Net Merit (NM$) for the same group of bulls in Table 1. You can see that when we consider the entire population of 2,307 bulls, our realized reliabilities are pretty close to the published reliabilities. They tend to be slightly lower, which likely reflects the fact that bulls with extremely poor gPTA are likely to not have any daughters, so the tail end of our distribution is cut off.

When we look at just the top 10%, we see the same general trend as we saw for fat yield. Fat percentage holds up a bit better than the other traits. Productive life had a low value, and the realized reliability for NM$ was only 6%. Keep in mind that the NM$ formula changes from time to time, which contributes to sire reranking and explains some of what we see. If we only looked at the top 100 NM$ from 2014, the realized reliability was zero.

Some may wonder if more recent results would tell a different story due to improvements in genomic predictions over time. When I do the same analysis using 2019 gPTA, accuracy for the top 10% is only marginally better across the board than reported in Table 1, and the realized reliability for NM$ was 7%.

Table 2 considers how daughter proven bulls fared. As you can see, the effect of concentrating on the very top bulls is the same. Realized reliabilities are higher compared with the bulls that did not have daughters, but they are still lower than what one may expect based on the published reliability. We also see the same trend when we consider Jersey sires. Values for yield were similar to that for Holsteins, whereas productive life and NM$ were near zero among the top 10%.

Opportunity awaits

Should we be concerned that our realized reliability is not robust when we just look at the top of the list? It is important to understand that genomic predictions are working as we expect. Commercial producers can be confident that bulls with high gPTA will generally sire good daughters and their herd will make progress as their breed improves.

Your perspective could be quite different if you invested a lot of money in a particular family that is no longer elite. Unfortunately, there is no easy solution to prevent that from happening. There are, however, opportunities that come when we have less certainty. If genetics were perfectly predictable, a handful of companies could buy the best genetics and essentially shut out other breeders. That is not easy to do when genetic reranking occurs.

A second opportunity relates to genetic diversity. If we can’t tell the best Holstein bull from the 100th best Holstein bull with a high degree of confidence, then we should be willing to expand our genetic base. The reality is that we can make the same amount of genetic progress without the extreme rise in inbreeding that we are currently experiencing.

Our gPTA are as accurate as expected based on our published reliability. Genomic testing certainly identifies a really good bull from a lousy bull. Just keep in mind that we are nowhere near being able to determine the best from the second best.