Heritability, Broad and Narrow Sense
Heritability is the proportion of phenotypic variance attributable to genetic variance. Broad-sense heritability (H squared) uses the total genotypic variance. Narrow-sense heritability (h squared) uses only the additive genetic variance. The two are different numbers and they answer different breeding questions.
Broad sense H squared
H squared equals sigma squared G divided by sigma squared P, where sigma squared P is the total phenotypic variance and sigma squared G is the total genetic variance (additive plus dominance plus epistatic). H squared is what an ANOVA on clonally replicated genotypes (or pure lines) gives directly. It is an upper bound on response to selection.
Narrow sense h squared
h squared equals sigma squared A divided by sigma squared P, where sigma squared A is the additive genetic variance only. This is the relevant quantity for response to selection because only the additive component is transmitted reliably from parent to offspring. Generation mean analysis (Mather and Jinks 1971), diallel analysis (Griffing 1956), and parent-offspring regression all give estimates of sigma squared A.
Common mistakes
First mistake: confusing H squared (broad sense) with h squared (narrow sense). Mathers and breeders use different conventions. Second mistake: inflating H squared by computing it from a single-replicate trial; sigma squared G in the numerator absorbs the residual error so the estimate is biased upward. Third mistake: treating heritability as a property of a trait. Heritability is a property of a population in a particular environment; the same trait can have h squared close to 0 in one population and h squared close to 0.7 in another. Always report the population, the environment, and the design from which the estimate came.