Background: Intellectual disability (ID) aggregates in families, but factors affecting individual risk and heritability estimates remain unknown. Methods: A population-based family cohort study of 4,165,785 individuals born 1973–2013 in Sweden, including 37,787 ID individuals and their relatives. The relative risks (RR) of ID with 95% confidence intervals (95% CI) were obtained from stratified Cox proportional-hazards models. Relatives of ID individuals were compared to relatives of unaffected individuals. Structural equation modeling was used to estimate heritability. Results: Relatives of ID individuals were at increased risk of ID compared to individuals with unaffected relatives. The RR of ID among relatives increased proportionally to the degree of genetic relatedness with ID probands; 256.70(95% CI 161.30–408.53) for monozygotic twins, 16.47(13.32–20.38) for parents, 14.88(12.19–18.16) for children, 7.04(4.67–10.61) for dizygotic twins, 8.38(7.97–8.83) for full siblings, 4.56(4.02–5.16) for maternal, 2.90(2.49–3.37) for paternal half-siblings, 3.03(2.61–3.50) for nephews/nieces, 2.84(2.45–3.29) for uncles/aunts, and 2.04(1.91–2.20) for cousins. Lower RRs were observed for siblings of probands with chromosomal abnormalities (RR 5.53, 4.74–6.46) and more severe ID (mild RR 9.15, 8.55–9.78, moderate RR 8.13, 7.28–9.08, severe RR 6.80, 5.74–8.07, and profound RR 5.88, 4.52–7.65). Male sex of relative and maternal line of relationship with proband was related to higher risk (RR 1.33, 1.25–1.41 for brothers vs. sisters and RR 1.49, 1.34–1.68 for maternal vs. paternal half-siblings). ID was substantially heritable with 0.95(95% CI 0.93–0.98) of the variance in liability attributed to genetic influences. Conclusions: The risk estimates will benefit researchers, clinicians, families in understanding the risk of ID in the family and the whole population. The higher risk of ID related to male sex and maternal linage will be of value for planning and interpreting etiological studies in ID. © 2021 The Authors. Journal of Child Psychology and Psychiatry published by John Wiley & Sons Ltd on behalf of Association for Child and Adolescent Mental Health