The purpose of this study was to investigate the relationship of light and strenuous exercise, and self-reported change in exercise status, with different components of cognitive function, and gender differences in this relation, in a large, representative sample included in the Swedish National study on Aging and Care (SNAC). Eight-hundred-and-thirteen participants in age-cohorts from 60-96 years completed a wide range of cognitive function tests, the Mini Mental State Exam (MMSE) and survey questions concerning exercise behaviour and exercise change with light or strenuous intensity. ANCOVA, controlling for age, education, depression, functional status and co-morbidity, demonstrated a main effect for light exercise, but not for strenuous exercise, on five of the six cognitive tests and the MMSE, for men but nor for women. A negative change in exercise status was associated with lower MMSE scores for men but not for women. Individuals exercising with light intensity several times a week had the highest cognitive test and MMSE scores and the inactive group had the lowest scores. The results of the study may contribute to increased knowledge in the exercise-mental health relationship for elderly and spawn new research specifically on gender differences in this relation.