In future planetary exploration missions, improvements in autonomous rover mobility have the potential to increase scientific data return by providing safe access to geologically interesting sites that lie in rugged terrain, far from landing areas. To improve rover-based terrain sensing, this paper proposes a self-supervised learning framework that will enable a robotic system to learn to predict mechanical properties of distant terrain, based on measurements of mechanical properties of similar terrain that has been traversed previously. In this framework, a proprioceptive terrain classifier is used to distinguish terrain classes based on features derived from rover-terrain interaction, and labels from this classifier are used to train an exteroceptive (i.e., vision-based) terrain classifier. Once trained, the vision-based classifier is able to recognize similar terrain classes in stereo imagery. This paper presents two distinct proprioceptive classifiers-a novel approach based on optimization of a traction force model and a previously described approach based on wheel vibration-as well as a vision-based terrain classification approach suitable for environments with unexpected appearances. The high accuracy of the self-supervised learning framework and its supporting algorithms is demonstrated using experimental data from a four-wheeled robot in an outdoor Mars-analogue environment. © 2012 Wiley Periodicals, Inc.