Psychosocial working conditions play a significant role for health during working age, as well as in post-retirement life. However, little is known about the impact of work stressors on cognitive and physical impairment assessed in combination. The aim was to study associations between midlife work stressors and physical and cognitive impairment in older age for women and men separately. The data were derived from two Swedish nationally representative surveys. The same participants were followed through three waves at the mean ages of 54, 61, and 81 (n=360). Multinomial logistic regression was used to assess the association between work stressors, and test of cognitive and physical function. Low control and passive jobs were associated with higher odds of cognitive impairment, and cognitive and physical impairment when assessed in combination. These findings were driven by associations found among men. Accumulated low control and passive jobs across midlife were associated with physical impairment in older age among men, but not among women. These results highlight the importance of work stressors for physical and cognitive impairment in older age. Improving psychosocial working conditions in midlife, in particular increasing the level of control among employees, may promote healthy aging.