The aim was to investigate associations of fitness and types and levels of physical activity with subsequent risk of cardiovascular disease. Four large-scale longitudinal cohort studies were used. The exposures were different measures related to physical activity and the outcomes were obtained through linkage to the Swedish In-Patient Register. In a cohort of 466 elderly men without pre-existing cardiovascular disease, we found that skeletal muscle morphology was associated with risk of cardiovascular events. A high amount of type I (slow-twitch, oxidative) skeletal muscle fibres was associated with lower risk of cardiovascular events and high amount of type IIx was associated with higher risk of cardiovascular events. This association was only seen among physically active men. Among 39,805 participants in a fundraising event, higher levels of both total and leisure time physical activity were associated with lower risk of heart failure. The associations were strongest for leisure time physical activity. In a cohort of 53,755 participants in the 90 km skiing event Vasaloppet, a higher number of completed races was associated with higher risk of atrial fibrillation and a higher risk of bradyarrhythmias. Further, better relative performance was associated with a higher risk of bradyarrhythmias. Among 1,26 million Swedish 18-year-old men, exercise capacity and muscle strength were independently associated with lower risk of vascular disease. The associations were seen across a range of major vascular disease events (ischemic heart disease, heart failure, stroke and cardiovascular death). Further, high exercise capacity was associated with higher risk of atrial fibrillation and a U-shaped association with bradyarrhythmias was found. Higher muscle strength was associated with lower risk of bradyarrhythmias and lower risk of ventricular arrhythmias. These findings suggest a higher rate of atrial fibrillation with higher levels of physical activity. The higher risk of atrial fibrillation does not appear to lead to a higher risk of stroke. In contrast, we found a strong inverse association of higher exercise capacity and muscle strength with vascular disease. Further, high exercise capacity and muscle strength are related to lower risk of cardiovascular death, including arrhythmia deaths. From a population perspective, the total impact of physical activity on cardiovascular disease is positive.