Physical activity and exercise beneficially link to brain properties and cognitive functions in older adults, but the findings concerning adolescents remain tentative. During adolescence, the brain undergoes significant changes, which are especially pronounced in white matter. Studies provide contradictory evidence regarding the influence of physical activity or aerobic-exercise on executive functions in youth. Little is also known about the link between both fitness and physical activity with the brain's white matter during puberty. We investigated the connection between aerobic fitness and physical activity with the white matter in 59 adolescents. We further determined whether white matter interacts with the connection of fitness or physical activity with core executive functions. Our results show that only the level of aerobic fitness, but not of physical activity relates to white matter. Furthermore, the white matter of the corpus callosum and the right superior corona radiata moderates the links of aerobic fitness and physical activity with working memory. Our results suggest that aerobic fitness and physical activity have an unequal contribution to the white matter properties in adolescents. We propose that the differences in white matter properties could underlie the variations in the relationship between either physical activity or aerobic fitness with working memory.