Previous cross-sectional research has largely associated binge drinking (BD) with changes in volume and thickness during adolescence and early adulthood. Nevertheless, the long-term alcohol-related effects on gray matter features in youths who had maintained a BD pattern over time have not yet been sufficiently explored. The present study aimed to assess group differences both cross-sectionally and longitudinally [using symmetric percent change (SPC)] on several structural measures (i.e., thickness, surface area, volume). For this purpose, magnetic resonance imaging was recorded twice within a 2-year interval; at baseline (18-19 years) and a follow-up (20-21 years). The sample included 44 university students who were classified as 16 stable binge drinkers (8 females) and 28 stable controls (13 females). Whole-brain analysis showed larger insular surface area in binge drinkers relative to controls at follow-up (cluster-wise = 0.045). On the other hand, region of interest (ROI) analyses on thickness also revealed a group by sex interaction at follow-up ( = 0.005), indicating that BD males had smaller right rostral middle frontal gyrus thickness than both control males ( = 0.011) and BD females ( = 0.029). Similarly, ROI-based analysis on longitudinal data showed a group by sex interaction in the right nucleus accumbens ( = 0.009) which revealed a decreased volume across time in BD males than in control males ( = 0.007). Overall, continued BD pattern during emerging adulthood appears to lead to gray matter abnormalities in regions intimately involved in reward processing, emotional regulation and executive functions. Notably, some anomalies varied significantly depending on sex, suggesting a sex-specific impact of BD on typical neurodevelopment processes.