The structural covariance network (SCN) has provided a perspective on the large-scale brain organization impairment in the Alzheimer's Disease (AD) continuum. However, the successive structural impairment across brain regions, which may underlie the disrupted SCN in the AD continuum, is not well understood. In the current study, we enrolled 446 subjects with AD, mild cognitive impairment (MCI) or normal aging (NA) from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database. The SCN as well as a casual SCN (CaSCN) based on Granger causality analysis were applied to the T1-weighted structural magnetic resonance images of the subjects. Compared with that of the NAs, the SCN was disrupted in the MCI and AD subjects, with the hippocampus and left middle temporal lobe being the most impaired nodes, which is in line with previous studies. In contrast, according to the 194 subjects with records on CSF amyloid and Tau, the CaSCN revealed that during AD progression, the CaSCN was enhanced. Specifically, the hippocampus, thalamus, and precuneus/posterior cingulate cortex (PCC) were identified as the core regions in which atrophy originated and could predict atrophy in other brain regions. Taken together, these findings provide a comprehensive view of brain atrophy in the AD continuum and the relationships among the brain atrophy in different regions, which may provide novel insight into the progression of AD.