Post-traumatic stress disorder (PTSD), obsessive-compulsive disorder (OCD), and social anxiety disorder (SAD) all bear the core symptom of anxiety and are separately classified in the new DSM-5 system. The aim of the present study is to obtain evidence for neuroanatomical difference for these disorders. We applied voxel-based morphometry (VBM) with Diffeomorphic Anatomical Registration Through Exponentiated Lie to compare gray matter volume (GMV) in magnetic resonance images obtained for 30 patients with PTSD, 29 patients with OCD, 20 patients with SAD, and 30 healthy controls. GMV across all four groups differed in left hypothalamus and left inferior parietal lobule and post hoc analyses revealed that this difference is primarily due to reduced GMV in the PTSD group relative to the other groups. Further analysis revealed that the PTSD group also showed reduced GMV in frontal lobe, temporal lobe, and cerebellum compared to the OCD group, and reduced GMV in frontal lobes bilaterally compared to SAD group. A significant negative correlation with anxiety symptoms is observed for GMV in left hypothalamus in three disorder groups. We have thus found evidence for brain structure differences that in future could provide biomarkers to potentially support classification of these disorders using MRI.