You’re looking at the resident memory – that is, the physical RAM consumed. See here for more info.
The virtual memory, however, is the memory consumed by your application, including the memory swapped out (to disk). You’ll see there’s a closer correspondance with the virtual memory and your -Xms settings.