Well my parents are from there and I have been there a bit, but basically the remained neutral in those wars is because they are just that- neutral. This doesnt mean they wouldnt have defended themselves, but it does mean that they didnt chose sides regardless of their position on the aggressor. And yes, it is fairly rugged terrain, and as far as I know Hitler had eventual plans to invade, but knew that for the time being Switzerland would not act towards them, so why waste the effort when his empire already surrounded it.
Also, if you are at all familiar with their road systems, as far as I know they Switzerland also had it planned so that when the German army was on one of two routes (at the time anyway) into Southern Switzerland, both containing really long tunnels, that they would just blow up the tunnels taking out. Thus taking out probably the whole cavalry or whatever, and making supply routes virtually impossible.
Side note: Every building made, is made with a bomb shelter.
How about just dang bro brah