I agree! My thoughts, you would drown in a Large! I Med would be a bigger fit on you, but it wouldn't be annoyingly huge.
To give you an idea of sizing I uploaded some very "bad" pictures of myself in my Jacket.
I'm sure someone will come along sooner or later and give you better insight on this.