It comes with extra padding and all that, so you can get all the fits in that range. I was right at the top end of medium and bottom of large with my actual head. So I got the large and with a toque it works and just on my head it works too with the padding in.
You sound like you're at the top end of small, so if you had a beanie on it would probably be a bit small on you. I'd get the medium. This helmet doesn't look bulky even if it is a bit big. Super nice super light.
I have the matte white by the way and it's my favorite helmet ever.