I went through the same thing as you (cleaning up my look). Last year I owned all large Jiberish. But by now my entire collection (aside from my flannel) is size medium. I'm 5'8" 135lbs and a medium fits nicely. Without being washed it is still kinda big, but once it is washed and dried it has a fitted look to it, but still looks sick cause it's JIberish.
Hopefully that helps