drop this dude you're with right now because he's clearly more interested in other things and although things may be ok for the time being, this thing you have with him is going to fail sooner or later without a doubt.
more importantly, if he sees that he can say shit like you mentioned, and almost convince you to go to a girl's bday party who he has slept with before, he'll just keep pushing the envelope because he's going to realize you'll stick around no matter what.
You can do better. Bottom line.