I think you underestimate the awesome potential of this sport. It would be like a mix of dodgeball, ultimate and basketball. The only real problem is the snitch and lets face it, that is a problem with the sport in the magical world too. 150 points and the end of the game is way to devastating.
So, you make some minor rule changes to fix the snitch problem and then with get people who have a bit of athleticism and boom, awesome time.
Also, you shouldn't have to hold a broom between your legs, that just seems dangerous.