It really depends on the stiffness of the grip itself, you may indeed break it.
Heat it up with the heatgun, and crush the whole thing in an actual vice. The crunched aluminum will hopefully be easier to pull out.
Hell, its worth a try. You already ruined the poles, might as well try and fix it.