bro, I know it's probably not what you want to hear.. but seriously, let it heal.
take it from me. A few years back I thrashed my body.. all kinds of injuries. I rode on that shit and never gave a damn... and because I never gave it enough time to recover... it's still screwed up to this day.
not worth it.
even if your wrist is in a cast... if you take a hard fall on it.. or jolt it rough.. you can screw it up.
but seriously. ask your doctor see what the word is.. If he's cool with you skiing around, then you're probably game.. If he wants you to sit out for a few weeks.. then definitely take the advice.
Also, you can try talking with some physical therapists/sports medicine staff and see if there's anything ya can do to help speed up recovery time.. if it's a bone break probably not shit ya can do but wait it out.. but ya never know.