DeepskierBump kind of want to get into climbing so I can summit some dicey peaks on backpacking trips. Not sure how that would go though. I'm a broke lil bitch and haven't looked into it at all.
The advice given in this thread is generally good. A harness and belay device are certainly not expensive. The Momentum package from Black Diamond is a good choice.
For shoes, you will need to spend a bit and you should not buy them online without trying them on first. They are like ski boots. Every brand, and even every model will fit and feel differently. La Sportiva and 5.10 are the two major brands. They also fit very differently. I cannot wear a La Sportiva at all - they just do not fit my foot (other than their ice boots, which fit well).
All you need to get started is a harness, belay device, climbing shoes, and chalk bag. Buy the cheapest chalk bag you can find - they all are fine.