I took it last year. Here is the bottom line when it comes to learning it.
1. Grammar is super easy, there is no verb conjugation like that of other languages, the difference between "I run" and "he runs" is that you change it from I to he. The verb stays the same.
2. Pronunciation is HUGE. Tones can mean the difference between saying something offensive or kind. The tones are a pain in the ass.
3. Memorizing characters is pretty difficult too, simply because there are so many. If you continually practice it and practice writing them you will get them down with ease, just like memorizing english vocabulary.
4. Speaking is not all that difficult, although you must be careful with the tones and pronunciation once again.
Yes, it is a difficult language. Yes, it is very rewarding. Quite frankly, I enjoyed it. It was a bit of work but fun to me.