See how the brick wall changes colors as well? It's because your orange shirt has some similar shades of color as the shirt. When you're trying to key anything, the thing you're keying needs to be extremely different than anything else. That's why they used the bright green color in green screens, because that color doesn't pop up that often.
The orange shirt would probably work better if you were doing it on a black background or something. That way, you can up the tolerance of what it changes and get better results.