Broadly speaking, there are two types of libertarians: rights theorists and consequentialists. Rights theorists (some of whom may be deontologists) assert that all persons are the absolute owners of their lives, and should be free to do whatever they wish with their persons or property, provided they do not infringe on the rights of other--i.e., they allow others the same liberty. They maintain that the initiation of force by any person or government, against another person or their property—with force meaning the use of physical force, the threat of it, or the commission of fraud against someone—who has not initiated physical force, threat, or fraud, is a violation of that principle. They do not oppose force used in response or resistance to initiatory aggressions such as violence, threat of violence, fraud or trespassing.
Consequentialist libertarians do not have a moral prohibition against "initiation of force," but believe that a society that allows individuals to enjoy a large scope of political and economic liberty is the most conducive environment for individuals, or society as a whole, to achieve maximum well-being and prosperity. Many of them maintain that a limited government is necessary for the maximization of liberty. This type of libertarianism is associated with Milton Friedman, Ludwig von Mises, Friedrich Hayek, and James M. Buchanan. Some of these writers who have been called libertarians have also been referred to as classical liberals, by others or themselves. Also, some use the phrase "the freedom philosophy" to refer to libertarianism, classical liberalism, or both. Libertarians may differ over particular issues, such as abortion, and some support the U.S. led coalition's invasion of Iraq while some oppose it. There is a distinction between a libertarian and a member of a Libertarian Party, the latter of which would be called a Libertarian with a capital "L", as not all libertarians agree with any particular libertarian organization's platform.
Well the majority of the people do go with labels so that is the status quo. Personally I don't like political ideologies as a whole and would rather everyone decides on each individual issue instead of fitting it into a belief system as a whole. I can't really see the system changing too much so you can stick to your idea of anti-label but to get things done in todays society you are forced to conform to a label even if you dont share all of its beliefs.