He wrote a great little examination about where the tipping point of elitism is. Why was Dubya not seen as an elitist, whereas John Kerry was? He goes through several examples and possible reasons, but I think it’s pretty darn simple.
If you know someone’s smarter, richer, or otherwise better off than you – AND YOU THINK THEY KNOW IT – then they’re called elitist.
Give that metric a try. I think you’ll find that it explains a lot – both about who gets called an elitist … and about the person doing the namecalling.