I don't get particularly annoyed at it, but it does come up a lot: the idea that social anxiety and introversion are the same thing. While I do need a certain amount of "me time", I'm primarily an extrovert. However, when I talk about social anxiety, people almost always assume that I'm an introvert.
I tend to prefer gatherings with small groups of friends, or events in familiar environments in order to get my fill of "people time" without triggering my anxiety too badly. I also enjoy meeting and talking to new people, even though it's a bit nerve-wracking.