I feel that privilege is a concept that should be applied to classes, not individuals. You can't just judge someone as privileged based on one aspect of their life.

The whole "punching up" concept just leads to it being accepted to make misogynistic jokes at white women, ablist and body-shaming jokes at "creepy" men, racist jokes at wealthy Asians, and so on.

Rather, I think the intent behind the joke, and the consequences of the stereotypes it reinforces, is what should be examined rather than the demographic on the receiving end.