Most of the time, it's safe to assume other people aren't talking about you. We're all living inside the feedback loop of our own neuroses, which means many of us walk around paranoid about other people's ideas about us, all the while, they're stuck in the same thought loops about themselves.
However, there are times when our names are stuck in the mouths of others, and while it can be hurtful, it can also be entertaining when the rumors are completely unfounded.
I'm of Arab descent and my parents are from a small town in Palestine so rumors run rampant. I was walking around in the major American city where we live and saw my father's cousin who came up to me very animated and excited to see me and smiling. After hugging and exchanging pleasantries he asked "So how long have you been out?" And when I told him I didn't understand he said "come on we all know you've been in jail for drug dealing, nothing to be ashamed of, we are just so happy you're out and free."