Better to have a bad, notorious, and famous reputation than none.
For example, everyone knows me at the gym now, and also, in the world of Street Photography, in photography, I might be one of the most hated photographers alive. But, this gives me insane pride.