It is fine and natural to let people judge

The nuance is this — just don’t let that get in the way of doing what you want to do!