Why weren’t slaves, servants, mechanics and farmers of Ancient Greece *not* allowed to go to the gumasium? Why were only free men, citizens and the aristocracy allowed to visit and utilize the gymnasium?

Is there some deeper and more philosophical and social/physical insights we have yet discovered about the gym, and the culture of the ancient gymnasium which makes us free men, noble, and virtuous?