rugby abuse

Though I’ve played and followed football for most of my life I’m also involved with rugby. Can someone please tell me why we allow players to abuse us like they do? Surely if a stand is taken and the players were to be penalised things would change very quickly.