Just my two cents on it.
It's weird how the players who bother me aren't the ones hurling abuse at me.
I've sent a few players off in my time calling me many things under the sun. However the strange thing is, most of those players have come back to apologise to me (and not your usual "don't put that card through ref," I felt like they were genuine apologies).
The ones that usually bother me at the quiet ones, those are the only times I've felt intimidated by players, when they don't necessarily say or do anything 'wrong' but carry a demeanour about them that they're out to hurt people on a Sunday or Saturday, or go into a game with the intention of doing whatever they want with no regard for the 21 other players playing the game.
I once had a player, when I called him over to dismiss him, he didn't even say a word, just advanced on me with a threatening stare, getting so close I had to take a few steps back.
Compare that to a few weeks later where I sent a player off for calling me a ****, he was effing and blinding as he left the field of play, kicked the changing room doors on the way off the pitch.
Next time I had him, he was good as gold, he apologised to me saying that his misses cheated on him the day before, that he lost his head, but that it was no excuse. Now I know he shouldn't have done it in the first place, but I couldn't help feel a bit sorry for him, and we both went into this game with the mentality that it is a new game.
Don't really know where I'm going with this post, but it's clear that this problem isn't black and white as it seems. Obviously if you lay hands on a referee, you're a troglodyte who has no place in the modern game. But as for the people who somewhat perpetuate the environment which allows this to happen, I don't think the majority of them are "bad people," maybe they're just caught up in it.
Some people turn to alcohol or drugs for an escape, but I think for most people sports is an escape too, and if you take away someone's escape from life, then they are going to be upset.
And just for the record, I'm not talking about people who assault referees here, but people who maybe abuse the referee or lose their heads. And I'm not condoning it either, I just find it interesting.