I agree with Padfoot (about the ignoring obvious dissent, not the other stuff - while I remember,
@Padfoot tone down the personal stuff - that comment about his professional life is not needed), someone tells me to f*ck off and they can expect repercussions, possibly an earlier than advertised solo shower while we carry on the match without him (context is important here as to whether it is a red or caution).
Ask yourself, Are you there to be abused? Is that what you think refereeing is about? Are you happy that bystanders and others around the pitch are hearing such a vile amount of foul language from a game you are refereeing? do you have so little self respect that you deserve so little respect on the field of play?
All them answers should be 100% no.
Just remember that aside from the obvious dissent and use of offensive, insulting or abusive language, part of your role as referee is to protect the image of the game - players swearing without any control from you is very poor and portrays football in a very poor light. I think you know this from your post. You say you are managing games well, but I would perhaps suggest you are just letting some idiots get away with murder. Making a rod for your own back mate - old referee wisdom: what you do not punish, you encourage and sooner or later the wheels are going to come off. Especially when you get better games when the play is faster and more intense and the behaviour from the players more difficult to manage once they realise they can take the p*ss.
Good refereeing is preventing problems - nipping this nonsense in the bud early in the game. A strong word from you, involving the captain to make clear your tolerance has been reached or resorting to cards - make sure you are not letting this happen without reaction and if a line is crossed, do not worry about going to the pocket. Players respect strong referees who are clear in their boundaries. They will take massive liberties with referees who just let them get away with murder.
One last point - I obviously cannot see what is happening in your matches, but you haven't had a single reckless challenge yet? lol just a thought but you are either the luckiest referee in the world or you haven't learned to identify them properly yet