A couple of points from me.
First try not to let what is said to you, impact your enjoyment. I know it's really hard, but you have to learn to shrug it off. Players and managers will rage against you unfairly, because they're p*ssed off. I once had a defender, with two hands, push an attacker in the back in the PA in the 91st minute. Clear as day penalty. Given, they score and win 2-1. I had a NAR on that side of the pitch, level 5, who said it was 100% a penalty and the right call. After the manager of the losing team said I was the worst referee he ever had and it was the worst decision he's ever seen! So he was just talking absolute nonsense. Doesn't make it right, but unfortunately you have to learn to deal with that part of the game and just shrug it off.
Also, what level are you refereeing at? Saturday football with players who can barely kick a ball and managers who have no tactical knowledge, but seem to think they know how to referee better than someone who is qualified?!
Second ask yourself why you're refereeing, everyone has different reasons (money, fitness, they enjoy it, like being in charge, all of that etc) and then ask if you're going to allow someone to ruin that. If you do it for fitness, do you care people get p*ssed off with you unjustly, as long as you get a good work out? If you do it for money, do you care about that they say when you have £30 in your pocket? That helps in the focus. Of course if you do it because you enjoy it, it's not enjoyable to get abused.