Level 6 Referee
I should imagine that this is a perpetual battle for referees at all levels!As Big cat says he is doing. Try and work this out of your game. You're only inviting conversation and dissent.
I too am trying to cut down on how much talking I am doing and saving it for when it's needed.
Save it for the less obvious fouls or if a player politely asks what it was for.