You are the ref, it's your level of tolerance.
I spent 10 years in the Army - and can take a lot of flack, if I so chose to do so, and it takes a lot for something I haven't heard.
BUT, there is a code for an offence of OFFINABUS - anything that, in the opinion of the referee is offensive, insuting or abusive. it does not have to be directed at the ref, or even be about the ref, but if it is heard, and refs opinion is that it is of the above, you have a right to santion respectively.
Now, I say that but there is also situation and context.
If i;ve pulled someone to the side for a quiet word, over a nothing challenge, and in a clam normal tone normal volume voice he wants to call me an idiot, and no=one else is about to hear it, I'd possibly add a bit of threatening behavour to the phrase, or it might be meant with a bit of intimidation.
and if others are around to hear it, well that's his stupidit for sying it within earshot of others.
So what you going to do when he actually DOES commit a foul tackle - he's already intimidated you once ......
From what is written above, you could sell it either way. I do no know your tolerence levels, or where your line is drawn butif he stepped over it, i wouldn't expect anything less.
Today I have a foul, stupid coming together, blew up pointed and got the hands thrust up and down, head back, eyes shut and a scream of "fuk's sake ref, for what?" shouting like that at me 3 yards away - yellow was up before he opened his eyes again.
aterwards he asked if I thought it was a bit hars "it;s not as if i called you a wan%er or anything was it?"
"not really - cos if you had called me a waker, i woud have found that offensive and you;d have been off.Did aynone else turn and shout at me for any other decision, inc the dnied penalty claims? No. Any you know why....
"cos you'd already given me a yellow. yeah yeah point taken ref. Thanks anyway, good game, see you in the bar........" shake of hands and walked off