and that one idiot might go on to atttack the referee
Yes, he might. So might any of the other players. If we were to abandon games on the basis of what might happen, we'd never finish one.
I would never criticise a referee for abandoning a match in which he or she feels threatened, but that's not what's happened here. OP issued a yellow and was later advised by somebody who wasn't there that he should have abandoned the match. I don't think that's very sound advice on the basis of the available evidence.