Rugby is way less subjective than football. Most of the decisions are factual, like forward pass, try over line, foot out of play before try. The only real one that isn't is dangerous tackling, but even that has very strict guidelines that the referees have to follow. "Spike tackles", where they pick someone up and drop them on their head are pretty obvious to see, and even the new "no arms tackling" is pretty obvious, if you tackle without using your arms and there is head to head contact you are sent off even if you haven't initiated the contact. Yes, there was controversy over the Tom Curry WC red card, but it was correct as for all the opponent pretty much landed on his head Curry had gone into a tackle without his arms.
There's probably a reason why the only sports that have a challenge system are the ones with black and white decisions. I seem to remember that rugby union in New Zealand brought in a trial where each captain got one trial per game, but don't think it was taken forward.