It's a matter of opinion as Dan says.
If I as the referee have decided to give a penalty, then that's my decision. I don't want my assistant to tell me I am wrong, in his/her opinion. (Unless I was incorrect in law) I'd kindly tell remind them to take their position.