At grass roots, unless the manager has done something appallingly bad out of the blue, I will have already told him that unless he cuts it out he will be going where I can't see him and he can't see the pitch. So if he carries on I don't really have any wiggle room having already made that threat. Of course, the standard argument is "it's a public park, you can't make me go anywhere", to which I have a standard reply of "well the game isn't restarting until I can't see you and I will abandon it if necessary".
At senior levels there will normally be rules as to what happens. Don't know if it is still the same, but in contrib leagues they had to go back to the changing room or leave the venue and cannot watch any more of the game. That policy certainly improved manager and coach behaviour when it came in, as previously they could just jump over the barrier, or sit a couple of rows back in the stand, and carry on managing the team.