It's always worth familiarising yourself with the competition rules in respect of the managers responsibilities and in particular in regards to each line. Every youth league I've officiated in or coached in make it very clear that it is up to the manager to manage their respective line and there are fines if they don't. Sometimes managers need to be reminded of that when they are ignoring an abusive parent.
A line I often use during youth games if I get an unruly parent/spectator is to simply say to the manager, I'm here to referee the children, not the adults, can you sort please.