Last year at a local high level game, I watched a GK handle outside of the area (by accident). He then dropped the ball, and the opposing player put the ball down and passed it into the vacant goal as the GK and another defender remonstrated with the AR about the call.
When the goal was scored, the GK realized what he'd done -- he'd disadvantaged his own team by not paying attention.
My approach is, on these types of situations is that if I see the fouled player (or team) looking to take that quick advantage of a quick free kick (potentially at an empty goal), I'm not going to stop them (same logic applies to a quick free kick near half that puts a player in alone on a breakaway, etc). If they start dilly dallying, and are obviously not taking that quick one, that when I'll step in. Typically my general approach to free kicks is to ask the kicker if they want to play it quickly or if they want me to deal with the distance. If they want me to deal with the distance, I hold the whistle up over my head and announce it to everyone, and then make a point of showing the whistle to the kicker and repeat it, just for them.
Of course, as noted above, if there's misconduct, sucks to be you guys... you're on my clock now.