No I've done the same plenty of times.
Couple of times a player was down injured so I went to get a drink and ended up playing snake on my Nokia 3310 for a good 15 minutes. There was another time, and you're not going to believe this but I was refereeing on the football pitch in Regents Park right behind London Zoo. One of the keepers was taking a Chimp called Elvis for a walk outside the boundary of the zoo and they had to pass the pitch. I strained my calf and asked if there was any qualified referees who could step in. The chimp put his hand up and apparently he was qualified so I gave him my kit and he refereed the game. None of the players even noticed and he even got a PK decision spot on and put in a 10/10 performance. I let him keep the match fee and he apparently walked across the road and bought a bulk bucket of nuts.