As has already been pointed out, it didn't have to be for 'time wasting' (unless of course, the referee said it was) - it could have been for showing a lack of respect for the game. It's not something I would probably do but the law is worded vaguely enough that a referee could use that provision if they really wanted to.
Now be honest, have you been winding him up prior to that incident? All jokes aside, you are right he would be wrong if the caution was for time wasting but as it was UB and probably using the pretext of not respecting the spirit of the game! Hence the vagueness of the wording to offer referees a bit of leeway in their interpretation of what that entails!
That being said, I fully understand why he would consider it as an attempt of time wasting but since he is not supported by law to caution you as such he found a loophole