Not really something you discuss with other refs really. I have worked with probably 100+ different refs over the last 10 years and the conversation has only come up once and they declared because they were a police officer and being prosecuted for not paying taxes likely result in dismissal, so I couldn't begin to guess how many do and don't.
All he is doing is raising a valid point about being careful. That's not saying you must/should declare, what it means is, if hmrc do come knocking, just have a record so you can pay what you owe, not an estimate.
You are right, one singular ref is not worth the effort but, let's say 14000 refs are earning and should be declaring that then becomes a much different figure, with a fine for each on top....
I do wonder how much hobbies/recreational earnings that are not declared are worth to HMRC though outside of refereeing. Would only take one big project to start bringing in revenue from those streams.