Appearance definitely plays a part. But that can be overridden by what you actually do once the game starts, you can still be close to play and in credible positions and change those initial perceptions, equally you can look the part but then be in the wrong positions.
As a level 3 I wasn't the youngest, perhaps didn't look the most athletic, and will readily admit I was sometimes carrying a bit of a beer belly, albeit a fairly small one. But fitness and positioning was by far the strongest point of my game and it drew a lot of positive comments and was always the highest marked competency. One observer said in the debrief that he was worried when he saw me but that I was the best he had seen all season for fitness and positioning, wasn't too sure what to make of that but I took it as a thinly veiled compliment.
Even now as a level 5 getting close to the big five o I still get compliments from players as to how close I am to decisions.
The argument is a bit pointless at professional level as those referees have all passed the fitness test, and I can assure you that if you aren't very fit you do not pass that test.