I would have no issue wearing a long sleeve baselayer under a short sleeve top. Nor would I as a player have changed my opinion of the ref if I saw one wearing them.
There was one ref from my playing days (let’s call him more experienced than others) who used to wear wooly hat, gloves, jacket...