It isn't that straight forward unfortunately, IR35 changed that. Prior to IR35 people working on contracts had zero protection, they could be released without any reason as long as they were given their notice period. IR35 saw the introduction of fixed term contracts (FTCs), and anyone on an FTC has the same employment rights as a full time employee. Using an example, if I recruit someone on a 12 month FTC I can choose not to renew it, but then legally I cannot bring someone else into that role as effectively I have made the role redundant. As I understand it SG1 and SG2 officials are on FTCs, therefore they would have to be performance managed out just like a permanent employee would.
Referees would fail the IR35 disguised employment test, so using them as purely casual contractors wouldn't work. The only option I can see is don't pay them a salary / retainer at all, just pay them per game and massively hike their match fees. That way there are no employment law blockers, they effectively just become casual employees. But that brings questions around what happens if they are injured, they'd have zero income, and might also prevent referees wanting to go to the professional levels.