Exactly this. When I played youth, we never had neutral appointed refs, and I recall no more than 5 yellows throughout U7s to U17s. Since refereeing, I’ve thought how many things would’ve been yellows or reds, but just never got given.
Also had the same feeling when I occasionally referee...