No right and wrong answer to this, but if I was the appointments officer for a youth league and had a limited pool of referees I would fill games down from the oldest age group. I'd much rather have an U7 game without a league referee than a U15 game, that is just common sense.
Personally speaking, I can't really see what benefit a qualified referee brings to an U7 game. Aside from the risk of parents being idiots, which isn't really for the referee to deal with anyway, what is a referee really going to be doing in those games.