You say 'you can't avoid', the law says nothing about intent. Many fouls can be accidental.
I don't understand how the question of intent comes up. The contact was unavoidable. That does not imply intent or a lack of intent. It was not possible to make that challenge without subsequently making contact with the attacker. Since he made contact with the ball before the man the inevitable contact, given it wasn't dangerous or reckless, is allowed, surely?