Absolutely there is a difference.
As a teacher with 20+ years experience in the classroom, I will chastise a student for swearing to a mate in such a manner, I would come down like a ton of bricks if it was directed at me, the teacher. I think that there are a lot of parallels between the classroom and pitch.
As an example, earlier this term, I had my tutor group in my classroom one morning. I was working one to one with one lad, helping him with his uni application. The rest were chatting/studying/ occupying themselves. All was fine. One lad dropped something and swore. I heard this, looked up, and told him not to swear. That would have, and should have been the end of it - we all swear, this was not the time and place for it, he got called out for it, accept it and move on.
But he didn't. He replied, directly to me: "Pi$$ off, I'm nearly 18, I can say what I like."
The room fell silent. Everyone knew he had overstepped the mark, by a long way. They all knew that swearing at someone in a position of authority is entirely different to swearing at each other. Players know that swearing at the referee is a significant step up from swearing at each other. It's not a football thing, it's a society thing.