Attacker is chasing a ball into a corner, putting pressure on the defender. Jumps in to try and block a clearance, doesn't manage to get there in time and gets caught by the defender in his (entirely natural) follow through.
For whatever reason, my spidey senses tingled and I kept half an eye on it, so I manage to see the attacker kick out from the floor in retaliation, catching the defender on the shin. "Easy red card" I think, so I blow and start to move in to disperse the mass con that is already looking likely. But I'm probably the 2nd or 3rd closest person to the two players in question, so am thinking that my chances of getting in with a quick red to calm things down might be pretty high.
And I find my way blocked by the giant centre-half captain of the defending side, who instead of going in to calm things down, has chosen to turn round so that his back is to the incident and move towards me to shout "Ref, he's kicked him, what are you gonna do about it??" over and over. By the time I manage to convince him to stop blocking me so I can get past, 4 or 5 other players have already piled in, so all I can do is stand back, blast the whistle, pick up on an extra yellow for each side and wait for it to calm down.
Frustrating as I feel like the opportunity was there to just make the right decision and get out without any major incident - instead, we had a mass con and a few bonus yellows on top. It occurred to me afterwards that I maybe could have sin-binned the captain for dissenting a decision I hadn't even had a chance to give yet....might technically have been a bit unfair, but would have felt well-deserved at the time!
Anyway, I agree you should have booked him.