Below Supply League level, the marking will likely be done by the manager, so there is more of a correlation between the mark and the result.
Even though it was a stonewall 90th minute penalty you gave against them, it cost them the game, so your mark goes down.
At Supply League and above, it's normally the secretary that will mark you. This topic came up with one of the secs from my Step 5 league on Twitter over the past few days. He's a good guy and an ex Ref, and he's really fair. He said that as a secretary he has so much to do and the thing that really annoys him are the referees that don't do the admin pre and post game in a proper or timely manner as it adds more work for him. He said he knows the refs he has to chase and guess what, he marks them lower than those who respond and do their admin. The 'doing a decent job' for 90 minutes is important, but not his only consideration.
Get into the habit of doing the basic admin right and everything else falls into place. Real life example today - got a county cup semi final middle Sunday morning (don't do Sunday morning football normally, so no idea on the teams), had to prompt them for their team colours and the home side said 'We play in all Navy Blue' which for me in Hampshire is a big issue as we are only allowed to wear black. Quick exchange and they are going to wear their change kit with no clash.
Think about what happens if I don't check and rock up Sunday morning with them in a kit that clashes and not having brought their spare kit - I'm immediately on the back foot and end up being the bad guy from before kickoff, what does that do for my club marks?