Observers (Step 6 and above) To No Longer See The Overall Mark Awarded To Referees

[ChasObserverRefDeveloper]

Feels like you're a fan of the system, and thats fair enough... are there any changes you would make?

I always feel uncomfortable talking to observers about average marks etc and it's something I avoid... but then it does feel like some observers are still marking to well below the group average and have no idea that they're doing so. I'm not sure, without introducing a very complicated system that takes an observers seasonal average mark and compares your mark against that, there is any better way to do it.

With all the checks and balances introduced into the system over the last three seasons, I believe that we are getting better at producing a fair Observation system. Not quite there yet.
The regional differences that used to be evident have been eroded (and nearly erased) as shown in the end of season results.
Your thought about comparing the individual Observer's average mark is quite rightly prefaced by "very complicated"!
The problem of using an average is that each Observer is going to report on what is in front of them. This season, for example, I have observed a number of newly promoted Level 4 referees, whose inexperience resulted in a higher number of developmental points than usual, and a mark within the 70-73 range, but probably below the average for that period of the season.
Expectations also play a part. Referees on another site have complained about getting a 72.0 "when I didn't get any development points" Reading or re-reading the Observer Report Writing Handbook may prove helpful.

 

[Russell Jones]

Feels like you're a fan of the system, and thats fair enough... are there any changes you would make?

I always feel uncomfortable talking to observers about average marks etc and it's something I avoid... but then it does feel like some observers are still marking to well below the group average and have no idea that they're doing so. I'm not sure, without introducing a very complicated system that takes an observers seasonal average mark and compares your mark against that, there is any better way to do it.

Systems based on averages will tend to work (generally) well if:

1) The sample size is big enough. Where officials are watched on every game / the majority of their games, the vagaries of individual matches and observers will tend to come out in the wash. Any one ‘rogue’ match day can’t unduly influence the whole
2) The system is knowingly set up to be as fair as possible, recognising the risks inherent in infrequent observations.

On the latter point, as an example, the 5-4 scheme in Herts is set up to ensure that all matches involve neutral assistants and all candidates are seen by the same four observers. Whilst this is challenging to administrate, it makes things as equitable as it can be for the refs trying to make this important step. I’d also be in favour of introducing adding complexity (in the way you suggest) at L4, as that’s the level where observations are least frequent and therefore the possibility of one ‘out of kilter’ observation killing any chance of progression is unreasonably high.

 

[one]

Been reading this thread with interest. A bit of a side track but in terms of worst and best observation (assessment for us), for me it hasn't been content quality, the mark, or if I felt that they had been right or wrong in their opinion of my performance. It has been the priority/goal of the observation/report. Worst ones I have had (luckily only a few) have been the ones with the goal of making themselves look good assuming for promotion purposes. They made me feel used and betrayed after reading the reports.
Best ones, you don't even need to read the report, you clearly feel you'll be a better referee for your next game after the debrief.

 

[Kes]

Expectations also play a part. Referees on another site have complained about getting a 72.0 "when I didn't get any development points" Reading or re-reading the Observer Report Writing Handbook may prove helpful.

Totally.

What a lot of referees seem to forget when reading their (below own expectations) report is that the report competencies under which they're marked are designed to deliver a mark of no higher than 70 if the referee is delivering at the standard expected of them at that level.

You can deliver in all areas and "shine" in 3 or 4 others and the form is still going to come up with a mark of 71-72 (depending on which competencies they were).

What seems to have evolved over the years is an expectation that meeting the requirements of any competency on the form might somehow invite a 7.5 in that area rather than a 7. This (if the observer is doing their job properly) just isn't the case but my belief is that this whole "expectation" thing has somehow crept it's way onto the shoulders of observers over the years and so in lots of cases, an over inflated mark can result.

Why on earth a first-season L4 referee, especially one who might only have been "in the game" 3-4 years would expect that they are "Above Standard Expected" in at least 4 or 5 competency areas is beyond me. It can of course happen if their particular game proves "difficult" or throws up any anomalies but that's just that particular game and any "above average" mark received as a result shouldn't then be an indicator of how subsequent reports (or games) are going to go.

I suppose that's just the way society has evolved in recent years ...

From my own POV as an observer, I will always reward wherever I feel able able to, but to get a 7.5 or higher from me in any competency, the positive match impact has to be obvious and "writable" without embellishment or pushing the boundaries of truth in order to meet expectation management.

I think this new initiative at Step 3-6 will prove beneficial.

 

[DavidObs]

Totally.



What a lot of referees seem to forget when reading their (below own expectations) report is that the report competencies under which they're marked are designed to deliver a mark of no higher than 70 if the referee is delivering at the standard expected of them at that level.

You can deliver in all areas and "shine" in 3 or 4 others and the form is still going to come up with a mark of 71-72 (depending on which competencies they were).

What seems to have evolved over the years is an expectation that meeting the requirements of any competency on the form might somehow invite a 7.5 in that area rather than a 7. This (if the observer is doing their job properly) just isn't the case but my belief is that this whole "expectation" thing has somehow crept it's way onto the shoulders of observers over the years and so in lots of cases, an over inflated mark can result.

Why on earth a first-season L4 referee, especially one who might only have been "in the game" 3-4 years would expect that they are "Above Standard Expected" in at least 4 or 5 competency areas is beyond me. It can of course happen if their particular game proves "difficult" or throws up any anomalies but that's just that particular game and any "above average" mark received as a result shouldn't then be an indicator of how subsequent reports (or games) are going to go.

I suppose that's just the way society has evolved in recent years ...

From my own POV as an observer, I will always reward wherever I feel able able to, but to get a 7.5 or higher from me in any competency, the positive match impact has to be obvious and "writable" without embellishment or pushing the boundaries of truth in order to meet expectation management.

I think this new initiative at Step 3-6 will prove beneficial.

I agree with much of what you said, but Observers shouldn’t need to know whether it’s a new L4 (or 3, or 2 for that matter), it’s all to do with what happens on the FoP (& sometimes off it - not that you have suggested this). Those who lack experience or skills will normally become clear & for the latter can easily include those who have been at their level for quite a few years.

 

[RefereeX]

I agree with much of what you said, but Observers shouldn’t need to know whether it’s a new L4 (or 3, or 2 for that matter), it’s all to do with what happens on the FoP (& sometimes off it - not that you have suggested this). Those who lack experience or skills will normally become clear & for the latter can easily include those who have been at their level for quite a few years.

This is another valid point. Last season (my first full season at 3) the only observer to give me a below average mark outright asked me how long I'd been a level 3. I didn't want to lie to him so I told him it was my first full season, to which he responded 'it doesn't show, you look perfectly comfortably out there' - a below average report then followed. And I do think that some observers will use knowing they're observing a first season as an excuse to bring their average down or whatever.

*Not saying this is what happened in my experience, as I absolutely accept that I missed a caution for reckless play during the match... but it does seem coincidental that the only one who knew gave me my only below average mark.

I don't think there's an answer to that problem though tbh... maybe this is the answer to that sort of problem.

 

[Kes]

I agree with much of what you said, but Observers shouldn’t need to know whether it’s a new L4 (or 3, or 2 for that matter), it’s all to do with what happens on the FoP (& sometimes off it - not that you have suggested this). Those who lack experience or skills will normally become clear & for the latter can easily include those who have been at their level for quite a few years.

Agree and disagree.
Of course it's all on the game in front of you, but generally speaking, an "old and bold" who has been operating at the same level for many seasons will normally have certain skills nailed on (and rightly so) whereas a "newbie" might still be acquiring them (at that level). You'd have to be a robot not to be mindful of that fact when observing.
Just my own opinion mind ...

 

[DavidObs]

Agree and disagree.
Of course it's all on the game in front of you, but generally speaking, an "old and bold" who has been operating at the same level for many seasons will normally have certain skills nailed on (and rightly so) whereas a "newbie" might still be acquiring them (at that level). You'd have to be a robot not to be mindful of that fact when observing.
Just my own opinion mind ...

You may be surprised. There is no template for anyone. I can see very young and very good, very young and not so good. Experienced very good and experienced and not so good. Sometimes in can just be because of attitudes.

 

[Big Cat]

It seems obvious the scheme was devised by referees who know nowt about maths/statistics

I don't think it matters too much however. I'm firmly of a mind that there's fook all to separate all refs at or above a certain level (Prob National League - Step 1) aside from having a face that fits and luck. So the mathematical side of the scoring scheme doesn't matter
The standard of refereeing at Step 1 and above is a collective thing, a function of the overall governance of the game, from FIFA down through IFAB to the Confederations and National FA's and Refereeing Bodies, all of whom take it upon themselves to do their own thing leading to an overall standard that falls a long way short of potential. As for meeting the expectations of the game, that cavernous gap is a function of the ignorance of everyone who has never refereed. They have no clue how difficult it is. Impossible actually

So none of it matters too much. It's futile. I've stopped giving any of it much thought. All that said, I still really enjoy refereeing games, pursuing the impossible every time I go out there. It's been one of the best things I've done with my life

 

[Big Cat]

You may be surprised. There is no template for anyone. I can see very young and very good, very young and not so good. Experienced very good and experienced and not so good. Sometimes in can just be because of attitudes.

Don't you find, you can assess a referee withing minutes or even seconds of seeing them out there? The things they do good or bad (if you stand and watch them for a longer period of time) are always consistent with that first impression. Just do away with the scheme and give a mark out of ten based what they look like and how they move. Then go home!

 

[DavidObs]

Don't you find, you can assess a referee withing minutes or even seconds of seeing them out there? The things they do good or bad (if you stand and watch them for a longer period of time) are always consistent with that first impression. Just do away with the scheme and give a mark out of ten based what they look like and how they move. Then go home!

Wouldn’t say seconds/minutes, but by half time certainly get a good feel about what they are about & how they are performing. However, more often than not the more challenging situations come during the 2nd half when players become fatigued, frustrated with themselves or their team-mates or are losing. Also, I find it often the case that there are more KMi during the 2nd half for the same reason, or penalty appeals - so need to be there for the entire game.

 

[RustyRef]

This is another valid point. Last season (my first full season at 3) the only observer to give me a below average mark outright asked me how long I'd been a level 3. I didn't want to lie to him so I told him it was my first full season, to which he responded 'it doesn't show, you look perfectly comfortably out there' - a below average report then followed. And I do think that some observers will use knowing they're observing a first season as an excuse to bring their average down or whatever.

*Not saying this is what happened in my experience, as I absolutely accept that I missed a caution for reckless play during the match... but it does seem coincidental that the only one who knew gave me my only below average mark.

I don't think there's an answer to that problem though tbh... maybe this is the answer to that sort of problem.

That's always been the case unfortunately, they just have to be cleverer about it now. I ran the line years ago to a 1st year L3 and I thought he was excellent, the observer came in after and tore him to pieces, all three of us were gobsmacked, he was talking about things that just simply had not happened. Step forward a year and I was the L3 and the same happened again, same ground, same observer, same outcome. Aside from this game my lowest ever mark as an L3 was 71, and I only got one of those, he gave me a 66. Then you speak to other referees and it turns out he had a reputation for it, he didn't just keep the mark low, he made things up to justify that low mark.

People say it doesn't happen even though we all know it does, referees cop an injury or a flu bout if they get an observer that they knew was going to screw them over. But there was no need to with this one, he was one of the highest average markers unless you were a 1st year L3.

 

[DavidObs]

That's always been the case unfortunately, they just have to be cleverer about it now. I ran the line years ago to a 1st year L3 and I thought he was excellent, the observer came in after and tore him to pieces, all three of us were gobsmacked, he was talking about things that just simply had not happened. Step forward a year and I was the L3 and the same happened again, same ground, same observer, same outcome. Aside from this game my lowest ever mark as an L3 was 71, and I only got one of those, he gave me a 66. Then you speak to other referees and it turns out he had a reputation for it, he didn't just keep the mark low, he made things up to justify that low mark.

People say it doesn't happen even though we all know it does, referees cop an injury or a flu bout if they get an observer that they knew was going to screw them over. But there was no need to with this one, he was one of the highest average markers unless you were a 1st year L3.

Whoever it was was doing other Obsevers an injustice.

 

[Kes]

You may be surprised. There is no template for anyone. I can see very young and very good, very young and not so good. Experienced very good and experienced and not so good. Sometimes in can just be because of attitudes.

No real difference to my own experiences gained so far. I accept that you've been in the Observer game longer than I have David but only speak as I've found up to now. I've been quite lucky (I believe) in so far as almost every referee I've observed since 2017 has had few if any Development Points and most of those would have been 5-4 observations. Maybe that's just me and my own way of observing? At L4, I've found the younger referees gaining marks in fitness, movement and LOTG knowledge whereas the older and often more experienced referees are showcasing game and player management skills.
Horses for courses ...

 

[DavidObs]

No real difference to my own experiences gained so far. I accept that you've been in the Observer game longer than I have David but only speak as I've found up to now. I've been quite lucky (I believe) in so far as almost every referee I've observed since 2017 has had few if any Development Points and most of those would have been 5-4 observations. Maybe that's just me and my own way of observing? At L4, I've found the younger referees gaining marks in fitness, movement and LOTG knowledge whereas the older and often more experienced referees are showcasing game and player management skills.
Horses for courses ...

I can relate to all of that.