how can you give someone a mark that you believe to be average if you don't actually know what the average mark across all refs / observers is? if i'd received an average mark i wouldnt have had an issue
Level 4 Observer Reports
- Thread starter Big Cat
- Start date
how can you give someone a mark that you believe to be average if you don't actually know what the average mark across all refs / observers is? if i'd received an average mark i wouldnt have had an issue
Regarding Observer Reports, these are the things which have struck me as obvious
1) There are too many sub-competencies
2) The scoring and calculation seems way over complicated
Because the referees ask what / how are we were marking them and want the detail. It used to be just a mark in each section, from our training.
On the level 4 development day we were told that steps are being then to train observers to bring them all on to the same hymn sheet. We weren't allowed to mention observers (presumably they are the main topic at these events usually usually!), but they were clear that steps are being taken so that's they're observing on a more uniform basis. Obviously that will take time but it must be so disheartening to referee really well and lose out on promotion due to factors out of your control.
I've far too short a fuse to have the situation @es1 had and just start again next year! I think everyone accepts that people see things differently but to be given an erroneous mark (accepted by the observer) and miss out would be a tough blow to take.
I've far too short a fuse to have the situation @es1 had and just start again next year! I think everyone accepts that people see things differently but to be given an erroneous mark (accepted by the observer) and miss out would be a tough blow to take.
How would people feel about this:
A system will calculate the average mark given by an observer over a period. Observers who tend to mark low would have all their marks adjusted upwards to make their average mark closer to the overall average. The same would happen to marks by observer who tend to mark high, but in the other direction.
It would mean that a referee is not penalised if he happens to be observed by observers who mark low (and doesn't benefit from getting generous ones).
A system will calculate the average mark given by an observer over a period. Observers who tend to mark low would have all their marks adjusted upwards to make their average mark closer to the overall average. The same would happen to marks by observer who tend to mark high, but in the other direction.
It would mean that a referee is not penalised if he happens to be observed by observers who mark low (and doesn't benefit from getting generous ones).
Some Observers will see lots of poor performances. Some Observers will see lots of good performances. Most Observers will see a combination of both. Just like any other bell-curve. By upgrading or downgrading Observer scores which lie towards one side or tother, you could just be upgrading/downgrading groups of poor/good Referees
A statistician would be able to identify when a particular Observer is scoring too high or too low over an extended period, but it wouldn't be right to for non-statisticians to make assumptions or false conclusions
Perhaps the FA employed statisticians when devising the scoring systems. Perhaps they use statisticians to identify Observers who are generally scoring too low/high. Quite unlikely they do however!
Doesn't really work as an observer could have a season where he ends up watching games where nothing really happens, whereas another could have a season with loads of firecracker games where the referees really have to excel.
The reports are all quality controlled and if the mark doesn't match the written text it is reviewed. And it is much better now than it used to be as there are few very low and very high marks that skew the system. In my fourth season at L4 I averaged over 79 and didn't get promoted, the following season I was over 80 and went up as top. Back then, and I'm only talking 10 years ago, you could get over 80 from one observer without really standing out, whereas the same performance with a different observer would get you closer to 70. I've always thought that I was perhaps protected by assessors in that final season because they were surprised I hadn't already gone up. I got an 84 in one game where the only really big decision I had to make I made a mess of, or rather I got the decision correct but then allowed the penalty to be taken with the sent off player still on the pitch and had to have it retaken. Trust me, it is much fairer now than it was back then.
ChasObserverRefDeveloper
Regular Contributor
A few assorted facts to help any discussion:
When the question was raised as the system changed, in April 2020 Lincs22 posted on RefChat:
I haven't done any yet, but the training briefings is that - standard is 70. What is expected at this level as been slightly redefined, so expect more 7.0 in the three main categories than previously. If your performance is not better than the expectation, then you will get 7.0.
We also expect more sub 70 scores. The two test scores done indicated a drop in the average mark by between 2.8 and 3.5 per game. And don't think you can make up the marks by doing things tidily.
Alan Wilkie said that a 74/75 now would be a rare as a 78/79 last year - you will need to have a brilliant performance to exceed 75 now.
All observers who cover matches with Level 3 and/or Level 4 referees received initial training, and each had their reports analysed and reviewed to improve consistency.
All observers watching Level 3 and/or Level 4 referees have attended (online) training events each season, with two separate sessions already this time round.
All observers at those levels get access to two LOTG quizzes at the start of the season, plus a monthly online LOTG quiz, with results published (not individual scores)
All observer reports at those levels are likely to be selected at random for quality review at any stage. Every observer gets at least one review per season.
Referees unhappy with an observation report can challenge it, based on the score not reflecting the words, or an error in law. Records are kept of these to arrive at figures showing what percentage are successful, etc.
Most 5/6 Leagues keep a record of observer marks, and share them as an average observer mark and some also publish each observer's average, but typically only at the end of the season.
Where a very low or very high mark is produced, many leagues require the observer to share the report with the Observer Co-ordinator before finalising it.
Each observer can only watch any referee a maximum of twice a season.
The above facts need to be seen in conjunction with the Covid situation, as in 2020 the FA felt that it would take a couple of seasons to get the new system working as they wish (a bit like VAR) but with two seasons curtailed the lead time will extend into this season.
When the question was raised as the system changed, in April 2020 Lincs22 posted on RefChat:
I haven't done any yet, but the training briefings is that - standard is 70. What is expected at this level as been slightly redefined, so expect more 7.0 in the three main categories than previously. If your performance is not better than the expectation, then you will get 7.0.
We also expect more sub 70 scores. The two test scores done indicated a drop in the average mark by between 2.8 and 3.5 per game. And don't think you can make up the marks by doing things tidily.
Alan Wilkie said that a 74/75 now would be a rare as a 78/79 last year - you will need to have a brilliant performance to exceed 75 now.
All observers who cover matches with Level 3 and/or Level 4 referees received initial training, and each had their reports analysed and reviewed to improve consistency.
All observers watching Level 3 and/or Level 4 referees have attended (online) training events each season, with two separate sessions already this time round.
All observers at those levels get access to two LOTG quizzes at the start of the season, plus a monthly online LOTG quiz, with results published (not individual scores)
All observer reports at those levels are likely to be selected at random for quality review at any stage. Every observer gets at least one review per season.
Referees unhappy with an observation report can challenge it, based on the score not reflecting the words, or an error in law. Records are kept of these to arrive at figures showing what percentage are successful, etc.
Most 5/6 Leagues keep a record of observer marks, and share them as an average observer mark and some also publish each observer's average, but typically only at the end of the season.
Where a very low or very high mark is produced, many leagues require the observer to share the report with the Observer Co-ordinator before finalising it.
Each observer can only watch any referee a maximum of twice a season.
The above facts need to be seen in conjunction with the Covid situation, as in 2020 the FA felt that it would take a couple of seasons to get the new system working as they wish (a bit like VAR) but with two seasons curtailed the lead time will extend into this season.
Good start... Happy with that? How does that translate to your fledgling Merit Table in MOAS
Good mark, well done!
I wouldn't know how to find it tbh.
Yes I'm pleased with it. First one at 4 so it's nice to have a steady start.
In MOAS go to Statistics and select Referee Ranking. You'll have a number at the bottom like... = 7/20
Meaning 7th out of the 20 Refs who've had at least one score
6/17. I'd be delighted to be in the top half at the end of the season for both observations and club marks and to enjoy the season, those are my ambitions.
When are club marks posted? I'd like to be top half just because that's an arbitrary goal I've given myself but what I'm ultimately interested in is whether the marks I get match with my opinion of where I think they should be based on how the games went etc if that makes sense. I've never once seen a club mark given to me from any match (and I've done almost 300 now).
Our pool releases club marks every 2 months. By that all you get is your position on the club mark list, not a breakdown of actual marks
Disappointed in my first report... .4 below league average in what my only point of note was not being 15-20 yards from the ball... The game was end to end with long balls over the top and I ran over 6.2 miles, I'm bound to not be within 15-20 yards when they are going from the middle of defence over the top wide?!
Local derby - 8 yellows, 1 red and no decisions wrong in the entire game. Match control never in doubt, 22 handshakes and an email from one of the clubs saying how well I did. Even the observer described it in the debrief as a good game.
Already feeling disillusioned.
Local derby - 8 yellows, 1 red and no decisions wrong in the entire game. Match control never in doubt, 22 handshakes and an email from one of the clubs saying how well I did. Even the observer described it in the debrief as a good game.
Already feeling disillusioned.
Standard fare I'm afraid. If you think there's anything worth appealing happy to assist, sounds like it should be 0.4 the other way at least
Your description of the game and the score don't seem to marry up. Assuming Mr O agreed with your sanctions, liked your discipline process and there was nothing obviously preventative you could've done, I'd have thought you'd get enough 'above expected' in the highly weighted sections to take you above the league average
Early days though... Quite honestly, I'm not sure I entertained the idea of further promotion as a first year Level 4, so I wouldn't get overly disillusioned. Four more assessments and mid-season reset etc... and just completely different experiences both in the middle and on the line
Rebounding from disappointment is par for the course in our game