Evaluation Scale

A Marking Scale for Modern Refereeing

Regular readers of my World Cup projects will know that the marks given at the bottom of each report are determined as per the UEFA scale, a really sound system, which is well-understood by referees and (most :)) observers alike. 

However, this project will experiment with a new system, which I would like to propose as a genuine alternative to, or perhaps more accurately, a revision of, the UEFA scale which is currently used at the highest level.

Below, you can read the rationale for trying to revise the way referee's performances are assessed, some practical examples to help explain this methodology, and looking at some potential contras of the change. But first, let's introduce this new mechanism on the most basic level.  

System

	Descriptor	≈ UEFA
10		>9,0
9	Excellent	9,0 - 8,8
8	Very good	8,7 - 8,6
7	Good	8,5 - 8,3
6	Satisfactory	8,2 - 8,1
5	Deficient	8,0 - 7,9/8
4	Disappointing	7,7 - 7,3
3	Very poor	7,2 - 7,0
2	Unacceptable	6,9 - 6,5
1		<6,5

As you might have guessed by the colour code - any mark of "5" and below roughly constitutes varying degrees of fell short; marks which are above that constitute a pass or better. 

The most important value in new the system is "5". This is the mark denoting a clear match error (see below). However, it also signifies a performance which failed the test (in the UEFA system roughly equating an 8,0 mark). 

Actually, you will note that this system borrows very heavily from what already exists. Probably, a conceptual understanding of the UEFA evaluation system is needed to really get what I'm trying to hit on here. 

That being said - this is not merely a converter between two systems, the whole purpose of this new mechanism is to approach formally assessing performances (in terms of marks) in a different way. 

Having grasped the two elements of the marking system (explained below) which I aimed to improve on from what's in the current mechanism, I hope that the scoring itself is relatively elementary to get one's head around.

Rationale

Video Assistant Refereeing has irrevocably changed football officiating - and it is my conviction that we should have a new way of evaluating official's performances. The current scale used by UEFA is really good, it's no accident at all that I've modelled this new system on it. 

However, there are two areas in which I've looked to develop the current system. 

Crucial mistakes 

Arguably the most brilliant thing about the current framework is the 7,9 system. This is something that simply everyone in UEFA refereeing understands very clearly. However, in our VAR era, perhaps it is a bit out of date. 

Nowadays, in games with live video assistant referees in use, it is (or should be) impossible to commit a totally egregious mistake in terms of Key Match Incidents. VAR has our priorities have shifted a bit - big calls have a different place in the complex arrangement of refereeing evaluation than yesteryear. 

Video Assistant Refereeing is a tool and part of the match officials team, not just a 'big brother' to correct referees when they have 'seriously messed up'. It is important that the way performances are evaluated reflects that. 

That being said, intervention takes time (matches are about football, not the officials!), and of course mistakes that trigger them should be reasonably avoided when possible. Certain KMIs simply should be evaluated correctly on the pitch, and not through intervention of the video match officials. 

Therefore, in matches with VAR in operation, the old idea of crucial mistakes (-> which require intervention) should form part of the assessment of performances and computing of the final mark, but is only one consideration, as opposed to the 7,9 system of yesteryear.
 

The current system of crucial mistakes does not fairly reflect this difference, in my view. Therefore, I propose a new notion - a Clear Match Error. Essentially, this denotes when a referee mistakes an error which directly influences the result of the match, in a blatant way.

It is the same spirit applied in games with and without VAR, too. Not all mistakes which in the old system which have resulted in a 7,9 mark, automatically yield in a "5". The new framework encourages observers to consider the bigger picture (how clear this mistake was to non-refereeing people, the match result, expectations in the respective competition) before assigning a clear match error.

Managing the Game

Anybody who watched the last two FIFA World Cups will note that referees were encouraged to handle the matches in a lenient way. Therefore, as an example, it is pretty inane having to deduct 0,1 from the mark for each theoretical missed caution. 

The focus nowadays is on managing the game, the event. Referees who do that successfully should be rewarded with strong marks. To state this clearly - this system is not a simple converter from the UEFA scale. It encourages an, at least partly, different way of thinking about refereeing.

Context matters; in modern refereeing, we should have an evaluation scheme that reflects that. This underlines quite a key point in my view: not all crucial mistakes are equal. The notion of clear match errors better reflects the realities of refereeing, in my view. 

VMO Marks

I would propose this system (which actually perhaps might already exist) for assessing the performances of the Video Match Officials team. 
 

V - excellent performance, with many (challenging) incidents solved correctly

IV - very good performance, usually with more-than-averagely incident to solve

 

III - standard expected, this is the base mark

II - below expectation, normally one missed intervention, incorrect intervention, or unnecessary / exaggerated delay of the game 

I - unacceptable, multiple important VAR mistakes, lost control of their work

Review

There are, of course, some drawbacks of this new proposal:

- too reductionist, the UEFA scale allows quite intricate differences (7,6 denotes something different from 7,7 eg.), whereas here you are bluntly reduced to mere integers 1-10

- offers way too much discretion from observers in clear match errors, whereas the crucial mistake system theoretically offers objectively and fairness [in reality, the new system is better for this reason in my view - consistency of what is, and isn't, a 7,9 mark in UEFA is quite shocking when you see some reports...]

- a rather fair charge is that there is no need to alter the system for assessing assistant referees; here, a direct conversion from the rough UEFA mark is probably in order

- evaluating the performances of video match officials is somewhat unsatisfactory without being in the VOR yourself; there is a reason why specific observers for that role exist, especially in terms of assessing individual officials

- in domestic competitions, marks are aggregated across the season to generate a league table for match officials; does this system fairly reflect performances when used in this way?

Balance

UEFA's evaluation scale remains an excellent way to assess refereeing performances, and my new system - though looking rather different in terms of the values generated - is really only an attempt to tweak this system. 

In my view, en bloc assigning 7,9s in games with VAR in operation, is to ignore important changes to the skills and aptitudes required to be a top class international referee (Pierluigi Collina speaks eloquently on this topic, and the old TTT blog published a visionary article on the subject too). 

Even besides VAR, refereeing is changing - the macro-picture matters more now, and it is unfair to assess referees in the same we did ten/twenty years ago. Of course, purely technical considerations matter too, but this new system, in my view, juggles these considerations better than the UEFA scale does currently. 

I hope you found this proposal useful and/or at least interesting, and I'd be happy to take questions on it in the comments if any have cropped up.