Employee Performance Appraisal Ranking Methods Lessons about the Flaws from “Arrow’s Paradox”

Many organizations have adopted forced ranking performance appraisal systems. Each employee is evaluated against peers and performance arranged from highest to lowest.

These end of the year performance appraisals resemble report cards for adults. There seems to be more angst though, since compensation, career direction, ego, prestige, and morale are all involved.

The crucial activity in the appraisal process is the procedure to determine the specific position of each employee relative to co-workers. The ranking methods are generally used empirically by the participants. A failure to understand and compensate for the inherent limitations of the method can limit the overall fairness of the process.

Ranked performance appraisals can be compared to ranked voting methods. Examples of a ranked votes are the college football and basketball polls published during the season. These rankings, using a method known as the Borda count, assign a different number of points for each position (i.e. 10 for the best team, 9 for the next etc.) Each voter ranks the teams according to his preference. Then, the total number of points each team received are summed for all of the voters and the overall team positions determined.

The major breakthrough in the theoretical understanding of ranked voting methods stems from the work of Nobel Price winner Kenneth Arrow, a mathematician and economist. In 1951, he published a proof of Arrow’s Impossibility Theorem (also known as Arrow’s Paradox). This finding was big news at the time and initiated a large body of work in understanding the validity and limitations of different voting methods.

A simple statement of the theorem: Each ranked voting method has inherent flaws. A slightly more detailed statement is that no voting system (more than 2 voters) based on rank preferences can possibly meet a certain set of reasonable criteria when there are 3 or more options to choose from. These reasonable criteria are detailed in the reference below.

The performance appraisal ranking process is not quite the same as a single winner election, but the method is subject to the flaws identified by the theorem: (1) Strategies can be employed by a subset of the electors to lead to an outcome that is not the choice of the majority; (2) Biases can be introduced if the voting methods are simplified. These factors are briefly examined below.

Distortion Strategies:

An example of a personal strategy to distort the overall results can be encountered in the sports’ polls. In a closed voting system, a voter can grossly change the evaluation of one team (i.e. deliberately ranking it well below its performance level) to benefit another. However, for performance appraisal applications, such selective strategies are mitigated by a collaborative discussion prior to the position assignment. There is some transparency if an elector is attempting to make a significant deviation to advance a personal objective. This discussion can address the potential flaw in the method. Obviously, if there is not a free discussion or some electors are unaware of these strategies, a distorted outcome can occur.

Simplification: Voting by Pairs

A more subtle bias, however, can be encountered during the actual ranking of larger groups by subsets. Generally, the voters have a group with many members to evaluate. In practice, these evaluations are often done using subsets, usually pairs. Comparisons are made in turn until there is agreement in the employee positions.

As an example, there is a group of 15 people whose performance must be ranked. The first six names are:

Adam Don Joe John Mike Sam (and 9 more)

Rather than evaluate the entire group and vote on all 15 at once (for example, using the Borda count), the first two, Adam and Don, may be considered. If Don is evaluated as the better, Don is moved ahead of Adam, the next comparison is with Adam and Joe. The process continues until the order is agreed.

The use of pairwise comparison would appear to get around the “3 option” condition of Arrow’s Paradox. However, this is not the case:

Another common way “around” the paradox is limiting the alternative set to two alternatives. Thus, whenever more than two alternatives should be put to the test, it seems very tempting to use a mechanism that pairs them and votes by pairs. As tempting as this mechanism seems at first glance, it is generally far from meeting (… the reasonable criteria …). The specific order by which the pairs are decided strongly influences the outcome.” (Reference below)

The counterintuitive assertion above is that there is a bias depending on the order of presentation.

In the earlier example with names, the people were listed alphabetically—Adam, Don, Joe etc. The names could be randomized. The bias still remains, it is just transferred to different individuals. Perhaps, the most common and biased case, is when an individual, drawing upon her own experience and opinion, submits the presentation list to the voter group. This approach introduces a subjective bias into the process. One person’s opinion may continue as artifact through to the final ranking.

There may be methods to minimize this flaw, but such methods are generally not known to either the participants or the human resource administrators. That expertise is held by others and is rarely sought.

Arrow’s paradox cannot be avoided. In order to obtain the fairest ranking evaluation process reasonably possible, the participants should some familiarity with the inherent strengths and weaknesses of the method in order to use it properly. The performance appraisal process is only as fair and unbiased as its weakest point.

Posts on Evaluations: Struggling to Give a Good Employee Performance Review–Maintaining Credibility

Dealing with a Bad Employee Performance Appraisal


Reference: http://en.wikipedia.org/wiki/Arrow’s_impossibility_theorem




One Response to Employee Performance Appraisal Ranking Methods Lessons about the Flaws from “Arrow’s Paradox”

  1. cynical says:

    I can’t imagine where you got this example, sir. : }

    Touches awful close to home though.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: