Blog / Why Highly Correlated Self-Test Scores May Still Not Agree

Why Highly Correlated Self-Test Scores May Still Not Agree

Correlation describes co-movement; agreement asks whether numerical results are close enough for the intended use.

October 14, 2026 · 5 min read · By Cepingku Editorial Team

A high correlation says that two scores move together. It does not say they give close numerical results or can be substituted for one another. Bland and Altman’s original paper explains that correlation measures linear relation, not agreement: points on any straight line can have perfect correlation, whereas numerical agreement requires points near the line of equality.

A fictional example: r = 1, yet a fixed bias

Let A = [10, 20, 30] and B = [12, 22, 32]. Because B is always A plus 2, their Pearson correlation is r = 1. Their ranks and spacing match perfectly. But paired differences B−A are [2, 2, 2], and the mean difference is +2. If these were intended to be interchangeable readings in the same unit, the constant two-point offset shows systematic bias. Whether they are still close enough for the intended use depends on whether a pre-specified acceptable difference permits two points.

A difference-versus-mean plot puts each paired difference on the vertical axis and the pair’s mean on the horizontal axis. Here the means are [11, 21, 31] and every difference is 2. The plot displays bias and can reveal whether differences grow, shrink, or follow a pattern as the score gets larger. Plotting differences against A or B alone is not a substitute because the shared value creates a statistical artefact.

Three pairs cannot establish limits of agreement

The sample standard deviation of the three fictional differences is zero. A mechanical calculation of mean difference ±1.96 times that standard deviation would put both limits at 2. That does not establish that the population limits are 2. With only three pairs, there is no reliable basis for estimating the difference distribution or limits of agreement. Bland and Altman stress that limits estimated from a sample would change in another sample.

Before looking at data, define what difference is acceptable for the intended use. A tolerable difference depends on whether it would change a report, screen, or research decision. Then inspect bias, spread, and estimated limits against that pre-specified range. If differences increase with the mean, a single constant limit may not fit; proportional error or a suitable transformation may need consideration.

Comparable scores come first

Agreement analysis presumes a meaningful comparison: the two results need to target the same construct, use comparable units, and support a similar interpretation. A raw total and a percentile, or scores with different item counts, scales, or constructs, cannot be made interchangeable merely because they correlate. State any conversion and the intended interpretation before applying an agreement method. The Big Five Personality Test organises one set of self-reported responses across five dimensions. This article does not compare it with another assessment or claim that this site’s scores have been validated for agreement, validity, or interchangeability.

Want to see where you land?

Take the Big Five Personality Test and get your results in minutes.

Take the Big Five Personality Test →

← Back to the blog