MAE vs MSE: The Nudge Argument
Why the median minimises absolute error and the mean minimises squared error, by nudging the guess.
Summarise data with a single guess . Which is best depends on the loss:
Nudge test: move by a small and ask whether the loss goes up or down. At the optimum, no nudge in either direction helps.
Absolute error: only the side matters
Each term changes by exactly . It does not matter how far away is, only which side it is on:
Points you move away from cost each; points you move toward save each.
At the median (odd ), the other points split on each side, so their changes cancel exactly. The medianβs own term goes from to , so the net change is in either direction. Any nudge hurts. For even there is no median term, and every between the two middle values is equally optimal.
Squared error: the size of the residual matters
The derivative of is , so each point pulls with a force proportional to its residual:
Two consequences:
- Leaving the zero-error point is almost free. The medianβs term goes from to , which is second order. Under leaving it cost , which is first order and enough to block the move.
- What drives the move is imbalance, not big residuals in general. If one sideβs residuals add up to more than the otherβs, that side pulls harder. The optimum is where , i.e. , the mean.
If the data is symmetric, the residuals balance, mean = median, and there is no incentive to move. Skew and outliers create the imbalance, and squaring makes large residuals count for more. That is why the mean follows outliers and the median does not.
Example
Take . The median is and the mean is .
| (median) | ||
| (mean) |
Moving from to under , the two points on the left cost each, the two on the right save each, and the median term costs : net . Under the single far point at pulls with residual , outweighing the three small residuals on the left combined (), so the loss falls until the residuals balance at the mean.
| Pull per point | Cost of leaving the zero-error point | Optimum | |
| (sign only) | (first order) | median: equal counts each side | |
| residual | (negligible) | mean: residuals sum to zero |