A reminder regarding p-values and their use

The use, and usefulness, of \(p\)-values is controversial. One of the reasons for this is that \(p\)-values often are misinterpreted. One such misinterpretation is that the \(p\)-value is a property of a particular hypothetical process. It is not, as

if the hypothetical process happens to be the true one, \(p\)-values are uniformly distributed in the case of a continuous and invertible cumulative density function.

What follows is a formal presentation of the above statement. Let \(X\) be a random variable with distribution function \(F_X = P(X \leq x)\), and let \(x\) be a realization of \(X\). We can think of \(x\) of an example value of \(X\). Consider for a moment the situation where we don’t know what \(F_X\) is, but we think it is \(F_0\). One thing we can do with \(x\) is to compute \(p = 1 - F_0(x)\), which is the probability of observing \(x\) or something more extreme under our assumption \(F_0\). The standard use is to compare \(p\) with a pre-defined threshold \(\tau\), and say that we reject our hypothesis \(F_0\) if \(p < \tau\).

The \(p\)-value can be seen as a realization of the random variable \(Y = 1 - F_0(X)\). In the case where \(F_0 = F_X\), and \(F_X\) is continuous and invertible, we get that:

\[ \begin{split} F_Y(y) &= P(Y \leq y) = P(1 - F_X(X) \leq y) \\ &= P(X > F^{-1}_X(1 - y)) = 1 - P(X \leq F^{-1}_X(1 - y))\\ &= 1 - F_X(F^{-1}_X(1 - y)) = 1 - (1 - y)\\ &= y \end{split} \] The uniform distribution on \([0,1]\) has \(F_Y\) as distribution function, hence the \(p\)-values in our case are uniformly distributed.