Covariance of Sample Mean and Sample Standard Deviation
Introduction
Let ${X_1, X_2, \ldots, X_n}$ be a random sample (i.e. $X_1, X_2, \ldots, X_n$ are independent and identically distributed). The sample mean and sample standard deviation are defined respectively as:
$$\bar{X}=\frac{1}{n}\sum_{i=1}^n X_i, \quad S=\sqrt{S^2}=\sqrt{\frac{1}{n-1}\sum_{i=1}^n(X_i-\bar{X})^2}.$$
If the sample is taken from a normal population, then $\bar{X}$ and $S^2$ are independent. Without the assumption of normality, Zhang (2007) shows that the covariance of $\bar{X}$ and $S^2$ is $\mu_3/n$, where $\mu_3$ is the third central moment of $X_1$ and $n$ denotes the sample size, and Sen (2012) provides the correlation between $\bar{X}$ and $S^2$. Now naturally it is of interest to know what the covariance of $\bar{X}$ and $S$ is without the assumption of normality. In the next section we present our main results; we give the proofs in Section 3 and an example in Section 4.Main Results
In what follows, denote the mean, variance, third and fourth central moments of $X_1$ by $\mu$, $\sigma^2$, $\mu_3$ and $\mu_4$, respectively.
Theorem 1. The asymptotic covariance of $\bar{X}$ and $S$ is
$$\frac{\mu_3}{2n\sigma}. $$
Corollary. The asymptotic correlation between $\bar{X}$ and $S$ is
$$\frac{\gamma_1}{\sqrt{\gamma_2+2}}, \label{corr}$$
where $$\gamma_1=\frac{\mu_3}{\sigma^3}, \ \hbox{and}\ \gamma_2=\frac{\mu_4}{\sigma^4}-3.$$Remark 1: The result shown in the Corollary has been given in Miller (1997), but there is no proof or derivation.
Proofs
Proof of Theorem 1. Let
$$Y_i=\frac{X_i-\mu}{\sigma}, \quad \hbox{for}\ i=1, 2, \ldots, n.$$
It follows that$$\bar{X}=\sigma \bar{Y} + \mu, \ \hbox{and}\ S=\sigma S_Y,$$
where $\bar{Y}$ and $S_Y$ are the sample mean and sample standard deviation of $\{Y_1, Y2, \ldots, Y_n\}$. Thus,$$\begin{array}{ccl} \hbox{cov}(\bar{X}, S) &=&\hbox{cov}(\sigma\bar{Y}+\mu, \sigma S_Y)\\ &=&\sigma^2 \hbox{cov}(\bar{Y}, S_Y)\\ &=&\sigma^2 E\left(\bar{Y}S_Y\right). \end{array}$$
Since the Taylor's expansion of function $f(x)=\sqrt{x}$ at $x=1$ is $$1+\frac{1}{2}(x-1)+o(x-1),$$ we have$$S_Y\approx 1+\frac{1}{2}(S_Y^2-1)=\frac{1}{2}(S_Y^2+1),$$
from which it follows that$$\begin{array}{ccl} E(\bar{Y}S_Y)&\approx & \frac{1}{2}E\left(\bar{Y}(S_Y^2+1)\right)\\ &=&\frac{1}{2}E(\bar{Y}S_Y^2)\\ &=&\frac{1}{2}\frac{E(Y_1^3)}{n}\ \ \hbox{(according to Zhang, 2007)}\\ &=&\frac{1}{2}\frac{\mu_3}{\sigma^3 n}. \end{array}$$
Therefore,$$\hbox{cov}(\bar{X}, S)\approx \sigma^2 \frac{1}{2}\frac{\mu_3}{\sigma^3 n}=\frac{\mu_3}{2n\sigma}.$$
Proof of the Corollary. We have obtained the asymptotic covariance of $\bar{X}$ and $S$, and we need to derive the variances of $\bar{X}$ and $S$. Since
$$\hbox{Var}(\bar{X})=\frac{\sigma^2}{n}, \ \hbox{and}\ \hbox{Var}(S)=E(S^2)-(E(S))^2=\sigma^2-(E(S))^2,$$
we only need to show how to derive the asymptotic mean of $S$. If we keep the quadratic term, then the Taylor's expansion of $\sqrt{x}$ at $x=1$ is$$\sqrt{x}=1+\frac{1}{2}(x-1)-\frac{1}{8}(x-1)^2+o((x-1)^2).$$ Now we have $$S_Y\approx 1+\frac{1}{2}(S_Y^2-1)-\frac{1}{8}(S_Y^2-1)^2.$$
Thus,$$\begin{array}{ccl} E(S)&=&E(\sigma S_Y)\\ &\approx& \sigma \left[1-\frac{1}{8}\hbox{Var}(S_Y^2)\right]\\ &=& \sigma\left[1-\frac{1}{8}\left(\frac{2}{n-1}+\frac{EY_1^4-3}{n}\right)\right]\\ &=&\sigma\left[1-\frac{1}{8}\left(\frac{2}{n-1}+\frac{\gamma_2}{n}\right) \right]; \end{array}$$
it follows that$$\hbox{Var}(S)\approx \frac{\sigma^2}{4}\left(\frac{2}{n-1}+\frac{\gamma_2}{n}\right),$$
if the $o(\frac{1}{n})$ terms are discarded. (Note that in the above for the variance of $S_Y^2$, we used the formula in Miller, 1997, p. 7. A direct derivation of the variance formula is available from the author upon request.)An Example
The accuracy of the result in Theorem 1 depends on the parent distribution and sample size $n$; in this section, we use an example to illustrate its accuracy.
Let $X_1, X_2, \ldots, X_n$ be independent and identically distributed Bernoulli random variables and $P(X_1=1)=p$, where $0<p<1$. In this case the third central moment of $X_1$ $$\mu_3=p(1-p)(1-2p),$$ and the standard deviation of $X_1$
$$\sigma=\sqrt{p(1-p)},$$
thus the asymptotic covariance of $\bar{X}$ and $S$ is$$\frac{\mu_3}{2n\sigma}=\frac{\sqrt{p(1-p)}(1-2p)}{2n}.$$
Noticing that $X_i^2=X_i$ for $i=1, 2, \ldots, n$, we have
$$\begin{array}{ccl} S^2&=&\frac{1}{n-1}\sum_{i=1}^n(X_i-\bar{X})^2 \nonumber \\ &=&\frac{1}{n-1}\left[\sum_{i=1}^nX_i-\frac{1}{n}\left(\sum_{i=1}^nX_i\right)^2\right] \nonumber \\ &\equiv& \frac{1}{n-1}\left(T_n - \frac{1}{n}T_n^2\right), \end{array}$$
where $T_n=\sum_{i=1}^nX_i$ and it has the binomial distribution $b(n, p)$. Now, we write$$\begin{array}{ccl} \hbox{cov}(\bar{X}, S)&=&\hbox{cov}\left(\displaystyle\frac{1}{n}T_n, \frac{1}{\sqrt{n-1}}\sqrt{T_n-\frac{1}{n}T_n^2}\right)\\ &=&\displaystyle\frac{1}{n\sqrt{n-1}}\left[E\left(T_n\sqrt{T_n-\frac{1}{n}T_n^2}\right)\right.\\& &\left.-E\left(T_n\right)E\left(\sqrt{T_n-\frac{1}{n}T_n^2}\right)\right]\\ &=&\displaystyle\frac{1}{n\sqrt{n-1}}\left[\sum_{j=1}^n j\sqrt{j-\frac{1}{n}j^2}p_j\right.\\ &&\left.-np\sum_{j=1}^n \sqrt{j-\frac{1}{n}j^2}p_j\right], \end{array}$$
where the binomial probability mass$$p_j=\left( \begin{array}{c} n\\ j \end{array} \right)p^j(1-p)^{n-j}, \ \hbox{for}\ j=1, 2, \ldots, n.$$
For some given values of $n$ and $p$, we are able to compute the exact and asymptotic covariance respectively; we present the results in Table 1. We see from Table 1 that the results given by Theorem 1 is accurate even for the sample size $n$ as small as $5$.Table 1: For various values of $n$ and $p$, we obtain exact (indicated by “Exact”) and asymptotic (indicated by “Asy”) covariances.
Remark 2: For $p=0.5$ regardless of the value of $n$, our numerical results suggest that the covariance is equal to $0$.
Reference
Miller, R. G. (1997). Beyond ANOVA. Chapman and Hall.
Sen, A. (2012). On the Interrelation Between the Sample Mean and the Sample Variance. The American Statistician, 66, 112-117.
Zhang, L. (2007). Sample Mean and Sample Variance: Their Covariance and Their (In)Dependence. The American Statistician, 61, 159-160.