Introduction

Let ${X_1, X_2, \ldots, X_n}$ be a random sample (i.e. $X_1, X_2, \ldots, X_n$ are independent and identically distributed). The sample mean and sample standard deviation are defined respectively as:

$$\bar{X}=\frac{1}{n}\sum_{i=1}^n X_i, \quad S=\sqrt{S^2}=\sqrt{\frac{1}{n-1}\sum_{i=1}^n(X_i-\bar{X})^2}.$$

If the sample is taken from a normal population, then $\bar{X}$ and $S^2$ are independent. Without the assumption of normality, Zhang (2007) shows that the covariance of $\bar{X}$ and $S^2$ is $\mu_3/n$, where $\mu_3$ is the third central moment of $X_1$ and $n$ denotes the sample size, and Sen (2012) provides the correlation between $\bar{X}$ and $S^2$. Now naturally it is of interest to know what the covariance of $\bar{X}$ and $S$ is without the assumption of normality. In the next section we present our main results; we give the proofs in Section 3 and an example in Section 4.

Main Results

In what follows, denote the mean, variance, third and fourth central moments of $X_1$ by $\mu$, $\sigma^2$, $\mu_3$ and $\mu_4$, respectively.

Theorem 1. The asymptotic covariance of $\bar{X}$ and $S$ is

$$\frac{\mu_3}{2n\sigma}. $$

Corollary. The asymptotic correlation between $\bar{X}$ and $S$ is

$$\frac{\gamma_1}{\sqrt{\gamma_2+2}}, \label{corr}$$

where $$\gamma_1=\frac{\mu_3}{\sigma^3}, \ \hbox{and}\ \gamma_2=\frac{\mu_4}{\sigma^4}-3.$$

Remark 1: The result shown in the Corollary has been given in Miller (1997), but there is no proof or derivation.

Proofs

Proof of Theorem 1. Let

$$Y_i=\frac{X_i-\mu}{\sigma}, \quad \hbox{for}\ i=1, 2, \ldots, n.$$

It follows that

$$\bar{X}=\sigma \bar{Y} + \mu, \ \hbox{and}\ S=\sigma S_Y,$$

where $\bar{Y}$ and $S_Y$ are the sample mean and sample standard deviation of $\{Y_1, Y2, \ldots, Y_n\}$. Thus,

$$\begin{array}{ccl} \hbox{cov}(\bar{X}, S) &=&\hbox{cov}(\sigma\bar{Y}+\mu, \sigma S_Y)\\ &=&\sigma^2 \hbox{cov}(\bar{Y}, S_Y)\\ &=&\sigma^2 E\left(\bar{Y}S_Y\right). \end{array}$$

Since the Taylor's expansion of function $f(x)=\sqrt{x}$ at $x=1$ is $$1+\frac{1}{2}(x-1)+o(x-1),$$ we have

$$S_Y\approx 1+\frac{1}{2}(S_Y^2-1)=\frac{1}{2}(S_Y^2+1),$$

from which it follows that

$$\begin{array}{ccl} E(\bar{Y}S_Y)&\approx & \frac{1}{2}E\left(\bar{Y}(S_Y^2+1)\right)\\ &=&\frac{1}{2}E(\bar{Y}S_Y^2)\\ &=&\frac{1}{2}\frac{E(Y_1^3)}{n}\ \ \hbox{(according to Zhang, 2007)}\\ &=&\frac{1}{2}\frac{\mu_3}{\sigma^3 n}. \end{array}$$

Therefore,

$$\hbox{cov}(\bar{X}, S)\approx \sigma^2 \frac{1}{2}\frac{\mu_3}{\sigma^3 n}=\frac{\mu_3}{2n\sigma}.$$

Proof of the Corollary. We have obtained the asymptotic covariance of $\bar{X}$ and $S$, and we need to derive the variances of $\bar{X}$ and $S$. Since

$$\hbox{Var}(\bar{X})=\frac{\sigma^2}{n}, \ \hbox{and}\ \hbox{Var}(S)=E(S^2)-(E(S))^2=\sigma^2-(E(S))^2,$$

we only need to show how to derive the asymptotic mean of $S$. If we keep the quadratic term, then the Taylor's expansion of $\sqrt{x}$ at $x=1$ is

$$\sqrt{x}=1+\frac{1}{2}(x-1)-\frac{1}{8}(x-1)^2+o((x-1)^2).$$ Now we have $$S_Y\approx 1+\frac{1}{2}(S_Y^2-1)-\frac{1}{8}(S_Y^2-1)^2.$$

Thus,

$$\begin{array}{ccl} E(S)&=&E(\sigma S_Y)\\ &\approx& \sigma \left[1-\frac{1}{8}\hbox{Var}(S_Y^2)\right]\\ &=& \sigma\left[1-\frac{1}{8}\left(\frac{2}{n-1}+\frac{EY_1^4-3}{n}\right)\right]\\ &=&\sigma\left[1-\frac{1}{8}\left(\frac{2}{n-1}+\frac{\gamma_2}{n}\right) \right]; \end{array}$$

it follows that

$$\hbox{Var}(S)\approx \frac{\sigma^2}{4}\left(\frac{2}{n-1}+\frac{\gamma_2}{n}\right),$$

if the $o(\frac{1}{n})$ terms are discarded. (Note that in the above for the variance of $S_Y^2$, we used the formula in Miller, 1997, p. 7. A direct derivation of the variance formula is available from the author upon request.)

An Example

The accuracy of the result in Theorem 1 depends on the parent distribution and sample size $n$; in this section, we use an example to illustrate its accuracy.

Let $X_1, X_2, \ldots, X_n$ be independent and identically distributed Bernoulli random variables and $P(X_1=1)=p$, where $0<p<1$. In this case the third central moment of $X_1$ $$\mu_3=p(1-p)(1-2p),$$ and the standard deviation of $X_1$

$$\sigma=\sqrt{p(1-p)},$$

thus the asymptotic covariance of $\bar{X}$ and $S$ is

$$\frac{\mu_3}{2n\sigma}=\frac{\sqrt{p(1-p)}(1-2p)}{2n}.$$

Noticing that $X_i^2=X_i$ for $i=1, 2, \ldots, n$, we have

$$\begin{array}{ccl} S^2&=&\frac{1}{n-1}\sum_{i=1}^n(X_i-\bar{X})^2 \nonumber \\ &=&\frac{1}{n-1}\left[\sum_{i=1}^nX_i-\frac{1}{n}\left(\sum_{i=1}^nX_i\right)^2\right] \nonumber \\ &\equiv& \frac{1}{n-1}\left(T_n - \frac{1}{n}T_n^2\right), \end{array}$$

where $T_n=\sum_{i=1}^nX_i$ and it has the binomial distribution $b(n, p)$. Now, we write

$$\begin{array}{ccl} \hbox{cov}(\bar{X}, S)&=&\hbox{cov}\left(\displaystyle\frac{1}{n}T_n, \frac{1}{\sqrt{n-1}}\sqrt{T_n-\frac{1}{n}T_n^2}\right)\\ &=&\displaystyle\frac{1}{n\sqrt{n-1}}\left[E\left(T_n\sqrt{T_n-\frac{1}{n}T_n^2}\right)\right.\\& &\left.-E\left(T_n\right)E\left(\sqrt{T_n-\frac{1}{n}T_n^2}\right)\right]\\ &=&\displaystyle\frac{1}{n\sqrt{n-1}}\left[\sum_{j=1}^n j\sqrt{j-\frac{1}{n}j^2}p_j\right.\\ &&\left.-np\sum_{j=1}^n \sqrt{j-\frac{1}{n}j^2}p_j\right], \end{array}$$

where the binomial probability mass

$$p_j=\left( \begin{array}{c} n\\ j \end{array} \right)p^j(1-p)^{n-j}, \ \hbox{for}\ j=1, 2, \ldots, n.$$

For some given values of $n$ and $p$, we are able to compute the exact and asymptotic covariance respectively; we present the results in Table 1. We see from Table 1 that the results given by Theorem 1 is accurate even for the sample size $n$ as small as $5$.

Table 1: For various values of $n$ and $p$, we obtain exact (indicated by “Exact”) and asymptotic (indicated by “Asy”) covariances.

Remark 2: For $p=0.5$ regardless of the value of $n$, our numerical results suggest that the covariance is equal to $0$.

Reference

Miller, R. G. (1997). Beyond ANOVA. Chapman and Hall.

Sen, A. (2012). On the Interrelation Between the Sample Mean and the Sample Variance. The American Statistician, 66, 112-117.

Zhang, L. (2007). Sample Mean and Sample Variance: Their Covariance and Their (In)Dependence. The American Statistician, 61, 159-160.