Introduction

Let $X_1,X_2,\dots ,X_n$ be a random sample (i.e. $X_1,X_2,\dots ,X_n$ are independent and identically distributed). The sample mean and sample standard deviation are defined respectively as:

$$\overline{X}=\frac{1}{n}\sum _{i=1}^n{X}_i,S=\sqrt{S^2}=\sqrt{\frac{1}{n-1}\sum _{i=1}^n(X_i-\overline{X}{)}^2}.$$

If the sample is taken from a normal population, then $\overline{X}$ and $S^2$ are independent. Without the assumption of normality, Zhang (2007) shows that the covariance of $\overline{X}$ and $S^2$ is ${\mu }_3/n$, where ${\mu }_3$ is the third central moment of $X_1$ and $n$ denotes the sample size, and Sen (2012) provides the correlation between $\overline{X}$ and $S^2$. Now naturally it is of interest to know what the covariance of $\overline{X}$ and $S$ is without the assumption of normality. In the next section we present our main results; we give the proofs in Section 3 and an example in Section 4.

Main Results

In what follows, denote the mean, variance, third and fourth central moments of $X_1$ by $\mu$, ${\sigma }^2$, ${\mu }_3$ and ${\mu }_4$, respectively.

Theorem 1. The asymptotic covariance of $\overline{X}$ and $S$ is

$$\frac{{\mu }_3}{2n\sigma }.$$

Corollary. The asymptotic correlation between $\overline{X}$ and $S$ is

$$\frac{{\gamma }_1}{\sqrt{{\gamma }_2+2}},$$

where $${\gamma }_1=\frac{{\mu }_3}{{\sigma }^3},\text{and}{\gamma }_2=\frac{{\mu }_4}{{\sigma }^4}-3.$$

Remark 1: The result shown in the Corollary has been given in Miller (1997), but there is no proof or derivation.

Proofs

Proof of Theorem 1. Let

$$Y_i=\frac{X_i-\mu }{\sigma },\text{for}i=1,2,\dots ,n.$$

It follows that

$$\overline{X}=\sigma \overline{Y}+\mu ,\text{and}S=\sigma S_Y,$$

where $\overline{Y}$ and $S_Y$ are the sample mean and sample standard deviation of $\{Y_1,Y2,\dots ,Y_n\}$. Thus,

$$\begin{array}{ccl}\text{cov}(\overline{X},S)& =& \text{cov}(\sigma \overline{Y}+\mu ,\sigma S_Y)\\ & =& {\sigma }^2\text{cov}(\overline{Y},S_Y)\\ & =& {\sigma }^{2}E\left(\overline{Y}{S}_Y\right).\end{array}$$

Since the Taylor's expansion of function $f(x)=\sqrt{x}$ at $x=1$ is $$1+\frac{1}{2}(x-1)+o(x-1),$$ we have

$$S_Y\approx 1+\frac{1}{2}(S_Y^2-1)=\frac{1}{2}(S_Y^2+1),$$

from which it follows that

$$\begin{array}{ccl}E(\overline{Y}{S}_Y)& \approx & \frac{1}{2}E(\overline{Y}(S_Y^2+1))\\ & =& \frac{1}{2}E(\overline{Y}{S}_Y^2)\\ & =& \frac{1}{2}\frac{E(Y_1^3)}{n}\text{(according to Zhang,2007)}\\ & =& \frac{1}{2}\frac{{\mu }_3}{{\sigma }^{3}n}.\end{array}$$

Therefore,

$$\text{cov}(\overline{X},S)\approx {\sigma }^2\frac{1}{2}\frac{{\mu }_3}{{\sigma }^{3}n}=\frac{{\mu }_3}{2n\sigma }.$$

Proof of the Corollary. We have obtained the asymptotic covariance of $\overline{X}$ and $S$, and we need to derive the variances of $\overline{X}$ and $S$. Since

$$\text{Var}(\overline{X})=\frac{{\sigma }^2}{n},\text{and}\text{Var}(S)=E(S^2)-(E(S){)}^2={\sigma }^2-(E(S){)}^2,$$

we only need to show how to derive the asymptotic mean of $S$. If we keep the quadratic term, then the Taylor's expansion of $\sqrt{x}$ at $x=1$ is

$$\sqrt{x}=1+\frac{1}{2}(x-1)-\frac{1}{8}(x-1{)}^2+o((x-1{)}^2).$$ Now we have $$S_Y\approx 1+\frac{1}{2}(S_Y^2-1)-\frac{1}{8}(S_Y^2-1{)}^2.$$

Thus,

$$\begin{array}{ccl}E(S)& =& E(\sigma S_Y)\\ & \approx & \sigma [1-\frac{1}{8}\text{Var}(S_Y^2)]\\ & =& \sigma [1-\frac{1}{8}(\frac{2}{n-1}+\frac{E{Y}_1^4-3}{n})]\\ & =& \sigma [1-\frac{1}{8}(\frac{2}{n-1}+\frac{{\gamma }_2}{n})];\end{array}$$

it follows that

$$\text{Var}(S)\approx \frac{{\sigma }^2}{4}(\frac{2}{n-1}+\frac{{\gamma }_2}{n}),$$

if the $o(\frac{1}{n})$ terms are discarded. (Note that in the above for the variance of $S_Y^2$, we used the formula in Miller, 1997, p. 7. A direct derivation of the variance formula is available from the author upon request.)

An Example

The accuracy of the result in Theorem 1 depends on the parent distribution and sample size $n$; in this section, we use an example to illustrate its accuracy.

Let $X_1,X_2,\dots ,X_n$ be independent and identically distributed Bernoulli random variables and $P(X_1=1)=p$, where $0<p<1$. In this case the third central moment of $X_1$ $${\mu }_3=p(1-p)(1-2p),$$ and the standard deviation of $X_1$

$$\sigma =\sqrt{p(1-p)},$$

thus the asymptotic covariance of $\overline{X}$ and $S$ is

$$\frac{{\mu }_3}{2n\sigma }=\frac{\sqrt{p(1-p)}(1-2p)}{2n}.$$

Noticing that $X_i^2=X_i$ for $i=1,2,\dots ,n$, we have

$$\begin{array}{}& \begin{array}{ccl}{S}^2& =& \frac{1}{n-1}\sum _{i=1}^n(X_i-\overline{X}{)}^2\\ & =& \frac{1}{n-1}[\sum _{i=1}^n{X}_i-\frac{1}{n}{\left(\sum _{i=1}^n{X}_i\right)}^2]\\ & \equiv & \frac{1}{n-1}(T_n-\frac{1}{n}{T}_n^2),\end{array}\end{array}$$

where $T_n=\sum _{i=1}^n{X}_i$ and it has the binomial distribution $b(n,p)$. Now, we write

$$\begin{array}{ccl}\text{cov}(\overline{X},S)& =& \text{cov}\left({\displaystyle \frac{1}{n}{T}_n,\frac{1}{\sqrt{n-1}}\sqrt{T_n-\frac{1}{n}{T}_n^2}}\right)\\ & =& {\displaystyle \frac{1}{n\sqrt{n-1}}[E\left(T_n\sqrt{T_n-\frac{1}{n}{T}_n^2}\right)}\\ & & -E\left(T_n\right)E\left(\sqrt{T_n-\frac{1}{n}{T}_n^2}\right)]\\ & =& {\displaystyle \frac{1}{n\sqrt{n-1}}[\sum _{j=1}^{n}j\sqrt{j-\frac{1}{n}{j}^2}{p}_j}\\ & & -np\sum _{j=1}^n\sqrt{j-\frac{1}{n}{j}^2}{p}_j],\end{array}$$

where the binomial probability mass

$$p_j=\left(\begin{array}{c}n\\ j\end{array}\right)p^j(1-p{)}^{n-j},\text{for}j=1,2,\dots ,n.$$

For some given values of $n$ and $p$, we are able to compute the exact and asymptotic covariance respectively; we present the results in Table 1. We see from Table 1 that the results given by Theorem 1 is accurate even for the sample size $n$ as small as $5$.

Table 1: For various values of $n$ and $p$, we obtain exact (indicated by “Exact”) and asymptotic (indicated by “Asy”) covariances.

Remark 2: For $p=0.5$ regardless of the value of $n$, our numerical results suggest that the covariance is equal to $0$.

Reference

Miller, R. G. (1997). Beyond ANOVA. Chapman and Hall.

Sen, A. (2012). On the Interrelation Between the Sample Mean and the Sample Variance. The American Statistician, 66, 112-117.

Zhang, L. (2007). Sample Mean and Sample Variance: Their Covariance and Their (In)Dependence. The American Statistician, 61, 159-160.