EE 364 Supplemental – Week 06
Moments, Variance, Distribution-Specific Results, Higher-Order Moments, Beta Function
Moments
(a) Population Statistics
Population mean: \[ \mu_X = E_X[X] = \begin{cases} \displaystyle\sum_x x \cdot f(x) & \text{discrete} \\[6pt] \displaystyle\int_{-\infty}^{\infty} x \cdot f(x)\, dx & \text{continuous} \end{cases} \]
if sum/integral exists (i.e. finite).
\(E_X[X]\) “averages out the randomness.”
(b) Sample Statistics
Sample mean: \(\bar{X}_n = \frac{1}{n}\sum_{k=1}^{n} X_k\) (r.v.)
\(\bar{X}_n\) is a r.v.; realizations: \(\bar{x}_n = \frac{1}{n}\sum_{k=1}^{n} x_k\).
\[ \mu_X = E_X[X] = \begin{cases} \displaystyle\sum_x x \cdot p_X(x) & X \text{ discrete} \\[6pt] \displaystyle\int_{-\infty}^{\infty} x \cdot f_X(x)\, dx & X \text{ continuous} \end{cases} \]
if sum/integral exists (is finite).
\(\mu_X\) = constant, \(X\) = r.v.
Example: Coin Flip
\(\Omega = \{H, T\}\), \(\mathcal{A} = 2^\Omega\). \(X\): \(H \mapsto +\$1\) (prob \(p\)), \(T \mapsto -\$1\) (prob \(1-p\)).
\((\Omega, \mathcal{A}, P)\) in background.
\[ E_X[X] = P[H] \cdot x_H + P[T] \cdot x_T = p(\$1) - q(\$1) \]
\(\therefore\) if \(p = q\): \(E_X[X] = 0\).
Aside: Casino: \(E_{\text{you}}[X] < 0\). Rational price = \(E[\text{gain}]\) (“break even” on average vs. \(\min[\text{loss}]\)).
LOTUS
\(X\) random variable, \(Y = g(X)\):
\[ E_Y[Y] = E_X[g(X)] = \begin{cases} \displaystyle\int_{-\infty}^{\infty} g(x) \cdot f_X(x)\, dx \\[6pt] \displaystyle\sum_x g(x) \cdot p_X(x) \end{cases} \]
if \(E[|X|] < \infty\).
Population Variance
\[ \sigma_X^2 = V_X[X] = E_X\!\left[(X - E_X[X])^2\right] \]
if finite (\(\sigma_X^2 < \infty\)).
Note: Cauchy has \(\sigma_X^2 = \infty\).
Binomial Moments
If \(X \sim b(n, k, p)\):
- \(E[X] = np\)
- \(V[X] = npq\)
Proof. (1) \(E[X]\):
\[ \begin{aligned} E[X] &= \sum_{k=0}^{n} k \cdot b(n,k,p) = \sum_{k=1}^{n} k \cdot \frac{n!}{k!(n-k)!}\, p^k q^{n-k} \\ &= \sum_{k=1}^{n} \frac{n!}{(k-1)!(n-k)!}\, p^k q^{n-k} \end{aligned} \]
Put \(j = k-1\), \(k = j+1\):
\[ = \sum_{j=0}^{n-1} \frac{n \cdot (n-1)!}{j!(n-1-j)!}\, p^{j+1} q^{(n-1)-j} = n \cdot p \underbrace{\sum_{j=0}^{n-1} \frac{(n-1)!}{j!(n-1-j)!}\, p^j q^{(n-1)-j}}_{= 1 \text{ (binomial theorem)}} = np \]
(2) \(E[X^2]\):
\[ E[X^2] = \sum_{k=0}^{n} k^2 \cdot \frac{n!}{k!(n-k)!}\, p^k q^{n-k} = np \cdot \sum_{j=0}^{n-1} (j+1) \frac{(n-1)!}{j!((n-1)-j)!}\, p^j q^{(n-1)-j} \]
\[ = np \left[\underbrace{\sum_{j=0}^{n-1} j \cdot \frac{(n-1)!}{j!((n-1)-j)!}\, p^j q^{(n-1)-j}}_{= (n-1)p \text{ by (1)}} + \underbrace{\sum_{j=0}^{n-1} \frac{(n-1)!}{j!((n-1)-j)!}\, p^j q^{(n-1)-j}}_{= 1 \text{ by B.T.}}\right] \]
\[ = np[(n-1)p + 1] = np(np + (1-p)) = np(np + q) = (np)^2 + npq \]
\[ \therefore V[X] = E[X^2] - E^2[X] = (np)^2 + npq - (np)^2 = npq \quad \square \]
Proposition 1: Affine Transformation
If \(Y = aX + b\) (\(a \neq 0\)) and \(X \sim f_X\), then:
- \(E_Y[Y] = a \cdot E_X[X] + b = a\mu_X + b\)
- \(V_Y[Y] = a^2 \cdot V_X[X] = a^2 \sigma_X^2\)
Proof. (Prove for continuous; discrete: replace integral with sum.)
(1)
\[ \begin{aligned} E_Y[Y] = E_X[aX + b] &= \int_{-\infty}^{+\infty}(ax + b) f_X(x)\, dx \\ &= a\underbrace{\int_{-\infty}^{+\infty} x \cdot f_X(x)\, dx}_{= E_X[X]} + b\underbrace{\int_{-\infty}^{\infty} f_X(x)\, dx}_{= 1} \\ &= a \cdot E_X[X] + b \end{aligned} \]
(2)
\[ \begin{aligned} V_Y[Y] = V_X[aX + b] &= E_X[(aX + b - E[aX + b])^2] \\ &= E_X[(aX + \cancel{b} - aE_X[X] - \cancel{b})^2] \\ &= E_X[(a(X - E_X[X]))^2] \\ &= a^2 \cdot \underbrace{E_X[(X - E_X[X])^2]}_{V_X[X]} \\ &= a^2 \cdot V_X[X] \end{aligned} \]
Proposition 2: Variance Shortcut
\[ V_X[X] = E_X[X^2] - E_X^2[X] \]
Proof. \[ \begin{aligned} V_X[X] = E[(X - E_X[X])^2] &= E_X[X^2 - 2X \cdot \underbrace{E_X[X]}_{\text{constant}} + \underbrace{(E_X[X])^2}_{\text{constant}}] \\ &= E_X[X^2] - 2 E_X[X] \cdot E_X[X] + (E_X[X])^2 \\ &= E_X[X^2] - (E_X[X])^2 \quad \square \end{aligned} \]
Proposition 3: Standardization
If \(Z = \frac{X - \mu}{\sigma}\) (\(\sigma_X^2 < \infty\)), then:
- \(E[Z] = 0\)
- \(V[Z] = 1\)
Proof. (1)
\[ E_Z[Z] = E_X\!\left[\frac{X - \mu}{\sigma}\right] = E_X\!\left[\frac{1}{\sigma} X - \frac{\mu}{\sigma}\right] = \frac{1}{\sigma} \underbrace{E_X[X]}_{\mu} - \frac{\mu}{\sigma} = \frac{\mu}{\sigma} - \frac{\mu}{\sigma} = 0 \]
(2)
\[ V_Z[Z] = V_X\!\left[\frac{1}{\sigma} X + \frac{\mu}{\sigma}\right] = \frac{1}{\sigma^2} \cdot \underbrace{V_X[X]}_{\sigma^2} = \frac{\sigma^2}{\sigma^2} = 1 \quad \square \]
Geometric / Negative Binomial Moments
If \(X \sim G(p)\) (Geometric, \(0 < p < 1\)):
- \(\mu_X = \frac{1}{p}\)
- \(\sigma_X^2 = \frac{q}{p^2}\)
\(\therefore\) If \(X \sim NB\) (Negative Binomial, sum of \(k\) independent \(G(p)\)):
- \(\mu_X = \frac{k}{p}\)
- \(\sigma_X^2 = \frac{kq}{p^2}\)
Since \(X = X_1 + \cdots + X_k\) and independent \(X_j \sim G(p)\).
Proof. (1) \(E[X]\):
\[ \mu_X = E[X] \stackrel{G(p)}{=} \sum_{n=1}^{\infty} n \cdot p \cdot q^{n-1} = p \sum_{n=1}^{\infty} n q^{n-1} \]
\[ = p \cdot \sum_{n=1}^{\infty} \frac{d}{dq} q^n = p \cdot \frac{d}{dq}\!\left(\sum_{n=1}^{\infty} q^n\right) \]
R.O.C.: \(p < 1\) (\(|q| < 1\)), \(\therefore\) uniform convergence (\(\therefore\) commute).
\[ = p \cdot \frac{d}{dq}\!\left(\frac{q}{1-q}\right) = p \cdot \frac{(1-q) - q(-1)}{(1-q)^2} = \frac{p}{(1-q)^2} \stackrel{q=1-p}{=} \frac{p}{p^2} = \frac{1}{p} \]
(2) \(E[X^2]\):
\[ E[X^2] = \sum_{n=1}^{\infty} n^2 \cdot p \cdot q^{n-1} = p \cdot \sum_{n=1}^{\infty} n^2 q^{n-1} \]
\[ = p \cdot \sum_{n=1}^{\infty} \frac{d}{dq}(n \cdot q^n) \stackrel{\text{R.O.C.}}{=} p \cdot \frac{d}{dq}\!\left(\sum_{n=1}^{\infty} n \cdot q^n\right) = p \cdot \frac{d}{dq}\!\left(q \sum_{n=1}^{\infty} n q^{n-1}\right) \]
\[ = p \cdot \frac{d}{dq}\!\left(\frac{q}{p} \cdot E[X]\right) \stackrel{(1)}{=} p \cdot \frac{d}{dq}\!\left(\frac{q}{p^2}\right) = p \cdot \frac{d}{dq}\!\left(\frac{1}{(1-q)^2}\right) \]
\[ = p \cdot \frac{(1-q)^2 + 2q(1-q)}{(1-q)^4} = \frac{p}{(1-q)^2} + \frac{2pq}{(1-q)^3} = \frac{1}{p} + \frac{2q}{p^2} \]
\(p > 0\) throughout.
\[ = \frac{2}{p^2} - \frac{1}{p} \]
\[ \therefore \sigma_X^2 = E[X^2] - E^2[X] = \frac{2}{p^2} - \frac{1}{p} - \frac{1}{p^2} = \frac{1}{p^2} - \frac{1}{p} = \frac{1-p}{p^2} = \frac{q}{p^2} \quad \square \]
Uniform Moments
If \(X \sim U[a, b]\):
- \(E[X] = \frac{a + b}{2}\)
- \(V[X] = \frac{(b - a)^2}{12}\)
Proof. (1)
\[ E_X[X] = \int_{-\infty}^{+\infty} x \cdot f_X(x)\, dx = \int_a^b x \cdot \frac{1}{b-a}\, dx = \frac{1}{b-a}\left[\frac{x^2}{2}\right]_a^b = \frac{1}{b-a} \cdot \frac{b^2 - a^2}{2} = \frac{a+b}{2} \]
(2)
\[ E_X[X^2] = \int_a^b x^2 \cdot \frac{1}{b-a}\, dx = \frac{1}{b-a}\left[\frac{x^3}{3}\right]_a^b = \frac{b^3 - a^3}{3(b-a)} = \frac{b^2 + ab + a^2}{3} \]
\[ \begin{aligned} V_X[X] &= E_X[X^2] - (E_X[X])^2 = \frac{b^2 + ab + a^2}{3} - \left(\frac{a+b}{2}\right)^2 \\ &= \frac{4b^2 + 4ab + 4a^2}{12} - \frac{3a^2 + 6ab + 3b^2}{12} = \frac{b^2 - 2ab + a^2}{12} = \frac{(b-a)^2}{12} \quad \square \end{aligned} \]
Cauchy Moments
If \(X \sim C(m, d)\) (Cauchy):
- \(E[X]\) does not exist
- \(V[X] = \infty\)
Proof. (1) \(E[X]\):
\[ E_X[X] = \int_{-\infty}^{\infty} x \cdot \frac{1}{\pi d(1 + (\frac{x-m}{d})^2)}\, dx \]
Let \(u = \frac{x-m}{d}\), \(du = \frac{dx}{d}\):
\[ = \int_{-\infty}^{\infty}(ud + m) \frac{1}{\pi(1+u^2)}\, du = \frac{d}{\pi}\int_{-\infty}^{+\infty}\frac{u}{1+u^2}\, du + \frac{m}{\pi}\underbrace{\int_{-\infty}^{+\infty}\frac{1}{1+u^2}\, du}_{= \pi \text{ (pdf integrates to 1)}} \]
The first integral: \(\int_{-\infty}^{\infty}\left|\frac{u}{1+u^2}\right| du \leq \int_{-\infty}^{\infty}\left|\frac{u}{u^2}\right| du = \int_{-\infty}^{+\infty}\left|\frac{1}{u}\right| du = \infty\).
\(\therefore E[X]\) does not exist.
(2) \(E[X^2]\):
\[ E_X[X^2] = \int_{-\infty}^{\infty} x^2 \cdot \frac{1}{\pi(1+x^2)}\, dx = \int_{-\infty}^{+\infty}\frac{x^2}{\pi(1+x^2)}\, dx \leq \int_{-\infty}^{+\infty}\frac{x^2}{\pi x^2}\, dx = \int_{-\infty}^{+\infty}\frac{1}{\pi}\, dx = \infty \]
Variance as Dispersion
Variance \(\sigma_X^2 = E_X[(X - \mu_X)^2]\) – type of dispersion (average “distance”\({}^2\)).
Variance \(\neq\) Dispersion. Could use \(|\cdot|\) (absolute value) instead of \((\cdot)^2\) – but math more difficult.
Higher-Order Moments
\[ E_X[X^k] = \int_{-\infty}^{\infty} x^k \cdot f_X(x)\, dx \qquad \text{(if } < \infty\text{)} \]
In general: \(k\)-th moment exists \(\implies\) \((k-1)\)-th moment exists (but not \(\Leftarrow\)).
\(\therefore \sigma_X^2 < \infty \implies |\mu_X| < \infty\).
If \(k \leq m\) and \(k \in \mathbb{Z}^+\) and \(m \in \mathbb{Z}^+\), then
\[ E[X^m] \text{ exists} \implies E[X^k] \text{ exists.} \]
Proof. (Continuous case; replace integrals with sums for discrete.)
\[ \begin{aligned} \int_{-\infty}^{\infty} |x|^k f_X(x)\, dx &= \int_{|x| \leq 1} |x|^k f_X(x)\, dx + \int_{|x| > 1} |x|^k f_X(x)\, dx \\ &\leq \int_{|x| \leq 1} |x|^k f_X(x)\, dx + \int_{|x| > 1} |x|^m f_X(x)\, dx &&\text{[since } k \leq m, |x| > 1\text{]} \\ &\leq \int_{|x| \leq 1} f_X(x)\, dx + \int_{|x| > 1} |x|^m f_X(x)\, dx &&\text{[since } |x| \leq 1 \Rightarrow |x|^k \leq 1\text{]} \\ &\leq \underbrace{\int_{-\infty}^{\infty} f_X(x)\, dx}_{= 1 \text{ (pdf)}} + \int_{|x| > 1} |x|^m f_X(x)\, dx &&\text{[since } \{|x| \leq 1\} \subset \mathbb{R}\text{]} \\ &\leq 1 + \int_{-\infty}^{\infty} |x|^m f_X(x)\, dx &&\text{[since } \{|x| > 1\} \subset \mathbb{R}\text{]} \\ &= 1 + E_X[|X|^m] < \infty &&\text{[since } E[X^m] \text{ exists by hypo.]} \end{aligned} \]
\(\therefore E[X^k]\) exists. \(\square\)
Normal Moments
If \(Z \sim N(0,1)\) (standard normal), then:
\[ E[Z^k] = \begin{cases} 0 & \text{if } k \text{ odd} \\ (k-1)(k-3)\cdots 5 \cdot 3 \cdot 1 & \text{if } k \text{ even} \end{cases} \]
In general, if \(X \sim N(\mu, \sigma^2)\), the \(k\)-th central moment:
\[ E[(X - \mu_X)^k] = \begin{cases} 0 & \text{if } k \text{ odd} \\ \sigma^k(k-1)(k-3)\cdots 5 \cdot 3 \cdot 1 & \text{if } k \text{ even} \end{cases} \]
Proof. Case 1, \(k\) even:
\[ E[Z^k] = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} z^k e^{-z^2/2}\, dz = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} z^{k-1}\underbrace{(z \cdot e^{-z^2/2}\, dz)}_{dv}\, \]
Integration by parts: \(u = z^{k-1}\), \(du = (k-1)z^{k-2}\, dz\); \(dv = z e^{-z^2/2}\, dz\), \(v = -e^{-z^2/2}\).
\[ = \frac{1}{\sqrt{2\pi}}\left(\underbrace{\left.\lim_{z\to\infty} z^{k-1} e^{-z^2/2}\right|_{-\infty}^{\infty}}_{= 0 \text{, L'Hopital's}} + (k-1)\int_{-\infty}^{\infty} z^{k-2} e^{-z^2/2}\, dz\right) \]
\[ = (k-1) \cdot E[Z^{k-2}] \]
\(k\) even: \(= (k-1)(k-3)(k-5)\cdots 5 \cdot 3 \cdot 1\).
Case 2, \(k\) odd:
\(e^{-z^2/2}\) is an even function, \(\therefore z^k e^{-z^2/2}\) is odd if \(k\) odd.
\[ \therefore \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} z^k e^{-z^2/2}\, dz = 0 \quad \square \]
Exponential Moments
\[ E[X^k] = k!\, \Theta^k \quad \text{if } X \sim \text{Exp}(\Theta) \]
Proof. \[ E[X^k] = \int_{-\infty}^{\infty} x^k f_X(x)\, dx = \int_0^{\infty} x^k \cdot \frac{1}{\Theta} e^{-x/\Theta}\, dx \]
\[ = \Theta^{k-1} \int_0^{\infty}\left(\frac{x}{\Theta}\right)^k e^{-x/\Theta}\, d\Theta \]
Let \(u = \frac{x}{\Theta}\), \(du = \frac{dx}{\Theta}\), \(dx = \Theta\, du\); \(x=0 \Rightarrow u=0\), \(x=\infty \Rightarrow u=\infty\).
\[ = \Theta^k \int_0^{\infty} u^k e^{-u}\, \Theta\, du \cdot \frac{1}{\Theta} = \Theta^k \underbrace{\int_0^{\infty} u^{(k+1)-1} e^{-u}\, du}_{\Gamma(k+1)} = \Theta^k \cdot \Gamma(k+1) = \Theta^k \cdot k! \quad \square \]
(since \(\Gamma(\alpha+1) = \alpha \cdot \Gamma(\alpha)\))
Corollary: \(k! = E[X^k]\) if \(X \sim \text{Exp}(1)\).
Ex: If \(X \sim \text{Exp}(\Theta)\), then \(\mu_X = E[X^1] = \Theta\), \(\therefore \sigma_X^2 = E[X^2] - E^2[X] = 2\Theta^2 - \Theta^2 = \Theta^2\).
Gamma Moments
\[ E[X^k] = \Theta^k \cdot \frac{\Gamma(\alpha + k)}{\Gamma(\alpha)} \quad \text{if } X \sim \Gamma(\alpha, \Theta), \quad k > 0, \; \alpha > 0, \; \Theta > 0 \]
Proof. \[ E[X^k] = \int_{-\infty}^{\infty} x^k f_X(x)\, dx = \frac{1}{\Gamma(\alpha)\Theta^\alpha}\int_0^{\infty} x^k \cdot x^{\alpha-1} e^{-x/\Theta}\, dx \]
\[ = \frac{1}{\Gamma(\alpha)\Theta^\alpha}\int_0^{\infty} x^{(\alpha+k)-1} e^{-x/\Theta}\, dx \]
Let \(x = u\Theta\), \(dx = \Theta\, du\):
\[ = \frac{1}{\Gamma(\alpha)\Theta^\alpha}\int_0^{\infty}(u\Theta)^{(\alpha+k)-1} e^{-u}\, \Theta\, du = \frac{\Theta^{(\alpha+k)}}{\Gamma(\alpha)\Theta^\alpha} \underbrace{\int_0^{\infty} u^{(\alpha+k)-1} e^{-u}\, du}_{\Gamma(\alpha+k)} \]
\[ = \Theta^k \cdot \frac{\Gamma(\alpha + k)}{\Gamma(\alpha)} \quad \square \]
Checking: \(\alpha = 1\): \(\Gamma(\alpha + k) = \Gamma(k+1) = k \cdot \Gamma(k) = k!\). \(\therefore E[X^k] = \Theta^k \cdot k!\) if \(X \sim \text{Exp}(\Theta)\).
Gamma Mean and Variance
If \(X \sim \Gamma(\alpha, \Theta)\):
- \(\mu_X = \alpha\Theta\)
- \(\sigma_X^2 = \alpha\Theta^2\)
Proof. \(\mu_X = E[X^1] = \Theta^1 \cdot \frac{\Gamma(\alpha+1)}{\Gamma(\alpha)} = \Theta \cdot \frac{\alpha\,\Gamma(\alpha)}{\Gamma(\alpha)} = \alpha\Theta\)
\[ \begin{aligned} \sigma_X^2 = E[X^2] - E^2[X] &= \Theta^2 \cdot \frac{\Gamma(\alpha+2)}{\Gamma(\alpha)} - \alpha^2\Theta^2 \\ &= \Theta^2\left[\frac{(\alpha+1)\Gamma(\alpha+1)}{\Gamma(\alpha)} - \alpha^2\right] \\ &= \Theta^2\left[\frac{(\alpha+1)\alpha\,\Gamma(\alpha)}{\Gamma(\alpha)} - \alpha^2\right] \\ &= \Theta^2[\alpha^2 + \alpha - \alpha^2] = \alpha\Theta^2 \quad \square \end{aligned} \]
Chi-Square Example
Ex: \(X \sim \chi^2(r) = \Gamma(\frac{r}{2}, 2)\):
\[ E[X^1] = \Theta^1 \cdot \frac{\Gamma(\alpha+1)}{\Gamma(\alpha)} = 2 \cdot \frac{\Gamma(\frac{r}{2}+1)}{\Gamma(\frac{r}{2})} = 2 \cdot \frac{r}{2} = r \]
\[ E[X^2] = \Theta^2 \cdot \frac{\Gamma(\alpha+2)}{\Gamma(\alpha)} = 4 \cdot \frac{(\frac{r}{2}+1)(\frac{r}{2})\Gamma(\frac{r}{2})}{\Gamma(\frac{r}{2})} = r(r+2) = r^2 + 2r \]
\[ \therefore V[X] = E[X^2] - (E[X])^2 = r^2 + 2r - r^2 = 2r \]
\(= \alpha\Theta^2 = \frac{r}{2} \cdot 2^2 = 2r\). Checks out.
Beta Moments
\[ B(\alpha, \beta) = \int_0^1 x^{\alpha-1}(1-x)^{\beta-1}\, dx = \frac{\Gamma(\alpha)\,\Gamma(\beta)}{\Gamma(\alpha+\beta)} \qquad \alpha > 0,\; \beta > 0 \]
(Optional) Moments via Beta Function
\[ E[X^k] = \frac{B(\alpha+k,\, \beta)}{B(\alpha, \beta)} \quad \text{if } X \sim \text{Beta}(\alpha, \beta) \text{ and } k > 0 \]
Proof. \[ E[X^k] = \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}\int_0^1 x^{(\alpha+k)-1}(1-x)^{\beta-1}\, dx = \frac{\underbrace{\int_0^1 x^{(\alpha+k)-1}(1-x)^{\beta-1}\, dx}_{B(\alpha+k,\, \beta)}}{B(\alpha, \beta)} \]
\[ E[X^k] = \frac{\alpha + k - 1}{\alpha + \beta + k - 1} \cdot E[X^{k-1}] \qquad (k \geq 1) \]
Proof. \[ \begin{aligned} E[X^k] = \frac{B(\alpha+k, \beta)}{B(\alpha, \beta)} &= \frac{\Gamma(\alpha+k)\,\Gamma(\beta)}{\Gamma(\alpha+\beta+k)} \cdot \frac{1}{B(\alpha,\beta)} \\ &= \frac{(\alpha+k-1)\,\Gamma(\alpha+k-1)\,\Gamma(\beta)}{(\alpha+\beta+k-1)\,\Gamma(\alpha+\beta+k-1)} \cdot \frac{1}{B(\alpha,\beta)} \\ &= \frac{\alpha+k-1}{\alpha+\beta+k-1} \cdot \frac{B(\alpha+k-1,\, \beta)}{B(\alpha, \beta)} = \frac{\alpha+k-1}{\alpha+\beta+k-1} \cdot E[X^{k-1}] \quad \square \end{aligned} \]
Beta Mean and Variance
If \(X \sim \text{Beta}(\alpha, \beta)\):
\(\mu_X = \dfrac{\alpha}{\alpha + \beta}\)
\(\sigma_X^2 = \dfrac{\alpha\beta}{(\alpha+\beta+1)(\alpha+\beta)^2}\)
Proof. \[ \mu_X = E[X^1] = \frac{B(\alpha+1, \beta)}{B(\alpha, \beta)} = \frac{\Gamma(\alpha+1)\,\Gamma(\beta)}{\Gamma(\alpha+\beta+1)} \cdot \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\,\Gamma(\beta)} = \frac{\alpha\,\cancel{\Gamma(\alpha)}\,\cancel{\Gamma(\beta)}}{(\alpha+\beta)\,\cancel{\Gamma(\alpha+\beta)}} \cdot \frac{\cancel{\Gamma(\alpha+\beta)}}{\cancel{\Gamma(\alpha)}\,\cancel{\Gamma(\beta)}} = \frac{\alpha}{\alpha+\beta} \]
\[ E[X^2] = \frac{B(\alpha+2, \beta)}{B(\alpha, \beta)} = \frac{\Gamma(\alpha+2)\,\Gamma(\beta)}{\Gamma(\alpha+\beta+2)} \cdot \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\,\Gamma(\beta)} = \frac{(\alpha+1)\alpha}{(\alpha+\beta+1)(\alpha+\beta)} \]
\[ \begin{aligned} \sigma_X^2 &= E[X^2] - E^2[X] = \frac{\alpha(\alpha+1)}{(\alpha+\beta+1)(\alpha+\beta)} - \frac{\alpha^2}{(\alpha+\beta)^2} \\[6pt] &= \frac{\alpha(\alpha+1)(\alpha+\beta) - \alpha^2(\alpha+\beta+1)}{(\alpha+\beta+1)(\alpha+\beta)^2} \\[6pt] &= \frac{\alpha^3 + \alpha^2\beta + \alpha^2 + \alpha\beta - \alpha^3 - \alpha^2\beta - \alpha^2}{(\alpha+\beta+1)(\alpha+\beta)^2} = \frac{\alpha\beta}{(\alpha+\beta+1)(\alpha+\beta)^2} \quad \square \end{aligned} \]
Beta Function Proof (Repeated)
(Optional) \(B(\alpha, \beta) = \frac{\Gamma(\alpha)\,\Gamma(\beta)}{\Gamma(\alpha+\beta)}\)
\[ B(\alpha, \beta) = \int_0^1 u^{\alpha-1}(1-u)^{\beta-1}\, du = \frac{\Gamma(\alpha)\,\Gamma(\beta)}{\Gamma(\alpha+\beta)} \qquad \alpha > 0,\; \beta > 0 \]
Proof. \[ \Gamma(\alpha) \cdot \Gamma(\beta) = \int_{x=0}^{\infty} e^{-x} x^{\alpha-1}\, dx \int_{y=0}^{\infty} e^{-y} y^{\beta-1}\, dy \]
By Fubini:
\[ = \int_{y=0}^{\infty}\!\int_{x=0}^{\infty} e^{-(x+y)} x^{\alpha-1} y^{\beta-1}\, dx\, dy \]
Double substitution: \(x = uv\), \(y = u(1-v)\).
Limits: \(0 < x < \infty\), \(0 < y < \infty\) \(\implies\) \(0 < u < \infty\), \(0 < v < 1\).
Jacobian:
\[ \left|\frac{\partial(x,y)}{\partial(u,v)}\right| = \begin{vmatrix} v & u \\ 1-v & -u \end{vmatrix} = |-vu - u + vu| = |-u| = u \]
By change of variable theorem:
\[ \begin{aligned} \Gamma(\alpha)\Gamma(\beta) &= \int_{v=0}^{1}\int_{u=0}^{\infty} e^{-u}(uv)^{\alpha-1}(u(1-v))^{\beta-1} u\, du\, dv \\[6pt] &= \left[\int_0^{\infty} e^{-u}\, u^{(\alpha+\beta)-1}\, du\right]\left[\int_0^1 v^{\alpha-1}(1-v)^{\beta-1}\, dv\right] \\[6pt] &= \Gamma(\alpha+\beta) \cdot B(\alpha, \beta) \end{aligned} \]
\[ \therefore B(\alpha, \beta) = \frac{\Gamma(\alpha)\,\Gamma(\beta)}{\Gamma(\alpha+\beta)} \quad \square \]
Indicator Function Moments
For any \(A \in \mathcal{A}\) in probability space \((\Omega, \mathcal{A}, P)\):
- \(E[I_A] = P(A)\)
- \(V[I_A] = P(A) \cdot P(A^c)\)
Proof. (1)
\[ \begin{aligned} E[I_A] &= \int_\Omega I_A(\omega)\, dP = \int_A \underbrace{I_A(\omega)}_{=1}\, dP + \int_{A^c}\underbrace{I_{A^c}(\omega)}_{=0}\, dP = \int_A 1\, dP + \int_{A^c} 0\, dP = \int_A dP = P(A) \end{aligned} \]
Since \(\Omega = A \cup A^c\).
(2)
\[ \begin{aligned} V[I_A] &= \int_\Omega (I_A(\omega) - E[I_A])^2\, dP = \int_\Omega (I_A(\omega) - P(A))^2\, dP \\ &= \int_\Omega (I_A^2(\omega) + P^2(A) - 2P(A) \cdot I_A(\omega))\, dP \\ &= \int_\Omega I_A^2(\omega)\, dP + P^2(A) \int_\Omega dP - 2P(A) \int_\Omega I_A(\omega)\, dP \\ &= \underbrace{\int_\Omega I_A(\omega)\, dP}_{= P(A)} + P^2(A) - 2P^2(A) \end{aligned} \]
\(I_A^2 = I_A\) for binary (indicator) sets (not fuzzy).
\[ = P[A] - P^2[A] = P(A)(1 - P(A)) = P(A) \cdot P(A^c) \quad \square \]