Central limit theorem and moment generating function

For a random variable $latex X$, we can simply define a moment generating function as $latex MG(t) \triangleq E[e^{t X}]$. Then, $latex MG(t)^{(n)}|_{t=0} = E[X^{n}e^{tX}]|_{t=0}=E[X^n]$ is simply the $latex n$-th moment of $latex X$.

Easy to verify that $latex \mathcal{N}(0,\sigma^2)$ has moment generating function of $latex e^{\frac{t^2\sigma^2}{2}}$ since

$latex E[e^{tX}]=\frac{1}{\sqrt{2\pi\sigma^2}}\int e^{-\frac{x^2}{2\sigma^2}}e^{tx}dx=\frac{1}{\sqrt{2\pi\sigma^2}}\int e^{-\frac{1}{2\sigma^2}[(x-t\sigma^2)^2-t^2\sigma^4]}dx=e^{\frac{t^2\sigma^2}{2}}$

Central limit theorem

The central limit theorem states that sum of independent variables will converge to Gaussian. For simplicity, let’s just show the case with variables are i.i.d. and zero mean. Denote $latex S_n = \frac{1}{n}\sum_{i=1}^n X_i$. The main idea of the proof here is to show a scaled version of $latex S_n$ will have the moment generating function of $latex \mathcal{N}(0,\sigma^2)$, where $latex \sigma^2$ is the variance of $latex X_i$.

Proof of central limit theorem

First, we should aware that $latex Var(S_n)=\frac{1}{n^2}\sum_{i=1}^n Var(X_i)=\frac{1}{{n}} Var(X)$. Therefore,  $latex S_n$ will converge to an impulse, however, $latex \sqrt{n} S_n$ instead will converge to $latex \mathcal{N}(0,\sigma^2)$.

Note that

$latex E[e^{t \sqrt{n}S_n}]=E[\prod_{i=1}^n e^{t\frac{X_i}{\sqrt{n}}}]=\prod_{i=1}^n E[e^{t\frac{X_i}{\sqrt{n}}}]=\prod_{i=1}^n MG(\frac{t}{\sqrt{n}})$

Now, consider the log of the expression and take the limit,

$latex \lim_{n\rightarrow \infty}\log E[e^{t \sqrt{n}S_n}]=\lim_{n\rightarrow \infty} n \log MG(\frac{t}{\sqrt{n}})=\lim_{y\rightarrow 0}\frac{ \log MG(ty)}{y^2}=\frac{t}{2}\lim_{y\rightarrow 0}\frac{ MG'(ty)}{y MG(ty)}=\frac{t^2}{2}\lim_{y\rightarrow 0}\frac{ MG”(ty)}{1}=\frac{t^2\sigma^2}{2}$

Therefore $latex E[e^{t \sqrt{n}S_n}]\rightarrow e^{\frac{t^2\sigma^2}{2}}$ and consequently $latex S_n \sim \mathcal{N}(0,\sigma^2)$ as $latex n$ goes to infinity.

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *