probability-and-statistics.bigb

= Probability and statistics
{wiki}

= Kullback-Leibler divergence
{title2=$D_{\mathrm{KL}}$}
{parent=Probability and statistics}
{c}
{wiki}

For probability densities $p$ and $q$, the Kullback-Leibler divergence from $q$ to $p$ is
$$
D_{\mathrm{KL}}(p\Vert q)
=\mathbb E_p\!\left[\log\frac{p(X)}{q(X)}\right].
$$

= Gibbs inequality
{c}
{parent=Kullback-Leibler divergence}
{wiki}

For <probability distribution>[probability distributions] $p$ and $q$ on the same finite set,
$$
D_{\mathrm{KL}}(p\Vert q)
=\sum_i p_i\log\frac{p_i}{q_i}\geq0,
$$
with equality exactly when $p=q$ on the support of $p$.

= Probability theory
{parent=Probability and statistics}
{wiki}

= Probability space
{title2=$(\Omega,\mathcal F,\mathbb P)$}
{parent=Probability theory}
{wiki}

A probability space is a <measure space> whose total measure is one. Its measurable subsets are <event>[events], and its measure $\mathbb P$ is their probability.

= Event
{parent=Probability space}
{wiki=Event_(probability_theory)}

An event is a measurable subset of a <probability space>.

= Uniform random point in a disk
{parent=Probability theory}

For a point uniformly distributed in the unit disk, its polar coordinates are independent with densities
$$
f_R(r)=2r\mathbf1_{[0,1]}(r),
\qquad
f_\Theta(\theta)=\frac1{2\pi}\mathbf1_{[0,2\pi)}(\theta).
$$

= Origin in a triangle of three radial random points
{parent=Uniform random point in a disk}

For three independent points from any rotationally symmetric planar distribution with nonzero radii, the origin lies in their triangle exactly when their three angles do not all lie in one semicircle. Three independent uniform angles have this property with probability $1/4$.

= Independent sigma-algebras
{parent=Probability theory}
{wiki=Independence_(probability_theory)}

Two sub-sigma-algebras $\mathcal F_1,\mathcal F_2$ are independent when
$$
\mathbb P(A\cap B)=\mathbb P(A)\mathbb P(B)
$$
for every $A\in\mathcal F_1$ and $B\in\mathcal F_2$.

= Independent events
{parent=Independent sigma-algebras}
{wiki=Independence_(probability_theory)}

Events $A_1,\ldots,A_n$ are mutually independent when
$$
\mathbb P\left(\bigcap_{j\in J}A_j\right)
=\prod_{j\in J}\mathbb P(A_j)
$$
for every finite index set $J$.

= Independence extended from generating pi-systems
{parent=Independent sigma-algebras}

If two pi-systems satisfy the probability factorization pairwise, their generated sigma-algebras are independent. For one event in the first family, the events in the second sigma-algebra satisfying the factorization form a Dynkin system; apply <Dynkin lemma>, then repeat with the roles reversed.

= Tail sigma-algebra
{title2=$\mathcal T$}
{parent=Independent sigma-algebras}
{wiki=Tail_event}

For a sequence of random variables $(X_n)$, the tail sigma-algebra is
$$
\mathcal T=\bigcap_{n\geq1}\sigma(X_n,X_{n+1},\ldots).
$$
Its events are unaffected by changing any finite initial segment.

= Kolmogorov zero-one law
{c}
{parent=Tail sigma-algebra}
{wiki=Kolmogorov%27s_zero%E2%80%93one_law}

The <tail sigma-algebra> of a sequence of <independent random variables> is trivial: every tail event has probability zero or one.

= Product measure
{parent=Probability theory}
{wiki=Product_measure}

The product measure $\bigotimes_n\mu_n$ is characterized on measurable rectangles depending on finitely many coordinates by the product of their component measures.

= Random variable
{parent=Probability theory}
{wiki}

A random variable is a measurable <function> from a probability space to a measurable space.

= Independent random variables
{parent=Random variable}
{wiki=Independence_(probability_theory)}

Random variables $X_1,\ldots,X_n$ are independent when their generated sigma-algebras are <independent sigma-algebras>. Equivalently, their joint distribution is the product of their marginal distributions.

= Independent random variable
{synonym}

= Independent and identically distributed random variables
{parent=Independent random variables}
{wiki=Independent_and_identically_distributed_random_variables}

A family of <independent random variables> is independent and identically distributed when every variable has the same <probability distribution>.

= IID random variables
{c}
{synonym}

= IID random variable
{c}
{synonym}

= Queueing theory
{parent=Probability theory}
{wiki=Queueing_theory}

= Poisson arrivals see time averages
{parent=Queueing theory}
{c}
{wiki=Poisson_arrivals_see_time_averages}

The Poisson arrivals see time averages property says that an external <Poisson process> independent of a stationary system sees the system-state distribution at its arrival times. Thus the fraction of arrivals that accept a state-dependent admission rule $p(n)$ is $\sum_n\pi_np(n)$.

= PASTA
{c}
{synonym}

= M-G-1 queue
{c}
{parent=Queueing theory}
{wiki=M/G/1_queue}

An $M/G/1$ queue has Poisson arrivals, <independent and identically distributed random variables>[independent identically distributed] service times, and one server. If service has mean $m$ and $\lambda m<1$, its mean busy period is
$$
\frac{m}{1-\lambda m}.
$$
This follows by decomposing the busy period into the first service and the busy-period descendants of arrivals during that service.

= Little law
{c}
{parent=Queueing theory}
{wiki=Little%27s_law}

In a stable regenerative queue,
$$
L=\lambda W,
$$
where $L$ is mean population, $\lambda$ is throughput, and $W$ is mean sojourn time. Renewal reward proves it by identifying the area under the population path with the sum of customer sojourn times.

= Convergence of random variables
{parent=Probability theory}
{wiki=Convergence_of_random_variables}

The main modes include almost-sure convergence, convergence in probability, convergence in $L^p$, and convergence in distribution.

= Convergence in distribution
{title2=$X_n\xrightarrow{d}X$}
{parent=Convergence of random variables}
{wiki=Convergence_in_distribution}

Random variables $X_n$ converge in distribution, or converge weakly, to $X$ when
$$
\mathbb E[f(X_n)]\longrightarrow\mathbb E[f(X)]
$$
for every bounded continuous function $f$. For real random variables this is equivalent to convergence of the distribution functions at every continuity point of the limiting distribution function.

= Weak convergence of random variables
{synonym}

= Almost sure convergence
{title2=$X_n\xrightarrow{\mathrm{a.s.}}X$}
{parent=Convergence of random variables}
{wiki}

A sequence of random variables converges almost surely when the set of outcomes on which pointwise convergence fails has probability zero.

= Strong law of large numbers
{parent=Almost sure convergence}
{wiki}

If $(X_n)$ are <independent and identically distributed random variables> with $\mathbb E|X_1|<\infty$, then
$$
\frac1n\sum_{j=1}^nX_j\longrightarrow\mathbb E[X_1]
$$
almost surely.

= Cramer-Wold theorem
{c}
{parent=Convergence of random variables}
{wiki=Cram%C3%A9r%E2%80%93Wold_theorem}

Random vectors $X_n$ converge weakly to $X$ exactly when every linear projection $u\cdot X_n$ converges weakly to $u\cdot X$. The reverse implication follows because these one-dimensional limits give pointwise convergence of the multivariate characteristic functions.

= Convergence in probability
{parent=Convergence of random variables}
{wiki}

Random variables $X_n$ converge in probability to $X$ when
$$
\mathbb P(|X_n-X|>\varepsilon)\to0
$$
for every $\varepsilon>0$.

= Bounded-metric characterization of convergence in probability
{parent=Convergence in probability}

Convergence in probability is equivalent to
$$
\mathbb E\big(|X_n-X|\wedge1\big)\to0.
$$
The forward implication splits at a fixed error threshold; the reverse implication is Markov's inequality.

= Almost-sure subsequence from convergence in probability
{parent=Convergence in probability}

From $X_n\to X$ in probability, choose $n_k$ with
$\mathbb P(|X_{n_k}-X|>2^{-k})<2^{-k}$. The first Borel--Cantelli lemma then gives $X_{n_k}\to X$ almost surely.

= Independent rare-event counterexample to almost-sure convergence
{parent=Convergence in probability}

Independent indicators with success probabilities $1/n$ converge to zero in probability, but successes occur infinitely often almost surely by the second Borel--Cantelli lemma.

= Convergence in Lp from convergence in probability and an Lr bound
{parent=Convergence of random variables}

If $X_n\to X$ in probability and $\sup_n\lVert X_n\rVert_r<\infty$, then $X_n\to X$ in $L^p$ for every $1\leq p<r$. The $L^r$ bound gives uniform integrability of the $p$th powers, while convergence in probability controls their bounded part.

= Central limit theorem
{parent=Convergence of random variables}
{wiki}

= Self-normalized central limit theorem with a second-moment denominator
{parent=Central limit theorem}

For independent identically distributed $Z_i$ with mean zero and variance one,
$$
\frac{n^{-1/2}\sum_{i=1}^nZ_i}{n^{-1}\sum_{i=1}^nZ_i^2}
\xrightarrow dN(0,1).
$$
The numerator obeys the central limit theorem, the denominator converges almost surely to one by the strong law, and Slutsky's theorem combines them.

= Multivariate central limit theorem
{parent=Central limit theorem}
{wiki=Central_limit_theorem#Multidimensional_CLT}

For independent identically distributed random vectors $Z_i$ with mean $\mu$ and finite covariance matrix $V$,
$$
\sqrt n\left(\frac1n\sum_{i=1}^nZ_i-\mu\right)\xrightarrow dN(0,V).
$$

= Poisson limit theorem
{parent=Convergence of random variables}
{c}
{wiki=Poisson_limit_theorem}

Binomial distributions with trial count tending to infinity and success probability tending to zero converge to a Poisson law when their product converges.

= Simple symmetric random walk
{parent=Probability theory}
{wiki=Random_walk}

A simple symmetric random walk starting from $S_0$ has
$$
S_n=S_0+\sum_{k=1}^n\xi_k,
$$
where the increments $\xi_k$ are independent and satisfy
$$
\mathbb P(\xi_k=1)=\mathbb P(\xi_k=-1)=\frac12.
$$

= Martingale
{parent=Probability theory}
{wiki}

An integrable adapted process $(M_n)$ is a martingale when $\mathbb E[M_{n+1}\mid\mathcal F_n]=M_n$.

= Symmetric signs forced by the martingale property
{parent=Martingale}

If every increment $\xi_n=M_n-M_{n-1}$ of a discrete martingale takes values in $\{-1,1\}$, then
$$
\mathbb P(\xi_n=1\mid\mathcal F_{n-1})
=\mathbb P(\xi_n=-1\mid\mathcal F_{n-1})=\frac12.
$$
Iterated conditioning shows that the increments are independent symmetric signs, so $M_n-M_0$ is a <simple symmetric random walk>.

= Supermartingale
{parent=Martingale}
{wiki}

An integrable adapted process $(V_n)$ is a supermartingale when $\mathbb E[V_{n+1}\mid\mathcal F_n]\leq V_n$.

= Supermartingale majorant bound
{parent=Supermartingale}

If a supermartingale $V$ dominates an adapted reward process $Z$, optional sampling gives $\mathbb E Z_\tau\leq V_0$ for every bounded stopping time $\tau$.

= Stopping time
{parent=Martingale}
{wiki}

A random time $\tau$ is a stopping time when $\{\tau\leq n\}\in\mathcal F_n$ for every $n$.

= Stopped martingale in discrete time
{parent=Stopping time}

For a discrete martingale and stopping time $T$,
$$
M_{(n+1)\wedge T}-M_{n\wedge T}
=\mathbf1_{\{T>n\}}(M_{n+1}-M_n).
$$
The indicator is measurable at time $n$, so the stopped process is a martingale. If it is uniformly bounded and $T<\infty$ almost surely, bounded convergence gives $\mathbb E M_T=\mathbb E M_0$ directly.

= Discounted symmetric random-walk exit transform
{parent=Stopped martingale in discrete time}

For simple symmetric random walk stopped on hitting $-a$ or $b$ and $0<z<1$, choose $w>1$ by
$$
z=\frac2{w+w^{-1}},
\qquad
w=\frac{1+\sqrt{1-z^2}}z.
$$
Then $z^nw^{\pm X_n}$ are martingales. Solving the two boundary equations gives
$$
\mathbb E_0z^T
=\frac{w^a+w^b-w^{-a}-w^{-b}}
{w^{a+b}-w^{-(a+b)}}.
$$

= Optional sampling theorem for a supermartingale
{parent=Stopping time}
{wiki=Optional_stopping_theorem}

For bounded stopping times $\sigma\leq\tau$, a supermartingale satisfies $\mathbb E[V_\tau\mid\mathcal F_\sigma]\leq V_\sigma$; equality holds for a martingale.

= Predictable process
{parent=Martingale}
{wiki=Predictable_process}

In discrete time, a process $A_n$ is predictable or previsible when $A_n$ is $\mathcal F_{n-1}$-measurable for every $n\geq1$.

= Previsible process
{synonym}

= Martingale transform
{title2=$(A\mathbin\cdot M)_n$}
{parent=Predictable process}
{wiki=Martingale_transform}

If $M$ is a martingale and $A$ is predictable, then, subject to integrability,
$$
X_n=X_0+\sum_{k=1}^nA_k(M_k-M_{k-1})
$$
is a martingale.

= Recovery of a martingale-transform integrand by conditional covariance
{parent=Martingale transform}

For a square-integrable martingale transform and positive conditional increment variance,
$$
A_n=\frac{\operatorname{Cov}(X_n,M_n\mid\mathcal F_{n-1})}
{\operatorname{Var}(M_n\mid\mathcal F_{n-1})}.
$$

= Stopped martingale
{title2=$M_{n\wedge T}$}
{parent=Martingale transform}
{wiki=Optional_stopping_theorem}

For a stopping time $T$,
$$
M_{n\wedge T}-M_{(n-1)\wedge T}
=\mathbf1_{\{T\geq n\}}(M_n-M_{n-1}).
$$
The indicator is predictable, so a stopped integrable martingale is a martingale.

= Reflection principle for simple symmetric random walk
{parent=Martingale transform}
{wiki=Reflection_principle_(Wiener_process)}

Let $S_0=0$ be a <simple symmetric random walk>, let $T_a$ be its first hitting time of the positive integer $a$, and reverse every increment after $T_a$. The resulting path is again a simple symmetric random walk. Consequently,
$$
\mathbb P\left(\max_{k\leq n}S_k\geq a\right)
=\mathbb P(S_n\geq a)+\mathbb P(S_n\geq a+1).
$$

= Point probability for the maximum of simple symmetric random walk
{parent=Reflection principle for simple symmetric random walk}

For $M_n=\max_{0\leq k\leq n}S_k$ and a positive integer $a$,
$$
\mathbb P(M_n=a)
=\mathbb P(S_n=a)+\mathbb P(S_n=a+1).
$$
Exactly one term can be nonzero because $S_n$ has the parity of $n$.

= Predictable representation in a Rademacher filtration
{parent=Martingale transform}

If $\mathcal F_n$ is generated by independent symmetric signs $\xi_1,\ldots,\xi_n$, every martingale has predictable coefficients $B_n$ such that
$$
M_n=M_0+\sum_{k=1}^nB_k\xi_k.
$$
One may take $B_n=\mathbb E[M_n\xi_n\mid\mathcal F_{n-1}]$.

= Stopped martingale isometry in a Rademacher filtration
{parent=Predictable representation in a Rademacher filtration}

For a bounded stopping time $T$ and a square-integrable representation $M_n=M_0+\sum_{k\leq n}B_k\xi_k$,
$$
\mathbb E[M_T^2]=M_0^2+\mathbb E\sum_{k=1}^TB_k^2.
$$
The cross terms vanish because predictable multiples of distinct independent signs are orthogonal martingale differences.

= Predictable compensator of a discrete supermartingale
{parent=Predictable process}

The increments $\Delta A_{n+1}=V_n-\mathbb E[V_{n+1}\mid\mathcal F_n]$ of a supermartingale are predictable and nonnegative.

= Doob decomposition in discrete time
{parent=Martingale}
{c}
{wiki=Doob_decomposition_theorem}

Every integrable discrete-time supermartingale has the form $V=M-A$, where $M$ is a martingale and $A$ is predictable, integrable, nondecreasing, and starts at zero.

= Snell envelope
{parent=Martingale}
{c}
{wiki}

The finite-horizon Snell envelope of rewards $Z_n$ is defined backward by $V_N=Z_N$ and $V_n=\max\{Z_n,\mathbb E[V_{n+1}\mid\mathcal F_n]\}$. It is the least supermartingale dominating $Z$.

= Complementarity for the Snell envelope compensator
{parent=Snell envelope}

For the Doob compensator of a Snell envelope, $(V_n-Z_n)(A_{n+1}-A_n)=0$: compensation grows only on the stopping region $V_n=Z_n$.

= Optimal stopping time from the compensator
{parent=Snell envelope}

The first time the Snell-envelope compensator is about to become positive is an optimal stopping time; before it the martingale part equals both value and reward at stopping.

= Borel-Cantelli lemmas
{c}
{parent=Probability theory}
{wiki=Borel–Cantelli_lemma}

If $\sum\mathbb P(A_n)<\infty$, only finitely many $A_n$ occur almost surely. For independent events, divergence implies infinitely many occur. Use a union bound for the first and $\prod(1-p_n)\le e^{-\sum p_n}$ for the second.

= Tail-sum formula for expectation
{parent=Borel-Cantelli lemmas}

For a nonnegative <random variable> $X$, <Tonelli theorem> gives
$$
\mathbb E[X]=\int_0^\infty\mathbb P(X>t)\,dt.
$$
Consequently, for each $a>0$, $\mathbb E[X]=\infty$ exactly when $\sum_{n\geq1}\mathbb P(X>an)=\infty$; compare the integral on successive intervals of length $a$.

= Extremes of independent exponential variables
{parent=Borel-Cantelli lemmas}

For independent rate-one exponential variables $X_n$ and $M_n=\max_{m\leq n}X_m$,
$$
\limsup_{n\to\infty}\frac{X_n}{\log n}=1,
\qquad
\liminf_{n\to\infty}\frac{M_n}{\log n}\geq1
$$
almost surely. The first follows by applying both Borel--Cantelli lemmas to thresholds $(1\pm\varepsilon)\log n$; the second uses
$\mathbb P(M_n\leq(1-\varepsilon)\log n)\leq e^{-n^\varepsilon}$.

= Stochastic process
{parent=Probability theory}
{wiki}

A stochastic process is a family of random variables indexed by time or another ordered parameter.

= Gaussian process
{parent=Stochastic process}
{wiki}

A Gaussian process is a <stochastic process> whose every finite vector of values has a <multivariate normal distribution>. Its law is determined by its mean and covariance functions.

= Brownian motion
{parent=Stochastic process}
{c}
{wiki}

A standard Brownian motion starts at zero, has almost surely continuous paths, and has independent Gaussian increments
$$
W_t-W_s\sim N(0,t-s)
\qquad(0\leq s<t).
$$

= Brownian bridge
{parent=Brownian motion}
{c}
{wiki}

A standard Brownian bridge is the centered Gaussian process $B(t)=W(t)-tW(1)$ on $[0,1]$. Its covariance is $\min(s,t)-st$ and it satisfies $B(0)=B(1)=0$.

= Exponential Brownian martingale
{parent=Brownian motion}

For every real $c$,
$$
M_t=\exp\left(cW_t-\frac12c^2t\right)
$$
is a martingale. Conditional expectation factors at time $s$ because the Gaussian increment $W_t-W_s$ is independent of the past and has exponential moment $e^{c^2(t-s)/2}$.

= Risk-neutral measure for the Black-Scholes model
{parent=Exponential Brownian martingale}
{c}

For $S_t=S_0e^{\mu t+\sigma W_t}$, shifting Brownian drift by
$$
\theta=\frac{\mu+\sigma^2/2-r}{\sigma}
$$
makes $S_t=S_0e^{(r-\sigma^2/2)t+\sigma W_t^Q}$ under an equivalent measure $Q$. The discounted stock is then a martingale.

= Power payoff in the Black-Scholes model
{parent=Risk-neutral measure for the Black-Scholes model}
{c}

The time-zero price of the payoff $S_T^p$ is
$$
S_0^p\exp\left[\left((p-1)r+\frac12p(p-1)\sigma^2\right)T\right].
$$

= Brownian time reversal for fixed-strike lookback extrema
{parent=Risk-neutral measure for the Black-Scholes model}

For a Brownian motion with drift $X$, the process $X_T-X_{T-t}$ on $[0,T]$ has the same law as $X_t$. Therefore
$$
X_T-\min_{t\leq T}X_t
\quad\text{and}\quad
\max_{t\leq T}X_t
$$
have the same distribution, which equates the corresponding geometric-Brownian lookback payoffs.

= Gaussian-process characterization of Brownian motion
{parent=Brownian motion}

A centered <Gaussian process> with almost surely continuous paths and covariance $\mathbb E[B_sB_t]=\min(s,t)$ is a standard <Brownian motion>.

= Time inversion of Brownian motion
{parent=Brownian motion}

If $W$ is <Brownian motion>, then
$$
B_0=0,
\qquad B_t=tW_{1/t}\quad(t>0)
$$
is also Brownian motion. Its covariance is $\min(s,t)$, and continuity at zero follows from $W_u/u\to0$ almost surely.

= Brownian motion with drift
{title2=$X_t=W_t+\mu t$}
{parent=Brownian motion}
{wiki}

A Brownian motion with drift $\mu$ is $X_t=W_t+\mu t$ for a standard <Brownian motion> $W$.

= Finite-horizon maximum of Brownian motion with negative drift
{parent=Brownian motion with drift}

For $a,b>0$,
$$
\mathbb P\!\left(\sup_{0\leq s\leq t}(W_s-as)\leq b\right)
=\Phi\!\left(\frac{b+at}{\sqrt t}\right)
-e^{-2ab}\Phi\!\left(\frac{at-b}{\sqrt t}\right).
$$
This follows from the <Brownian reflection principle> weighted by the <Cameron-Martin theorem for a linear drift>.

= Infinite-horizon crossing probability for Brownian motion with negative drift
{parent=Brownian motion with drift}

For $a,y>0$,
$$
\mathbb P\!\left(\sup_{u\geq0}(W_u-au)>y\right)=e^{-2ay}.
$$

= Last passage time above a level for Brownian motion with negative drift
{title2=$T=\sup\{t\geq0:W_t-at>b\}$}
{parent=Brownian motion with drift}

For $a,b>0$ and the convention $\sup\varnothing=0$,
$$
\mathbb P(T\leq t)
=\Phi\!\left(a\sqrt t+\frac b{\sqrt t}\right)
-e^{-2ab}\Phi\!\left(\frac b{\sqrt t}-a\sqrt t\right),
\qquad t>0.
$$

= Brownian reflection principle
{c}
{parent=Brownian motion}
{wiki=Reflection_principle_(Wiener_process)}

Reflecting a Brownian path after its first hit of a level $b>0$ preserves Wiener measure and maps an endpoint $x\leq b$ to $2b-x\geq b$.

= Cameron-Martin theorem for a linear drift
{c}
{parent=Brownian motion}
{wiki=Cameron%E2%80%93Martin_theorem}

On paths through time $t$, translating Brownian motion by the linear drift $\mu s$ changes Wiener measure by the density
$$
\exp\!\left(\mu W_t-\frac12\mu^2t\right).
$$

= Brownian transition semigroup
{parent=Brownian motion}
{c}
{wiki=Weierstrass_transform}

For suitable $f$,
$$
(P_tf)(x)=\mathbb E[f(x+\sqrt tZ)]
$$
defines the Brownian transition semigroup. Its generator identity is
$$
\frac d{dt}P_tf=\frac12P_tf''.
$$

= Brownian compensator martingale
{parent=Brownian transition semigroup}
{c}

For a twice differentiable integrable test function $f$ and Brownian motion $W$,
$$
f(W_t)-\frac12\int_0^tf''(W_s)\,ds
$$
is a martingale. This is the one-dimensional generator form of Dynkin's formula.

= Exponential test-function characterization of Brownian motion
{parent=Brownian motion}

Let $W$ be continuous with $W_0=0$. If
$$
e^{cW_t}-\frac{c^2}{2}\int_0^te^{cW_s}\,ds
$$
is a martingale for every real $c$, then conditional expectations solve
$$
\mathbb E[e^{cW_t}\mid\mathcal F_s]
=e^{cW_s+c^2(t-s)/2}.
$$
Thus $W_t-W_s$ is independent of $\mathcal F_s$ and distributed as $N(0,t-s)$, so $W$ is Brownian motion.

= Characteristic-function convergence theorem
{parent=Probability theory}
{wiki=Lévy%27s_continuity_theorem}

Pointwise convergence of characteristic functions to a function continuous at zero implies convergence in distribution.

= Poisson process
{c}
{parent=Probability theory}
{wiki}

A rate-$\lambda$ Poisson process has independent stationary increments. Conditional on $N_t=n$, arrival times are the order statistics of $n$ independent uniform points on $[0,t]$.

= Poisson processes
{c}
{synonym}

= Poisson thinning
{c}
{parent=Poisson process}
{wiki=Thinning_(probability)}

Independently retaining a point at time $s$ with probability $p(s)$ gives a Poisson count of mean $\lambda\int p(s)\,ds$.

= Binomial thinning conditional on a total
{parent=Poisson thinning}

Conditioned on a total of $n$ Poisson points, the number falling in a region of relative intensity $p$ is $\operatorname{Binomial}(n,p)$.

= Infinite-server queue transient distribution
{parent=Poisson thinning}
{wiki=M/M/∞_queue}

With empty initial state, Poisson arrivals of rate $\lambda$ and exponential lifetimes of rate $\mu$ produce a Poisson occupancy of mean $\lambda(1-e^{-\mu t})/\mu$ at time $t$.

= Kolmogorov forward equation for a Poisson process
{parent=Poisson process}
{c}

The state probabilities satisfy $p_0'=-\lambda p_0$ and $p_n'=\lambda p_{n-1}-\lambda p_n$, yielding the Poisson law of mean $\lambda t$.

= Poisson process conditional arrival times
{parent=Poisson process}
{c}

Conditioned on $N_t=n$, the arrival times are distributed as the order statistics of $n$ independent uniform points on $[0,t]$. Their conditional joint density on
$$
0<s_1<\cdots<s_n<t
$$
is $n!/t^n$, obtained by dividing the Poisson-process density $\lambda^ne^{-\lambda t}$ by $\mathbb P(N_t=n)$.

= Number of one Poisson type before another
{parent=Poisson process}

For independent Poisson processes of rates $\lambda_1$ and $\lambda_2$, the number $K$ of type-two arrivals before the first type-one arrival satisfies
$$
\mathbb P(K=k)
=\left(\frac{\lambda_2}{\lambda_1+\lambda_2}\right)^k
\frac{\lambda_1}{\lambda_1+\lambda_2}.
$$
This follows by marking the superposed process by its arrival type.

= Exponential formula for a marked Poisson sum
{parent=Poisson process}

Let $J_i$ be the points of a rate-$\lambda$ Poisson process and let independent identically distributed marks $X_i$ be independent of the process. Then
$$
\mathbb E\exp\left(\theta\sum_{i=1}^{N_t}g(J_i,X_i)\right)
=\exp\left\{\lambda\int_0^t
\left(\mathbb E e^{\theta g(s,X_1)}-1\right)ds\right\}.
$$
Condition on $N_t$, use uniform order statistics, and sum the resulting exponential series.

= Age of a Poisson process
{parent=Poisson process}

With $J_0=0$, the age $A(t)=t-J_{N_t}$ of a rate-$\lambda$ Poisson process has mean
$$
\mathbb EA(t)=\frac{1-e^{-\lambda t}}{\lambda}.
$$
Given $N_t=n$, the last arrival is the maximum of $n$ uniform points and $\mathbb E[A(t)\mid N_t=n]=t/(n+1)$, including $n=0$.

= Fokker-Planck equation
{c}
{parent=Probability theory}
{wiki=Fokker–Planck_equation}

A Fokker-Planck equation evolves a density by drift and diffusion in conservation-law form.

= Fokker-Planck probability current
{title2=$J$}
{parent=Fokker-Planck equation}
{c}
{wiki=Fokker%E2%80%93Planck_equation}

Writing a Fokker-Planck equation as $\partial_tP=-\partial_xJ$ defines its probability current. For $\partial_tP=\partial_x(gP)+\tfrac12\partial_x^2(hP)$,
$$
J=-gP-\frac12\partial_x(hP).
$$

= Wald identity
{c}
{parent=Probability theory}
{wiki=Wald%27s_equation}

For an integrable stopping time and i.i.d. integrable increments, the expected stopped sum equals expected stopping time times mean increment.

= Poisson point process
{parent=Probability theory}
{c}
{wiki}

A Poisson point process has independent counts on disjoint sets and Poisson count in each set with mean given by its intensity measure.

= Superposition theorem for Poisson point processes
{parent=Poisson point process}
{c}
{wiki=Poisson_point_process\#Superposition_theorem}

The union of independent Poisson point processes with diffuse intensity measures $\mu_1,\ldots,\mu_m$ is a Poisson point process with intensity measure
$$
\mu_1+\cdots+\mu_m.
$$
On every finite collection of disjoint measurable sets, this follows by adding independent <Poisson distribution>[Poisson counts] and checking that the resulting counts remain independent.

= Campbell theorem
{c}
{parent=Poisson point process}
{wiki=Campbell%27s_theorem_(probability)}

For a Poisson point process $\Pi$ with intensity measure $\mu$ and a square-integrable measurable function $g$,
$$
\mathbb E\sum_{x\in\Pi}g(x)=\int g\,d\mu,
\qquad
\operatorname{Var}\left(\sum_{x\in\Pi}g(x)\right)=\int g^2\,d\mu.
$$
The second identity follows because distinct points contribute the square of the mean, while the diagonal contribution is $\int g^2\,d\mu$.

= Non-homogeneous Poisson point process
{c}
{parent=Poisson point process}
{wiki=Poisson_point_process}

Let $\lambda$ be a nonnegative measurable intensity function on a space with reference measure $dx$. A Non-homogeneous Poisson point process has independent counts on disjoint measurable sets and
$$
N(A)\sim\operatorname{Poisson}\left(\int_A\lambda(x)\,dx\right)
$$
whenever the integral is finite.

= Intensity function of a point process
{title2=$\lambda(x)$}
{parent=Non-homogeneous Poisson point process}

The intensity function $\lambda$ is the density of the expected counting measure: $\mathbb E[N(A)]=\int_A\lambda(x)\,dx$.

= Mapping theorem for Poisson point processes
{parent=Poisson point process}
{wiki=Mapping_theorem_(point_process)}

Let $\Pi$ be a Poisson point process with intensity measure $\mu$ and let $f$ be measurable. If the pushforward
$$
f_*\mu(B)=\mu(f^{-1}(B))
$$
is locally finite, then the image counting measure is Poisson with intensity $f_*\mu$. If one requires a simple point process, the pushforward must also be diffuse, so distinct source points do not produce coincident image points with positive probability.

= Radial volume transform of a homogeneous Poisson point process
{parent=Mapping theorem for Poisson point processes}

For a homogeneous Poisson point process of intensity $\lambda$ in $\mathbb R^d$, mapping $x$ to $v_d|x|^d$, where $v_d$ is the unit-ball volume, gives a rate-$\lambda$ Poisson process on $[0,\infty)$. Mapping to $|x|^d$ instead gives rate $\lambda v_d$.

= Kth-nearest-neighbour distance in a homogeneous Poisson point process
{parent=Radial volume transform of a homogeneous Poisson point process}

For intensity $\lambda$ in $\mathbb R^d$, let $R_k$ be the distance to the $k$th closest point and put $\rho=\lambda v_d$. Then $R_k^d$ is the $k$th arrival time of a rate-$\rho$ Poisson process, so
$$
f_{R_k}(r)=\frac{d\rho^k}{(k-1)!}\,
r^{dk-1}e^{-\rho r^d},
\qquad r>0.
$$

= Independent marking theorem for Poisson point processes
{c}
{parent=Poisson point process}
{wiki=Poisson_point_process#Marked_Poisson_point_processes}

Independently assigning a mark to each point of a <Poisson point process>, with mark probabilities allowed to depend measurably on the point, produces independent Poisson point processes for the different marks. If the original intensity is $\lambda(x)\,dx$ and mark $j$ has probability $p_j(x)$, its intensity is $\lambda(x)p_j(x)\,dx$.

= Poisson thinning theorem
{c}
{parent=Poisson point process}
{wiki=Poisson_point_process#Thinning}

Independently retaining a point at $x$ with probability $p(x)$ thins a Poisson point process of intensity $\lambda(x)$ to one of intensity $p(x)\lambda(x)$; retained and rejected processes are independent.

= Order statistic
{parent=Probability theory}
{wiki}

= Order symmetry of independent random variables
{parent=Order statistic}

<Independent and identically distributed random variables> with a <continuous probability distribution> realize each of their possible strict orderings with equal probability.

= Probability distribution
{parent=Probability theory}
{wiki}

A probability distribution assigns probabilities to the possible values of a random variable.

= Beta distribution
{title2=$\operatorname{Beta}(\alpha,\beta)$}
{c}
{parent=Probability distribution}
{wiki}

The beta distribution on $[0,1]$ has density proportional to $x^{\alpha-1}(1-x)^{\beta-1}$ and mean $\alpha/(\alpha+\beta)$.

= Cumulative distribution function
{title2=$F_X(x)$}
{parent=Probability distribution}
{wiki}

The cumulative distribution function of a real <random variable> $X$ is
$$
F_X(x)=\mathbb P(X\leq x).
$$
It is nondecreasing, right-continuous, tends to zero at $-\infty$, and tends to one at $+\infty$.

= Uniform convergence of distribution functions to a continuous limit
{parent=Cumulative distribution function}

If distribution functions $F_n$ converge pointwise to a continuous distribution function $F$, then the convergence is uniform. Choose finitely many points whose successive $F$-increments and two tails are small, then use monotonicity to squeeze $F_n-F$ between its values at adjacent points.

= Cauchy distribution
{title2=$\operatorname{Cauchy}(x_0,\gamma)$}
{c}
{parent=Probability distribution}
{wiki}

The Cauchy distribution with location $x_0$ and scale $\gamma>0$ has <probability density function>
$$
f(x)=\frac{1}{\pi\gamma\left(1+((x-x_0)/\gamma)^2\right)}.
$$
The standard Cauchy distribution has $x_0=0$, $\gamma=1$, <characteristic function> $e^{-|t|}$, and infinite first absolute moment.

= Stability of the Cauchy distribution
{parent=Cauchy distribution}

If $X_1,\ldots,X_n$ are independent standard Cauchy random variables, then
$$
\frac{X_1+\cdots+X_n}{n}
$$
is again standard Cauchy. Indeed, its characteristic function is $(e^{-|t|/n})^n=e^{-|t|}$.

= Stable distribution
{parent=Cauchy distribution}
{wiki}

A probability distribution is stable when every linear combination of independent copies has the same distribution up to a location and scale change. The <Cauchy distribution> is stable with stability parameter one.

= Probability generating function
{title2=$G_X(s)=\mathbb E[s^X]$}
{parent=Probability distribution}
{wiki}

For a nonnegative integer-valued <random variable> $X$, its probability generating function is
$$
G_X(s)=\mathbb E[s^X]=\sum_{n\geq0}\mathbb P(X=n)s^n.
$$
The generating function of a sum of <independent random variables> is the product of their generating functions.

= Probability mass function
{title2=$p_X(x)$}
{parent=Probability distribution}
{wiki}

For a discrete <random variable> $X$, its probability mass function is
$$
p_X(x)=\mathbb P(X=x).
$$
It is nonnegative and sums to one over the possible values of $X$.

= Quantile function
{parent=Probability distribution}
{wiki}

For a distribution function $F$, its generalized inverse or quantile function is
$$
F^{-1}(p)=\inf\{t:F(t)\geq p\}.
$$

= Quantile
{synonym}

= Inverse transform sampling
{parent=Quantile function}
{wiki}

If $U$ is uniform on $[0,1]$, then $F^{-1}(U)$ has distribution function $F$. Indeed, right continuity gives
$$
F^{-1}(p)\leq t\quad\Longleftrightarrow\quad p\leq F(t)
$$
apart from irrelevant endpoint cases.

= Quantile coupling for convergence in distribution
{parent=Quantile function}

If $F_n\to F$ at every continuity point of $F$, then the generalized inverses satisfy
$$
F_n^{-1}(u)\to F^{-1}(u)
$$
at every continuity point of $F^{-1}$. Since a monotone function has only countably many discontinuities, evaluating all quantiles at one uniform random variable gives an almost-sure coupling of the corresponding convergence in distribution.

= Discrete probability distribution
{parent=Probability distribution}
{wiki}

A discrete probability distribution is supported on a finite or countable set.

= Geometric distribution
{parent=Discrete probability distribution}
{wiki}

On the convention supported by $1,2,\ldots$, a geometric variable of parameter $p$ has mass $(1-p)^{k-1}p$, mean $1/p$, and variance $(1-p)/p^2$.

= Coupon collector problem
{parent=Geometric distribution}
{wiki}

When independent uniform draws are made from $n$ types, the time $T_n$ until every type has appeared decomposes as a sum of independent geometric waiting times with parameters
$$
1,\frac{n-1}{n},\ldots,\frac1n.
$$
Consequently
$$
\mathbb ET_n=nH_n,
\qquad
\operatorname{var}(T_n)\leq n^2\sum_{j=1}^{\infty}\frac1{j^2},
$$
and Chebyshev's inequality gives $T_n/(n\log n)\to1$ in probability.

= Negative binomial distribution
{parent=Discrete probability distribution}
{wiki}

On the failures-before-the-$r$th-success convention, a negative binomial variable with success probability $p$ has probability mass function
$$
\mathbb P(X=x)=\binom{x+r-1}{x}(1-p)^x p^r,
\qquad x=0,1,\ldots.
$$
It has expected value $r(1-p)/p$ and variance $r(1-p)/p^2$.

= Negative binomial stopping argument
{parent=Negative binomial distribution}

For the $r$th success to occur after exactly $x$ failures, the final trial must be a success, while the preceding $x+r-1$ trials contain $x$ failures and $r-1$ successes. Choosing the failure locations gives the factor $\binom{x+r-1}{x}$.

= Binomial distribution
{title2=$\operatorname{Bin}(n,p)$}
{parent=Discrete probability distribution}
{wiki}

The binomial distribution counts successes in independent Bernoulli trials with common success probability.

= Multinomial distribution
{title2=$\operatorname{Mult}(n;p_1,\ldots,p_k)$}
{parent=Discrete probability distribution}
{wiki}

The multinomial distribution gives the category counts from $n$ independent trials with category probabilities $p_1,\ldots,p_k$.

= Poisson distribution
{title2=$\operatorname{Pois}(\lambda)$}
{parent=Discrete probability distribution}
{c}
{wiki}

The Poisson distribution with mean lambda assigns mass $e^{-\lambda}\lambda^k/k!$.

= Poisson distributions
{synonym}

= Poisson central limit theorem
{c}
{parent=Poisson distribution}

If $N_\lambda\sim\operatorname{Pois}(\lambda)$ and $\lambda\to\infty$, then
$$
\frac{N_\lambda-\lambda}{\sqrt\lambda}
\ \xrightarrow{\mathrm d}\ N(0,1).
$$
Indeed, its <characteristic function> is
$$
\exp\!\left(\lambda\left(e^{it/\sqrt\lambda}-1-\frac{it}{\sqrt\lambda}\right)\right)
\longrightarrow e^{-t^2/2}.
$$

= Poisson-multinomial conditioning
{parent=Poisson distribution}
{c}
{wiki}

Independent Poisson counts conditioned on their sum have a multinomial distribution.

= Poisson trick
{c}
{parent=Poisson-multinomial conditioning}

For each group $g$, let independent cell counts have Poisson means $\mu_{gj}$. Conditional on their total $n_g$, the cells have a <multinomial distribution> with probabilities
$$
\pi_{gj}=\frac{\mu_{gj}}{\sum_k\mu_{gk}}.
$$
Including a free group main effect in the Poisson log-linear model makes its profiled likelihood for the remaining parameters proportional to the corresponding multinomial likelihood. The two fits therefore give the same fitted proportions and likelihood-ratio comparisons.

= Normal approximation to the Poisson distribution
{parent=Poisson distribution}

For large $\lambda$, a Poisson random variable of mean $\lambda$ is approximately normal with mean and variance $\lambda$.

= Continuous probability distribution
{parent=Probability distribution}
{wiki}

A continuous probability distribution is described by a density with respect to Lebesgue measure.

= Probability density function
{title2=$f_X(x)$}
{parent=Continuous probability distribution}
{wiki}

A probability density function is a nonnegative measurable function $f_X$ satisfying
$$
\mathbb P(X\in A)=\int_A f_X(x)\,dx.
$$
Its integral over the whole sample space is one.

= Gamma distribution
{parent=Continuous probability distribution}
{wiki}

The gamma distribution with shape $a>0$ and rate $r>0$ has density
$$
f(x)=\frac{r^a}{\Gamma(a)}x^{a-1}e^{-rx},
\qquad x>0.
$$

= Erlang distribution
{parent=Gamma distribution}
{c}
{wiki}

The sum of $n$ independent exponential variables of rate $r$ has the Erlang distribution with density
$$
f_n(x)=\frac{r^nx^{n-1}}{(n-1)!}e^{-rx},
\qquad x>0.
$$
Convolving with one more exponential proves the formula inductively.

= Sampling an integer-shape gamma distribution
{parent=Gamma distribution}

If $m$ is a positive integer and $U_1,\ldots,U_m$ are independent uniform variables, then
$$
-\frac1r\sum_{j=1}^m\log U_j
$$
has the gamma distribution with shape $m$ and rate $r$, because each summand is exponential with rate $r$.

= Exponential distribution
{title2=$\operatorname{Exp}(\lambda)$}
{parent=Continuous probability distribution}
{wiki}

The exponential distribution with rate $c>0$ has density
$$
f(x)=ce^{-cx}\mathbf 1_{\{x\geq0\}}.
$$
It is the gamma distribution with shape one.

= Memorylessness of the exponential distribution
{parent=Exponential distribution}

If $T\sim\operatorname{Exp}(\mu)$, then
$$
\mathbb P(T>s+t\mid T>s)=\mathbb P(T>t)=e^{-\mu t}.
$$
Thus an exponential remaining lifetime has the same law regardless of the elapsed lifetime.

= Uniform ratio of independent exponential variables
{parent=Exponential distribution}

If $A$ and $B$ are independent exponential variables with the same rate, then $A/(A+B)$ is uniform on $(0,1)$ and is independent of $A+B$.

= Competing exponential clocks
{parent=Exponential distribution}

For independent exponential variables $S_i$ of rates $q_i$ and $Q=\sum_iq_i$, let $T=\min_iS_i$ and let $K$ be the minimizing index. Then
$$
\mathbb P(K=k,T\geq t)=\frac{q_k}{Q}e^{-Qt}.
$$
Consequently $T$ is exponential with rate $Q$, $\mathbb P(K=k)=q_k/Q$, and $K$ and $T$ are independent.

= Geometric sum of exponential variables
{parent=Exponential distribution}

If $N$ is geometric on $\{1,2,\ldots\}$ with parameter $p$, independently of unit-rate exponential variables $S_i$, then
$$
\sum_{i=1}^NS_i
$$
is exponential with rate $p$. Its moment-generating function is $p/(p-\theta)$ for $\theta<p$.

= Uniform distribution
{title2=$U(a,b)$}
{parent=Continuous probability distribution}
{wiki=Continuous_uniform_distribution}

A continuous uniform distribution has constant density on its interval of support.

= Sum of two independent uniform variables
{parent=Uniform distribution}

For independent $U_1,U_2\sim\operatorname{Uniform}(0,1)$, the sum has triangular density
$$
f(s)=
\begin{cases}
s,&0\leq s\leq1,\\
2-s,&1\leq s\leq2,\\
0,&\text{otherwise}.
\end{cases}
$$
Its upper tail for $1\leq c\leq2$ is $(2-c)^2/2$.

= Uniform split of a random total
{parent=Uniform distribution}

Let $X\geq0$ have density $g$, let $U$ be independent and uniform on $[0,1]$, and set
$$
Y=XU,\qquad Z=X(1-U).
$$
Then
$$
f_{Y,Z}(y,z)=\frac{g(y+z)}{y+z}\mathbf 1_{\{y,z\geq0\}},
\qquad
f_Y(y)=f_Z(y)=\int_y^\infty\frac{g(t)}t\,dt.
$$

= Exponential characterization by a uniform random split
{parent=Uniform split of a random total}

Write $h(y)=\int_y^\infty g(t)/t\,dt$. The two pieces in a uniform split are independent exactly when
$$
-h'(y+z)=h(y)h(z).
$$
Setting $z=0$ shows that $h(y)=ce^{-cy}$ for some $c>0$. Thus the pieces are independent exactly when they are independent exponential variables of the same rate; equivalently, the total has gamma density $c^2xe^{-cx}$.

= Laplace distribution
{parent=Continuous probability distribution}
{c}
{wiki}

The centred Laplace distribution with rate $\theta$ has density $\theta e^{-\theta|x|}/2$.

= Laplace regression
{parent=Laplace distribution}

In the model
$$
Y_i\mid X_i\sim\operatorname{Laplace}(X_i^T\beta,\sigma),
$$
maximum likelihood minimizes the sum of absolute residuals
$$
S(\beta)=\sum_i|Y_i-X_i^T\beta|.
$$
The scale estimate is $\widehat\sigma=S(\widehat\beta)/n$.

= Least absolute deviations
{synonym}

= Symmetric distribution
{parent=Probability distribution}

A distribution symmetric about $c$ is invariant under $x\mapsto2c-x$; when its mean exists, it equals $c$.

= Median
{parent=Probability distribution}
{wiki=Median}

A median is a point with at least half the probability on either side.

= Moment-generating function
{title2=$M_X(t)$}
{parent=Probability distribution}
{wiki}

The moment-generating function is $M_X(t)=E[e^{tX}]$ where finite and encodes moments through derivatives at zero.

= Joint distribution function
{parent=Probability distribution}

The joint distribution function is $F(x,y)=P(X\le x,Y\le y)$; mixed differentiation gives a joint density when sufficiently smooth.

= Normal distribution
{title2=$N(\mu,\sigma^2)$}
{parent=Probability distribution}
{wiki}

The normal distribution has density proportional to $\exp(-(x-\mu)^2/(2\sigma^2))$.

= Gaussian distribution
{synonym}

= Half-normal distribution
{parent=Normal distribution}
{wiki=Half-normal_distribution}

The absolute value of a standard normal random variable has density
$$
\sqrt{\frac2\pi}e^{-x^2/2}\mathbf1_{x\geq0}.
$$

= Log-normal distribution
{parent=Normal distribution}
{wiki=Log-normal_distribution}

A positive random variable is log-normal when its logarithm has a <normal distribution>. If $Z\sim N(0,1)$ and $X=e^Z$, then $\mathbb E[X^n]=e^{n^2/2}$.

= Gaussian integration by parts
{parent=Normal distribution}
{c}
{wiki=Stein%27s_lemma}

For $Z\sim N(0,1)$ and a sufficiently regular function $g$,
$$
\mathbb E[Zg(Z)]=\mathbb E[g'(Z)].
$$
It follows by integrating $zg(z)e^{-z^2/2}$ by parts.

= Standard normal distribution
{title2=$N(0,1)$}
{parent=Normal distribution}
{wiki}

The standard normal distribution is $N(0,1)$ and has cumulative distribution function $\Phi$.

= Standard normal distribution function
{title2=$\Phi$}
{parent=Standard normal distribution}

The standard normal distribution function is
$$
\Phi(x)=\mathbb P(Z\leq x),
\qquad Z\sim N(0,1).
$$

= Chi-squared distribution
{title2=$\chi_k^2$}
{parent=Probability distribution}
{wiki}

A $\chi_k^2$ variable is the sum of squares of $k$ independent standard normal variables.

= Chi-squared stochastic ordering
{parent=Chi-squared distribution}

If $r<s$, coupling by extra independent squared normals gives $\chi_s^2=\chi_r^2+W$ with $W\geq0$, so every fixed upper quantile increases with the degrees of freedom.

= Weibull distribution
{parent=Probability distribution}
{c}
{wiki}

The Weibull distribution has density $k\lambda^{-1}(y/\lambda)^{k-1}e^{-(y/\lambda)^k}$ on $y>0$.

= Characteristic function of a uniform distribution
{parent=Probability distribution}

For $X$ uniform on $[-a,a]$, $\phi_X(t)=\sin(at)/(at)$.

= Small-argument sine product asymptotic
{parent=Characteristic function of a uniform distribution}

The expansion $\log(\sin x/x)=-x^2/6+O(x^4)$ converts products of small sinc factors into Gaussian characteristic-function limits.

= Variance of a uniform distribution
{parent=Probability distribution}

A uniform variable on $[-a,a]$ has mean zero and variance $a^2/3$.

= Exponential survival probability
{parent=Probability distribution}

A lifetime exponential with rate $\mu$ survives for at least duration $u$ with probability $e^{-\mu u}$.

= Expected value
{title2=$\mathbb E[X]$}
{parent=Probability theory}
{wiki}

The expected value is the probability-weighted average of a random variable, defined by a sum or an integral when it exists.

= Variance
{title2=$\operatorname{Var}(X)$}
{parent=Expected value}
{wiki}

The variance of a square-integrable random variable is
$$
\operatorname{var}(X)=\mathbb E[(X-\mathbb EX)^2]
=\mathbb E[X^2]-(\mathbb EX)^2.
$$

= Variance additivity for independent random variables
{parent=Variance}

For independent square-integrable random variables,
$$
\operatorname{Var}\left(\sum_iX_i\right)
=\sum_i\operatorname{Var}(X_i).
$$

= Standard deviation
{title2=$\sigma_X$}
{parent=Variance}
{wiki}

The standard deviation is the nonnegative square root $\sigma_X=\sqrt{\operatorname{Var}(X)}$ of the <variance>.

= Covariance
{title2=$\operatorname{Cov}(X,Y)$}
{parent=Variance}
{wiki}

The covariance of square-integrable random variables is
$$
\operatorname{Cov}(X,Y)
=\mathbb E[(X-\mathbb EX)(Y-\mathbb EY)].
$$
It determines the variance of a difference through
$$
\operatorname{Var}(X-Y)
=\operatorname{Var}(X)+\operatorname{Var}(Y)-2\operatorname{Cov}(X,Y).
$$

= Covariance matrix
{parent=Covariance}
{wiki}

For a random vector $X$ with finite second moments, its covariance matrix is
$$
\operatorname{cov}(X)=\mathbb E[(X-\mathbb EX)(X-\mathbb EX)^T].
$$
It is positive semidefinite, and the variance of $a^TX$ is $a^T\operatorname{cov}(X)a$.

= Correlation coefficient
{title2=$\rho_{X,Y}$}
{parent=Covariance matrix}
{wiki=Pearson_correlation_coefficient}

For random variables with finite positive variances,
$$
\rho_{X,Y}=\frac{\operatorname{Cov}(X,Y)}{\sqrt{\operatorname{Var}(X)\operatorname{Var}(Y)}}.
$$

= Indicator random variable
{parent=Expected value}
{wiki}

= Tail-sum formula
{parent=Expected value}

For nonnegative $Y$, $E[Y]=\int_0^\infty P(Y\ge t)\,dt$; for nonnegative integer $N$, $E[N]=\sum_{k\ge1}P(N\ge k)$.

= Linearity of expectation
{parent=Expected value}
{wiki=Expected_value\#Linearity}

Expectation is linear without requiring independence: $E[aX+bY]=aE[X]+bE[Y]$ whenever the expectations exist.

= Variance of a sample mean
{parent=Expected value}

For independent identically distributed variables of variance $\sigma^2$, the mean of $n$ samples has variance $\sigma^2/n$.

= Probability inequality
{parent=Probability theory}
{wiki=Concentration_inequality}

Probability inequalities bound event probabilities using moments or other tractable quantities.

= Chebyshev inequality
{parent=Probability inequality}
{c}
{wiki}

For a random variable $X$ of finite variance and $a>0$,
$$
\mathbb P(|X-\mathbb EX|\geq a)
\leq\frac{\operatorname{var}(X)}{a^2}.
$$

= First moment method
{parent=Probability inequality}
{wiki=First_moment_method}

For a nonnegative integer-valued random variable $N$, <Markov inequality> gives
$$
\mathbb P(N\geq1)\leq\mathbb EN.
$$
Consequently, $\mathbb EN<1$ proves that some outcome has $N=0$, while $\mathbb EN\to0$ shows that $N=0$ with probability tending to one.

= Second moment method
{parent=Probability inequality}
{wiki}

For a nonnegative random variable $N$, Chebyshev's inequality gives
$$
\mathbb P(N=0)
\leq\frac{\operatorname{var}(N)}{(\mathbb EN)^2}.
$$
Hence $\operatorname{var}(N)/(\mathbb EN)^2\to0$ implies that $N>0$ with probability tending to one.

= Markov inequality
{parent=Probability inequality}
{c}
{wiki}

For a nonnegative random variable $Y$ and $a>0$,
$$
\Pr(Y\geq a)\leq\frac{\mathbb EY}{a}.
$$

= Exponential Markov bound
{parent=Markov inequality}

Applying Markov's inequality to $e^{tX}$ gives
$$
\Pr(X\geq x)\leq e^{-tx}M_X(t),
\qquad t>0.
$$
Optimizing this expression is the basic Chernoff-bound method.

= Union bound
{parent=Probability inequality}
{wiki=Boole%27s_inequality}

The union bound says $P(\bigcup_i A_i)\le\sum_iP(A_i)$.

= Conditional probability
{title2=$P(A\mid B)$}
{parent=Probability theory}
{wiki}

Conditional probability is $P(A\mid B)=P(A\cap B)/P(B)$ when $P(B)>0$.

= Bayes theorem
{c}
{parent=Conditional probability}
{wiki=Bayes%27_theorem}

For a finite partition $(B_j)$ with positive probabilities,
$$
\mathbb P(B_j\mid A)
=\frac{\mathbb P(A\mid B_j)\mathbb P(B_j)}
{\sum_k\mathbb P(A\mid B_k)\mathbb P(B_k)}.
$$

= Conditional distribution
{parent=Conditional probability}
{wiki}

A conditional distribution is the probability law of one random quantity given information about another.

= Random partition
{parent=Probability theory}
{wiki}

A random partition is a probability distribution on partitions of a finite or countable set.

= Restricted-growth string
{parent=Random partition}

A restricted-growth string encodes a set partition by numbering blocks in order of first appearance.

= Singleton block
{parent=Random partition}

A singleton block is a block containing exactly one element.

= Exchangeable random partition
{parent=Random partition}
{wiki=Exchangeable_random_partition}

An exchangeable random partition has a law invariant under finite permutations of the underlying labels.

= Random path in a binary tree
{parent=Probability theory}

In a fair random root-to-leaf path, a specified leaf at depth $d$ is reached with probability $2^{-d}$.

= Markov chain
{parent=Probability theory}
{c}
{wiki}

A Markov chain has a future conditional distribution depending on the present state alone.

= Birth-death chain
{parent=Markov chain}
{wiki=Birth%E2%80%93death_process}

A birth-death chain is a discrete-time <Markov chain> on nonnegative integers that can move only to the same state or a neighbouring state.

= Product Markov chain
{parent=Markov chain}

Independent Markov chains with transition matrices $P$ and $Q$ form a product chain with transition matrix $P\otimes Q$. If their stationary distributions are $\pi$ and $\rho$, the product distribution $\pi\otimes\rho$ is stationary.

= Transition matrix
{parent=Markov chain}
{wiki=Stochastic_matrix}

The transition matrix of a finite <Markov chain> has entries
$$
P_{ij}=\mathbb P(X_{n+1}=j\mid X_n=i).
$$

= Irreducible Markov chain
{parent=Markov chain}
{wiki}

A <Markov chain> is irreducible when every state can reach every other state along a path of positive-probability transitions.

= Aperiodic Markov chain
{parent=Markov chain}
{wiki=Markov_chain\#Periodicity}

A state is aperiodic when the greatest common divisor of its positive-probability return times is one. A <Markov chain> is aperiodic when all its states are aperiodic.

= Recurrent state
{parent=Markov chain}
{wiki=Recurrence_(probability)}

A state is recurrent when a <Markov chain> started there returns to it with probability one.

= Transient state
{parent=Markov chain}
{wiki=Recurrence_(probability)}

A state is transient when its probability of ever returning is less than one.

= First return time
{parent=Markov chain}
{wiki=Hitting_time}

The first return time to a state $i$ is
$$
T_i^+=\inf\{n\geq1:X_n=i\}.
$$

= Mean recurrence time
{parent=First return time}
{wiki=Kac%27s_lemma}

The mean recurrence time of $i$ is $\mathbb E_iT_i^+$. In a finite irreducible <Markov chain> with <stationary distribution> $\pi$, it equals $1/\pi_i$.

= Preservation under enlargement of transition support
{parent=Markov chain}

If two finite <Markov chains> satisfy
$$
P_{ij}>0\Longrightarrow Q_{ij}>0,
$$
then every positive-probability path and return cycle for $P$ also exists for $Q$. Consequently irreducibility and aperiodicity pass from $P$ to $Q$. Recurrence and mean return time need not do so, because the added transitions and changed probabilities can create escape routes or make returns less frequent.

= Running minimum of an independent sequence is Markov
{parent=Markov chain}

For <independent random variables> with a common law, $L_n=\min_{j\leq n}X_j$ satisfies $L_{n+1}=\min(L_n,X_{n+1})$. The next-state law therefore depends on the past only through $L_n$.

= Overlapping moving sum need not be Markov
{parent=Markov chain}

For i.i.d. Bernoulli variables, $K_n=X_{n-1}+X_n$ hides which summand is the newest one. Given $K_n=1$, the previous value can reveal $X_n$ and thereby change the conditional law of $K_{n+1}$, so $(K_n)$ need not be a Markov chain.

= Reversible Markov chain
{parent=Markov chain}
{wiki}

A Markov chain is reversible when its stationary flow satisfies detailed balance.

= Detailed balance
{parent=Reversible Markov chain}
{wiki}

Detailed balance is the identity pi_i P_ij = pi_j P_ji for every pair of states.

= Hitting probability
{parent=Markov chain}
{wiki}

A hitting probability is the probability of reaching one set before another and solves a discrete harmonic boundary problem.

= Harmonic hitting probability for the balanced-difference urn
{parent=Hitting probability}

Suppose an urn starts with $m$ green balls and $m+2$ red balls. Drawing green removes one green ball, while drawing red adds one green and one red ball. The difference remains two, and $h(g)=1/(g+1)$ is harmonic for the green-ball chain. Stopping $h$ on hitting $0$ or $N$ and then letting $N$ tend to infinity gives termination probability
$$
\mathbb P_m(\tau_0<\infty)=\frac1{m+1}.
$$

= Expected hitting time
{parent=Markov chain}
{wiki=Hitting_time}

For a target set $A$, the expected hitting time $g(i)=\mathbb E_iT_A$ satisfies $g=0$ on $A$ and
$$
g(i)=1+\sum_jP_{ij}g(j)
$$
outside $A$, whenever the expectation is finite.

= Expected occupation count before absorption
{parent=Markov chain}

The expected number $u(i)$ of visits to a state $k$ before absorption satisfies the reward equation
$$
u(i)=\mathbf1_{\{i=k\}}+\sum_jP_{ij}u(j),
$$
with zero boundary data at absorbing states where counting stops.

= Four-state reflecting absorbing random walk
{parent=Expected occupation count before absorption}

For the chain with $0$ absorbing, equal-probability nearest-neighbour moves between $1,2,3$, and a probability-$1/2$ self-loop at $3$, starting from $1$ the expected absorption time is $6$, the probability of visiting $3$ before $0$ is $1/3$, and the expected number of visits to $3$ before absorption is $2$.

= Conditional hitting time
{parent=Markov chain}
{wiki}

A conditional hitting time measures time to a target under a specified successful hitting event and can be computed by weighted first-step equations.

= Periodic Markov chain
{parent=Markov chain}
{wiki}

A periodic chain can return to a state only at times sharing a common divisor greater than one.

= Lazy Markov chain
{parent=Markov chain}
{wiki}

A lazy chain stays put with positive probability, removing periodicity without changing invariant distributions.

= Convergence to an invariant distribution
{parent=Markov chain}
{wiki}

A finite irreducible aperiodic Markov chain converges to its unique invariant distribution.

= Recurrent Markov chain
{parent=Markov chain}
{wiki}

A state is recurrent when the chain returns to it almost surely after starting there.

= Positive recurrent state
{parent=Recurrent Markov chain}
{wiki=Markov_chain#Recurrence}

A recurrent state is positive recurrent when its expected return time is finite.

= Null recurrent state
{parent=Recurrent Markov chain}
{wiki=Markov_chain#Recurrence}

A recurrent state is null recurrent when its expected return time is infinite.

= Recurrence criterion by return probabilities
{parent=Recurrent Markov chain}
{wiki}

A state is recurrent exactly when the sum over time of its return probabilities diverges.

= Biased random walk hitting probability
{parent=Recurrent Markov chain}

For nearest-neighbour random walk on $\mathbb Z_+$ with upward probability $p>q$, the probability of ever hitting zero from $i$ is $(q/p)^i$.

= Random walk on a graph
{parent=Markov chain}
{wiki}

A random walk on a graph moves along an incident edge at each step.

= Simple random walk
{parent=Random walk on a graph}
{wiki=Random_walk}

A simple random walk on a locally finite graph chooses each neighbouring vertex with equal probability.

= Simple random walk on the integer line
{parent=Random walk on a graph}
{wiki=Random_walk}

If the independent increments are $+1$ with probability $p$ and $-1$ with probability $q=1-p$, then
$$
S_n=2B_n-n,
\qquad B_n\sim\operatorname{Binomial}(n,p).
$$
Thus $\mathbb ES_n=n(p-q)$ and $\operatorname{var}(S_n)=4npq$.

= Expected visits to the origin by a simple random walk
{parent=Simple random walk on the integer line}

For the symmetric walk and $V_{2n}=|\{0\leq j\leq2n:S_j=0\}|$,
$$
\mathbb EV_{2n}=\sum_{k=0}^n\frac{\binom{2k}{k}}{4^k}.
$$
Since the central binomial coefficient satisfies $\binom{2k}{k}/4^k\asymp k^{-1/2}$, this expectation is of order $\sqrt n$.

= Simultaneous returns of independent simple random walks
{parent=Simple random walk on the integer line}

For $d$ <independent random variables>[independent] symmetric walks on $\mathbb Z$, the probability that all are at the origin at time $2n$ is
$$
\left(\frac{\binom{2n}{n}}{4^n}\right)^d\asymp n^{-d/2}.
$$
The <Borel-Cantelli lemmas> therefore show that simultaneous returns occur only finitely often almost surely when $d\geq3$.

= Stationary distribution of a graph random walk
{parent=Random walk on a graph}
{wiki}

On a finite undirected graph, stationary mass is proportional to vertex degree.

= Biased random walk
{parent=Random walk on a graph}
{wiki}

A biased nearest-neighbor walk steps right and left with unequal probabilities.

= Reflected biased random walk on the nonnegative integers
{parent=Biased random walk}

Let a discrete-time chain move up with probability $q$ and down with probability $p=1-q$ away from zero, while at zero it moves up with probability $q$ and stays with probability $p$. It is transient for $q>p$, null recurrent for $q=p$, and positive recurrent for $q<p$. In the positive-recurrent case its invariant distribution is
$$
\pi_k=(1-r)r^k,
\qquad r=\frac qp,
$$
and aperiodicity from the self-loop at zero implies convergence to this distribution.

= Gambler's ruin
{parent=Markov chain}
{wiki=Gambler%27s_ruin}

Gambler's ruin studies a nearest-neighbour random walk stopped on reaching either endpoint of a finite interval.

= Expected duration of symmetric gambler's ruin
{parent=Gambler's ruin}

For a symmetric random walk started at $i\in\{-a,\ldots,b\}$ and stopped on first reaching $-a$ or $b$, the mean duration is
$$
\mathbb E_iT=(i+a)(b-i).
$$
In particular, $\mathbb E_0T=ab$. This follows either from the first-step recurrence or by stopping the martingale $S_n^2-n$.

= Strong Markov property
{parent=Markov chain}
{c}
{wiki}

At an almost surely finite stopping time, a Markov process restarts from its stopped state with the same transition law and independently of the history conditional on that state.

= Two-state excursion visit law
{parent=Strong Markov property}

Let $a=\mathbb P_i(T_j<T_i)$ and $b=\mathbb P_j(T_i<T_j)$. Before returning to $i$, the chain visits $j$ zero times with probability $1-a$ and, for $k\geq1$, exactly $k$ times with probability
$$
ab(1-b)^{k-1}.
$$
When $b>0$, the expected number of visits is $a/b$.

= Stationary cycle occupation formula
{parent=Markov chain}

For an irreducible positive-recurrent Markov chain with invariant distribution $\pi$, the expected number of visits to $j$ during one return cycle from $i$ to $i$ is
$$
\frac{\pi(j)}{\pi(i)}.
$$
Combining this with the two-state excursion visit law gives
$$
\frac{\mathbb P_i(T_j<T_i)}{\mathbb P_j(T_i<T_j)}
=\frac{\pi(j)}{\pi(i)}.
$$

= Markov property
{parent=Markov chain}
{wiki=Markov_property}

The Markov property says that, conditional on the present state, the future evolution is independent of the past.

= Continuous-time Markov chain
{parent=Markov chain}
{wiki}

A continuous-time Markov chain waits an exponential time of rate $q_i$ in state $i$ and then jumps according to probabilities $p_{ij}=q_{ij}/q_i$.

= Martingale problem for a continuous-time Markov chain
{parent=Continuous-time Markov chain}

A finite-state process has generator $Q$ exactly when, for every function $f$ on the state space,
$$
f(X_t)-f(X_0)-\int_0^t(Qf)(X_s)\,ds
$$
is a martingale with respect to the process filtration.

= Q-matrix
{c}
{parent=Continuous-time Markov chain}
{wiki=Transition_rate_matrix}

A Q-matrix has nonnegative off-diagonal entries and row sums zero. Its diagonal entry is the negative total rate out of the state.

= Transition semigroup of a continuous-time Markov chain
{title2=$P(t)$}
{parent=Continuous-time Markov chain}

For a <continuous-time Markov chain>, $P_{ij}(t)=\mathbb P_i(X_t=j)$ and the matrices satisfy $P(s+t)=P(s)P(t)$. On a finite state space with <Q-matrix> $Q$, $P(t)=e^{tQ}$.

= Kolmogorov equations
{c}
{parent=Transition semigroup of a continuous-time Markov chain}
{wiki=Kolmogorov_equations}

For a finite-state <continuous-time Markov chain> with <Q-matrix> $Q$, the backward and forward equations are $P'(t)=QP(t)$ and $P'(t)=P(t)Q$, respectively.

= Markov jump-process generator
{parent=Continuous-time Markov chain}

For transition rates $q_{nm}$, the generator acts on a function $f$ by
$$
(Lf)(n)=\sum_{m\ne n}q_{nm}\bigl(f(m)-f(n)\bigr).
$$
Whenever the expectations are finite,
$$
\frac d{dt}\mathbb E[f(N_t)]=\mathbb E[(Lf)(N_t)].
$$
Choosing polynomial functions $f$ gives differential equations for the moments.

= Holding-time reversal along a fixed CTMC jump path
{parent=Continuous-time Markov chain}

Conditioned on a finite jump path $i_0,\ldots,i_n$, the holding times are independent exponentials with the corresponding rates. Multiplying the probability of occupying $i_n$ at time $t$ by $q_{i_n}$ turns its simplex density into a product symmetric under reversal of the time portions and the state sequence.

= Recurrence equivalence for a CTMC and its fixed-time skeleton
{parent=Continuous-time Markov chain}

For $h>0$, the fixed-time skeleton has matrix $P(h)$. If $q_i$ is the holding rate, then for $nh\leq t\leq(n+1)h$,
$$
e^{-q_ih}p_{ii}(nh)\leq p_{ii}(t)
\leq e^{q_ih}p_{ii}((n+1)h).
$$
Thus $\int_0^\infty p_{ii}(t)dt$ diverges exactly when $\sum_np_{ii}(nh)$ does, proving recurrence equivalence for an irreducible chain.

= Jump chain
{parent=Continuous-time Markov chain}
{wiki=Continuous-time_Markov_chain\#Embedded_Markov_chain}

The jump chain records the successive states visited by a continuous-time Markov chain while discarding its holding times.

= Reflected nearest-neighbour random walk
{parent=Jump chain}

On $\mathbb N$, let the chain jump from every $i\geq1$ to $i+1$ with probability $p$ and to $i-1$ with probability $1-p$, and jump from zero to one. It is recurrent exactly when $p\leq1/2$; at $p=1/2$ it is null recurrent.

= Invariant-measure transfer between a jump chain and a CTMC
{parent=Jump chain}

For holding rates $q_i$, a measure $\pi$ satisfies $\pi Q=0$ exactly when $\mu_i=q_i\pi_i$ is invariant for the jump chain. Thus $\pi_i=\mu_i/q_i$ when normalization is possible.

= Slow holding rates destroy positive recurrence
{parent=Invariant-measure transfer between a jump chain and a CTMC}

Even when the jump chain has invariant probability $\mu$, the continuous-time invariant measure $\mu_i/q_i$ may have infinite mass if holding rates become too small, producing infinite mean return time.

= Fast holding rates create continuous-time positive recurrence
{parent=Invariant-measure transfer between a jump chain and a CTMC}

A recurrent jump chain with infinite invariant measure $\mu$ can yield a positive-recurrent continuous-time chain when $\sum_i\mu_i/q_i<\infty$.

= Nonexplosion from recurrent visits to a slow state
{parent=Jump chain}

If the jump chain visits a state of finite holding rate infinitely often, the sum of its independent positive holding times at that state diverges almost surely, preventing explosion.

= Explosion of a continuous-time Markov chain
{parent=Continuous-time Markov chain}
{wiki=Explosive_Markov_chain}

Explosion occurs when infinitely many jumps take place in finite time.

= Explosion by summable holding times
{parent=Explosion of a continuous-time Markov chain}

Conditional on a jump path $(Y_n)$, if $\sum_n1/q_{Y_n}<\infty$, then the sum of exponential holding times has finite expectation and is finite almost surely.

= Explosion of an upward-biased walk with geometrically increasing rates
{parent=Explosion of a continuous-time Markov chain}

Suppose a transient nearest-neighbour jump chain on $\mathbb N$ has uniformly bounded expected visits to each state and the holding rate at $i$ grows geometrically. Then
$$
\mathbb E\sum_n\frac1{q_{Y_n}}
=\sum_i\frac{\mathbb E V_i}{q_i}<\infty,
$$
so the continuous-time chain explodes almost surely.

= Invariant distribution of a continuous-time Markov chain
{parent=Continuous-time Markov chain}

Under the generator convention, an invariant distribution satisfies $\pi Q=0$.

= Invariant distribution of an explosive chain
{parent=Invariant distribution of a continuous-time Markov chain}

An explosive chain can have a summable solution of $\pi Q=0$ despite being transient. Nonexplosion is needed for the usual equivalence between invariant distributions and positive recurrence.

= Batch-birth linear-death process
{parent=Continuous-time Markov chain}

A batch-birth linear-death process has transitions
$$
n\longrightarrow n+k\quad\hbox{at rate }\lambda,
\qquad
n\longrightarrow n-1\quad\hbox{at rate }\beta n.
$$
Writing $P_n(t)=\mathbb P(N_t=n)$ and taking $P_j=0$ for $j<0$, its master equation is
$$
\dot P_n=\lambda P_{n-k}+\beta(n+1)P_{n+1}-(\lambda+\beta n)P_n.
$$

= Moments of a batch-birth linear-death process
{parent=Batch-birth linear-death process}

For $m=\mathbb E N$ and $v=\operatorname{var}N$, the Markov jump-process generator applied to $n$ and $n^2$ gives
$$
m'=k\lambda-\beta m,
\qquad
v'=k^2\lambda+\beta m-2\beta v.
$$
Consequently
$$
m\longrightarrow\frac{k\lambda}{\beta},
\qquad
v\longrightarrow\frac{k(k+1)\lambda}{2\beta}.
$$

= Birth-death process
{parent=Continuous-time Markov chain}
{wiki=Birth%E2%80%93death_process}

A birth-death process moves only between neighbouring nonnegative integer states, with state-dependent birth and death rates.

= Yule process
{c}
{parent=Birth-death process}
{wiki=Yule_process}

The Yule process is the pure birth process with rate $n\lambda$ in population state $n$. Starting from one individual, it is nonexplosive because the independent holding times have divergent total almost surely.

= Mean population of a Yule process
{title2=$\mathbb E X_t=e^{\lambda t}$}
{parent=Yule process}

For the Yule process starting from one individual, the generator applied to $f(n)=n$ gives
$$
\frac d{dt}\mathbb E X_t=\lambda\mathbb E X_t,
\qquad \mathbb E X_0=1,
$$
and hence $\mathbb E X_t=e^{\lambda t}$.

= Conditional Yule birth times
{parent=Yule process}

Given $X_t=n+1$, the $n$ birth times in a Yule process have the law of the order statistics of independent variables with density
$$
f(x)=\frac{\lambda e^{\lambda x}}{e^{\lambda t}-1},
\qquad0\leq x\leq t.
$$
This follows by dividing the joint holding-time density by
$\mathbb P(X_t=n+1)=e^{-\lambda t}(1-e^{-\lambda t})^n$.

= Extinction probability of a linear birth-death process
{parent=Birth-death process}

For individual birth and death rates $\lambda,\mu$ and one initial individual, the finite-time extinction probability satisfies
$$
h'=(h-1)(\lambda h-\mu),\qquad h(0)=0.
$$
When $\lambda\ne\mu$,
$$
h(t)=\frac{\mu(1-e^{-(\lambda-\mu)t})}
{\lambda-\mu e^{-(\lambda-\mu)t}}.
$$

= Detailed balance for a birth-death process
{parent=Birth-death process}

For birth rates $b_i$ and death rates $d_i$, a reversible measure satisfies
$$
\pi_i b_i=\pi_{i+1}d_{i+1}.
$$
Thus $\pi_{i+1}/\pi_i=b_i/d_{i+1}$, and it is an invariant distribution when the resulting measure is summable and the chain is nonexplosive.

= Birth-death master equation
{parent=Birth-death process}

For birth rates $b_n$, death rates $d_n$, and state probabilities $P_n$,
$$
\dot P_n=b_{n-1}P_{n-1}+d_{n+1}P_{n+1}-(b_n+d_n)P_n.
$$

= First-moment equation of a birth-death process
{parent=Birth-death master equation}

Since births and deaths change the state by $+1$ and $-1$,
$$\frac d{dt}\mathbb{E} N_t=\mathbb{E}[b_{N_t}-d_{N_t}].$$

= Factorial moment
{parent=Birth-death master equation}
{wiki=Factorial_moment}

The $r$th falling factorial moment is $\mathbb{E}[(N)_r]$, where $(N)_r=N(N-1)\cdots(N-r+1)$.

= Poisson factorial moments
{parent=Factorial moment}
{c}

If $N$ is Poisson with mean $\lambda$, then $\mathbb{E}[(N)_r]=\lambda^r$; in particular, $\mathbb{E}N=\lambda$ and $\mathbb{E}[N(N-1)]=\lambda^2$.

= Poisson moment closure for a birth-death process
{parent=Birth-death process}
{c}

Approximating the population law by a Poisson law replaces its first two factorial moments by $\lambda$ and $\lambda^2$, closing a quadratic first-moment equation as a Riccati equation for $\lambda$.

= Diffusion approximation of a birth-death process
{parent=Birth-death process}

For unit jumps with birth and death rates $b(x),d(x)$, the Fokker--Planck drift and diffusion in the convention
$$\partial_tP=-\partial_x(uP)+\partial_x^2(DP)$$
are $u=b-d$ and $D=(b+d)/2$.

= Linear noise approximation
{parent=Diffusion approximation of a birth-death process}
{wiki}

Near a stable deterministic equilibrium, replace the drift by its linearization and the diffusion coefficient by its equilibrium value. The resulting Ornstein-Uhlenbeck approximation has a stationary <normal distribution>.

= Mean drift from a Fokker--Planck equation
{parent=Diffusion approximation of a birth-death process}

If boundary fluxes decay sufficiently fast, two integrations by parts give
$$\frac d{dt}\mathbb{E} X_t=\mathbb{E}[u(X_t)],$$
while the diffusion term contributes zero to the first moment.

= Birth-death recurrence classification
{parent=Birth-death process}

For constant birth rate $\lambda$ and death rate $\delta$ away from zero, the chain is transient when $\lambda>\delta$, null recurrent when $\lambda=\delta$, and positive recurrent when $\lambda<\delta$.

= M-M-1 queue
{parent=Birth-death process}
{c}
{wiki=M/M/1_queue}

The queue length in an $M/M/1$ queue is a birth-death process with constant arrival rate $\lambda$ and service rate $\mu$ while nonempty.

= M-M-1 queue with state-dependent admission
{parent=M-M-1 queue}
{c}

If potential arrivals form a Poisson process of rate $\lambda$ and an arrival seeing $n$ customers joins with probability $p(n)$, the queue length is a <birth-death process> with birth rate $b_n=\lambda p(n)$ and death rate $d_n=\mu$ for $n\geq1$. Its stationary ratios, when normalizable, are
$$
\frac{\pi_{n+1}}{\pi_n}=\frac{\lambda p(n)}\mu.
$$

= Positive recurrence of an M-M-1 queue
{parent=M-M-1 queue}

An $M/M/1$ queue is positive recurrent exactly when its traffic intensity $\rho=\lambda/\mu$ is below one.

= Stationary distribution of an M-M-1 queue
{parent=M-M-1 queue}

When $\rho<1$, the equilibrium queue length has distribution $\pi_n=(1-\rho)\rho^n$ and mean $\rho/(1-\rho)$.

= Burke theorem
{parent=M-M-1 queue}
{c}
{wiki=Burke%27s_theorem}

At equilibrium, the departure process of a stable $M/M/1$ queue is Poisson with the same rate as the arrival process.

= M-M-1 queue with Bernoulli feedback
{parent=M-M-1 queue}
{c}

If a served customer returns with probability $p$, thinning service completions gives effective external departure rate $\mu(1-p)$ for the queue-length process.

= Poisson output theorem for a feedback queue
{parent=M-M-1 queue with Bernoulli feedback}
{c}

In equilibrium, a Bernoulli-feedback $M/M/1$ queue has the same queue-length process as an ordinary queue with service rate $\mu(1-p)$, so its external departures form a Poisson process of rate $\lambda$.

= Mean busy period of an M-M-1 queue
{parent=M-M-1 queue}

Starting with one customer, the mean busy period of an $M/M/1$ queue is
$$
\mathbb EB=\frac1{\mu-\lambda}
$$
when $\lambda<\mu$. It is infinite when $\lambda\geq\mu$.

= M-M-infinity queue
{parent=Birth-death process}
{c}
{wiki=M/M/∞_queue}

An $M/M/\infty$ queue has <Poisson process>[Poisson] arrivals of rate $\lambda$, <independent random variables>[independent] exponential service times of rate $\mu$, and infinitely many servers. Its occupancy is a <birth-death process> with birth rate $\lambda$ and death rate $n\mu$ in state $n$.

= Stationary distribution of an M-M-infinity queue
{parent=M-M-infinity queue}

For every $\lambda,\mu>0$, an $M/M/\infty$ queue is positive recurrent and has the <Poisson distribution> with mean $\rho=\lambda/\mu$ as its unique stationary distribution:
$$
\pi_n=e^{-\rho}\frac{\rho^n}{n!}.
$$

= Burke theorem for an M-M-infinity queue
{parent=M-M-infinity queue}
{c}

In stationarity, the departure process of an $M/M/\infty$ queue is a <Poisson process> of rate $\lambda$. By time reversibility, the departures before any fixed time are independent of the occupancy at that time.

= Transient distribution of an M-M-infinity queue
{parent=M-M-infinity queue}

If $X_0=n$ and $p_t=e^{-\mu t}$, then
$$
X_t\overset{d}{=}
\operatorname{Binomial}(n,p_t)
+\operatorname{Poisson}\!\left(\frac\lambda\mu(1-p_t)\right),
$$
with independent summands. The binomial term counts surviving initial customers; <Poisson thinning> gives the surviving later arrivals.

= Mean busy period of an M-M-infinity queue
{parent=M-M-infinity queue}

Writing $\rho=\lambda/\mu$, the stationary empty probability is $e^{-\rho}$. Alternating idle periods of mean $1/\lambda$ with busy periods and applying the renewal-reward theorem gives
$$
\mathbb EB=\frac{e^\rho-1}{\lambda}.
$$

= Finite Markov chain
{parent=Markov chain}
{wiki}

= Communicating class
{parent=Markov chain}
{wiki}

= Communicating states
{parent=Communicating class}
{wiki=Markov_chain#Communicating_classes}

Two states of a <Markov chain> communicate when each can be reached from the other with positive probability in finitely many steps.

= Communicating state
{synonym}

= Closed communicating class
{parent=Communicating class}

A communicating class is closed when no transition from it can reach a state outside it.

= Transient communicating class
{parent=Communicating class}

In a finite chain, a communicating class that is not closed is transient: after it is left, it need not be visited again and is eventually left almost surely.

= Stationary distribution
{parent=Markov chain}
{wiki}

A stationary distribution $\pi$ satisfies $\pi P=\pi$.

= Invariant distribution
{synonym}

= Renewal process
{parent=Probability theory}
{wiki}

A renewal process with independent identically distributed nonnegative interarrival times $\xi_1,\xi_2,\ldots$ has renewal epochs
$$
S_0=0,\qquad S_n=\xi_1+\cdots+\xi_n,
$$
and counting process $N_t=\max\{n:S_n\leq t\}$.

= Renewal-reward theorem
{parent=Renewal process}
{wiki}

For independent identically distributed regenerative cycles with finite mean length, the long-run reward rate equals expected reward per cycle divided by expected cycle length.

= Renewal counting law of large numbers
{parent=Renewal process}

If $0<\mathbb E\xi_1=1/\lambda<\infty$, then the strong law $S_n/n\to1/\lambda$ and the inequalities $S_{N_t}\leq t<S_{N_t+1}$ imply $N_t/t\to\lambda$ almost surely.

= Renewal interval containing a fixed time
{parent=Renewal process}

The renewal interval containing $t$ has length
$$
L(t)=S_{N_t+1}-S_{N_t}=\xi_{N_t+1}.
$$
Selection by a fixed observation time favours longer interarrival times.

= Size-biased distribution
{parent=Renewal interval containing a fixed time}
{wiki}

For a nonnegative random variable $X$ with $0<\mathbb EX<\infty$, its size-biased version $\widehat X$ is defined by
$$
\mathbb E h(\widehat X)=\frac{\mathbb E[Xh(X)]}{\mathbb EX}
$$
for bounded measurable $h$. If $X$ has density $f$, then $\widehat X$ has density $xf(x)/\mathbb EX$.

= Exponential renewal interval limit
{parent=Renewal interval containing a fixed time}

For exponential interarrival times of rate $\lambda$, the age and residual life at time $t$ converge jointly to independent rate-$\lambda$ exponential variables. Their sum therefore converges to the gamma distribution with density $\lambda^2xe^{-\lambda x}$, the size-biased exponential law.

= Stochastic length bias of a renewal interval
{parent=Renewal interval containing a fixed time}

At every deterministic $t>0$, the containing interval stochastically dominates an unselected interarrival time:
$$
\mathbb P(L(t)\geq x)\geq\mathbb P(\xi_1\geq x),\qquad x>0.
$$
Conditioning on the first interarrival gives a renewal equation; subtracting the constant tail probability leaves a renewal equation with a nonnegative forcing term.

= Residual lifetime process
{parent=Renewal process}
{wiki}

The residual lifetime is the time remaining until the next renewal.

= Renewal excess limit theorem
{parent=Residual lifetime process}

For non-arithmetic inter-renewal times with finite positive mean $\mu$, the excess $E(t)=S_{N(t)+1}-t$ converges in distribution to $E_\infty$ with
$$
\mathbb P(E_\infty>x)
=\frac1\mu\int_x^\infty\mathbb P(\xi_1>y)\,dy.
$$

= Exponential first interarrival in a renewal superposition containing a Poisson process
{parent=Renewal excess limit theorem}

If an independent superposition of a Poisson process and a non-arithmetic finite-mean renewal process is itself a renewal process, then its first interarrival is exponential. Equating the limiting excess law computed as a renewal excess with the minimum of the Poisson and second-process excesses yields an integral equation whose survival-function solutions are exponential.

= Residual lifetime Markov chain
{parent=Residual lifetime process}
{wiki}

With bounded integer lifetimes, residual lifetime decreases deterministically and resets randomly at zero.

= Equilibrium residual-life distribution
{parent=Residual lifetime process}
{wiki}

The invariant residual-life mass at i is proportional to the probability that a lifetime exceeds i.

= Law of total variance
{parent=Probability theory}
{wiki}

Variance equals the variance of a conditional mean plus the mean conditional variance.

= Characteristic function
{title2=$\varphi_X(t)$}
{parent=Probability theory}
{wiki=Characteristic_function_(probability_theory)}

The characteristic function of a real <random variable> $X$ is
$$
\varphi_X(t)=\mathbb E[e^{itX}].
$$
It determines the <probability distribution> of $X$.

= Characteristic function of a sum of independent variables
{parent=Characteristic function}

Independence makes the characteristic function of a sum equal the product of the individual characteristic functions.

= Variance normalization
{parent=Probability theory}

Dividing a centered sum by the square root of its total variance gives unit variance and is the natural central-limit scaling.

= Information theory
{parent=Probability and statistics}
{wiki}

= Alphabet
{parent=Information theory}
{wiki=Alphabet_(formal_languages)}

An alphabet is a finite nonempty set of symbols from which strings or source outputs are formed.

= Conditional entropy
{parent=Information theory}
{wiki}

For discrete random variables, conditional entropy is
$$
H(Y\mid X)=\sum_x\mathbb P(X=x)H(Y\mid X=x).
$$

= Conditioning reduces entropy
{parent=Conditional entropy}

For discrete random variables, $H(U\mid V)\leq H(U)$, with equality exactly when $U$ and $V$ are <independent random variables>[independent].

= Entropy of a sum of independent finite-group variables
{parent=Conditional entropy}

If <independent random variables> $X,Y$ take values in a <finite additive group>, then
$$
H(X+Y)\geq\max\{H(X),H(Y)\}.
$$
Indeed, $H(X+Y)\geq H(X+Y\mid Y)=H(X)$, and symmetrically for $Y$. Independence is essential: taking $Y=-X$ makes the sum constant.

= Mutual information
{parent=Information theory}
{wiki}

The mutual information between discrete random variables is
$$
I(X;Y)=H(Y)-H(Y\mid X)
=H(X)+H(Y)-H(X,Y).
$$

= Chain rule for mutual information
{parent=Mutual information}

The information supplied by two observations decomposes as
$$
I(X;Y_1,Y_2)=I(X;Y_1)+I(X;Y_2\mid Y_1).
$$

= Incremental information in a twofold binary repetition code
{parent=Mutual information}

Send a uniform bit twice through independent binary symmetric channels of crossover probability $p$. The first received bit supplies $1-h_2(p)$ bits. Since the received bits disagree with probability $2p(1-p)$, the additional information in the second bit is
$$
h_2(2p(1-p))-h_2(p).
$$

= Channel capacity
{parent=Information theory}
{wiki}

The capacity of a discrete memoryless channel is
$$
C=\max_{P_X}I(X;Y).
$$

= Weakly symmetric channel capacity
{parent=Channel capacity}

If every row of an $m$-output channel matrix is a permutation of every other row and all column sums agree, the uniform input achieves a uniform output and
$$
C=\log_2m-H(\text{one row}).
$$

= Binary symmetric channel capacity
{parent=Weakly symmetric channel capacity}

A binary symmetric channel with crossover probability $p$ has capacity
$$
C=1-h_2(p)
$$
bits per channel use.

= Ternary symmetric channel capacity
{parent=Weakly symmetric channel capacity}

For diagonal transition probability $1-2\alpha$ and each off-diagonal probability $\alpha$,
$$
C=\log_2 3
 +(1-2\alpha)\log_2(1-2\alpha)
 +2\alpha\log_2\alpha,
$$
with $0\log 0=0$.

= Information entropy
{title2=$H(X)$}
{parent=Information theory}
{wiki=Entropy_(information_theory)}

For a discrete random variable with probabilities $p_i$, its Shannon information entropy is
$$
H=-\sum_i p_i\log p_i.
$$

= Shannon entropy
{c}
{synonym}

= Binary entropy
{parent=Information entropy}
{wiki=Binary_entropy_function}

The binary entropy function is
$$
h_2(p)=-p\log_2p-(1-p)\log_2(1-p).
$$

= Huffman coding
{c}
{parent=Information theory}
{wiki}

Huffman's algorithm repeatedly combines the $D$ least weights for a $D$-ary prefix code and yields an optimal expected length.

= Huffman sibling property
{parent=Huffman coding}
{c}

Some optimal prefix tree places the two least probable symbols as sibling leaves of maximum depth, which is the inductive basis of Huffman's algorithm.

= Optimal prefix code
{parent=Huffman coding}
{wiki=Huffman_coding}

An optimal prefix code minimizes probability-weighted codeword length among all prefix codes for the source.

= Expected codeword length
{title2=$L(c)$}
{parent=Optimal prefix code}

For a source symbol $X$ and a code $c$, the expected codeword length is
$$
L(c)=\mathbb E[|c(X)|]=\sum_xP(X=x)|c(x)|.
$$

= Probability monotonicity of optimal prefix-code lengths
{parent=Optimal prefix code}

If symbols of probabilities $p_i>p_j$ have codeword lengths $l_i$ and $l_j$ in an optimal prefix code, then $l_i\leq l_j$. Otherwise, exchanging their codewords preserves the prefix property and changes the expected length by
$$
(p_i-p_j)(l_j-l_i)<0,
$$
contradicting optimality.

= Deepest sibling property of an optimal prefix code
{parent=Optimal prefix code}

Every optimal binary prefix code has two codewords of maximal length that differ only in their final bit. Indeed, choose a deepest leaf in the prefix tree. If its sibling were not a codeword, maximality of the depth would make that sibling subtree empty, so the chosen word could be shortened by deleting its final bit. This would preserve prefix-freeness and reduce expected length.

= Nonunique optimal prefix code
{parent=Optimal prefix code}

Distinct tree shapes can have the same minimum expected length, particularly when Huffman merges have ties.

= Huffman codeword length bounds
{parent=Huffman coding}
{c}

For a largest binary-symbol probability $p_1$, $p_1<1/3$ prevents a length-one word, while $p_1>2/5$ forces one in a Huffman tree.

= Bernoulli source
{parent=Information theory}

A Bernoulli information source emits <independent and identically distributed random variables>[independent symbols with one fixed probability distribution].

= Information rate
{parent=Information theory}
{wiki}

The information rate is the limiting entropy per emitted symbol, when that limit exists.

= Reliable source encoding at a rate
{parent=Information rate}

A source is reliably encodable at rate $r$ if there are block encoders with at most $2^{nr}$ outputs and corresponding decoders whose probability of reconstructing the first $n$ source symbols incorrectly tends to zero as $n\to\infty$.

= Information rate under fixed-length blocking
{parent=Information rate}

For a source $(X_n)$ and fixed positive integer $N$, form the block source $Y_i=(X_{(i-1)N+1},\ldots,X_{iN})$. Whenever the original information rate $h$ exists, the blocked source has information rate $Nh$, because
$$
\frac1mH(Y_1,\ldots,Y_m)
=N\frac1{mN}H(X_1,\ldots,X_{mN}).
$$

= Asymptotic equipartition property
{parent=Information theory}
{wiki}

For a stationary source, typical long blocks have probability approximately $2^{-nH}$; equivalently normalized self-information converges to the entropy rate.

= Shannon noiseless coding theorem
{c}
{parent=Information theory}
{wiki=Shannon%27s_source_coding_theorem}

For binary prefix codes, the optimal expected length $L$ obeys $H(X)\le L<H(X)+1$.

= Unicity distance
{parent=Information theory}
{wiki}

Unicity distance is the ciphertext length at which key equivocation reaches zero and the key becomes uniquely determined under the source model.

= Key equivocation
{parent=Unicity distance}
{wiki=Unicity_distance\#Key_equivocation}

Key equivocation is the conditional entropy $H(K\mid C^n)$ remaining about a key after observing $n$ ciphertext symbols.

= Source redundancy in cryptanalysis
{parent=Unicity distance}

For ciphertext alphabet $\Sigma$ and source entropy $H$, each symbol has redundancy $\log|\Sigma|-H$, leading in Shannon's ideal model to $U=\log|K|/(\log|\Sigma|-H)$.

= Infinite unicity distance from a source symmetry
{parent=Unicity distance}

If distinct keys transform every ciphertext into equally probable plaintexts through a probability-preserving source symmetry, key equivocation never vanishes and unicity distance is infinite.

= Statistical modelling
{parent=Probability and statistics}
{wiki=Statistical_model}

= Unbiased estimator
{parent=Statistical modelling}
{wiki=Bias_of_an_estimator}

An estimator $T$ of a parameter $\theta$ is unbiased when $\mathbb E_\theta T=\theta$ for every parameter value in the model.

= Heteroscedastic
{parent=Statistical modelling}
{wiki=Homoscedasticity_and_heteroscedasticity}

Observations or errors are heteroscedastic when their variances are not all equal.

= Akaike information criterion
{parent=Statistical modelling}
{c}
{wiki=Akaike_information_criterion}

For a model with $k$ fitted parameters and maximized likelihood $L(\widehat\theta)$, the Akaike information criterion is
$$
\operatorname{AIC}=-2\log L(\widehat\theta)+2k.
$$

= Mallows Cp
{parent=Akaike information criterion}
{c}
{wiki=Mallows%27s_Cp}

For a normal linear model with known error variance $\sigma^2$, the scaled Akaike criterion differs by a model-independent constant from
$$
C_p=\|Y-\widehat\mu\|^2+2p\sigma^2.
$$

= Bias-variance decomposition for linear prediction
{parent=Mallows Cp}

If $Y,Y^*$ are independent $N(\mu,\sigma^2I_n)$ vectors and $H$ is a rank-$p$ orthogonal projection, then
$$
\mathbb E\|HY-Y^*\|^2
=\|(I-H)\mu\|^2+(n+p)\sigma^2.
$$
The first term is squared approximation bias, while $p\sigma^2$ is fitted-model variance and $n\sigma^2$ is irreducible new-response noise.

= Unbiased prediction-error identity for ordinary least squares
{parent=Mallows Cp}

If $H$ is the rank-$p$ orthogonal projection onto a normal linear model's column space, then
$$
\mathbb E\|(I-H)Y\|^2
=\|(I-H)\mu\|^2+(n-p)\sigma^2.
$$
Adding $2p\sigma^2$ gives $\|(I-H)\mu\|^2+(n+p)\sigma^2$, so Mallows' $C_p$ is unbiased for independent-copy prediction error even when the projection model is misspecified.

= Shape parameter
{parent=Statistical modelling}
{wiki}

A shape parameter changes the form of a <probability distribution> without merely translating it or rescaling its <random variable>.

= Exponential family
{parent=Statistical modelling}
{wiki}

A density $h(y)\exp\{\theta T(y)-A(\theta)\}$ has sufficient statistic $T$ and cumulant function $A$.

= Natural parameter of an exponential family
{parent=Exponential family}

In the canonical representation
$$
f_\theta(y)=h(y)\exp\{\theta T(y)-A(\theta)\},
$$
$\theta$ is the natural parameter and $T$ is the natural statistic.

= Cumulant function of an exponential family
{parent=Exponential family}

The cumulant function, or log-partition function, is the normalizing term
$$
A(\theta)=\log\int h(y)e^{\theta T(y)}\,dy,
$$
with a sum in the discrete case.

= Exponential-family derivative identities
{parent=Cumulant function of an exponential family}

Differentiating the normalization identity gives
$$
A'(\theta)=\mathbb E_\theta T,
\qquad
A''(\theta)=\operatorname{var}_\theta T.
$$

= Mean parameter of an exponential family
{title2=$\mu$}
{parent=Cumulant function of an exponential family}

The mean parameter corresponding to the natural parameter $\theta$ is
$$
\mu(\theta)=\mathbb E_\theta[T(Y)]=A'(\theta).
$$

= Inverse Gaussian distribution
{c}
{parent=Exponential family}
{wiki}

For an <expected value> parameter $\mu>0$ and a <shape parameter> $\lambda>0$, the inverse Gaussian distribution has <probability density function>
$$
f(x;\mu,\lambda)
=\left(\frac{\lambda}{2\pi x^3}\right)^{1/2}
\exp\left(-\frac{\lambda(x-\mu)^2}{2\mu^2x}\right),
\qquad x>0.
$$
Its mean is $\mu$ and its variance is $\mu^3/\lambda$.

= Exponential-family deviance
{title2=$D(\theta_1,\theta_2)$}
{parent=Exponential family}

The deviance from $\theta_1$ to $\theta_2$ is twice the <Kullback-Leibler divergence>:
$$
D(\theta_1,\theta_2)
=2\mathbb E_{\theta_1}\!\left[
\log\frac{f_{\theta_1}(Y)}{f_{\theta_2}(Y)}
\right].
$$

= Negative binomial exponential family
{parent=Exponential family}

For fixed positive integer $r$, the failures-before-the-$r$th-success distribution is a one-parameter exponential family with
$$
\theta=\log(1-p)<0,
\quad
T(x)=x,
\quad
A(\theta)=-r\log(1-e^\theta),
$$
and carrier $h(x)=\binom{x+r-1}{x}$.

= Exponential dispersion family
{parent=Exponential family}
{wiki=Exponential_dispersion_model}

With dispersion $\phi$ and known weight $w$, an exponential dispersion density has exponent
$$\frac w\phi\{y\theta-b(\theta)\}+c(y,\phi/w).$$
Its mean is $b'(\theta)$ and its variance is $(\phi/w)b''(\theta)$.

= Generalized linear model
{parent=Statistical modelling}
{wiki}

A generalized linear model takes independent responses from an exponential dispersion family and relates their means to linear predictors by $g(\mu_i)=x_i^T\beta$. The response distribution, systematic component, link, dispersion, and known observation weights together specify the model.

= Linear predictor
{title2=$\eta=X\beta$}
{parent=Generalized linear model}

The linear predictor of a generalized linear model is the linear combination $\eta_i=x_i^T\beta$ of the explanatory variables.

= Link function
{parent=Generalized linear model}
{wiki=Generalized_linear_model#Link_function}

A link function connects the response mean to the linear predictor through $g(\mu_i)=\eta_i$.

= Binomial regression
{parent=Generalized linear model}

Binomial regression models independent $Y_i\sim\operatorname{Bin}(m_i,p_i)$ and relates $p_i$ to a linear predictor. Logistic and probit regression use the logit and inverse-normal links respectively.

= Pearson dispersion estimator
{title2=$\widehat\phi$}
{parent=Generalized linear model}

For a generalized linear model with variance function $V$, the Pearson estimator is
$$
\widehat\phi=\frac1{n-p}\sum_i\frac{(Y_i-\widehat\mu_i)^2}{V(\widehat\mu_i)}.
$$

= Canonical link function
{parent=Generalized linear model}
{wiki=Generalized_linear_model\#Link_function}

The canonical link identifies the linear predictor with the exponential family's natural parameter: $g(\mu)=\theta$, where $\mu=b'(\theta)$.

= Poisson canonical link
{parent=Canonical link function}
{c}

For a Poisson mean $\mu$, the natural parameter is $\theta=\log\mu$, so the canonical link is the logarithmic link $g(\mu)=\log\mu$.

= Poisson regression
{parent=Generalized linear model}
{c}
{wiki}

A Poisson regression models independent count responses by
$$
Y_i\sim\operatorname{Pois}(\mu_i),
\qquad \log\mu_i=x_i^T\beta.
$$
Its likelihood is $\prod_i e^{-\mu_i}\mu_i^{y_i}/y_i!$.

= Poisson exposure model
{parent=Generalized linear model}
{c}

For fixed exposures $x_i\geq0$, a Poisson exposure model takes independent observations
$$
Y_i\sim\operatorname{Poisson}(\theta x_i).
$$
It has mean and variance $\theta x_i$ and is therefore heteroscedastic when the exposures differ.

= Estimators for a Poisson exposure model
{parent=Poisson exposure model}

For $S_r=\sum_i x_i^r$, unweighted least squares through the origin and maximum likelihood give
$$
\widehat\theta_{LS}=\frac{\sum_i x_iY_i}{S_2},
\qquad
\widehat\theta_{MLE}=\frac{\sum_iY_i}{S_1}.
$$
Both estimators are unbiased.

= Variance comparison for Poisson exposure estimators
{parent=Estimators for a Poisson exposure model}

The two variances are
$$
\operatorname{var}(\widehat\theta_{LS})
=\frac{\theta S_3}{S_2^2},
\qquad
\operatorname{var}(\widehat\theta_{MLE})
=\frac\theta{S_1}.
$$
Since $S_2^2\leq S_1S_3$ by Cauchy-Schwarz, maximum likelihood has no larger variance, with equality when all positive exposures are equal.

= Normal-approximation tests for a Poisson exposure model
{parent=Poisson exposure model}

For testing $H_0:\theta=1$ against a larger alternative, approximate size-$0.05$ upper-tail tests reject when
$$
\widehat\theta_{MLE}>1+\frac{1.645}{\sqrt{S_1}}
$$
or, respectively,
$$
\widehat\theta_{LS}>1+1.645\frac{\sqrt{S_3}}{S_2}.
$$

= Logistic regression
{parent=Generalized linear model}
{wiki}

Logistic regression models the conditional log odds as an affine function of predictors.

= Bernoulli logistic-regression model
{parent=Logistic regression}

For binary independent responses, logistic regression sets
$$
Y_i\sim\operatorname{Bernoulli}(p_i),
\qquad
\log\frac{p_i}{1-p_i}=\beta_0+x_i^T\beta.
$$

= Fitted-mean balance for logistic regression with an intercept
{parent=Bernoulli logistic-regression model}

At an interior maximum-likelihood fit of a logistic-regression model containing an intercept, the intercept <score function> is
$$
\frac{\partial\ell}{\partial\beta_0}=\sum_i(y_i-\widehat p_i)=0.
$$
Consequently the sum of the fitted probabilities equals the number of observed successes exactly.

= Logistic loss
{parent=Logistic regression}
{wiki=Logistic_regression\#Logistic_model}

For a signed margin $u=yh(x)$, the logistic loss is
$$
\phi(u)=\log(1+e^{-u}).
$$
It is convex and has derivative $\phi'(u)=-e^{-u}/(1+e^{-u})$.

= Positive semidefinite quadratic-form classifier
{parent=Logistic loss}

For a positive semidefinite trace ball $\mathcal S_s$, the quadratic-form hypothesis class is
$$
\mathcal H_s=\{h_M:x\mapsto x^TMx,\ M\in\mathcal S_s\}.
$$
Its empirical logistic risk is convex as a function of $M$.

= Grouped-binomial logistic regression
{parent=Logistic regression}

For independent grouped counts $Y_i\sim\operatorname{Bin}(n_i,p_i)$, grouped-binomial logistic regression specifies
$$
\log\frac{p_i}{1-p_i}=x_i^T\beta.
$$
Fitting the proportions $Y_i/n_i$ with binomial weights $n_i$ gives the same likelihood.

= Reference level in a regression factor
{parent=Logistic regression}

With treatment coding, the intercept describes the reference factor level and each displayed factor coefficient is a contrast against that level. Changing the reference level reparametrizes the same fitted model: fitted probabilities and likelihood do not change, but coefficients representing different pairwise contrasts can have different standard errors and p-values.

= Probit regression
{parent=Generalized linear model}
{wiki=Probit_model}

Binary probit regression sets $p_i=\Phi(\beta_0+x_i^T\beta)$, where $\Phi$ is the <standard normal distribution> function. Its intercept score is a weighted sum of residuals, so an intercept does not generally force $\sum_i\widehat p_i=\sum_i y_i$.

= Iteratively reweighted least squares
{parent=Generalized linear model}
{wiki}

IRLS fits a generalized linear model by repeatedly solving a weighted least-squares approximation to its score equations.

= Independent Poisson conditioning
{parent=Statistical modelling}

Independent Poisson variables conditioned on their sum are multinomial, with cell probabilities proportional to their rates.

= Normal linear model
{parent=Statistical modelling}

$Y=X\beta+\varepsilon$, $\varepsilon\sim N(0,\sigma^2I)$, and full-rank $X$ gives $\widehat\beta=(X^TX)^{-1}X^TY$.

= Cochran's theorem
{parent=Normal linear model}
{c}
{wiki}

Orthogonal projections of an isotropic Gaussian vector are independent. If $P$ is an orthogonal projection of rank $r$ and $Z\sim N(0,I)$, then
$$
\|PZ\|^2\sim\chi_r^2.
$$

= Linear regression
{parent=Normal linear model}
{wiki}

Linear regression models a response mean as a linear combination $X\beta$ of predictors. Ordinary least squares chooses $\beta$ to minimize the sum of squared residuals.

= Design matrix
{title2=$X$}
{parent=Linear regression}
{wiki}

The design matrix has one row for each observation and one column for each regression coefficient, so the linear predictor is $X\beta$.

= Regression coefficient
{title2=$\beta$}
{parent=Linear regression}
{wiki=Regression_analysis}

A regression coefficient is a component of $\beta$ measuring the change in the linear predictor associated with its design-matrix column.

= Interaction term
{parent=Linear regression}
{wiki=Interaction_(statistics)}

An interaction term lets the effect of one predictor depend on another. In
$$
\mathbb E[Y\mid X,Z]
=\beta_0+\beta_1X+\beta_2Z+\beta_3XZ,
$$
the slope with respect to $X$ is $\beta_1+\beta_3Z$.

= Attenuation bias from classical measurement error
{parent=Linear regression}
{wiki=Regression_dilution}

If $Y=\beta Z+\varepsilon$ but the predictor is observed as $X=Z+\eta$, with independent centred errors, the population regression slope is
$$
\frac{\operatorname{Cov}(X,Y)}{\operatorname{Var}(X)}
=\beta\frac{\operatorname{Var}(Z)}
{\operatorname{Var}(Z)+\operatorname{Var}(\eta)}.
$$
Measurement error in the predictor therefore shrinks the slope toward zero, while independent response noise changes its uncertainty but not its population value.

= Regression dilution
{synonym}

= R linear-model formula
{c}
{parent=Normal linear model}
{wiki=R_(programming_language)}

In R, `lm(y ~ 1, data=d)` fits an intercept-only linear model, `lm(y ~ x1 + x2, data=d)` adds quantitative predictors, and a factor predictor is expanded into indicator columns.

= Degrees of freedom of a factor predictor
{parent=R linear-model formula}

With an intercept present, a categorical predictor having $m$ represented levels adds $m-1$ linearly independent indicator columns and therefore $m-1$ model degrees of freedom.

= Hat matrix
{parent=Normal linear model}
{wiki=Hat_matrix}

The hat matrix
$$
H=X(X^TX)^{-1}X^T
$$
is the orthogonal projection onto the column space of a full-rank design matrix, and the fitted response is $\widehat Y=HY$.

= Normal linear-model confidence ellipsoid
{parent=Normal linear model}

With $\widehat\sigma^2=\|Y-X\widehat\beta\|^2/(n-p)$, a simultaneous $(1-\alpha)$ confidence region for $\beta$ is
$$
(\beta-\widehat\beta)^TX^TX(\beta-\widehat\beta)
\leq p\widehat\sigma^2F_{p,n-p}(1-\alpha).
$$

= Cook's distance
{parent=Normal linear-model confidence ellipsoid}
{c}
{wiki=Cook%27s_distance}

If $\widehat\beta_{(i)}$ is the least-squares estimate after deleting observation $i$, Cook's distance is
$$
D_i=\frac{(\widehat\beta_{(i)}-\widehat\beta)^TX^TX
(\widehat\beta_{(i)}-\widehat\beta)}{p\widehat\sigma^2}
=\frac{\|\widehat Y_{(i)}-\widehat Y\|^2}{p\widehat\sigma^2}.
$$
It measures the deleted estimate in the same quadratic metric as the coefficient confidence ellipsoid.

= Multicollinearity
{parent=Normal linear model}
{wiki}

Multicollinearity means that predictor columns are strongly linearly related. It inflates coefficient variances and can make individual effects imprecise even when a joint test strongly rejects that all corresponding coefficients vanish.

= Ordinary least squares estimators
{parent=Normal linear model}
{wiki=Ordinary_least_squares}

In simple linear regression with a nonconstant predictor,
$$
\widehat\beta=\frac{\sum_i(x_i-\bar x)(Y_i-\bar Y)}
{\sum_i(x_i-\bar x)^2},
\qquad
\widehat\alpha=\bar Y-\widehat\beta\bar x.
$$

= Residual sum of squares in simple linear regression
{parent=Ordinary least squares estimators}

For a nonconstant predictor, put
$$
S_{xx}=\sum_i(x_i-\bar x)^2,
\quad
S_{xy}=\sum_i(x_i-\bar x)(y_i-\bar y),
\quad
S_{yy}=\sum_i(y_i-\bar y)^2.
$$
The minimized residual sum of squares is
$$
\operatorname{RSS}=S_{yy}-\frac{S_{xy}^2}{S_{xx}}.
$$
Each centered sum can be recovered in constant time from the five raw sums $\sum x_i$, $\sum y_i$, $\sum x_i^2$, $\sum x_iy_i$, and $\sum y_i^2$.

= Gauss-Markov theorem
{parent=Normal linear model}
{c}
{wiki}

Among linear unbiased estimators in a homoscedastic linear model, least squares has minimum variance.

= Weighted least squares
{parent=Normal linear model}
{wiki}

Weighted least squares minimizes $\sum_iw_i(Y_i-X_i^T\beta)^2$; inverse-variance weights give the efficient estimator under known heteroscedasticity.

= Generalized least squares
{parent=Weighted least squares}
{wiki}

Generalized least squares minimizes $(Y-X\beta)^T\Omega^{-1}(Y-X\beta)$ for a known covariance shape $\Omega$.

= Whitening transformation
{parent=Generalized least squares}
{wiki=Whitening_transformation}

Whitening multiplies a model by a square root of its precision matrix so that the transformed errors have identity covariance.

= Normal equation
{parent=Normal linear model}
{wiki=Linear_least_squares\#Derivation_of_the_normal_equations}

Least-squares differentiation gives $X^TX\widehat\beta=X^TY$, or its precision-weighted analogue.

= Consistency of least squares
{parent=Normal linear model}

Under finite moments and a nonsingular limiting design moment, laws of large numbers make least-squares estimators converge to the true coefficient.

= One-way normal linear model
{parent=Normal linear model}
{wiki=One-way_analysis_of_variance}

A one-way normal model assigns each factor level its own mean while assuming independent Gaussian errors with a common variance.

= Cell-means parametrization
{parent=One-way normal linear model}

A cell-means model uses one coefficient per factor level and no intercept, so each coefficient directly equals a group mean.

= Linear contrast of cell means
{parent=Cell-means parametrization}

A linear contrast is a coefficient-weighted sum of cell means whose coefficients total zero, used to test interpretable differences among groups.

= Full-dominance mean constraint
{parent=One-way normal linear model}

For genotypes $aa,Aa,AA$, full dominance of allele $A$ imposes $\mu_{Aa}=\mu_{AA}$.

= Additive allele-count model
{parent=One-way normal linear model}

With allele counts $0,1,2$, no dominance makes the genotype mean affine in count and imposes $2\mu_{Aa}=\mu_{aa}+\mu_{AA}$.

= Regression leverage
{parent=Statistical modelling}

For $H=X(X^TX)^{-1}X^T$, leverage is $h_{ii}$ and residual variance is $\sigma^2(1-h_{ii})$.

= Risk ratio
{title2=$\operatorname{RR}$}
{parent=Statistical modelling}
{wiki=Relative_risk}

The risk ratio compares event probabilities in two groups:
$$
\operatorname{RR}=\frac{P(\text{event}\mid\text{exposed})}
{P(\text{event}\mid\text{reference})}.
$$

= Relative risk
{synonym}

= Drug efficacy as a risk reduction
{parent=Risk ratio}

When worsening is the adverse event, drug efficacy relative to a control group is
$$
1-\frac{P(\text{worse}\mid\text{treatment})}
{P(\text{worse}\mid\text{control})}
=1-\operatorname{RR}.
$$

= Odds ratio
{title2=$\operatorname{OR}$}
{parent=Statistical modelling}
{wiki}

The odds ratio is the ratio of two event odds. For rare events it is close to the corresponding <risk ratio>, because $p/(1-p)\simeq p$ when $p$ is small.

= Log-linear model
{parent=Statistical modelling}

Additive main effects encode mutual independence in contingency-table Poisson models. An interaction permits association of its factors.

= Saturated log-linear model
{parent=Log-linear model}

A saturated log-linear model has enough parameters to reproduce every observed contingency-table cell count and therefore has zero residual deviance.

= Equal-efficacy Poisson log-linear model
{parent=Log-linear model}

For treatment groups Control, LD, and SD and binary outcomes, equal LD and SD efficacy can be imposed while retaining separate group totals by using separate treatment main effects but one shared treated-versus-control interaction with outcome. Comparing this five-parameter model with the saturated six-parameter model gives a one-degree-of-freedom <analysis of deviance for nested generalized linear models>.

= Maximum-likelihood estimator
{parent=Statistical modelling}
{wiki=Maximum_likelihood_estimation}

An MLE maximizes sample likelihood. Regularly, $\sqrt n(\widehat\theta-\theta)\Rightarrow N(0,I(\theta)^{-1})$; parameter-dependent support can change both rate and limit.

= Likelihood function
{title2=$L(\theta;x)$}
{parent=Maximum-likelihood estimator}
{wiki}

For observed data $x$, the likelihood function is the joint probability mass or density $p_\theta(x)$ regarded as a function of the model parameter $\theta$.

= Log-likelihood
{parent=Maximum-likelihood estimator}
{wiki=Likelihood_function\#Log-likelihood}

The log-likelihood is the logarithm of the likelihood as a function of the model parameter. Its maximizers are the same because the logarithm is strictly increasing.

= Binomial proportion maximum-likelihood estimator
{parent=Maximum-likelihood estimator}

For $X\sim\operatorname{Bin}(n,\theta)$, maximizing $\theta^X(1-\theta)^{n-X}$ gives
$$
\widehat\theta=\frac Xn.
$$

= Exponential distribution rate estimator
{parent=Maximum-likelihood estimator}

For independent exponential observations with rate $\theta$, the MLE is $1/\overline X$ and has asymptotic variance $\theta^2/n$.

= Asymptotic normality of a maximum likelihood estimator
{parent=Maximum-likelihood estimator}

In a regular $p$-parameter model with one-observation Fisher information $I(\theta_0)$,
$$
\sqrt n(\widehat\theta_n-\theta_0)
\xrightarrow d N_p(0,I(\theta_0)^{-1}).
$$

= Logistic distribution
{parent=Statistical modelling}
{wiki}

The logistic location family has density and distribution function
$$
f(x\mid\theta)=\frac{e^{x-\theta}}{(1+e^{x-\theta})^2},
\qquad
F_\theta(x)=\frac{e^{x-\theta}}{1+e^{x-\theta}}.
$$

= Gaussian conditional expectation
{c}
{parent=Statistical modelling}
{wiki=Conditional_expectation}

For jointly Gaussian $(S,Y)$ with $\operatorname{cov}(S)=V$,
$$
\mathbb E[Y\mid S]=\mathbb EY+\operatorname{cov}(Y,S)V^{-1}(S-\mathbb ES).
$$

= Linear discriminant analysis
{parent=Statistical modelling}
{wiki}

LDA assumes class-conditional Gaussian distributions with a shared covariance matrix, producing affine log odds.

= Quadratic discriminant analysis
{parent=Statistical modelling}
{wiki}

Quadratic discriminant analysis uses class-conditional <multivariate normal distribution>[Gaussian distributions] with class-dependent covariance matrices. Their log density ratio is a quadratic polynomial in the observation.

= Bootstrapping
{disambiguate=statistics}
{parent=Statistical modelling}
{wiki=Bootstrapping_(statistics)}

The bootstrap approximates a sampling distribution by repeatedly sampling with replacement from the empirical distribution.

= Bootstrap
{synonym}

= Risk function
{parent=Statistical modelling}
{wiki=Risk_function}

The risk of an estimator is its expected loss as a function of the unknown parameter. Under quadratic loss,
$$
R(\theta,\delta)=\mathbb E_\theta[(\delta-\theta)^2]
=\operatorname{Var}_\theta(\delta)
+(\mathbb E_\theta\delta-\theta)^2.
$$

= Mean squared error
{title2=$\operatorname{MSE}$}
{parent=Risk function}
{wiki}

The mean squared error of an estimator $\widehat\theta$ is
$$
\operatorname{MSE}_\theta(\widehat\theta)
=\mathbb E_\theta[(\widehat\theta-\theta)^2].
$$

= Bias-variance decomposition of mean squared error
{parent=Mean squared error}

Every estimator with finite second moment satisfies
$$
\operatorname{MSE}_\theta(\widehat\theta)
=\operatorname{Var}_\theta(\widehat\theta)
+\operatorname{Bias}_\theta(\widehat\theta)^2.
$$

= Affine shrinkage estimator for a binomial proportion
{parent=Mean squared error}

For $X\sim\operatorname{Bin}(n,\theta)$ and
$$
\widetilde\theta=wX/n+(1-w)\theta_0,
$$
the mean squared error is
$$
\frac{w^2}{n}\theta(1-\theta)
+(1-w)^2(\theta-\theta_0)^2.
$$
When $\theta_0=1/2$, its maximal risk is at $\theta=1/2$ or at an endpoint of $[0,1]$.

= Admissible estimator
{parent=Risk function}
{wiki=Admissible_decision_rule}

An estimator is admissible when no other estimator has risk no larger at every parameter value and strictly smaller at at least one value.

= Minimax estimator
{parent=Risk function}
{wiki=Minimax_estimator}

A minimax estimator minimizes the supremum of its risk over the parameter space.

= Cramer-Rao bound
{c}
{parent=Statistical modelling}
{wiki=Cramér–Rao_bound}

The Cramer-Rao inequality lower-bounds estimator variance by inverse Fisher information, with a derivative correction for bias.

= Score function
{parent=Statistical modelling}
{wiki=Score_(statistics)}

The score is the gradient of the log-likelihood, $S_n(\theta)=\nabla_\theta\ell_n(\theta)$.

= Mean-zero score identity
{parent=Score function}

Under regularity permitting differentiation under the integral, $\mathbb E_\theta S_1(\theta)=\nabla_\theta\int f(x,\theta)dx=0$.

= Fisher information matrix
{parent=Score function}
{c}
{wiki=Fisher_information}

The Fisher information is $I_n(\theta)=\mathbb E[S_nS_n^T]$ and equals $nI_1(\theta)$ for an independent identically distributed sample.

= Fisher information
{synonym}

= Tensorization of Fisher information
{parent=Fisher information matrix}
{c}

Fisher information tensorizes when the information in an $n$-observation model is the sum of the information contributions from its observations. For identically distributed observations this means
$$
I_n(\theta)=nI_1(\theta).
$$
Independence and the <mean-zero score identity> make the cross terms vanish.

= Fisher information in a stationary Gaussian autoregressive location model
{parent=Fisher information matrix}
{c}

For
$$
X_1=\theta+\varepsilon_1,\qquad
X_i=\theta(1-\sqrt\gamma)+\sqrt\gamma X_{i-1}
+\sqrt{1-\gamma}\,\varepsilon_i,
$$
where the $\varepsilon_i$ are independent $N(0,1)$ and $0\leq\gamma<1$,
$$
I_n(\theta)
=1+(n-1)\frac{1-\sqrt\gamma}{1+\sqrt\gamma}.
$$
The information tensorizes exactly when $\gamma=0$.

= Fisher scoring
{c}
{parent=Fisher information matrix}
{wiki=Scoring_algorithm}

Fisher scoring replaces the observed negative Hessian in Newton iteration by the Fisher information:
$$
\theta^{(r+1)}=\theta^{(r)}+I(\theta^{(r)})^{-1}S(\theta^{(r)}).
$$

= Binomial-proportion Fisher scoring
{parent=Fisher scoring}

For $Y=n^{-1}\operatorname{Bin}(n,p)$, the score and information are
$$
S(p)=\frac{n(Y-p)}{p(1-p)},
\qquad I(p)=\frac n{p(1-p)}.
$$
Consequently one Fisher-scoring step sends every interior starting value directly to the MLE $\widehat p=Y$.

= Information identity
{parent=Fisher information matrix}

Under standard regularity conditions, $I_n(\theta)=-\mathbb E_\theta[\nabla_\theta^2\ell_n(\theta)]$.

= Normal location-scale score
{parent=Fisher information matrix}

For $N(\mu,v)$, the one-observation score is $((x-\mu)/v,-1/(2v)+(x-\mu)^2/(2v^2))$ and the information matrix is $\operatorname{diag}(v^{-1},(2v^2)^{-1})$.

= Statistical hypothesis testing
{parent=Statistical modelling}
{wiki=Statistical_hypothesis_test}

Statistical hypothesis testing compares data against a null hypothesis using a controlled rejection probability.

= Null hypothesis
{title2=$H_0$}
{parent=Statistical hypothesis testing}
{wiki}

The null hypothesis is the set of parameter values or probability distributions against which a statistical test controls its probability of rejection.

= Significance level
{title2=$\alpha$}
{parent=Statistical hypothesis testing}
{wiki=Statistical_significance}

The significance level is the chosen upper bound on a test's probability of rejecting the null hypothesis when it is true.

= P-value
{parent=Statistical hypothesis testing}
{wiki=P-value}

A p-value is the probability, computed under a null hypothesis, of obtaining a test statistic at least as unfavorable to that hypothesis as the observed value.

= Joint hypothesis test
{parent=Statistical hypothesis testing}

A joint hypothesis test imposes several parameter restrictions simultaneously and accounts for dependence among their estimators; separate one-parameter confidence intervals do not generally determine its result.

= Power function of a statistical test
{parent=Statistical hypothesis testing}
{wiki=Statistical_power}

For a test with rejection region $R$, its power function is
$$
\pi(\theta)=\mathbb P_\theta(X\in R).
$$
Its supremum over the null parameter space is the size of the test.

= Power function
{synonym}

= Uniformly most powerful test
{parent=Statistical hypothesis testing}
{wiki=Uniformly_most_powerful_test}

A level-$\alpha$ test is uniformly most powerful when its power is at least that of every other level-$\alpha$ test at every parameter value in the alternative.

= Crossing-power obstruction to a uniformly most powerful test
{parent=Uniformly most powerful test}

If two tests have the same size and each has strictly greater power than the other at some alternative, neither test is uniformly most powerful.

= Pearson chi-squared goodness-of-fit test
{c}
{parent=Statistical hypothesis testing}
{wiki=Pearson%27s_chi-squared_test}

For observed cell counts $O_j$ and null expected counts $E_j$, the statistic is
$$
X^2=\sum_j\frac{(O_j-E_j)^2}{E_j}.
$$
With $k$ specified null cell probabilities and sufficiently large expected counts, its null limit is $\chi^2_{k-1}$.

= Pearson chi-squared test of independence
{parent=Pearson chi-squared goodness-of-fit test}
{c}
{wiki=Pearson%27s_chi-squared_test#Testing_for_statistical_independence}

For an $r$ by $c$ contingency table, the statistic
$$
X^2=\sum_{i,j}\frac{(O_{ij}-E_{ij})^2}{E_{ij}}
$$
has asymptotically a chi-squared distribution with $(r-1)(c-1)$ degrees of freedom under independence.

= Wald test
{parent=Statistical hypothesis testing}
{c}
{wiki}

For an approximately normal estimator, the one-parameter Wald statistic
$$
Z=\frac{\widehat\theta-\theta_0}{\operatorname{se}(\widehat\theta)}
$$
is approximately standard normal under $H_0:\theta=\theta_0$. A two-sided test reports $2\Phi(-|Z|)$.

= Wald statistic
{synonym}

= Multivariate Wald statistic
{parent=Wald test}

For a regular $p$-parameter model, the Wald statistic for a candidate $\theta$ is
$$
W_n(\theta)
=n(\widehat\theta_n-\theta)^T
I(\widehat\theta_n)
(\widehat\theta_n-\theta).
$$
Under the true parameter it converges in distribution to $\chi_p^2$.

= Wald statistic for linear restrictions
{parent=Multivariate Wald statistic}

For a full-row-rank $k\times p$ matrix $R$ and the null hypothesis $R\theta=r$, put
$$
W_{n,R}=n(R\widehat\theta_n-r)^T
\left[R I(\widehat\theta_n)^{-1}R^T\right]^{-1}
(R\widehat\theta_n-r).
$$
Under the null, asymptotic normality and <Slutsky theorem> give $W_{n,R}\xrightarrow d\chi_k^2$.

= Wald and likelihood-ratio asymptotic equivalence
{c}
{parent=Wald test}

In a regular scalar model under the null, Taylor expansion about the MLE gives
$$
2\{\ell_n(\widehat\theta)-\ell_n(\theta_0)\}
=-\ell_n''(\widetilde\theta)(\widehat\theta-\theta_0)^2.
$$
A uniform law of large numbers for the observed information makes
$-\ell_n''(\widetilde\theta)/n$ and the Fisher information used in the Wald statistic converge to the same positive limit. Their ratio therefore converges in probability to one.

= Two-sided Gaussian p-value
{parent=Wald test}

For $\overline X\sim N(\mu,1/n)$ under $H_0:\mu=0$, the two-sided level-$\alpha$ test rejects when $|\sqrt n\,\overline X|>\Phi^{-1}(1-\alpha/2)$, and the observed p-value is
$$
p(\overline x)=2[1-\Phi(\sqrt n|\overline x|)].
$$

= Likelihood-ratio test
{parent=Statistical hypothesis testing}
{wiki}

A likelihood-ratio test rejects where the alternative likelihood is sufficiently large relative to the null likelihood.

= Likelihood ratio
{parent=Likelihood-ratio test}

For two parameter values $\theta_1$ and $\theta_0$, the likelihood ratio is $f(x\mid\theta_1)/f(x\mid\theta_0)$.

= Generalized likelihood-ratio test
{parent=Likelihood-ratio test}
{wiki=Likelihood-ratio_test\#Generalized_likelihood-ratio}

A generalized likelihood-ratio test compares the likelihood maximized under the null with the likelihood maximized over the full parameter space.

= Analysis of deviance for nested generalized linear models
{parent=Generalized likelihood-ratio test}

The deviance of a fitted generalized linear model is
$$
D=2\{\ell_{\mathrm{sat}}-\ell_{\mathrm{fit}}\}.
$$
For regular nested models, the deviance reduction $D_0-D_1=2(\ell_1-\ell_0)$ is asymptotically chi-squared with degrees of freedom equal to the difference in fitted dimensions.

= Likelihood-ratio test for equality of two normal means
{parent=Generalized likelihood-ratio test}

For independent unit-variance samples, testing equal means uses $Z=\sqrt{mn/(m+n)}(\bar X-\bar Y)$ and rejects for large $|Z|$.

= Nested likelihood-ratio rejection regions
{parent=Generalized likelihood-ratio test}

Likelihood-ratio tests of different null subspaces can have crossing rejection regions; comparing their quadratic-form geometry exhibits observations accepted by one and rejected by another.

= Score test
{parent=Statistical hypothesis testing}
{wiki=Score_test}

The score test evaluates the likelihood gradient at the restricted estimator and scales its quadratic form by inverse Fisher information.

= Restricted maximum-likelihood estimator
{parent=Score test}

A restricted maximum-likelihood estimator maximizes likelihood over the null parameter space rather than the full model.

= Score test under a simple null
{parent=Score test}

Under a regular simple $p$-parameter null, the normalized score converges to $N_p(0,I_1)$, so its information-standardized squared norm converges to $\chi_p^2$.

= Quadratic form of a standard normal vector
{parent=Score test under a simple null}

If $Z\sim N_p(0,I_p)$, then $Z^TZ=\sum_{j=1}^pZ_j^2\sim\chi_p^2$.

= Residual sum of squares in a normal sample
{parent=Score test}

For independent $N(\mu,\sigma^2)$ observations, $\sum_i(X_i-\bar X)^2/\sigma^2$ has the $\chi_{n-1}^2$ distribution.

= Chi-square central limit theorem
{parent=Residual sum of squares in a normal sample}

If $Q_m\sim\chi_m^2$, then $(Q_m-m)/\sqrt{2m}$ converges in distribution to $N(0,1)$.

= Neyman-Pearson lemma
{parent=Statistical hypothesis testing}
{c}
{wiki}

For testing one simple hypothesis against another, a test that rejects for sufficiently large likelihood ratio has greatest power among all tests of no greater size.

= Monotone likelihood ratio
{parent=Neyman-Pearson lemma}
{wiki}

A one-parameter family has a monotone likelihood ratio in a statistic $T$ when $f(x\mid\theta_1)/f(x\mid\theta_0)$ is nondecreasing in $T(x)$ whenever $\theta_1>\theta_0$.

= Karlin-Rubin theorem
{parent=Monotone likelihood ratio}
{c}
{wiki=Monotone_likelihood_ratio}

For a family with a <monotone likelihood ratio> in $T$, a level-$\alpha$ upper-tail test in $T$ is a <uniformly most powerful test> for the corresponding one-sided hypothesis.

= Uniformly most powerful upper-tail test for a logistic location
{parent=Karlin-Rubin theorem}

For one observation from the <logistic distribution>, the size-$\alpha$ test of $H_0:\theta\leq0$ against $H_1:\theta>0$ rejects when
$$
X>\log\frac{1-\alpha}{\alpha}.
$$

= Grouped exponential observation
{parent=Statistical modelling}

Flooring an exponential variable of rate $\theta$ produces a geometric variable on the nonnegative integers with success probability $1-e^{-\theta}$.

= Asymptotic information loss from grouping
{parent=Grouped exponential observation}

Coarsening continuous observations changes the delta-method variance; for floored exponential data the rate estimator has limiting variance $(e^\theta-1)^2/e^\theta$.

= Augmented Markov chain
{parent=Probability and statistics}
{wiki}

= Bayesian normal model
{parent=Probability and statistics}
{c}
{wiki}

= Normal-normal conjugacy with known observation variance
{parent=Bayesian normal model}

For independent $X_i\sim N(\theta,\sigma^2)$ and prior $\theta\sim N(\mu,\nu^2)$, the posterior has variance and mean
$$
v_n=\left(\frac n{\sigma^2}+\frac1{\nu^2}\right)^{-1},
\qquad
m_n=v_n\left(\frac{n\overline X}{\sigma^2}+\frac\mu{\nu^2}\right).
$$
As $n\to\infty$, $m_n-\overline X=O_p(n^{-1})$ and $v_n^{1/2}\sim\sigma/\sqrt n$, so fixed-level posterior credible intervals agree asymptotically with the corresponding known-variance confidence intervals.

= Branching-process extinction criterion
{parent=Probability and statistics}
{wiki}

= Cauchy-Schwarz inequality
{parent=Probability and statistics}
{c}
{wiki}

= Cover time
{parent=Probability and statistics}
{wiki}

= Expected energy
{parent=Probability and statistics}
{wiki}

= F-test
{parent=Probability and statistics}
{c}
{wiki}

= Nested-model F-test
{parent=F-test}

For nested Gaussian linear models differing by $r$ parameters, the statistic
$$\frac{(RSS_0-RSS_1)/r}{RSS_1/(n-p_1)}$$
has an $F_{r,n-p_1}$ distribution under the reduced model.

= Galton-Watson process
{parent=Probability and statistics}
{c}
{wiki}

A Galton--Watson process forms each generation by giving every current individual an independent number of offspring with a common distribution.

= Galton-Watson extinction fixed point
{parent=Galton-Watson process}
{c}

If $G$ is the offspring probability-generating function, the eventual extinction probability is the smallest fixed point of $G$ in $[0,1]$.

= Quadratic Galton-Watson extinction bound
{parent=Galton-Watson process}

For offspring mean $\mu>1$ and variance $\sigma^2$, Taylor expansion at one gives
$$
d\leq1-\frac{2(\mu-1)}{\sigma^2+\mu^2-\mu}
$$
for the extinction probability $d$.

= Gaussian tail bound
{parent=Probability and statistics}
{c}
{wiki}

For $Z\sim N(0,1)$ and $v>0$, exponential Markov optimization gives
$$
\mathbb P(Z\geq v)\leq e^{-v^2/2}.
$$

= Gronwall inequality
{parent=Probability and statistics}
{c}
{wiki}

= Discrete Gronwall inequality
{parent=Gronwall inequality}
{c}

If nonnegative quantities satisfy
$$
e_{n+1}\leq(1+hL)e_n+h b_n,
$$
then iteration gives
$$
e_n\leq(1+hL)^ne_0
+h\sum_{j=0}^{n-1}(1+hL)^{n-1-j}b_j.
$$
For $nh\leq T$, the growth factor is bounded by $e^{LT}$.

= Kac's lemma
{parent=Probability and statistics}
{c}
{wiki}

For an irreducible positive recurrent <Markov chain> with <stationary distribution> $\pi$, the expected return time to state $i$ is $1/\pi_i$. More generally, expected occupation rewards in a return cycle equal their stationary rate times the expected cycle length.

= Kac lemma
{synonym}

= Markov additive process
{parent=Probability and statistics}
{c}
{wiki}

A Markov additive process couples a Markov chain to an additive functional whose increments depend on the chain.

= Multiple hypothesis testing
{parent=Probability and statistics}
{wiki}

= Multivariate normal distribution
{parent=Probability and statistics}
{wiki}

= Multivariate normal density
{parent=Multivariate normal distribution}

For a <positive-definite matrix> $\Sigma$, the distribution $N_p(\mu,\Sigma)$ has density
$$
f(x)=\frac{
\exp\!\left[-\frac12(x-\mu)^T\Sigma^{-1}(x-\mu)\right]
}{(2\pi)^{p/2}(\det\Sigma)^{1/2}}.
$$

= Linear image of a multivariate normal vector
{parent=Multivariate normal distribution}

If $X\sim N(\mu,\Sigma)$ and $A$ and $c$ are deterministic, then
$$
AX+c\sim N(A\mu+c,A\Sigma A^T).
$$
In particular, $a^TX$ is normal with mean $a^T\mu$ and variance $a^T\Sigma a$.

= Independence of uncorrelated jointly normal variables
{parent=Multivariate normal distribution}

Two subvectors of a jointly <multivariate normal distribution> are independent exactly when their cross-covariance matrix is zero.

= Bivariate normal distribution
{parent=Multivariate normal distribution}
{wiki}

For means $\mu_i$, standard deviations $\sigma_i$, and correlation $\rho$, the standardized quadratic form in the density is
$$
\frac1{1-\rho^2}
\left[
\frac{(x_1-\mu_1)^2}{\sigma_1^2}
-\frac{2\rho(x_1-\mu_1)(x_2-\mu_2)}{\sigma_1\sigma_2}
+\frac{(x_2-\mu_2)^2}{\sigma_2^2}
\right].
$$

= Maximum-likelihood covariance estimator for centered Gaussian data
{title2=$\widehat\Sigma$}
{parent=Bivariate normal distribution}

For independent $Z_i\sim N_p(0,\Sigma)$ with known zero mean,
$$
\widehat\Sigma=\frac1n\sum_{i=1}^nZ_iZ_i^T.
$$

= Asymptotic covariance of the bivariate Gaussian covariance estimator
{parent=Maximum-likelihood covariance estimator for centered Gaussian data}

Writing $a=\Sigma_{11}$, $c=\Sigma_{12}$, and $b=\Sigma_{22}$, the vector $(\widehat\Sigma_{11},\widehat\Sigma_{12},\widehat\Sigma_{22})$ has asymptotic covariance
$$
\begin{pmatrix}
2a^2&2ac&2c^2\\
2ac&ab+c^2&2bc\\
2c^2&2bc&2b^2
\end{pmatrix}.
$$

= Conditional distribution of a bivariate normal variable
{parent=Bivariate normal distribution}

Conditionally on $X_2=x_2$,
$$
X_1\sim N\left(
\mu_1+\rho\frac{\sigma_1}{\sigma_2}(x_2-\mu_2),
\sigma_1^2(1-\rho^2)\right).
$$

= Uncorrelated jointly normal variables are independent
{parent=Multivariate normal distribution}

Any jointly normal random variables with zero covariance are independent.

= Orthogonal projection of a Gaussian vector
{parent=Multivariate normal distribution}
{wiki}

Orthogonal projections of an isotropic Gaussian vector onto orthogonal subspaces are independent.

= Paley-Zygmund inequality
{parent=Probability and statistics}
{c}
{wiki}

= Pivotal quantity
{parent=Probability and statistics}
{wiki}

= Rao-Blackwell theorem
{parent=Probability and statistics}
{c}
{wiki}

If $T$ is a <sufficient statistic> and $U$ is an estimator with finite variance, then $U^*=\mathbb E[U\mid T]$ has the same expectation and no greater variance. Indeed, the <tower property of conditional expectation> preserves the mean and the <law of total variance> gives
$$
\operatorname{Var}(U)
=\operatorname{Var}(U^*)+\mathbb E[\operatorname{Var}(U\mid T)].
$$

= Rao-Blackwell estimator of a squared normal mean
{parent=Rao-Blackwell theorem}
{c}

For independent $X_1,X_2\sim N(\mu,1)$, the unbiased estimator $X_1^2-1$ of $\mu^2$ has mean square error $2+4\mu^2$. Conditioning on the sufficient statistic $T=X_1+X_2$ gives
$$
\mathbb E[X_1^2-1\mid T]=\frac{T^2}{4}-\frac12,
$$
whose mean square error is $1/2+2\mu^2$.

= Regression residual
{parent=Probability and statistics}
{wiki}

= Sufficient statistic
{parent=Probability and statistics}
{wiki}

A statistic $T$ is sufficient for a parameter when the conditional distribution of the full data given $T$ is independent of that parameter.

= Normal sample sum with known variance
{parent=Sufficient statistic}

For independent observations $X_i\sim N(\mu,\sigma^2)$ with known variance, $\sum_iX_i$ is minimal sufficient for $\mu$. Factorization proves sufficiency, while the likelihood ratio for two samples is independent of $\mu$ exactly when their sums agree.

= Minimal sufficient statistic
{parent=Sufficient statistic}
{wiki=Sufficient_statistic\#Minimal_sufficiency}

A sufficient statistic is minimal sufficient when it is a function of every other sufficient statistic, up to null sets. Its level sets form the coarsest sufficient partition of the sample space.

= Likelihood-ratio criterion for minimal sufficiency
{parent=Minimal sufficient statistic}

For a dominated family with positive densities $p_\theta$, a statistic $T$ is minimal sufficient if
$$
T(x)=T(y)
\quad\Longleftrightarrow\quad
\frac{p_\theta(x)}{p_\theta(y)}
\text{ is independent of }\theta.
$$

= Fisher-Neyman factorization theorem
{parent=Sufficient statistic}
{c}
{wiki}

A statistic is sufficient exactly when the likelihood factors into a parameter-dependent function of the statistic times a parameter-free function of the sample.

= Weierstrass M-test
{parent=Probability and statistics}
{c}
{wiki}

= Monte Carlo method
{parent=Probability and statistics}
{wiki}

A Monte Carlo method approximates a deterministic quantity using averages of simulated random variables and controls the approximation with probabilistic limit theorems.

= Rejection sampling
{parent=Monte Carlo method}
{wiki=Rejection_sampling}

Suppose $f$ is a target probability density and $h$ is a proposal density with $f(x)\leq Mh(x)$. Draw $X\sim h$ and an independent $U\sim\operatorname{Uniform}[0,1]$, and accept $X$ when
$$
U\leq\frac{f(X)}{Mh(X)}.
$$
The accepted value has density $f$, the acceptance probability is $1/M$, and the expected number of proposals is $M$.

= Importance sampling
{parent=Monte Carlo method}
{wiki}

To estimate $\mathbb E_f[g(X)]$, sample $Y_1,\ldots,Y_m$ independently from a reference density $h$ whose support covers that of $gf$, and use
$$
\frac1m\sum_{i=1}^m g(Y_i)\frac{f(Y_i)}{h(Y_i)}.
$$
The summands have expectation $\int g(x)f(x)\,dx$, and the <strong law of large numbers> gives almost-sure convergence under integrability.

= Self-normalized importance sampling
{parent=Importance sampling}
{wiki=Importance_sampling#Self-normalized_importance_sampling}

For an unnormalized target density, self-normalized importance sampling uses weights $w_i=f(Y_i)/h(Y_i)$ and estimates expectations by $\sum_iw_ig(Y_i)/\sum_iw_i$. Resampling a point $Y_i$ with probability proportional to $w_i$ produces the associated weighted empirical distribution.

= Statistical inference
{parent=Probability and statistics}
{wiki}

Statistical inference estimates unknown parameters and quantifies uncertainty from observed data.

= Kolmogorov-Smirnov theorem
{c}
{parent=Statistical inference}
{wiki=Kolmogorov–Smirnov_test}

For an empirical distribution function based on an independent sample from a continuous distribution $F$,
$$
\sqrt n\sup_x|\widehat F_n(x)-F(x)|
\xrightarrow{d}\sup_{0\leq t\leq1}|B(t)|,
$$
where $B$ is a <Brownian bridge>.

= Bias of a kernel density estimator
{parent=Statistical inference}

For a symmetric bandwidth-$h$ averaging kernel and a density with bounded first derivative, the interior pointwise bias is at most a constant times $h\|f'\|_\infty$. Near a support boundary, an uncorrected symmetric kernel can instead have nonvanishing bias.

= Standard error
{parent=Statistical inference}
{wiki}

The standard error of an estimator is the standard deviation of its sampling distribution, or an estimate of that standard deviation.

= Wilks theorem
{parent=Statistical inference}
{c}
{wiki=Likelihood-ratio_test#Distribution:_Wilks%27_theorem}

Under regularity conditions, minus twice the logarithm of a generalized likelihood ratio converges under the null hypothesis to a chi-squared distribution whose degrees of freedom equal the difference in parameter dimensions.

= Chi-squared test of independence
{parent=Statistical inference}
{c}
{wiki=Chi-squared_test}

For an $r$ by $c$ contingency table, Pearson's sum of squared observed-minus-fitted counts divided by fitted counts is asymptotically $\chi^2_{(r-1)(c-1)}$ under independence.

= Statistical decision theory
{parent=Statistical inference}
{wiki}

Statistical decision theory compares decision rules through the expected loss they incur under each parameter value.

= Bayes classifier
{parent=Statistical decision theory}
{wiki=Bayes_classifier}

Under zero-one loss, a Bayes classifier assigns an observation to a class of greatest posterior probability. For two classes with prior probabilities $\pi_0,\pi_1$ and densities $f_0,f_1$, it chooses class one exactly when $\pi_1f_1(x)\geq\pi_0f_0(x)$.

= Gaussian Bayes classifier
{parent=Bayes classifier}

For two <multivariate normal density>[multivariate normal densities], the Bayes log posterior odds are a quadratic polynomial. Equal covariance matrices cancel the quadratic term and give <linear discriminant analysis>; unequal covariance matrices give <quadratic discriminant analysis>.

= Uniqueness of a Bayes classifier
{parent=Bayes classifier}

A binary Bayes classifier under zero-one loss is unique up to null sets when the posterior class probabilities tie only on a null set. For two distinct nonsingular Gaussian distributions, the tie set is the zero set of a nonzero quadratic polynomial and has Lebesgue measure zero.

= Risk of a decision rule
{parent=Statistical decision theory}

For a model $\{P_\theta:\theta\in\Theta\}$, loss $L$, and decision rule $\delta$, the risk is
$$
R(\theta,\delta)=\mathbb E_\theta L(\delta(X),\theta).
$$

= Bayes risk
{parent=Statistical decision theory}
{c}
{wiki}

For a prior $\pi$, the integrated risk is $r(\pi,\delta)=\int R(\theta,\delta)\,\pi(d\theta)$. Its infimum over decision rules is the Bayes risk, attained by a Bayes rule when one exists.

= Least favorable prior
{parent=Bayes risk}

A least favorable prior maximizes the Bayes risk over the allowed priors. If a Bayes rule has constant risk equal to the minimax value, its prior is least favorable.

= Minimax decision rule
{parent=Statistical decision theory}
{wiki}

A minimax rule minimizes the worst-case risk $\sup_{\theta\in\Theta}R(\theta,\delta)$.

= Equalizer rule
{parent=Minimax decision rule}

An equalizer rule has the same risk at every parameter value. A Bayes equalizer rule is minimax when its constant risk bounds from below the worst-case risk of every competing rule.

= Minimax Gaussian Bayes classifier from equal class errors
{parent=Equalizer rule}

For two distinct nonsingular Gaussian class distributions, vary the class-zero prior continuously. The class-zero and class-one error probabilities of the corresponding <Bayes classifier> cross. At a crossing they are equal, making the classifier an <equalizer rule>, a <minimax decision rule>, and the associated prior a <least favorable prior>.

= Constant-risk limit-of-Bayes-risk criterion
{parent=Minimax decision rule}

If a rule has constant risk $r$ and a sequence of priors has Bayes risks tending to $r$, then the rule is minimax. Every competing rule has worst-case risk at least every one of those Bayes risks.

= Student t confidence interval
{parent=Statistical inference}
{c}
{wiki}

A Student t confidence interval replaces unknown Gaussian scale by an independent residual estimate.

= Student t confidence interval for a centered regression intercept
{parent=Student t confidence interval}
{c}

In a two-parameter Gaussian simple linear regression centered at $\bar x$, the intercept estimator is $\bar Y$. With residual standard deviation $s$,
$$
\frac{\bar Y-\alpha'}{s/\sqrt n}\sim t_{n-2},
$$
which gives the usual two-sided interval from the corresponding quantiles.

= Bayesian statistics
{parent=Statistical inference}
{c}
{wiki}

Bayesian statistics updates a prior distribution to a posterior distribution using likelihood.

= Prior probability
{parent=Bayesian statistics}
{wiki}

A prior probability describes uncertainty about an event or parameter before the current observation is incorporated.

= Bayesian posterior
{parent=Bayesian statistics}
{c}
{wiki}

The posterior density is proportional to likelihood times prior density.

= Point-null mixture prior
{parent=Bayesian posterior}

A point-null mixture prior assigns positive mass to one parameter value and distributes the remaining mass continuously over alternatives. Posterior point mass is obtained by dividing the null component of the marginal density by the full marginal density.

= Marginal likelihood for a Gaussian point-null mixture
{parent=Point-null mixture prior}

For $\overline X\mid\mu\sim N(\mu,1/n)$ and an equal mixture of $\mu=0$ and $\mu\sim N(0,\tau^2)$,
$$
m(x)=\frac12\phi_{1/n}(x)+\frac12\phi_{\tau^2+1/n}(x).
$$

= Posterior probability of a Gaussian point null
{parent=Marginal likelihood for a Gaussian point-null mixture}

For the equal point-null mixture,
$$
\mathbb P(\mu=0\mid\overline X=x)
=\left[1+\frac{1}{\sqrt{1+n\tau^2}}
\exp\left(\frac{n^2\tau^2x^2}{2(1+n\tau^2)}\right)
\right]^{-1}.
$$

= Jeffreys-Lindley paradox for a Gaussian point null
{parent=Posterior probability of a Gaussian point null}
{c}

With a fixed diffuse alternative prior, a frequentist p-value and the posterior probability of a point null can order evidence differently. For $n=100$ and $\tau=1$, the p-value exceeds the null posterior near zero, while sufficiently far in the tails the null posterior exceeds the p-value because it has a slightly slower Gaussian decay.

= Improper prior
{parent=Bayesian posterior}
{wiki}

An improper prior is a nonnegative prior kernel with infinite total mass. It can still produce a proper posterior after multiplication by the likelihood and normalization.

= Poisson-gamma conjugacy
{parent=Bayesian posterior}
{c}

For a Poisson observation of mean $\theta$ and a gamma prior with shape $\alpha$ and rate $\lambda$, the posterior is gamma with shape $\alpha+X$ and rate $\lambda+1$.

= Gamma-exponential conjugacy
{parent=Bayesian posterior}
{wiki}

A gamma prior combined with exponential observations gives a gamma posterior.

= Beta-binomial conjugacy
{c}
{parent=Bayesian posterior}
{wiki=Conjugate_prior#Table_of_conjugate_distributions}

A beta prior $\operatorname{Beta}(\alpha,\beta)$ combined with $s$ Bernoulli successes and $f$ failures gives posterior $\operatorname{Beta}(\alpha+s,\beta+f)$.

= Bayes estimator
{parent=Bayesian statistics}
{c}
{wiki}

A Bayes estimator minimizes posterior expected loss.

= Bayes estimator under squared error loss
{parent=Bayes estimator}
{c}
{wiki}

Under squared error loss, the Bayes estimator is the posterior mean.

= Bayes estimator under parameter-weighted squared error
{parent=Bayes estimator}
{c}

For loss $L(\theta,a)=q(\theta)(\theta-a)^2$ with $q(\theta)>0$, the Bayes estimator is the mean under the posterior reweighted by $q$:
$$
\widehat\theta_B
=\frac{\mathbb E[\theta q(\theta)\mid X]}
{\mathbb E[q(\theta)\mid X]}.
$$

= Bayes estimator under reciprocal weighted quadratic loss
{parent=Bayes estimator}
{c}

Under $L(d,\theta)=\theta^{-1}(\theta-d)^2$, the posterior expected loss is minimized at
$$
d=\big(\mathbb E[\theta^{-1}\mid X]\big)^{-1},
$$
provided the conditional inverse moment is finite and positive.

= Posterior expected loss
{parent=Bayes estimator}
{wiki}

Posterior expected loss averages the loss over the posterior and is minimized pointwise in the data.

= Markov chain Monte Carlo
{parent=Bayesian statistics}
{c}
{wiki}

Markov chain Monte Carlo constructs an ergodic Markov chain whose stationary distribution is the target law and uses its post-burn-in states as approximate dependent samples.

= Gibbs sampler
{parent=Markov chain Monte Carlo}
{c}
{wiki}

For a joint density $f_{XY}$, a two-coordinate Gibbs update draws
$$
Y_m\sim f_{Y\mid X}(\,\cdot\mid X_{m-1}),
\qquad
X_m\sim f_{X\mid Y}(\,\cdot\mid Y_m).
$$

= Stationarity of the two-coordinate Gibbs sampler
{parent=Gibbs sampler}

The joint target density is stationary for a full Gibbs sweep, since
$$
\int f_X(x)f_{Y\mid X}(y'\mid x)
f_{X\mid Y}(x'\mid y')\,dx
=f_Y(y')f_{X\mid Y}(x'\mid y')
=f_{XY}(x',y').
$$

= Normal mean-precision Gibbs sampler
{parent=Gibbs sampler}

For independent $Z_i\mid\mu,\omega\sim N(\mu,\omega^{-1})$, a flat prior on $\mu$, and an exponential prior of rate $\lambda$ on $\omega$, the full conditionals are
$$
\mu\mid\omega,z\sim N\left(\bar z,\frac1{n\omega}\right),
$$
and
$$
\omega\mid\mu,z\sim
\operatorname{Gamma}\left(\frac n2+1,\,
\lambda+\frac12\sum_i(z_i-\mu)^2\right),
$$
where the gamma distribution is parametrized by shape and rate.

= Jackknife bias correction
{parent=Statistical inference}
{wiki=Jackknife_resampling}

For an estimator $T_n$ and leave-one-out versions $T_{(-i)}$, the jackknife bias estimate and corrected estimator are
$$
\widehat B_n=(n-1)\left(\frac1n\sum_iT_{(-i)}-T_n\right),
\qquad
\widetilde T_{\rm JACK}=T_n-\widehat B_n.
$$
If $B_n=a/n+b/n^2+O(n^{-3})$, the correction leaves bias $O(n^{-2})$.

= Sample variance
{title2=$s_n^2$}
{parent=Jackknife bias correction}
{wiki=Variance#Sample_variance}

For observations $X_1,\ldots,X_n$, the unbiased sample variance is
$$
s_n^2=\frac1{n-1}\sum_{i=1}^n(X_i-\overline X_n)^2.
$$
For independent identically distributed observations with finite variance $\sigma^2$, it converges in probability to $\sigma^2$.

= Asymptotic normality
{parent=Statistical inference}
{wiki}

An estimator is asymptotically normal when a rescaled estimation error converges in distribution to a normal law.

= Slutsky theorem
{parent=Asymptotic normality}
{c}
{wiki}

Slutsky's theorem combines convergence in distribution with convergence in probability through continuous algebraic operations.

= Sandwich covariance matrix
{parent=Asymptotic normality}
{wiki=Heteroskedasticity-consistent_standard_errors}

For estimating equations, asymptotic covariance often has the form $A^{-1}BA^{-1}$, called a sandwich covariance.

= Delta method
{parent=Asymptotic normality}
{wiki}

If $\sqrt n(T_n-\theta)$ is asymptotically normal, differentiability gives $\sqrt n(g(T_n)-g(\theta))$ the variance multiplied by $g'(\theta)^2$.

= Asymptotic distribution of the Gaussian sample correlation
{title2=$\sqrt n(\widehat\rho-\rho)$}
{parent=Delta method}

For a centered bivariate normal sample,
$$
\sqrt n(\widehat\rho-\rho)\xrightarrow dN\bigl(0,(1-\rho^2)^2\bigr).
$$
This is the multivariate delta method for $g(a,c,b)=c/\sqrt{ab}$.

= Endpoint asymptotics of the symmetric-uniform maximum likelihood estimator
{parent=Asymptotic normality}

For an independent sample from $\operatorname{Uniform}[-\theta,\theta]$, the maximum likelihood estimator is $M_n=\max_i|X_i|$ and
$$
n(\theta-M_n)\xrightarrow d\operatorname{Exp}(1/\theta).
$$
The parameter-dependent support makes the convergence rate $n$ rather than $\sqrt n$, so the regular maximum-likelihood central limit theorem does not apply.

= Asymptotic relative efficiency
{parent=Statistical inference}
{wiki}

Asymptotic relative efficiency compares the leading asymptotic variances of two consistent estimators.

= Wald confidence interval
{parent=Statistical inference}
{c}
{wiki}

A Wald interval centers at an asymptotically normal estimator and uses a consistent estimated standard error times a normal quantile.

= Linear least-squares projection
{parent=Probability and statistics}
{wiki=Linear_least_squares}

The best affine predictor has slope covariance divided by predictor variance and intercept chosen to match means.

= Schur complement covariance
{parent=Linear least-squares projection}
{c}

The residual variance after projection onto one factor is $\operatorname{Var}(Y)-\operatorname{Cov}(X,Y)^2/\operatorname{Var}(X)$, a covariance Schur complement.