Shirt a Phobia

In statistics, point estimation involves the use of sample data to calculate a single value (known as a point estimate, since it identifies a point rather than an interval), which serves as a “best guess” or “best estimate” of an unknown quantity, for example, the population mean, the variance of a distribution, or a model parameter (in a parametric model).

Point estimation can be contrasted with interval estimation: interval estimates are typically either confidence intervals, in the case of frequentist inference, or credible intervals, in the case of Bayesian inference. More generally, a point estimator can be contrasted with a set estimator. Examples are given by confidence sets or credible sets. A point estimator can also be contrasted with a distribution estimator. Examples are given by confidence distributions, randomized estimators, and Bayesian posteriors.

Properties of point estimators

Biasedness

The bias is defined as the difference between the expected value of the estimator and the true value of the population parameter being estimated. It can also be described that the closer the expected value of a parameter is to the measured parameter, the lesser the bias. When the estimated number and the true value is equal, the estimator is considered unbiased. This is called an unbiased estimator. The estimator will become a best unbiased estimator if it has minimum variance. However, a biased estimator with a small variance may be more useful than an unbiased estimator with a large variance.^[1]

If we let T = h(X₁,X₂, . . . , X_n) be an estimator based on a random sample X₁,X₂, . . . , X_n, the estimator T is called an unbiased estimator for the parameter θ if E[T] = θ, irrespective of the value of θ.^[1] For example, from the same random sample we have E(x̄) = μ (mean) and E(s²) = σ² (variance), then x̄ and s² would be unbiased estimators for μ and σ². The difference E[T ] − θ is called the bias of T ; if this difference is nonzero, then T is called biased.

The concept of (un-)biasedness can be generalized to other metrics than the mean. An unbiased estimator $T$ of $X=(X_{1},\dots ,X_{n})$ fulfills^[2] $\mathbb {E} _{\theta }[T(X)]=\mathrm {arg} \min _{\theta '}\mathbb {E} _{\theta }[(T(X)-\theta ')^{2}]=\theta .$ Thus, a more general condition for unbiasedness is given by^[3] $\mathrm {arg} \min _{\theta '}\mathbb {E} _{\theta }{\big [}W{\big (}T(X),\theta '{\big )}{\big ]}=\theta ,$ for some function $W$ . For example, if $W(T(X),\theta ')=|T(X)-\theta '|$ , then the estimator is called median-unbiased^[4], since the median is a minimiser of the mean absolute error.

Consistency

A point estimator is called consistent, if the probability that the estimate is close to the true value tends to 1 as the sample size grows to infinity. If the estimate (almost) surely gets arbitrarily close to the true value, eventually, as the sample size grows to infinity, then the estimator is even called strongly consistent. Intuitively, a consistent estimator will be “probably approximately correct”, and a strongly consistent estimator even “surely approximately correct”, if the sample size is large enough. In the special case, where an estimator is unbiased, it is already consistent, if its variance decreases to zero as the sample size grows to infinity.

Efficiency

Let T₁ and T₂ be two unbiased estimators for the same parameter θ. The estimator T₂ would be called more efficient than estimator T₁ if Var(T₂) < Var(T₁), irrespective of the value of θ.^[1] We can also say that the most efficient estimators are the ones with the least variability of outcomes. Therefore, if the estimator has smallest variance among sample to sample, it is both most efficient and unbiased. We extend the notion of efficiency by saying that estimator T₂ is more efficient than estimator T₁ (for the same parameter of interest), if the MSE (mean square error) of T₂ is smaller than the MSE of T₁.^[1]

Generally, we must consider the distribution of the population when determining the efficiency of estimators. For example, in a normal distribution, the mean is considered more efficient than the median, but the same does not apply in asymmetrical, or skewed, distributions.

Sufficiency

In statistics, the job of a statistician is to interpret the data that they have collected and to draw statistically valid conclusion about the population under investigation. But in many cases the raw data, which are too numerous and too costly to store, are not suitable for this purpose. Therefore, the statistician would like to condense the data by computing some statistics and to base their analysis on these statistics so that there is no loss of relevant information in doing so, that is the statistician would like to choose those statistics which exhaust all information about the parameter, which is contained in the sample. We define sufficient statistics as follows: Let X =( X₁, X₂, … ,X_n) be a random sample. A statistic T(X) is said to be sufficient for θ (or for the family of distribution) if the conditional distribution of X given T is free from θ.^[5]

Asymptotic properties

Often, one is interested in the behaviour of a statistical estimation procedure as the sample size $n$ tends to infinity. Additionally to consistency, desirable asymptotic properties of an estimator $T_{n}$ are:

asymptotic unbiasedness: The bias of $T_{n}$ converges to zero as $n$ tends to infinity (though the estimator might be biased for finite sample sizes).
asymptotic efficiency: The variance of the $T_{n}$ times the sample size $n$ converges to the Cramér–Rao bound, that is, the smallest possible variance of an estimator based on one data point. The multiplication of the variance with $n$ is required to account for the decrease in variance due to the increasing sample size.
asymptotic normality: Roughly speaking, the distribution of $T_{n}$ approaches a normal distribution as $n$ tends to infinity. That is to say, the rescaled estimator $\sigma _{n}(T_{n}-\mu _{n})$ , for some sequences $(\mu _{n})_{n\in \mathbb {N} }$ and $(\sigma _{n})_{n\in \mathbb {N} }$ , converges in distribution to a standard normal distribution.

Estimation methods

Below are some commonly used methods for estimating unknown parameters. The methods vary in their domain of applicability and on the underlying statistical paradigm (frequentist or Bayesian).

Frequentist estimation

Maximum likelihood estimation (MLE)

The method of maximum likelihood, due to R.A. Fisher, is arguably the most important general method for estimating the parameters $\theta _{1},\dots ,\theta _{p}$ of a parametric model. According to this method, the best estimate for the unknown model parameter is the one that maximizes the probability (or probability density) of observing the data that has been observed. This probability as a function of the model parameters is called the likelihood, giving the method its name.^[6]

In mathematical terms, the methods works as follows: Let $X=(X_{1},\dots ,X_{n})$ denote a random data sample with joint probability density function or probability mass function $f_{\theta }$ depending on the vector $\theta =(\theta _{1},\dots ,\theta _{p})$ of model parameters. The function $\theta \mapsto f_{\theta }(X)$ is called the likelihood function, often denoted by $L$ . Assuming that true model parameter lies in a set $\Theta \subset \mathbb {R} ^{p}$ , a maximum likelihood estimator ${\hat {\theta }}_{\mathrm {MLE} }$ fulfills $L({\hat {\theta }}_{\mathrm {MLE} })=\max _{\theta \in \Theta }L(\theta )=\max _{\theta \in \Theta }f_{\theta }(X).$ In practice, the likelihood function is often differentiable, in which case ${\hat {\theta }}_{\mathrm {MLE} }$ is a solution of the equations^[5] ${\frac {\partial L(\theta )}{\partial \theta _{i}}}=0,\quad i=1,\dots ,k.$ Under certain assumptions on the likelihood, the MLE is strongly consistent, asymptotically efficient and asymptotically normal.

Method of moments (MOM)

The method of moments was introduced by K. Pearson and P. Chebyshev in 1887, and it is one of the oldest methods of estimation. This method is based on law of large numbers, which uses all the known facts about a population and apply those facts to a sample of the population by deriving equations that relate the population moments to the unknown parameters. We can then solve with the sample mean of the population moments.^[7] However, due to the simplicity, this method is not always accurate and can be easily biased.

Let (X₁, X₂,…X_n) be a random sample from a population having p.d.f. (or p.m.f) f(x,θ), θ = (θ₁, θ₂, …, θ_k). The objective is to estimate the parameters θ₁, θ₂, …, θ_k. Further, let the first k population moments about zero exist as explicit function of θ, i.e. μ_r = μ_r(θ₁, θ₂,…, θ_k), r = 1, 2, …, k. In the method of moments, we equate k sample moments with the corresponding population moments. Generally, the first k moments are taken because the errors due to sampling increase with the order of the moment. Thus, we get k equations μ_r(θ₁, θ₂,…, θ_k) = m_r, r = 1, 2, …, k. Solving these equations we get the method of moment estimators (or estimates) as^[5] $m_{r}={\frac {1}{n}}\sum _{i=1}^{n}X_{i}^{r}.$ See also generalized method of moments.

Score matching estimation

Score matching is a rather new method for parameter estimation that is less intuitive than MLE, but is more advantageous for complicated distributions with a large number of parameters.^[8]

The data is assumed to be drawn from a distribution with probability density $f(\mathbf {x} ,{\boldsymbol {\theta }})$ , $\mathbf {x} \in \mathbb {R} ^{d}$ , parameterized by ${\boldsymbol {\theta }}$ . The key quantity, that is needed for the score matching estimation, is the score $\nabla _{\mathbf {x} }\log f(\mathbf {x} ,{\boldsymbol {\theta }})$ . The true parameter (vector) ${\boldsymbol {\theta }}$ is characterized by ${\begin{aligned}{\boldsymbol {\theta }}&=\mathrm {arg} \min _{{\boldsymbol {\theta }}'}{\frac {1}{2}}\int _{\mathbb {R} ^{d}}\lVert \nabla _{\mathbf {y} }\log f(\mathbf {y} ,{\boldsymbol {\theta }}')-\nabla _{\mathbf {y} }\log f(\mathbf {y} ,{\boldsymbol {\theta }})\rVert ^{2}f(\mathbf {y} ,{\boldsymbol {\theta }})\mathrm {d} \mathbf {y} \\&=\mathrm {arg} \min _{{\boldsymbol {\theta }}'}\sum _{i=1}^{d}\int _{\mathbb {R} ^{d}}\left(\partial _{y_{i}}^{2}\log f(\mathbf {y} ,{\boldsymbol {\theta }}')+{\frac {1}{2}}(\partial _{y_{i}}\log f(\mathbf {y} ,{\boldsymbol {\theta }}')^{2}\right)f(\mathbf {y} ,{\boldsymbol {\theta }})\mathrm {d} \mathbf {y} .\end{aligned}}$ In line with the second equality, for observations $\mathbf {x} _{1},\dots ,\mathbf {x} _{n}$ , the score matching estimate ${\hat {\boldsymbol {\theta }}}$ is defined by ${\hat {\boldsymbol {\theta }}}=\mathrm {arg} \min _{{\boldsymbol {\theta }}'}\sum _{j=1}^{n}\sum _{i=1}^{d}\left(\partial _{x_{i}}^{2}\log f(\mathbf {x} _{j},{\boldsymbol {\theta }}')+{\frac {1}{2}}(\partial _{x_{i}}\log f(\mathbf {x} _{j},{\boldsymbol {\theta }}')^{2}\right).$ The main advantage of score matching estimation compared to the maximum likelihood estimation is the following. Typically, the probability density is of the form $f(\mathbf {x} ,{\boldsymbol {\theta }})={\frac {1}{Z({\boldsymbol {\theta }})}}q(\mathbf {x} ,{\boldsymbol {\theta }}),$ with an unnormalized density $q(\mathbf {x} ,{\boldsymbol {\theta }})$ and a normalization constant $Z({\boldsymbol {\theta }})$ . The computation of latter is often the bottleneck of maximum likelihood estimation, especially if the dependence on ${\boldsymbol {\theta }}$ is non-analytic and the parameter vector is high-dimensional. By taking the derivatives of the logarithm of $f$ , $Z({\boldsymbol {\theta }})$ does not occur in the score matching estimation procedure, thus eliminating this issue.^[8]

Additionally, under mild restrictions on $f(\mathbf {x} ,{\boldsymbol {\theta }})$ , the score matching estimate can be shown to be weakly consistent.

Least square estimation (LSE)

In the method of least squares, we consider the estimation of parameters using some specified form of the expectation and second moment of the observations. To fit a curve of the form y = f( x, β₀, β₁, ,,,, β_p) to the data (x_i, y_i), i = 1, 2,…n, we may use the method of least squares. This method consists of minimizing the sum of squares.

When f(x, β₀, β₁, ,,,, β_p) is a linear function of the parameters and the x-values are known, least square estimators will be best linear unbiased estimator (BLUE). Again, if we assume that the least square estimates are independently and identically normally distributed, then a linear estimator will be minimum-variance unbiased estimator (MVUE) for the entire class of unbiased estimators. See also minimum mean squared error (MMSE).^[5]

Bayesian estimation

Bayesian inference is typically based on the posterior distribution. Many Bayesian point estimators are the posterior distribution’s statistics of central tendency, e.g., its mean, median, or mode:

Posterior mean, which minimizes the (posterior) risk (expected loss) for a squared-error loss function; in Bayesian estimation, the risk is defined in terms of the posterior distribution, as observed by Gauss.^[9]
Posterior median, which minimizes the posterior risk for the absolute-value loss function, as observed by Laplace.^[9]^[10]
maximum a posteriori (MAP), which finds a maximum of the posterior distribution; for a uniform prior probability, the MAP estimator coincides with the maximum-likelihood estimator;

The MAP estimator has good asymptotic properties, even for many difficult problems, on which the maximum-likelihood estimator has difficulties. For regular problems, where the maximum-likelihood estimator is consistent, the maximum-likelihood estimator ultimately agrees with the MAP estimator.^[11]^[12]^[13] Bayesian estimators are admissible, by Wald’s theorem.^[12]^[14]

The Minimum Message Length (MML) point estimator is based in Bayesian information theory and is not so directly related to the posterior distribution.

Special cases of Bayesian filters are important:

Several methods of computational statistics have close connections with Bayesian analysis:

particle filter
Markov chain Monte Carlo (MCMC)

Point estimate v.s. confidence interval estimate

There are two major types of estimates: point estimate and confidence interval estimate. In the point estimate we try to choose a unique point in the parameter space which can reasonably be considered as the true value of the parameter. On the other hand, instead of unique estimate of the parameter, we are interested in constructing a family of sets that contain the true (unknown) parameter value with a specified probability. In many problems of statistical inference we are not interested only in estimating the parameter or testing some hypothesis concerning the parameter, we also want to get a lower or an upper bound or both, for the real-valued parameter. To do this, we need to construct a confidence interval.

Confidence interval describes how reliable an estimate is. We can calculate the upper and lower confidence limits of the intervals from the observed data. Suppose a dataset x₁, . . . , x_n is given, modeled as realization of random variables X₁, . . . , X_n. Let θ be the parameter of interest, and γ a number between 0 and 1. If there exist sample statistics L_n = g(X₁, . . . , X_n) and U_n = h(X₁, . . . , X_n) such that P(L_n < θ < U_n) = γ for every value of θ, then (l_n, u_n), where l_n = g(x₁, . . . , x_n) and u_n = h(x₁, . . . , x_n), is called a 100γ% confidence interval for θ. The number γ is called the confidence level.^[1] In general, with a normally-distributed sample mean, Ẋ, and with a known value for the standard deviation, σ, a 100(1-α)% confidence interval for the true μ is formed by taking Ẋ ± e, with e = z_1-α/2(σ/n^1/2), where z_1-α/2 is the 100(1-α/2)% cumulative value of the standard normal curve, and n is the number of data values in that column. For example, z_1-α/2 equals 1.96 for 95% confidence.^[15]

Here two limits are computed from the set of observations, say l_n and u_n and it is claimed with a certain degree of confidence (measured in probabilistic terms) that the true value of γ lies between l_n and u_n. Thus we get an interval (l_n and u_n) which we expect would include the true value of γ(θ). So this type of estimation is called confidence interval estimation.^[5] This estimation provides a range of values which the parameter is expected to lie. It generally gives more information than point estimates and are preferred when making inferences. In some way, we can say that point estimation is the opposite of interval estimation.

References

^ ^a ^b ^c ^d ^e A Modern Introduction to Probability and Statistics. F.M. Dekking, C. Kraaikamp, H.P. Lopuhaa, L.E. Meester. 2005.
^ Casella, George; Berger, Roger L. (2002). Statistical Inference (2nd ed.). Duxbury. p. 58.
^ Lehmann, Erich Leo (1951). “A General Concept of Unbiasedness”. Annals of Mathematical Statistics. 22 (4): 587–592 – via JSTOR.
^ Brown, George W. (1947). “On Small-Sample Estimation”. Annals of Mathematical Statistics. 18 (4): 582–585 – via JSTOR.
^ ^a ^b ^c ^d ^e Estimation and Inferential Statistics. Pradip Kumar Sahu, Santi Ranjan Pal, Ajit Kumar Das. 2015.
^ Categorical Data Analysis. John Wiley and Sons, New York: Agresti A. 1990.
^ The Concise Encyclopedia of Statistics. Springer: Dodge, Y. 2008.
^ ^a ^b Hyvärinen, Aapo (2005). “Estimation of Non-Normalized Statistical Models by Score Matching”. Journal of Machine Learning Research. 6: 695–709.
^ ^a ^b Dodge, Yadolah, ed. (1987). Statistical data analysis based on the L1-norm and related methods: Papers from the First International Conference held at Neuchâtel, August 31–September 4, 1987. North-Holland Publishing.
^ Jaynes, E. T. (2007). Probability Theory: The logic of science (5. print. ed.). Cambridge University Press. p. 172. ISBN 978-0-521-59271-0.
^ Ferguson, Thomas S. (1996). A Course in Large Sample Theory. Chapman & Hall. ISBN 0-412-04371-8.
^ ^a ^b Le Cam, Lucien (1986). Asymptotic Methods in Statistical Decision Theory. Springer-Verlag. ISBN 0-387-96307-3.
^ Ferguson, Thomas S. (1982). “An inconsistent maximum likelihood estimate”. Journal of the American Statistical Association. 77 (380): 831–834. doi:10.1080/01621459.1982.10477894. JSTOR 2287314.
^ Lehmann, E. L.; Casella, G. (1998). Theory of Point Estimation (2nd ed.). Springer. ISBN 0-387-98502-6.
^ Experimental Design – With Applications in Management, Engineering, and the Sciences. Springer: Paul D. Berger, Robert E. Maurer, Giovana B. Celli. 2019.

Sample Page