Статистикалық модельдерде максималды ықтималдық бағалауларға итеративтік әдіс

Iterative method for finding maximum likelihood estimates in statistical models

In statistics, an expectation–maximization (EM) algorithm is an iterative method to find (local) maximum likelihood or maximum a posteriori (MAP) estimates of parameters in statistical models, where the model depends on unobserved latent variables. The EM iteration alternates between performing an expectation (E) step, which creates a function for the expectation of the log likelihood evaluated using the current estimate for the parameters, and a maximization (M) step, which computes parameters maximizing the expected log likelihood found on the E step. These parameter estimates are then used to determine the distribution of the latent variables in the next E step. It can be used, for example, to estimate a mixture of gaussians, or to solve the multiple linear regression problem.

EM is frequently used for parameter estimation of mixed models, notably in quantitative genetics. In psychometrics, EM is an important tool for estimating item parameters and latent abilities of item response theory models. With the ability to deal with missing data and observe unidentified variables, EM is becoming a useful tool to price and manage risk of a portfolio. The EM algorithm (and its faster variant ordered subset expectation maximization) is also widely used in medical image reconstruction, especially in positron emission tomography, single photon emission computed tomography, and x ray computed tomography. See below for other faster variants of EM. In structural engineering, the Structural Identification using Expectation Maximization (STRIDE) algorithm is an output only method for identifying natural vibration properties of a structural system using sensor data (see Operational Modal Analysis). EM is also used for data clustering. In natural language processing, two prominent instances of the algorithm are the Baum–Welch algorithm for hidden Markov models, and the inside outside algorithm for unsupervised induction of probabilistic context free grammars. In the analysis of intertrade waiting times i. e. the time between subsequent trades in shares of stock at a stock exchange the EM algorithm has proved to be very useful.

A number of methods have been proposed to accelerate the sometimes slow convergence of the EM algorithm, such as those using conjugate gradient and modified Newton's methods (Newton–Raphson). Also, EM can be used with constrained estimation methods. Parameter expanded expectation maximization (PX EM) algorithm often provides speed up by "us[ing] a `covariance adjustment' to correct the analysis of the M step, capitalising on extra information captured in the imputed complete data". Expectation conditional maximization (ECM) replaces each M step with a sequence of conditional maximization (CM) steps in which each parameter θi is maximized individually, conditionally on the other parameters remaining fixed. Itself can be extended into the Expectation conditional maximization either (ECME) algorithm. This idea is further extended in generalized expectation maximization (GEM) algorithm, in which is sought only an increase in the objective function F for both the E step and M step as described in the As a maximization–maximization procedure section. It is also possible to consider the EM algorithm as a subclass of the MM (Majorize/Minimize or Minorize/Maximize, depending on context) algorithm, and therefore use any machinery developed in the more general case.

The Q function used in the EM algorithm is based on the log likelihood. Therefore, it is regarded as the log EM algorithm. The use of the log likelihood can be generalized to that of the α log likelihood ratio. Then, the α log likelihood ratio of the observed data can be exactly expressed as equality by using the Q function of the α log likelihood ratio and the α divergence. Obtaining this Q function is a generalized E step. Its maximization is a generalized M step. This pair is called the α EM algorithm
which contains the log EM algorithm as its subclass. Thus, the α EM algorithm by Yasuo Matsuyama is an exact generalization of the log EM algorithm. No computation of gradient or Hessian matrix is needed. The α EM shows faster convergence than the log EM algorithm by choosing an appropriate α. The α EM algorithm leads to a faster version of the Hidden Markov model estimation algorithm α HMM.

EM is a partially non Bayesian, maximum likelihood method. Its final result gives a probability distribution over the latent variables (in the Bayesian style) together with a point estimate for θ (either a maximum likelihood estimate or a posterior mode). A fully Bayesian version of this may be wanted, giving a probability distribution over θ and the latent variables. The Bayesian approach to inference is simply to treat θ as another latent variable. In this paradigm, the distinction between the E and M steps disappears. If using the factorized Q approximation as described above (variational Bayes), solving can iterate over each latent variable (now including θ) and optimize them one at a time. Now, k steps per iteration are needed, where k is the number of latent variables. For graphical models this is easy to do as each variable's new Q depends only on its Markov blanket, so local message passing can be used for efficient inference.

In information geometry, the E step and the M step are interpreted as projections under dual affine connections, called the e connection and the m connection; the Kullback–Leibler divergence can also be understood in these terms.

Let be a sample of independent observations from a mixture of two multivariate normal distributions of dimension , and let be the latent variables that determine the component from which the observation originates. Special cases of this model include censored or truncated observations from one normal distribution. or the so called spectral techniques. Moment based approaches to learning the parameters of a probabilistic model are of increasing interest recently since they enjoy guarantees such as global convergence under certain conditions unlike EM which is often plagued by the issue of getting stuck in local optima. Algorithms with guarantees for learning can be derived for a number of important models such as mixture models, HMMs etc. For these spectral methods, no spurious local optima occur, and the true parameters can be consistently estimated under some regularity conditions.