Статистикалық модельдерді бағалау критерийі: Акайке ақпараттық критерийі

Estimator for quality of a statistical model
The Akaike information criterion (AIC) is an estimator of prediction error and thereby relative quality of statistical models for a given set of data. Given a collection of models for the data, AIC estimates the quality of each model, relative to each of the other models. Thus, AIC provides a means for model selection. AIC is founded on information theory. When a statistical model is used to represent the process that generated the data, the representation will almost never be exact; so some information will be lost by using the model to represent the process. AIC estimates the relative amount of information lost by a given model: the less information a model loses, the higher the quality of that model. In estimating the amount of information lost by a model, AIC deals with the trade off between the goodness of fit of the model and the simplicity of the model. In other words, AIC deals with both the risk of overfitting and the risk of underfitting. The Akaike information criterion is named after the Japanese statistician Hirotsugu Akaike, who formulated it. It now forms the basis of a paradigm for the foundations of statistics and is also widely used for statistical inference.

Suppose that we have a statistical model of some data. Let k be the number of estimated parameters in the model. Let be the maximized value of the likelihood function for the model. Then the AIC value of the model is the following. As an example, suppose that there are three candidate models, whose AIC values are 100, 102, and 110. Then the second model is times as probable as the first model to minimize the information loss. Similarly, the third model is times as probable as the first model to minimize the information loss. In this example, we would omit the third model from further consideration. We then have three options: (1) gather more data, in the hope that this will allow clearly distinguishing between the first two models; (2) simply conclude that the data is insufficient to support selecting one model from among the first two; (3) take a weighted average of the first two models, with weights proportional to 1 and 0.368, respectively, and then do statistical inference based on the weighted multimodel. The quantity exp((AICmin − AICi)/2) is known as the relative likelihood of model i. It is closely related to the likelihood ratio used in the likelihood ratio test. Indeed, if all the models in the candidate set have the same number of parameters, then using AIC might at first appear to be very similar to using the likelihood ratio test. There are, however, important distinctions. In particular, the likelihood ratio test is valid only for nested models, whereas AIC (and AICc) has no such restriction.

Every statistical hypothesis test can be formulated as a comparison of statistical models. Hence, every statistical hypothesis test can be replicated via AIC. Two examples are briefly described in the subsections below. Details for those examples, and many more examples, are given by and .

As an example of a hypothesis test, consider the t test to compare the means of two normally distributed populations. The input to the t test comprises a random sample from each of the two populations. To formulate the test as a comparison of models, we construct two different models. The first model models the two populations as having potentially different means and standard deviations. The likelihood function for the first model is thus the product of the likelihoods for two distinct normal distributions; so it has four parameters: μ1, σ1, μ2, σ2. To be explicit, the likelihood function is as follows (denoting the sample sizes by n1 and n2). The second model models the two populations as having the same means but potentially different standard deviations. The likelihood function for the second model thus sets in the above equation; so it has three parameters. We then maximize the likelihood functions for the two models (in practice, we maximize the log likelihood functions); after that, it is easy to calculate the AIC values of the models. We next calculate the relative likelihood. For instance, if the second model was only 0.01 times as likely as the first model, then we would omit the second model from further consideration: so we would conclude that the two populations have different means. The t test assumes that the two populations have identical standard deviations; the test tends to be unreliable if the assumption is false and the sizes of the two samples are very different (Welch's t test would be better). Comparing the means of the populations via AIC, as in the example above, has an advantage by not making such assumptions.

For another example of a hypothesis test, suppose that we have two populations, and each member of each population is in one of two categories—category #1 or category #2. Each population is binomially distributed. We want to know whether the distributions of the two populations are the same. We are given a random sample from each of the two populations. Let m be the size of the sample from the first population. Let m1 be the number of observations (in the sample) in category #1; so the number of observations in category #2 is m − m1. Similarly, let n be the size of the sample from the second population. Let n1 be the number of observations (in the sample) in category #1. Let p be the probability that a randomly chosen member of the first population is in category #1. Hence, the probability that a randomly chosen member of the first population is in category #2 is 1 − p. Note that the distribution of the first population has one parameter. Let q be the probability that a randomly chosen member of the second population is in category #1. Note that the distribution of the second population also has one parameter. To compare the distributions of the two populations, we construct two different models. The first model models the two populations as having potentially different distributions. The likelihood function for the first model is thus the product of the likelihoods for two distinct binomial distributions; so it has two parameters: p, q. To be explicit, the likelihood function is as follows. The second model models the two populations as having the same distribution. The likelihood function for the second model thus sets in the above equation; so the second model has one parameter. We then maximize the likelihood functions for the two models (in practice, we maximize the log likelihood functions); after that, it is easy to calculate the AIC values of the models. We next calculate the relative likelihood. For instance, if the second model was only 0.01 times as likely as the first model, then we would omit the second model from further consideration: so we would conclude that the two populations have different distributions.

Statistical inference is generally regarded as comprising hypothesis testing and estimation. Hypothesis testing can be done via AIC, as discussed above. Regarding estimation, there are two types: point estimation and interval estimation. Point estimation can be done within the AIC paradigm: it is provided by maximum likelihood estimation. Interval estimation can also be done within the AIC paradigm: it is provided by likelihood intervals. Hence, statistical inference generally can be done within the AIC paradigm. The most commonly used paradigms for statistical inference are frequentist inference and Bayesian inference. AIC, though, can be used to do statistical inference without relying on either the frequentist paradigm or the Bayesian paradigm: because AIC can be interpreted without the aid of significance levels or Bayesian priors. In other words, AIC can be used to form a foundation of statistics that is distinct from both frequentism and Bayesianism.

When the sample size is small, there is a substantial probability that AIC will select models that have too many parameters, i. e. that AIC will overfit. To address such potential overfitting, AICc was developed: AICc is AIC with a correction for small sample sizes. The formula for AICc depends upon the statistical model. Assuming that the model is univariate, is linear in its parameters, and has normally distributed residuals (conditional upon regressors), then the formula for AICc is as follows. —where n denotes the sample size and k denotes the number of parameters. Thus, AICc is essentially AIC with an extra penalty term for the number of parameters. Note that as n → ∞, the extra penalty term converges to 0, and thus AICc converges to AIC. If the assumption that the model is univariate and linear with normal residuals does not hold, then the formula for AICc will generally be different from the formula above. For some models, the formula can be difficult to determine. For every model that has AICc available, though, the formula for AICc is given by AIC plus terms that includes both k and k2. In comparison, the formula for AIC includes k but not k2. In other words, AIC is a first order estimate (of the information loss), whereas AICc is a second order estimate. Further discussion of the formula, with examples of other assumptions, is given by and by In particular, with other assumptions, bootstrap estimation of the formula is often feasible. To summarize, AICc has the advantage of tending to be more accurate than AIC (especially for small samples), but AICc also has the disadvantage of sometimes being much more difficult to compute than AIC. Note that if all the candidate models have the same k and the same formula for AICc, then AICc and AIC will give identical (relative) valuations; hence, there will be no disadvantage in using AIC, instead of AICc. Furthermore, if n is many times larger than k2, then the extra penalty term will be negligible; hence, the disadvantage in using AIC, instead of AICc, will be negligible.

The Akaike information criterion was formulated by the statistician Hirotsugu Akaike. It was originally named "an information criterion". The 1973 publication, though, was only an informal presentation of the concepts. The first formal publication was a 1974 paper by Akaike, which as of lc=y, has received more than 35,100 citations in the Web of Science database and more than 64,000 according to Google Scholar. AIC has become common enough that it is often used without citing Akaike's 1974 paper; indeed, as of lc=y, there are around 289,000 scholarly articles and books that use AIC (as estimated by Google Scholar). The initial derivation of AIC relied upon some strong assumptions. showed that the assumptions could be made much weaker. Takeuchi's work, however, was in Japanese and was not widely known outside Japan for many years. (Translated in )

AICc was originally proposed for linear regression (only) by That instigated the work of , and several further papers by the same authors, which extended the situations in which AICc could be applied. The first general exposition of the information theoretic approach was the volume by It includes an English presentation of the work of Takeuchi. The volume led to far greater use of AIC, and it now has more than 64,000 citations on Google Scholar. Akaike called his approach an "entropy maximization principle", because the approach is founded on the concept of entropy in information theory. Indeed, minimizing AIC in a statistical model is effectively equivalent to maximizing entropy in a thermodynamic system; in other words, the information theoretic approach in statistics is essentially applying the Second Law of Thermodynamics. As such, AIC has roots in the work of Ludwig Boltzmann on entropy. For more on these issues, see and .

The AIC values of the candidate models must all be computed with the same data set. Sometimes, though, we might want to compare a model of the response variable, y, with a model of the logarithm of the response variable, log(y). More generally, we might want to compare a model of the data with a model of transformed data. Following is an illustration of how to deal with data transforms (adapted from : "Investigators should be sure that all hypotheses are modeled using the same response variable"). Suppose that we want to compare two models: one with a normal distribution of y and one with a normal distribution of log(y). We should not directly compare the AIC values of the two models. Instead, we should transform the normal cumulative distribution function to first take the logarithm of y. To do that, we need to perform the relevant integration by substitution: thus, we need to multiply by the derivative of the (natural) logarithm function, which is 1/y. Hence, the transformed distribution has the following probability density function:

—which is the probability density function for the log normal distribution. We then compare the AIC value of the normal model against the AIC value of the log normal model. For misspecified model, Takeuchi's Information Criterion (TIC) might be more appropriate. However, TIC often suffers from instability caused by estimation errors.

The critical difference between AIC and BIC (and their variants) is the asymptotic property under well specified and misspecified model classes. Their fundamental differences have been well studied in regression variable selection and autoregression order selection problems. In general, if the goal is prediction, AIC and leave one out cross validations are preferred. If the goal is selection, inference, or interpretation, BIC or leave many out cross validations are preferred. A comprehensive overview of AIC and other popular model selection methods is given by Ding et al. (2018)

Leave one out cross validation is asymptotically equivalent to AIC, for ordinary linear regression models. Asymptotic equivalence to AIC also holds for mixed effects models.

Mallows's Cp is equivalent to AIC in the case of (Gaussian) linear regression.