Robust statistics

Type of statistics
Robust statistics are statistics which maintain their properties even if the underlying distributional assumptions are incorrect. Robust statistical methods have been developed for many common problems, such as estimating location, scale, and regression parameters. One motivation is to produce statistical methods that are not unduly affected by outliers. Another motivation is to provide methods with good performance when there are small departures from a parametric distribution. For example, robust methods work well for mixtures of two normal distributions with different standard deviations; under this model, non robust methods like a t test work poorly.

Robust statistics seek to provide methods that emulate popular statistical methods, but are not unduly affected by outliers or other small departures from model assumptions. In statistics, classical estimation methods rely heavily on assumptions that are often not met in practice. In particular, it is often assumed that the data errors are normally distributed, at least approximately, or that the central limit theorem can be relied on to produce normally distributed estimates. Unfortunately, when there are outliers in the data, classical estimators often have very poor performance, when judged using the breakdown point and the influence function, described below. The practical effect of problems seen in the influence function can be studied empirically by examining the sampling distribution of proposed estimators under a mixture model, where one mixes in a small amount (1–5% is often sufficient) of contamination. For instance, one may use a mixture of 95% a normal distribution, and 5% a normal distribution with the same mean but significantly higher standard deviation (representing outliers). Robust parametric statistics can proceed in two ways:
by designing estimators so that a pre selected behaviour of the influence function is achieved
by replacing estimators that are optimal under the assumption of a normal distribution with estimators that are optimal for, or at least derived for, other distributions: for example using the t distribution with low degrees of freedom (high kurtosis) or with a mixture of two or more distributions. Robust estimates have been studied for the following problems:
estimating location parameters
estimating scale parameters
estimating regression coefficients
estimation of model states in models expressed in state space form, for which the standard method is equivalent to a Kalman filter.

There are various definitions of a "robust statistic". Strictly speaking, a robust statistic is resistant to errors in the results, produced by deviations from assumptions (e. g., of normality). This means that if the assumptions are only approximately met, the robust estimator will still have a reasonable efficiency, and reasonably small bias, as well as being asymptotically unbiased, meaning having a bias tending towards 0 as the sample size tends towards infinity. Usually, the most important case is distributional robustness robustness to breaking of the assumptions about the underlying distribution of the data.

Although this article deals with general principles for univariate statistical methods, robust methods also exist for regression problems, generalized linear models, and parameter estimation of various distributions.

The basic tools used to describe and measure robustness are the breakdown point, the influence function and the sensitivity curve.

Intuitively, the breakdown point of an estimator is the proportion of incorrect observations (e. g. arbitrarily large observations) an estimator can handle before giving an incorrect (e. g., arbitrarily large) result. Usually, the asymptotic (infinite sample) limit is quoted as the breakdown point, although the finite sample breakdown point may be more useful. For example, given independent random variables and the corresponding realizations , we can use to estimate the mean. Such an estimator has a breakdown point of 0 (or finite sample breakdown point of ) because we can make arbitrarily large just by changing any of
The higher the breakdown point of an estimator, the more robust it is. Intuitively, we can understand that a breakdown point cannot exceed 50% because if more than half of the observations are contaminated, it is not possible to distinguish between the underlying distribution and the contaminating distribution Therefore, the maximum breakdown point is 0.5 and there are estimators which achieve such a breakdown point. For example, the median has a breakdown point of 0.5. The X% trimmed mean has a breakdown point of X%, for the chosen level of X. and contain more details. The level and the power breakdown points of tests are investigated in
Statistics with high breakdown points are sometimes called resistant statistics.

In the speed of light example, removing the two lowest observations causes the mean to change from 26.2 to 27.75, a change of 1.55. The estimate of scale produced by the Qn method is 6.3. We can divide this by the square root of the sample size to get a robust standard error, and we find this quantity to be 0.78. Thus, the change in the mean resulting from removing two outliers is approximately twice the robust standard error. The 10% trimmed mean for the speed of light data is 27.43. Removing the two lowest observations and recomputing gives 27.67. The trimmed mean is less affected by the outliers and has a higher breakdown point. If we replace the lowest observation, −44, by −1000, the mean becomes 11.73, whereas the 10% trimmed mean is still 27.43. In many areas of applied statistics, it is common for data to be log transformed to make them near symmetrical. Very small values become large negative when log transformed, and zeroes become negatively infinite. Therefore, this example is of practical interest.

This value, which looks a lot like a Lipschitz constant, represents the effect of shifting an observation slightly from to a neighbouring point , i. e., add an observation at and remove one at .

(The mathematical context of this paragraph is given in the section on empirical influence functions.) Historically, several approaches to robust estimation were proposed, including R estimators and L estimators. However, M estimators now appear to dominate the field as a result of their generality, their potential for high breakdown points and comparatively high efficiency. See
M estimators are not inherently robust. However, they can be designed to achieve favourable properties, including robustness. M estimator are a generalization of maximum likelihood estimators (MLEs) which is determined by maximizing or, equivalently, minimizing In 1964, Huber proposed to generalize this to the minimization of , where is some function. MLE are therefore a special case of M estimators (hence the name: "Maximum likelihood type" estimators). Minimizing can often be done by differentiating and solving , where (if has a derivative). Several choices of and have been proposed. The two figures below show four functions and their corresponding functions. For squared errors, increases at an accelerating rate, whilst for absolute errors, it increases at a constant rate. When Winsorizing is used, a mixture of these two effects is introduced: for small values of x, increases at the squared rate, but once the chosen threshold is reached (1.5 in this example), the rate of increase becomes constant. This Winsorised estimator is also known as the Huber loss function. Tukey's biweight (also known as bisquare) function behaves in a similar way to the squared error function at first, but for larger errors, the function tapers off.

M estimators do not necessarily relate to a probability density function. Therefore, off the shelf approaches to inference that arise from likelihood theory can not, in general, be used. It can be shown that M estimators are asymptotically normally distributed so that as long as their standard errors can be computed, an approximate approach to inference is available. Since M estimators are normal only asymptotically, for small sample sizes it might be appropriate to use an alternative approach to inference, such as the bootstrap. However, M estimates are not necessarily unique (i. e., there might be more than one solution that satisfies the equations). Also, it is possible that any particular bootstrap sample can contain more outliers than the estimator's breakdown point. Therefore, some care is needed when designing bootstrap schemes. Of course, as we saw with the speed of light example, the mean is only normally distributed asymptotically and when outliers are present the approximation can be very poor even for quite large samples. However, classical statistical tests, including those based on the mean, are typically bounded above by the nominal size of the test. The same is not true of M estimators and the type I error rate can be substantially above the nominal level. These considerations do not "invalidate" M estimation in any way. They merely make clear that some care is needed in their use, as is true of any other method of estimation.

In many practical situations, the choice of the function is not critical to gaining a good robust estimate, and many choices will give similar results that offer great improvements, in terms of efficiency and bias, over classical estimates in the presence of outliers. Theoretically, functions are to be preferred, and Tukey's biweight (also known as bisquare) function is a popular choice. recommend the biweight function with efficiency at the normal set to 85%.

A pivotal quantity is a function of data, whose underlying population distribution is a member of a parametric family, that is not dependent on the values of the parameters. An ancillary statistic is such a function that is also a statistic, meaning that it is computed in terms of the data alone. Such functions are robust to parameters in the sense that they are independent of the values of the parameters, but not robust to the model in the sense that they assume an underlying model (parametric family), and in fact, such functions are often very sensitive to violations of the model assumptions. Thus test statistics, frequently constructed in terms of these to not be sensitive to assumptions about parameters, are still very sensitive to model assumptions.

Replacing missing data is called imputation. If there are relatively few missing points, there are some models which can be used to estimate values to complete the series, such as replacing missing values with the mean or median of the data. Simple linear regression can also be used to estimate missing values. In addition, outliers can sometimes be accommodated in the data through the use of trimmed means, other scale estimators apart from standard deviation (e. g., MAD) and Winsorization. In calculations of a trimmed mean, a fixed percentage of data is dropped from each end of an ordered data, thus eliminating the outliers. The mean is then calculated using the remaining data. Winsorizing involves accommodating an outlier by replacing it with the next highest or next smallest value as appropriate. However, using these types of models to predict missing values or outliers in a long time series is difficult and often unreliable, particularly if the number of values to be in filled is relatively high in comparison with total record length. The accuracy of the estimate depends on how good and representative the model is and how long the period of missing values extends. When dynamic evolution is assumed in a series, the missing data point problem becomes an exercise in multivariate analysis (rather than the univariate approach of most traditional methods of estimating missing values and outliers). In such cases, a multivariate model will be more representative than a univariate one for predicting missing values. The Kohonen self organising map (KSOM) offers a simple and robust multivariate model for data analysis, thus providing good possibilities to estimate missing values, taking into account their relationship or correlation with other pertinent variables in the data record. Standard Kalman filters are not robust to outliers. To this end have recently shown that a modification of Masreliez's theorem can deal with outliers. One common approach to handle outliers in data analysis is to perform outlier detection first, followed by an efficient estimation method (e. g., the least squares). While this approach is often useful, one must keep in mind two challenges. First, an outlier detection method that relies on a non robust initial fit can suffer from the effect of masking, that is, a group of outliers can mask each other and escape detection. Second, if a high breakdown initial fit is used for outlier detection, the follow up analysis might inherit some of the inefficiencies of the initial estimator.