Көптік гипотеза тестілеуде I қателіктерінің ықтималдығы

Probability of making type I errors when performing multiple hypotheses tests

In statistics, family wise error rate (FWER) is the probability of making one or more false discoveries, or type I errors when performing multiple hypotheses tests.

John Tukey developed in 1953 the concept of a familywise error rate as the probability of making a Type I error among a specified group, or "family," of tests. Ryan (1959) proposed the related concept of an experimentwise error rate, which is the probability of making a Type I error in a given experiment. Hence, an experimentwise error rate is a familywise error rate where the family includes all the tests that are conducted within an experiment. As Ryan (1959, Footnote 3) explained, an experiment may contain two or more families of multiple comparisons, each of which relates to a particular statistical inference and each of which has its own separate familywise error rate.

Some classical solutions that ensure strong level FWER control, and some newer solutions exist.

Testing each hypothesis at level is Sidak's multiple testing procedure. This procedure is more powerful than Bonferroni but the gain is small. This procedure can fail to control the FWER when the tests are negatively dependent.

Tukey's procedure is only applicable for pairwise comparisons. It assumes independence of the observations being tested, as well as equal variation across observations (homoscedasticity). The procedure calculates for each pair the studentized range statistic: where is the larger of the two means being compared, is the smaller, and is the standard error of the data in question. Tukey's test is essentially a Student's t test, except that it corrects for family wise error rate.

Charles Dunnett (1955, 1966) described an alternative alpha error adjustment when k groups are compared to the same control group. Now known as Dunnett's test, this method is less conservative than the Bonferroni adjustment.

The procedures of Bonferroni and Holm control the FWER under any dependence structure of the p values (or equivalently the individual test statistics). Essentially, this is achieved by accommodating a `worst case' dependence structure (which is close to independence for most practical purposes). But such an approach is conservative if dependence is actually positive. To give an extreme example, under perfect positive dependence, there is effectively only one test and thus, the FWER is uninflated. Accounting for the dependence structure of the p values (or of the individual test statistics) produces more powerful procedures. This can be achieved by applying resampling methods, such as bootstrapping and permutations methods. The procedure of Westfall and Young (1993) requires a certain condition that does not always hold in practice (namely, subset pivotality). The procedures of Romano and Wolf (2005a,b) dispense with this condition and are thus more generally valid.

The harmonic mean p value (HMP) procedure provides a multilevel test that improves on the power of Bonferroni correction by assessing the significance of groups of hypotheses while controlling the strong sense family wise error rate. The significance of any subset of the tests is assessed by calculating the HMP for the subset, where are weights that sum to one (i. e. ). An approximate procedure that controls the strong sense family wise error rate at level approximately rejects the null hypothesis that none of the p values in subset are significant when (where ). This approximation is reasonable for small (e. g. ) and becomes arbitrarily good as approaches zero. An asymptotically exact test is also available (see main article).

FWER control exerts a more stringent control over false discovery compared to false discovery rate (FDR) procedures. FWER control limits the probability of at least one false discovery, whereas FDR control limits (in a loose sense) the expected proportion of false discoveries. Thus, FDR procedures have greater power at the cost of increased rates of type I errors, i. e., rejecting null hypotheses that are actually true. On the other hand, FWER control is less stringent than per family error rate control, which limits the expected number of errors per family. Because FWER control is concerned with at least one false discovery, unlike per family error rate control it does not treat multiple simultaneous false discoveries as any worse than one false discovery. The Bonferroni correction is often considered as merely controlling the FWER, but in fact also controls the per family error rate.