Omnibus tests are a kind of statistical test. They test whether the explained variance in a set of data is significantly greater than the unexplained variance, overall. One example is the F-test in the analysis of variance. There can be legitimate significant effects within a model even if the omnibus test is not significant. For instance, in a model with two independent variables, if only one variable exerts a significant effect on the dependent variable and the other does not, then the omnibus test may be non-significant. This fact does not affect the conclusions that may be drawn from the one significant variable. In order to test effects within an omnibus test, researchers often use contrasts.
Omnibus test, as a general name, refers to an overall or a global test. Other names include F-test or Chi-squared test. It is a statistical test implemented on an overall hypothesis that tends to find general significance between parameters' variance, while examining parameters of the same type, such as: Hypotheses regarding equality vs. inequality between k expectancies vs. at least one pair , where and , in Analysis Of Variance (ANOVA); or regarding equality between k standard deviations vs. at least one pair in testing equality of variances in ANOVA; or regarding coefficients vs. at least one pair in Multiple linear regression or in Logistic regression.
Usually, it tests more than two parameters of the same type and its role is to find general significance of at least one of the parameters involved.
Definitions Omnibus test commonly refers to either one of those statistical tests:
ANOVA F test to test significance between all factor means and/or between their variances equality in Analysis of Variance procedure; The omnibus multivariate F Test in ANOVA with repeated measures; F test for equality/inequality of the regression coefficients in multiple regression; Chi-Square test for exploring significance differences between blocks of independent explanatory variables or their coefficients in a logistic regression.
These omnibus tests are usually conducted whenever one tends to test an overall hypothesis on a quadratic statistic (like sum of squares or variance or covariance) or rational quadratic statistic (like the ANOVA overall F test in Analysis of Variance or F Test in Analysis of covariance or the F Test in Linear Regression, or Chi-Square in Logistic Regression).
While significance is founded on the omnibus test, it doesn't specify exactly where the difference has occurred, meaning it doesn't bring specification on which parameter is significantly different from the other, but it statistically determines that there is a difference. Thus, at least two of the tested parameters are statistically different. If significance was met, none of those tests will tell specifically which mean differs from the others (in ANOVA), which coefficient differs from the others (in regression) etc.
In one-way analysis of variance The F-test in ANOVA is an example of an omnibus test, which tests the overall significance of the model. A significant F test means that among the tested means, at least two of the means are significantly different, but this result doesn't specify exactly which means are different one from the other. Actually, testing means' differences is done by the quadratic rational F statistic ( F=MSB/MSW). In order to determine which mean differs from another mean or which contrast of means are significantly different, Post Hoc tests (Multiple Comparison tests) or planned tests should be conducted after obtaining a significant omnibus F test. It may be considered to use the simple Bonferroni correction or another suitable correction. Another omnibus test we can find in ANOVA is the F test for testing one of the ANOVA assumptions: the equality of variance between groups. In One-Way ANOVA, for example, the hypotheses tested by omnibus F test are:
H0: μ1=μ2=....= μk
H1: at least one pair μj≠μj'
These hypotheses examine model fit of the most common model: yij = μj + εij, where yij is the dependent variable, μj is the j-th independent variable's expectancy, which usually is referred to as "group expectancy" or "factor expectancy"; and εij are the errors results on using the model.
The F statistics of the omnibus test is:
Where, is the overall sample mean, is the group j sample mean, k is the number of groups and nj is sample size of group j.
The F statistic is distributed F(k-1,n-k),(α) under assumption of null hypothesis and normality assumption. F test is considered robust in some situations, even when the normality assumption isn't met.
Model assumptions in one-way ANOVA
Random sampling. Normal or approximately normal distribution of in each group. Equal variances between groups.
If the assumption of equality of variances is not met, Tamhane's test is preferred. When this assumption is satisfied we can choose amongst several tests. Although the LSD (Fisher's least significant difference) is a very strong test in detecting pairs of means differences, it is applied only when the F test is significant, and it is mostly less preferable since its method fails in protecting low error rate. Bonferroni test is a good choice due to its correction suggested by his method. This correction states that if n independent tests are to be applied then the α in each test should be equal to α /n. Tukey's method is also preferable by many statisticians because it controls the overall error rate. On small sample sizes, when the assumption of normality is not met, a nonparametric analysis of variance can be made by the Kruskal-Wallis test.
An alternative option is to use bootstrap methods to assess whether the group means are different. Bootstrap methods do not have any specific distributional assumptions and may be an appropriate tool to use like using re-sampling, which is one of the simplest bootstrap methods. A person can extend the idea to the case of multiple groups and estimate p-values.
Example A cellular survey on customers' time-wait was reviewed on 1,963 different customers during 7 days on each one of 20 in-sequential weeks. Assuming none of the customers called twice and none of them have customer relations among each other, One Way ANOVA was run on SPSS to find significant differences between the days time-wait:
ANOVA
Dependent variable: time minutes to respond
Source Sum of Squares df Mean Square F Sig. Between Groups 12823.921 6 2137.320 158.266 .000 Within Groups 26414.958 1956 13.505 Total 39238.879 1962
The omnibus F ANOVA test results above indicate significant differences between the days time-wait (P-Value =0.000 < 0.05, α =0.05).
The other omnibus tested was the assumption of Equality of Variances, tested by the Levene F test:
Test of Homogeneity of Variances
Dependent variable: time minutes to respond
Levene Statistic df1 df2 Sig. 36.192 6 1956 .000
The results suggest that the equality of variances assumption can't be made. In that case Tamhane's test can be made on Post Hoc comparisons.
Considerations A significant omnibus F test in ANOVA procedure, is an in advance requirement before conducting the Post Hoc comparison, otherwise those comparisons are not required. If the omnibus test fails to find significant differences between all means, it means that no difference has been found between any combinations of the tested means. In such, it protects family-wise Type I error, which may be increased if overlooking the omnibus test. Some debates have occurred about the efficiency of the omnibus F Test in ANOVA.
In a paper Review of Educational Research (66(3), 269-306) which reviewed by Greg Hancock, those problems are discussed:
William B. Ware (1997) claims that the omnibus test significance is required depending on the Post Hoc test is conducted or planned: "... Tukey's HSD and Scheffé's procedure are one-step procedures and can be done without the omnibus F having to be significant. They are "a posteriori" tests, but in this case, "a posteriori" means "without prior knowledge", as in "without specific hypotheses." On the other hand, Fisher's Least Significant Difference test is a two-step procedure. It should not be done without the omnibus F-statistic being significant."
William B. Ware (1997) argued that there are a number of problems associated with the requirement of an omnibus test rejection prior to conducting multiple comparisons. Hancock agrees with that approach and sees the omnibus requirement in ANOVA in performing planned tests an unnecessary test and potentially detrimental, hurdle unless it is related to Fisher's LSD, which is a viable option for k=3 groups.
Other reason for relating to the omnibus test significance when it is concerned to protect family-wise Type I error.
The publication "Review of Educational Research" discusses four problems in the omnibus F test requirement:
First, in a well planned study, the researcher's questions involve specific contrasts of group means' while the omnibus test, addresses each question only tangentially and it is rather used to facilitate control over the rate of Type I error.
Secondly, this issue of control is related to the second point: the belief that an omnibus test offers protection is not completely accurate. When the complete null hypothesis is true, weak family-wise Type I error control is facilitated by the omnibus test; but, when the complete null is false and partial nulls exist, the F-test does not maintain strong control over the family-wise error rate.
A third point, which Games (1971) demonstrated in his study, is that the F-test may not be completely consistent with the results of a pairwise comparison approach. Consider, for example, a researcher who is instructed to conduct Tukey's test only if an alpha-level F-test rejects the complete null. It is possible for the complete null to be rejected but for the widest ranging means not to differ significantly. This is an example of what has been referred to as non-consonance/dissonance (Gabriel, 1969) or incompatibility (Lehmann, 1957). On the other hand, the complete null may be retained while the null associated with the widest ranging means would have been rejected had the decision structure allowed it to be tested. This has been referred to by Gabriel (1969) as incoherence. One wonders if, in fact, a practitioner in this situation would simply conduct the MCP contrary to the omnibus test's recommendation.