In the design of experiments, optimal experimental designs (or optimum designsThe adjective "optimum" (and not "optimal") "is the slightly older form in English and avoids the construction 'optim(um) + al´—there is no 'optimalis' in Latin" (page x in Optimum Experimental Designs, with SAS, by Atkinson, Donev, and Tobias).) are a class of experimental designs that are optimal with respect to some statistical criterion. The creation of this field of statistics has been credited to Danish statistician Kirstine Smith.
In the design of experiments for estimating statistical models, optimal designs allow parameters to be estimated without bias and with minimum variance. A non-optimal design requires a greater number of experimental runs to estimate the parameters with the same precision as an optimal design. In practical terms, optimal experiments can reduce the costs of experimentation.
The optimality of a design depends on the statistical model and is assessed with respect to a statistical criterion, which is related to the variance-matrix of the estimator. Specifying an appropriate model and specifying a suitable criterion function both require understanding of statistical theory and practical knowledge with designing experiments.
Advantages Optimal designs offer three advantages over sub-optimal experimental designs:These three advantages (of optimal designs) are documented in the textbook by Atkinson, Donev, and Tobias. Optimal designs reduce the costs of experimentation by allowing statistical models to be estimated with fewer experimental runs. Optimal designs can accommodate multiple types of factors, such as process, mixture, and discrete factors. Designs can be optimized when the design-space is constrained, for example, when the mathematical process-space contains factor-settings that are practically infeasible (e.g. due to safety concerns).
Minimizing the variance of estimators Experimental designs are evaluated using statistical criteria.Such criteria are called objective functions in optimization theory.
It is known that the least squares estimator minimizes the variance of mean-unbiased estimators (under the conditions of the Gauss–Markov theorem). In the estimation theory for statistical models with one real parameter, the reciprocal of the variance of an ("efficient") estimator is called the "Fisher information" for that estimator.The Fisher information and other "information" functionals are fundamental concepts in statistical theory. Because of this reciprocity, minimizing the variance corresponds to maximizing the information.
When the statistical model has several parameters, however, the mean of the parameter-estimator is a vector and its variance is a matrix. The inverse matrix of the variance-matrix is called the "information matrix". Because the variance of the estimator of a parameter vector is a matrix, the problem of "minimizing the variance" is complicated. Using statistical theory, statisticians compress the information-matrix using real-valued summary statistics; being real-valued functions, these "information criteria" can be maximized.Traditionally, statisticians have evaluated estimators and designs by considering some summary statistic of the covariance matrix (of a mean-unbiased estimator), usually with positive real values (like the determinant or matrix trace). Working with positive real-numbers brings several advantages: If the estimator of a single parameter has a positive variance, then the variance and the Fisher information are both positive real numbers; hence they are members of the convex cone of nonnegative real numbers (whose nonzero members have reciprocals in this same cone). For several parameters, the covariance-matrices and information-matrices are elements of the convex cone of nonnegative-definite symmetric matrices in a partially ordered vector space, under the Loewner (Löwner) order. This cone is closed under matrix-matrix addition, under matrix-inversion, and under the multiplication of positive real-numbers and matrices.
An exposition of matrix theory and the Loewner-order appears in Pukelsheim. The traditional optimality-criteria are invariants of the information matrix; algebraically, the traditional optimality-criteria are functionals of the eigenvalues of the information matrix.
A-optimality ("average" or trace) One criterion is A-optimality, which seeks to minimize the trace of the inverse of the information matrix. This criterion results in minimizing the average variance of the estimates of the regression coefficients. C-optimality This criterion minimizes the variance of a best linear unbiased estimator of a predetermined linear combination of model parameters. D-optimality (determinant) A popular criterion is D-optimality, which seeks to minimize |(X'X)−1|, or equivalently maximize the determinant of the information matrix X'X of the design. This criterion results in maximizing the differential Shannon information content of the parameter estimates. E-optimality (eigenvalue) Another design is E-optimality, which maximizes the minimum eigenvalue of the information matrix. S-optimality This criterion maximizes a quantity measuring the mutual column orthogonality of X and the determinant of the information matrix. T-optimality This criterion maximizes the discrepancy between two proposed models at the design locations.
Other optimality-criteria are concerned with the variance of predictions: G-optimality A popular criterion is G-optimality, which seeks to minimize the maximum entry in the diagonal of the hat matrix X(X'X)−1X'. This has the effect of minimizing the maximum variance of the predicted values. I-optimality (integrated) A second criterion on prediction variance is I-optimality, which seeks to minimize the average prediction variance over the design space. V-optimality (variance) A third criterion on prediction variance is V-optimality, which seeks to minimize the average prediction variance over a set of m specific points.The above optimality-criteria are convex functions on domains of symmetric positive-semidefinite matrices: See an on-line textbook for practitioners, which has many illustrations and statistical applications: (book in pdf) Boyd and Vandenberghe discuss optimal experimental designs on pages 384–396.
Contrasts
In many applications, the statistician is most concerned with a "parameter of interest" rather than with "nuisance parameters". More generally, statisticians consider linear combinations of parameters, which are estimated via linear combinations of treatment-means in the design of experiments and in the analysis of variance; such linear combinations are called contrasts. Statisticians can use appropriate optimality-criteria for such parameters of interest and for contrasts.Optimality criteria for "parameters of interest" and for contrasts are discussed by Atkinson, Donev and Tobias.
Implementation Catalogs of optimal designs occur in books and in software libraries.
In addition, major statistical systems like SAS and R have procedures for optimizing a design according to a user's specification. The experimenter must specify a model for the design and an optimality-criterion before the method can compute an optimal design.Iterative methods and approximation algorithms are surveyed in the textbook by Atkinson, Donev and Tobias and in the monographs of Fedorov (historical) and Pukelsheim, and in the survey article by Gaffke and Heiligers.
Practical considerations Some advanced topics in optimal design require more statistical theory and practical knowledge in designing experiments.
Model dependence and robustness Since the optimality criterion of most optimal designs is based on some function of the information matrix, the 'optimality' of a given design is model dependent: While an optimal design is best for that model, its performance may deteriorate on other models. On other models, an optimal design can be either better or worse than a non-optimal design.See Kiefer ("Optimum Designs for Fitting Biased Multiresponse Surfaces" pages 289–299). Therefore, it is important to benchmark the performance of designs under alternative models.Such benchmarking is discussed in the textbook by Atkinson et al. and in the papers of Kiefer. Model-robust designs (including "Bayesian" designs) are surveyed by Chang and Notz.
Choosing an optimality criterion and robustness The choice of an appropriate optimality criterion requires some thought, and it is useful to benchmark the performance of designs with respect to several optimality criteria. Cornell writes that
Indeed, there are several classes of designs for which all the traditional optimality-criteria agree, according to the theory of "universal optimality" of Kiefer.An introduction to "universal optimality" appears in the textbook of Atkinson, Donev, and Tobias. More detailed expositions occur in the advanced textbook of Pukelsheim and the papers of Kiefer. The experience of practitioners like Cornell and the "universal optimality" theory of Kiefer suggest that robustness with respect to changes in the optimality-criterion is much greater than is robustness with respect to changes in the model.
Flexible optimality criteria and convex analysis High-quality statistical software provide a combination of libraries of optimal designs or iterative methods for constructing approximately optimal designs, depending on the model specified and the optimality criterion. Users may use a standard optimality-criterion or may program a custom-made criterion.
All of the traditional optimality-criteria are convex (or concave) functions, and therefore optimal-designs are amenable to the mathematical theory of convex analysis and their computation can use specialized methods of convex minimization.Computational methods are discussed by Pukelsheim and by Gaffke and Heiligers. The practitioner need not select exactly one traditional, optimality-criterion, but can specify a custom criterion. In particular, the practitioner can specify a convex criterion using the maxima of convex optimality-criteria and nonnegative combinations of optimality criteria (since these operations preserve convex functions). For convex optimality criteria, the Kiefer-Wolfowitz equivalence theorem allows the practitioner to verify that a given design is globally optimal.The Kiefer-Wolfowitz equivalence theorem is discussed in Chapter 9 of Atkinson, Donev, and Tobias. The Kiefer-Wolfowitz equivalence theorem is related with the Legendre-Fenchel conjugacy for convex functions.Pukelsheim uses convex analysis to study Kiefer-Wolfowitz equivalence theorem in relation to the Legendre-Fenchel conjugacy for convex functions
The minimization of convex functions on domains of symmetric positive-semidefinite matrices is explained in an on-line textbook for practitioners, which has many illustrations and statistical applications: (book in pdf) Boyd and Vandenberghe discuss optimal experimental designs on pages 384–396.
If an optimality-criterion lacks convexity, then finding a global optimum and verifying its optimality often are difficult.
Model uncertainty and Bayesian approaches Model selection
When scientists wish to test several theories, then a statistician can design an experiment that allows optimal tests between specified models. Such "discrimination experiments" are especially important in the biostatistics supporting pharmacokinetics and pharmacodynamics, following the work of Cox and Atkinson.See Chapter 20 in Atkinison, Donev, and Tobias.
Bayesian experimental design