Even though many researchers have worked on the theoretical aspects of the MDDO approach, there are still some unresolved issues that we encountered in the industrial practice. In this section, we mention the most urgent ones that we have come across.
Concerning the third step (Design of Computer Experiments), there are four important issues:
• It is very difficult to find a good experimental design when many design parameter constraints are present in the design optimization problem. This problem is specifically difficult when the design space is relatively small compared to the smallest box containing the design space.
• It is not clear which criterion for the DoCE leads the best meta-models. Some authors mention that Maximin designs work well, which is contradicted by others. See Liu and Wakeland [2005].
• Since black box evaluations are time -consuming, we want to use as few as possible evaluations. Therefore, we may want to start with a small set of design sites, and add more design sites later. When new design sites are added to the set of experiments, the Design of Computer Experiments should still be optimal in some sense. Husslage et al. [2005] describe a construction method for LHDs that take possible future extensions into account. See also Kleijnen and Van Beers [2004]. However, both articles describe specific situations (in the first article, only two dimensional DoCE is treated, and in the second article only a one dimensional DoCE is treated in combination with Kriging models).
• Taking the simulation cost into account (e.g., when the black box can be evaluated in less time in one sub-area of the design space compared to another subarea) when constructing a DoCE is also an interesting open issue. An example of such a situation can be found in queueing theory, where the design parameters define the number of servers in a system. When there are few servers, it takes less time to simulate the average waiting time compared to the situation when many servers exist.
Concerning the fifth step (meta-model creation), there are four important issues that need attention.
• The meta-models that are frequently used for modeling of highly non-linear relations have a difficult interpretation. Kriging models are very popular for such situations. It is not trivial to extract the most important design parameters from such models. In many cases, transformations on design parameters could be used. It may be possible to create models that can easily be interpreted on the transformed data. However, finding good transformations is often a matter of trial and error.
• It is not clear how we can preserve a-priori known characteristics of the black box into the meta-model. For example, a-priori knowledge may be present about the relation between response parameters and design parameters, such as positivity (a response parameter should always have a positive value independent of the design parameter value), monotonicity (e.g., a response parameter value should increase when a design parameter value decreases) or convexity (the relation between a response parameter and design parameters should be convex). Preserving these characteristics leads to better user acceptance of the meta-model, but also to better meta-models. See Siem et al. [2005], where such situations are described for specific meta-model types.
• Estimating a meta-model for a multivariate response parameter in a deterministic setting is also an open issue. All techniques described in Section 1.3 estimate a meta-model per individual response parameter, ignoring the fact that the response parameters may be correlated, which could be estimated from the data. This information could be used to create meta-models more efficiently and to make better predictions. To the best of our knowledge, there is no literature available that treats this idea in a deterministic setting. For multivariate analysis in the non-deterministic setting, see Arnold [1981], Breiman and Friedman [1997] and Wackernagel [2003].
• Another interesting issue is dealing with uncertainty in the black box output. As mentioned earlier, simulation tools often suffer from numerical noise, which leads to inaccuracy of the simulation model in comparison with the real system. Taking these uncertainties into account will lead to better meta-models and eventually to final product designs that are more robust against the noise in the simulation tool.
In the analysis step (step 6) of the MDDO method, optimization and robustness analysis with respect to implementation errors are usually treated sequentially; first, a design is optimized with
to treat these problems integrally. Next to the implementation error, it would be good to take uncertainty about the meta-model and the simulation outcome into account when searching for an optimal design.
Besides these issues, coping with a large number of design parameters is difficult for both steps 3 and 5. The number of black box evaluations that is needed to find a good meta-model increases exponentially when the number of design parameters increases, which limits the use of the MDDO method. Screening is one answer to this problem, but still many design parameters may turn out to be relatively important. This issue is particularly of interest when complex products are designed that consist of many parts that interact, since in such cases all relevant design parameters should be considered in one design optimization problem.