Model validation is an important exercise in any empirical analysis. The overall purpose of the validation process is to test how well a model serves its intended purpose. In the case of predictive models, validation tests usually involve comparing the model predictions to real world results. For prescriptive models, decision maker reliance is the ultimate validation test. Unfortunately, however, these tests (especially the comparison of the model predictions to real world results) are rarely used. This is mainly because they are expensive and time-consuming. Consequently, linear programming models in most cases are superficially validated (McCarl and Spreen, 1997).
Although a model may have a broad range of potential uses, it may be valid only for a few of those uses. The validation process usually results in the identification of valid applications. Model validation is fundamentally subjective. Modellers choose the validity tests, the criteria for passing those tests, what model outputs to validate, what setting to test in and what data to use. Thus, the assertion "the model was judged valid" can mean almost anything. Nonetheless, a model validation effort will reveal model strengths and weaknesses which is valuable to users and those who extract information from the model’s results. This section provides a brief description of the main approaches for validating linear programming models.
Approaches for Validating Linear Programming Models
There is a wide variation in approaches for validating linear programming models. But the most widely used techniques are: validation by construct and validation by results. Whereas validation by construct asserts that the model was built properly therefore it is valid, validation by results refers to exercises where the model outputs are systematically compared against real world observations (McCarl and Spreen, 1997).
Validation by Construct
This is the most widely used validation approach. Although, theoretically, validation by construct is supposed to be only the starting point for the process of validating a linear programming model, in most studies it is also the end of the validation exercise. The linear programming model used in the present study has been validated by construct. The use of validation by construct, as the sole method of validation, is justified by the following assertions about modelling:
i) The right procedures have been used in the process of building the model. Usually this entails the assertion that the approach is consistent with the industry, previous research and/or theory; and that the data have been specified using reasonable scientific estimation or accounting procedures. Most of the parameters of the model in the present study have been estimated by using data collected through a detail survey of producers of potential feedstocks for biofuels production which was conducted in 2005.
ii) Trial results indicate that the model is behaving satisfactorily. This arises from a nominal examination of model results which indicates they do not contradict the modeller’s, user's, and/or associated experts perceptions of reality.
iii) Constraints have been imposed to restrict the model to realistic solutions. Some exercises use constraints to limit adjustment possibilities and force the model to give results very close to historically observed outcomes. In the present study we imposed minimum acreage constraints for crops such as cassava to ensure their availability for their traditional food use.
iv) The data have been set up in a manner that ensures that real world outcome would be replicated. In some models one can assure replication of a real world outcome through the model structure and data calculation procedures.
Validation by Results
Validation by results involves the comparison of the model solutions with real world outcomes. Models used in such a comparison will always have been built relying on experience, precedence, theory, appropriate data estimation and measurement procedures. Thus, validation by construct will always precede validation by results.
The process of validating a model by results entails five main phases: first, a set of real world outcomes and the data causing that outcome is gathered; second, a validation experiment is selected; third, the model is set up with the appropriate data, the experiment is implemented and a solution is generated; fourth, the degree of association between model output and the real world outcome is tested; and, finally, a decision is made regarding model validity (McCarl and Spreen, 1997).
Common Causes of Validation Failures for Linear Programming Models
From a practical standpoint, models do not always pass validation tests. Since models always involve many assumptions, failure to validate, likely indicates that improper assumptions have been used. Consequently, when models fail validation tests, modellers often ask: What assumptions should be corrected? (McCarl and Spreen, 1997).
As discussed in the previous sections, linear programming models embody assumptions about both mathematical relationships and the model structure. The mathematical relationships assumptions are: additivity, divisibility, certainty, and proportionality. These assumptions, when severely violated, will cause the model to fail validation tests. In such situations, the model designer has to consider whether the assumptions are the real cause of the failure. If so, the use of techniques such as separable, integer, nonlinear, or stochastic programming may be considered in constructing a new model. Modelling assumptions may also cause a linear programming model to fail a validation test. These assumptions embody the correctness of the objective function, variables, equations included, coefficients, and equation specification. Programming algorithms are quite useful in discovering the violation of linear programming assumptions. Given an optimal solution, one may easily discover what resources have been used, how they have been used, and their marginal values. Thus, when a model fails a validation test, resource usage and valuation should be investigated. Models are most often invalid
because of inconsistent data, bad coefficient calculation, bad equation specification, or an incorrect objective function. Thus, common fixes for a model failing a validation test involve data respecification and/or structural corrections (McCarl and Spreen, 1997). Models may also fail validation tests because of improperly formulated objective function. While the specification of the constraints identifies the set of possible solutions, the objective function determines the single optimal solution. Thus, if the model fails the validation test, the objective function must be carefully reviewed. Another phenomenon which may cause models to fail validation tests is ignoring the fact that operations, quite often, are performed over several time periods. Consequently, an annual model depicting operations of this type may well be invalid because it ignores initial conditions or does not recognise that parameter expectations may change over time. Thus, unless the model has initial conditions identical to those in the real world, it may be very difficult for it to pass validation tests (McCarl and Spreen, 1997).