In practice, it may happen that not all parts of a given code have been properly verified prior to its use in V&V or even in end-user activities. This can happen for a variety of reasons such as (1) overlooking a relevant part of an algorithm, (2) improperly testing it, (3) insufficiently testing it, (4) not adequately documenting the verification, or (5) not including the tests within the set of regression tests. Therefore, it behooves anyone engaged in solution verification, validation, uncertainty quantification, and anyone who is an end-user to identify the specific algorithms within the code that their application will use.4 Identifying the relevant parts of the code should be done prior to use and it should then be determined whether or not the relevant parts have been verified and to what extent. If testing or documentation deficiencies are found, then the relevant algorithms in the code must be additionally verified, through appropriate use of NAVP and NAAV, prior to use of the code.
Within the context of a specific intended use of the code (say, within a solution verification exercise), one can say that a collection of algorithms used in one or more simulations that are performed for the purpose of the prediction and its verification is adequate if, within the memory and efficiency limits of the selected computing platform, the algorithms produce correct numerical solutions that are both converged and asymptotic. In this definition, “converged”
means that both the solver and non-linear iteration loops were terminated by satisfying acceptably low relative error tolerances on the residual or on other numerical quantities of interest; and
“asymptotic” means that the numerical solution, as a function of mesh-discretization size, can be shown via mesh refinement studies to be in the asymptotic range of mesh convergence. Under this narrow definition, adequacy is directly determined during the process of creating numerical solutions for the calculation of numerically precise error estimates and error bars, (i.e., in a validation exercise or in the process of using the code as an end-user). In this context, the activities involved in both NAVP and NAAV are often considered to be part of the solution verification process.
Code users, including those engaged in various V&V processes desire evidence that provides them with confidence that mathematical model and its solution algorithms are working correctly.
Although there are technical software engineering and code verification drivers, another driver of the need for confidence is due to the fact that there is often a division of labor in which one group of people function as code developers and another group functions as code users. The first group develops and tests code capabilities, while the second group (sometimes referred to as analysts) performs the predictions and their calculation verification. Both groups require evidence that gives them confidence in the code and algorithms. The preferred evidence is frequently based on running the code on nearby problems, which involve performing a test that resembles the ultimate
4 This determination can be systematically addressed through code coverage tables, which explain the mapping between the tests in the verification test suite and the portions of the code that are exercised in a particular prediction simulation [5, 6].
application in one or more of its essential aspects, but removes non-essential features of the ultimate application so that important behavior can be isolated and the testing is less cumbersome. In this context, passing a series of nearby problem tests provides evidence that the code is working correctly.5 Examples of algorithm adequacy testing performed by developers include: efficiency tests, robustness tests, iterative convergence tests, numerical artifact tests, symmetry tests, benchmark tests [8], and many others. The confidence issue arises in several contexts:
1. The decision by the users regarding the selection of the code that is to be used in the V&V exercise and beyond.
2. The hand-off between the code developers and the analysts who will perform calculation verification.
3. The subsequent use, improvement, and advertisement of the code after a V&V exercise are completed.
References
1. American Society of Mechanical Engineers, Guide for Verification and Validation in Computational Solid Mechanics, American Society of Mechanical Engineers (ASME) V&V 10 – 2006, 36p, 2006.
2. Institute of Electrical and Electronics Engineers, Standard for Software Verification and Validation, IEEE Standard 1012, IEEE Computer Society, 2004.
3. Bezier, B., Software Testing Techniques, International Thompson Press, 1990.
4. Sommerville, I., Software Engineering, Addison-Wesley, 2005.
5. Dowding, K., Blackwell, B., and Knupp, P., “ Demonstrating Code Verification Methods with Calore,” Sandia National Laboratories report SAND2007-5612, Albuquerque NM, 2007.
6. Knupp, P., and Ober, C., “ A Code-Verification Evidence-Generation Process Model and Checklist,” Sandia National Laboratories report SAND2008-4832, Albuquerque NM, 2008.
7. Oberkampf, W. L., and Trucano, T. G., “Verification and Validation in Computational Fluid Dynamics,” Sandia National Laboratories report SAND2002-0529, 2002.
8. Oberkampf, W., and Trucano, T., “ Verification and Validation Benchmarks,” Sandia National Laboratories report SAND2007-0853, 2007.
9. Knupp, P., Ober, C., and Bond, R., “Measuring Progress in Order-Verification within Software Development Projects,” Engineering with Computers, Vol. 23, 2007.
10. Roache, P. J., “Verification of Codes and Calculations,” AIAA Journal, Vol. 36, No. 5, May 1998.
5 It should be noted, however, that passing a set of nearby problem tests does not guarantee that the code will then provide an acceptable solution to the simulations to which they are supposedly “near.”
11. Roache, P. J., Verification and Validation in Computational Science and Engineering, Hermosa Publishers, 1998.
12. Knupp, P., and Salari, K., Verification of Computer Codes in Computational Science and Engineering, Chapman & Hall/CRC, Boca Raton FL, 2003.
13. Roache, P., “ Building Codes to be Verifiable and Validable,” Computing in Science &
Engineering, pp. 30-38, September/October 2004.