Session: 02-01 Development and Application of Verification, Validation, Uncertainty Quantification Standards

Paper Number: 139260

139260 - Developing a Universal Error Assessment Framework for Modeling and Simulation 

The U.S. Nuclear Regulatory Commission (NRC) has advanced its regulatory processes through the implementation of a robust assessment framework, primarily focused on data-driven models. The introduction of this framework has led to an improvement in the consistency and completeness of those technical reviews. Notably, it has also achieved a substantial reduction in the time and resources required for those evaluations. Owing to these substantial benefits, the NRC is now transitioning this specific framework into a more comprehensive and universal error assessment framework for modeling and simulation (M&S).

The presentation will focus on three main objectives: developing a universal, developing a “complete set” of errors, and ranking the techniques used to estimate each error. To make the framework universal, it is defined in terms of basic components which are common to practically all of M&S: the system, conceptual model, mathematical model, computational model, and empirical data. Sets of errors are defined purely in terms of these components. Different use cases will have different components and therefore will result in different errors. For example, physics-based models have a different set of errors than machined learned models. While both models are expressible in this system, it is important to recognize their differences. 

No matter the specific use case, one key requirement is that the final set of errors chosen to evaluate is a “complete set”. A “complete set” of errors is defined as any set of errors, such that if each error were known exactly, the simulation could be corrected to exactly match the real-world system. This is a crucial aspect as it provides a theoretical foundation for achieving perfect simulation accuracy if perfect knowledge were available.

Because perfect knowledge is not available, many errors will not be able to be directly calculated, but must be determined through some estimation process.  However, there are generally multiple methods for estimating each error.  Therefore, these methods are compared with each other and ranked according to their level of maturity (i.e., ranked according to which methods provide more trust worthy estimates of the error). This ranking process is crucial as each estimation of an error results in an error itself, and the error in that estimation process is often difficult to quantify and may be ignored in the final analysis. 

Finally, the presentation will conclude with a discussion on the future directions and developments required to fully establish this framework as a credibility assessment tool. While the error assessment framework will enable analysts to successful “measure” M&S, by itself it does not provide any indication that a given M&S is “credible”.  However, if trained on past credible simulations, the credibility levels for each error assessment can be determined and used going forward in decision making processes.                 

Presenting Author: Joshua Kaizer U.S. Nuclear Regulatory Commission

Presenting Author Biography: Dr. Joshua Kaizer has been a regulator at the U.S. Nuclear Regulatory Commission since 2006. He has spent his entire career reviewing the Verification, Validation, and Uncertainty Quantification (VVUQ) analysis which supports reactor safety models and simulations. He has performed over 30 reviews of modeling and simulation, primarily focused on thermal-hydraulics, data-driven modeling, and uncertainty analysis. He is Vice-Chair of ASME’s VVUQ standards committee, a member of the VVUQ subcommittees on nuclear systems and machine learning, a member of NAFEMS working group on simulation governance, an associate editor of the Journal of VVUQ.