Session: Artificial Intelligence and Machine Learning Models

Paper Number: 152225

152225 - Wholistic Credibility Assessment of Machine Learning-Based Surrogate Model in an Aerodynamics Application 

Abstract: 

Computational physics models offer many advantages in science and engineering. These include enabling a deeper understanding of phenomena through detailed simulation, exploration of design space with significantly lower cost than physical experimentation, and wise investment in design due to robust simulation-based understanding of systems. Yet, computational models of relatively high accuracy often have high costs in terms of computational resources and time to execute a simulation. Surrogate models are often used instead of high-fidelity physics-based models in situations that call for many model evaluations, such as design optimization and uncertainty quantification. Such surrogate models have been used to great effect in science and engineering. However, the development of established practices for credibility assessment of surrogate models lags the development of the models themselves. By comparison with established credibility assessment practices for computational physics models, which have standards published by major companies and professional organizations, best practices for surrogate model credibility assessment need significant development. In high-consequence applications, it is imperative that these best practices for surrogate model credibility assessment be established. In the present study, a deep neural network-based surrogate model is used to predict the coefficients of lift and drag on a NACA 0012 airfoil in subsonic turbulent flow. The base set of conditions in the parameter space defined by Mach number, Reynolds number, and angle of attack corresponds to a NASA turbulence modeling resource validation case. The parameter space is expanded significantly and additional points are chosen via an LHS sampling strategy.  Simulations are run using the RANS-SST model implemented in Ansys Fluent. The simulation results form the dataset used for training and testing of the surrogate model. A credibility analysis is conducted for the surrogate model. Elements to include in the credibility assessment are guided by a PCMM-based framework. The credibility analysis done in the present study includes a datasheet describing the dataset, cross-validation of the surrogate model, quantification of uncertainty from both the parent CFD model and the surrogate model, and an ASME V&V 20-style validation analysis at experimental points. The performance of the surrogate model near the edges of the training data space is compared with that further inside of the training space. A validation error regression exercise is also conducted for the surrogate model. Several points are held out of the training dataset and used to evaluate an estimate of the validation error and uncertainty. Overall, this study demonstrates rigorous credibility assessment practices for surrogate models and drives toward a more comprehensive set of best practices for such assessments.

Presenting Author: Jared Kirsch Texas A&M University

Presenting Author Biography: Jared Kirsch is a Ph.D. candidate in the Texas A&M Department of Mechanical Engineering and a graduate student intern at Sandia National Laboratories. He holds bachelor's and master's degrees in mechanical engineering from the University of New Mexico. His research involves the core areas of CFD and VVUQ and exists at the intersection of hypersonic aerothermodynamics, machine learning, and surrogate model credibility analysis.

Authors:

Jared Kirsch Texas A&M University
William Rider Sandia National Laboratories
Nima Fathi Texas A&M University