Session: Validation Methods

Paper Number: 151299

151299 - “Vvuq Metrics and Applications to Improve Your Research” 

Abstract: 

The VVUQ 10.30 Metrics for Verification and Validation standard is almost complete.  This presentation will go through how each proposed metric is used and how to better communicate validation levels using the metrics.

Examples from defense programs and other data sets will highlight how to use the information for calibration of models, model development/improvement, and explanation of physical variabilities.  Other process details like master lists of sources of variability and factor tables will be shown and explained.

The goal here is to stop the use of, “The data looks similar,” that is seen in much of engineering research and publications and replace it with, “The metric improved from X to Y which is below our acceptable level for variability.”  This increase in rigor for the VVUQ process in engineering will result in higher test coverage and increased test power throughout multiple industries and applications.

Other fields of study such as psychology and political science must validate their findings with statistical methods, but engineering regularly publishes paper without this rigor.  The reasoning is generally assumed to start with a lack of statistics training in the engineering curriculum and extends to the practices of academia and industry to allow publication and presentations without statistical rigor.  Another reason is that engineering projects are generally governed by repeatable physics and, “Just need to work once,” (compared to the extension of findings from other fields to subjects outside the original population).   

Fortunately, engineers possess the mathematical skills to learn statistics relatively easily and an increase in master’s degrees in uncertainty quantification and applied statistics within engineering programs now exist.  At the least, larger companies can employ statistical engineers to help design experiments and consult on statistical/VVUQ analysis as needed.   In my experience, engineers often pick up on many of the processes after two or three experimental efforts.

This talk will serve as a basic process for organizing research with the goal of making the final VVUQ statistical analysis align with one of the VVUQ 10.30 metrics.  The talk will cover continuous vs discrete variables for inputs and outputs as well as experimental design for models and physical testing with UQ in mind.  With a good foundation and test plan, the statistics can be very simple, especially when using a tool like JMP or SmartUQ.  The goal is that the process in this talk becomes an important method for raising the bar for all work in future VVUQ conference. 

Presenting Author: David Harrison Lockheed Martin

Presenting Author Biography: David Harrison is the Scientific Test and Analysis Techniques and Uncertainty Quantification (STAT/UQ) lead for the Lockheed Martin corporation. He is in charge of improving the modeling, simulation, and physical test processes using improved statistical test design methods. David his a lean/six-sigma master black belt and a professor of process improvement at the Colorado School of Mines. He holds a Bachelor of Science degree in Mechanical Engineering from Kansas State University, a Masters of Science in Materials Science from the Colorado School of Mines, and a Masters of Science in Engineering Management from the University of Colorado.

Authors:

David R. Harrison Lockheed Martin
Kelsey Cannon Lockheed Martin