Thoughts about the sources of uncertainty in engineering simulations
This is just a little thought experiment about the field of CFD (Computational Fluid Dynamics). Specifically, I’d like to think about the possible sources of uncertainty of engineering simulations and what aspects are critical to delivering successful projects.
I’ll be talking primarily about CFD but similar principles are valid also for other virtual prototyping disciplines like FEA, optimization, and whole CAE (Computer-Aided Engineering).
The unavoidable issue of CFD is it’s rather a science than a calculator. Rather a craft than a tool. And this is a common misunderstanding. That’s why CFD is often associated with disappointment.
Consider a parallel with a typical scientific tool – a calculator. While the calculator directly executes the arithmetic operations and it gives exact answers on the exact inputs, and its results don’t bring any discussion about their correctness, the engineering simulations are different.
In CFD, there is always a certain amount of uncertainty involved. The reason is the solving method. What is solved is Navier-Stokes equations, which are technically parabolic nonlinear nonhomogeneous partial differential equations. On such a highly dynamic system is first applied a set of physical models and then it must be discretized in time and space using various schemes and then it is converted to the linear algebra problem and solved. The final result is always an approximation. Even if everything has been done perfectly, there is always a certain error involved which is quite impossible to quantify.
What is the error?
We understand the error of the results as their difference from the perfect results. In the field of fluid dynamics, the reference results are always associated with the physical measurements (experiment). Maybe you know this old engineering joke:
“Nobody trusts CFD simulation, except the one who made it. Everybody trusts the measurement, except the one who made it.”
CFD is not perfect, neither is the measurement. But in the fluid dynamics world, it’s the CFD who has to chase the measurement results and never the other way around. This is the game we play and these are the rules, no matter if we like it or not.
Total Error
First of all, any CFD workflow is a set of various techniques (methods). The total error is composed of several errors that arise from those applied techniques. Let’s take a look at a few most significant factors that contribute to the total error of the CFD results.
A. CAD Model Error
Let’s start with the basic and the most obvious source of error. It is the non-realistic CAD model for the simulation. The CAD model is a virtual representation of the real object to be simulated. The issue is that the simulation-ready CAD model is always a certain simplification of the real object. It is always a model that is simulated. Not the real object. The model has to be clean, watertight, all the tiny, irrelevant, and problematic model parts must be removed, and all the holes must be sealed up. The CAD modeling phase is an extremely important part of each simulation workflow. It sets up all the simulation potential and limitations. It should never be underestimated. Mistakes or poor quality engineering in the CAD modeling phase can be hardly compensated later in the simulation phase and postprocessing phase.
B. Mathematical Model Error
Every simulation project needs a connection between Physics and Mathematics. We call it a mathematical model – it turns physical theory into mathematical formulas. Let’s consider the following project example: We have a medical device – a blood pump. Our goal is to simulate the total pressure the pump generates at specific flow rates and rotation speeds. We have to set up our mathematical model. A couple of decisions have to be made. Just briefly, we could go with 1. Steady-state flow 2. Incompressible flow 3. Newtonian viscosity model 4. Not elastic fluid 5. No chemistry flow 6. Standard RANS Turbulence 7. Glassy walls. That means we consciously put aside the effects of transient behavior, compressibility, shear-thinning effects, blood elasticity, chemical reactions, resolved turbulence, wall roughness, elastic walls, and many more. All these decisions are connected with the resulting mathematical model – particular equations to be solved. This mathematical model carries many specific simplifications, but it is fine enough to give results that meet our goal. Despite the fact, the blood flow is much more complex, our model still can be very accurate – in terms of meeting our goal – pump total pressure. There we know the physical error is relatively low. On the other hand, the above model would be totally inconvenient if our goal were, for example, blood chemistry where the reacting transient effects are critical. The mathematical model is extremely important and always has to be created with respect to the goal of the simulation.
C. Boundary Conditions Error
While inside the simulation domain, in the volumes, there are the equations to be solved (mathematical model). On the domain boundary, the boundary conditions are applied. There are two major issues with boundary conditions. 1. The first problem of the boundary equations is the magnitude (typical for fixed, Dirichlet BC) and distribution of the quantities. 2. The second problem (typical for extrapolated (Neumann BC) and modeled quantities) is the way of their application. In the real world, there’s nothing like a perfectly closed domain, boundary conditions are just a simplification that enables creating a finite Eulerian region. Boundary conditions are an extremely important part of every simulation. We recommend paying the very best attention to them. We know from our technical support that boundary conditions are the source of most errors in CFD.
D. Discretization Error
To make this article simple enough, I allowed myself to place the mesh error and numerical scheme error in one section called discretization error. The simulation domain needs to be discretized in time and space. That means that continuous functions are virtually chopped on the discrete points because then numerical mathematics can be applied. In CFD, most typically the Eulerian domain is covered with the mesh – the domain is split into a number of little volumes. On every individual discrete volume (cell) and every quantity (or term), the numerical scheme is applied and summarized in the resulting standard linear system of equations Ax=b. The Linear system is then solved relatively easily with an exact algebraic method.
Again, there are many critical decisions to be made here. 1. How fine mesh to be made? 2. Mesh topology? 3. Mesh quality? 4. Boundary layer? 5. Numerical scheme? A finer mesh typically (not always) gives lower error. But the price you pay for fine mesh is enormous, the CPU time grows very quickly with the number of cells. Most often mesh setup ends up with a painful compromise between fine-enough-mesh and CPU-time. The choice of the numerical scheme is very critical too. While the first-order schemes are quite stable and less accurate, the second-order schemes (and higher) are more accurate but way less stable. Additionally, there is a large variety of mixed-order schemes which are somewhere in the middle both in accuracy and robustness. On top of that, every combination of mathematical model and mesh may lead to a different set of suitable numerical schemes.
E. Numerical Error
Especially at the large simulations (large meshes lead to large linear systems), the numerical error takes effect. An issue of numerical mathematics (linear algebra) is it makes a gigantic number of simple operations, but even a very little numerical error which is repeated many times, spreads and grows. Eventually, it may kill your results. The critical is the real number precision (the way your code is compiled). I remember some years ago, we did a couple of benchmarks on various precisions for OpenFOAM. The Single precision (32-bits, 8 digits) is super fast but for the meshes starting at a few million cells, the numerical error completely ruins the results. For productive CFD simulations, Double precision (64-bits, 16 digits) is recommended. Large, multi-million meshes which are used for example at properly resolved LES or DNS simulations, produce linear systems where the Quadruple precision (128-bits, 34 digits) has to be used. It’s next to impossible to enumerate the numerical error for each particular case apriori and one has to rely on special tests and experience. All in all, you can be sure that there will be some numerical error involved in every simulation. The only question is how big it will be.
F. Human Error
Although the tools we use are imperfect the way we use them is still even worse. We’re humans. Nature made us smart but not perfect at all. In fact, human error is perhaps the most common source of suffering in CFD. The reason is obvious. Every engineering simulation has many parameters. Even a very simple simulation forces us to make many decisions. Either conscious or unconscious. And unfortunately, we keep making mistakes. For many reasons, carelessness, lack of knowledge, too much knowledge (lol), lack of experience, etc. My colleagues from our technical support could tell you many stories. Some of them are funny, some of them are less funny. Yes, the problem is most of the time right in front of the screen. Human error is certainly a true killer of success.
G. Postprocessing Error
I personally hate this one the most. You would be surprised how often it happens that the raw simulation results are correct but their evaluation or interpretation is wrong. Sometimes the postprocessing includes many sophisticated mathematical operations where it’s very easy to make a mistake. It sounds crazy but many times, especially at new engineering projects, postprocessing is the most challenging discipline. And again, mistakes or poor engineering in the postprocessing phase can ruin otherwise correct results. We’ve seen it in countless cases in the past, to name a few: physical units confusion, total vs. static quantities, flux-weighted vs. area-weighted averaging, etc.
H. X-Factor Error
At every new project, there is always something we do not know. You have to be very careful with new projects because even if you did many similar projects before, the practice shows us again and again that even little different circumstances lead to tremendous differences in results. In CFD, more than anywhere else: The devil is in the detail. Every single project has its X-Factor and has to be treated with maximum care possible.
Fight the Error
All the above-mentioned errors can be diminished by engineering skills. The experience is critical. No one gets great CFD results, for the first time, the same way no one can ride a bike for the first time. It’s knowledge, focus, trained judgment, patience, dedication, open mind, tons of coffee, and the will to learn new things that help you to get great results. BTW, did you know it takes five years to raise a senior CFD engineer?
Ph.D. needed? Not necessarily.
Sometimes I see in online discussions people arguing if a Ph.D. degree is needed to produce excellent results. I am one of those who think that it is NOT a condition. In my career, I’ve seen excellent engineers without Ph.D. as well as poor engineers with the Ph.D. degree. Sure, I admit that the best engineers I ever met did have a Ph.D. degree. In my opinion, the Ph.D. degree is rather a symptom than a requirement. It tells about the personality and one’s ability to deal with a single problem down to a particular level rather than some extra theoretical knowledge. And that’s what really matters in CFD.
Can skills and experience be build-in into the simulation software?
This question is popping up regularly. Is it possible that some very experienced engineer implements his experience into a CFD code in a way that is easy to use, robust, and accurate enough even for non-experienced users? I think the answer is yes, but with many but-s. But only for a single application. But only for a simple application. But only to some basic level of accuracy. But only for a very limited range of user options.
Can CFD Code Be Quick, Robust, and Accurate at The Same Time?
All in all, the choice of the physical model together with the discretization method is critical and always leads to some kind of compromise. The simulation method can be either accurate or robust, but not all at once. Similarly, a method can be either quick or accurate. I like the quote from Prof. Tabor, I saw it somewhere in an internet discussion, he once commented: “Simulation can be quick, robust, and accurate. You can pick two of them.”
CFD is a very complex field consisting of Math, Physics, and software engineering. None of these three can’t be left behind. All of the above-mentioned ideas are supported by our 18-year’s experience with real projects in the field of numerical simulations. Based on all of that, I have to finish with: CFD(CAE) is rather a science than a calculator. Rather a craft than a tool.
