A few weeks ago, when I first discussed the ESAS Appendices on this blog, I had to be somewhat vague about my concerns, since the information wasn’t out in the public, and I didn’t want to get Chris in trouble with his sources. However, since nobody has commented further on the appendices since they went public, I figured I ought to do a follow-up post. While I haven’t had a chance to dive much more into that 630pg document, I wanted to elaborate on some of the flaws I’ve seen so far, as many of them demonstrate fundamentally rookie engineering mistakes.
Gaming the Assumptions to Give You the “Right” Results
Probably the most egregious of the flaws I saw was the exception given in the “ground rules and assumptions” to the Stick concepts. Now, as I’ve discussed before on this blog, tweaking assumptions to make sure that “the right answer” looks favorable is standard fare for trade studies. Now, this isn’t always a case of deliberate larceny. Many times honest engineers can have an opinion about the best route, and when the numbers don’t come out quite they way they expected, they go back and try and see if they “made a mistake” in their assumptions. That said, typically when a honest person is unintentionally massaging the assumptions–or when a dishonest but competent person is intentionally gaming the assumptions–they aren’t as blatant as this (found on page 28, my emphasis added):
Max dynamic pressure = 800 psf (undispersed), except for certain In-line Crew (ILC) configuration-Solid Rocket Motor (SRM)-In-line cases where the limit was raised to 1,000 psf due to very high accelerations early in the ascent profile.
Max dynamic pressure = 1000 psf (dispersed), except for certain ILC-SRM-In-line cases where the limit was raised to 1,200 psf due to very high accelerations early in the ascent profile.
Dynamic pressure (or “q”) is the component of total pressure due to fluid kinetic energy, ie in this case it is pressure felt by trying to shove a high speed rocket through the air. Dynamic pressure is proportional to the air density and the square of the vehicle’s velocity. The maximum dynamic pressure (“max-q”) is an important factor for designing manned launch vehicles, and there are legitimate safety reasons for wanting to keep max-q within reasonable levels. Higher max-q can rapidly drive up the required thrust of an LAS, it drives up the structural stress, especially bending loads, and in the case of a controls problem could lead to tumbling and rapid failure of the stack.
So, what is it about an SRM-based vehicle that makes it safer to fly at higher max-q? Why do they get an exception, when no other approaches do? To me, the fact that an exception was given only to “ILC-SRM In-line” (aka variants on der Griffenshaft) is what really marks this as an amateur job of assumptions gaming. A clever assumptions gamer would’ve just upped the max-q value for everyone. After all, even though no other alternatives come anywhere close to needing an exception, at least by giving some technobabble excuse like “on further analysis we found that the original GR&A numbers were too conservative” it looks like you’re being honest.
The other thing I find amusing about this exception, is that after carving out a special exemption just for the boss’s preferred option, someone felt they needed to at least give some sort of rationalization. Of course, you’d think that when making an exception to a safety-based requirement, you’d come up with, you know, a safety-based justification. But, the justification given was that this safety requirement should be modified–because the preferred vehicles weren’t capable of meeting the requirement! Note to future assumptions gamers: if you’re going to blatantly carve out an exception like this, don’t openly admit that your ideas would’ve failed without the exception!
It’s worth noting that of all the options compared on pages 3-11, all of the vehicles requiring this exception are Stick-like vehicles, and that every one of the 5-segment SRB designs requires the exemption. In fact, some of the 5-segment designs closest to the one NASA’s trying to build now almost fail even this exception.
Now, there’s another relatively innocuous-sounding assumption on page 29 that, when combined with this exception, really biases things against EELVs and in favor of the stick. The assumption about LES mass:
LES mass = 9,300 lb for vehicles sized under Block 2 analysis. 9,172 lb for vehicles analyzed under Block 1.
Now, ignore the Block 1 vs. Block 2 bit, as that is fairly irrelevant. While it sure makes calculations easier, why should all LES’s weigh the same? In reality, vehicles with higher max-qs, first stages or boosters that can’t be turned off easily, and with stages whose failures tend to be more catastrophic (like solids), would tend to require more beefy LES systems. The thrust requirement of the LES is driven by a transonic abort (typically around max-q). In this situation, you have the maximum aerodynamic force trying to shove your capsule back into the first stage, especially since a low-pressure region tends to form between the capsule and the stage during separation. The higher the max-q, the more force you need to provide just to stand-still relative to the first stage. If the first stage also can’t be shut down, you now need to provide more acceleration…
It’s illustrative to note that current LES weights for Ares-I are somewhere between 1.5-2x heavier than the number used in the ESAS study.
There are also a few other examples of GR&A’s that may favor Shuttle Derived Vehicles at the expense of existing boosters. On page 30, there’s a long list of propellant-related margins and reserves. In each case, SDVs get to use a number from the “MPS propellant inventory from STS-117 TDDP”, while the non-SDV systems are given an equation to use (which produces numbers significantly higher than those measured in their flights). Not having access to that Trajectory Design Data Package, I have no way of telling how the two compare, but it’s a potential source of bias.
There may be others that I’ve also missed, but that’s enough on that topic.
Why It’s Important to Validate Models
One of the most common rookie engineering mistake when using computational models is to trust the pretty pictures without doing proper validation. The scary thing about things like CFD (which a cynical friend calls “Colorful Fluid Drawings”) is that the computer will often give you a plausible-looking, bug completely bogus answer if you screw up in your assumptions. One of the keys to properly using FEA or CFD is that you need to validate your code and your processes. If you can’t get your analysis to match up with a known, physically measured result, either your code is broken, or you’re making bad assumptions. I ought to repeat that–If Your Model and Reality Don’t Agree, Your Model Is Wrong. Now, in school, when the deadlines are near, professors will often give partial credit even if you screwed up and your model doesn’t match reality…but in the real world, a little bit more is expected.
So, coming back to ESAS, the a group at Marshall had developed a cool software tool for predicting the performance of new rocket stages, called INTROS (INTegrated ROcket Sizing program), that they decided to use for doing an apples-to-apples comparison of all the vehicles. Now, being from Marshall, a lot of assumptions were built into the design based on how Marshall would design a stage, and based on shuttle experience. The problem is that when you ran an existing launch vehicle designed by another group through the wringer, the result that came out was often…slightly reality impaired.
Take the Dual Engine Centaur (DEC) upper stage as an example. On pages 35-37, the appendix goes into details about the “existing” Atlas V and Delta IV stages (as replicated by INTROS), as well Atlas V’s and Delta IVs with a new-and-improved MSFC designed, man-rated upper stage. Here’s the relevant stats according to them:
Usable propellant: 43,840lbm
Dry Mass: 7,159 lbm
Burnout Mass: 9,222 lbm
Now, if you add the difference between the burnout mass and the dry mass (2063lbm) to the “Usable Propellant” number, you get about 45,903lbm, which is right in the ballpark of the numbers typically given for a Centaur stage. The problem is with the Dry Mass figure. 7,159 is substantially higher than any number I’ve ever seen given for a dual engine centaur. Most numbers I’ve seen (including the ones on the Lockheed Martin Atlas V page) give a dry weight for the DEC in the 4700-5400lbm range. The DEC that flew on Titan IV, and which had a slightly higher propellant load (and a more complicated shape) was on the order of 6,600lb. When your model mispredicts the weight of existing stages by almost 50%, it suggests that the model wasn’t really ready for primetime. More importantly, it suggests that the model probably shouldn’t have been used to evaluate existing stages since it was so far out of whack with even publicly available numbers for those stages. I know that some publicly available stage numbers might be inaccurate (notice the range given for DEC weights), but the mere fact that the model was that far off should’ve led to more investigation.
This, by the way, perfectly corroborates the testimony of several people on the LM (now ULA) side of the story, as well as some people working on the NASA side of the EELV program–that the ESAS guys really badly botched the dry weight numbers for the existing EELV upper stages. The explanation I heard at the time was that the NASA guys were given insufficient time during the prep-work for ESAS to get the model calibrated, and once the study started they weren’t allowed to communicate with the EELV companies. The sad thing is that when the ULA guys pointed out that the numbers were wrong, the MSFC manager really believed his own models more than the “as-weighed” numbers from ULA.
When you combine unrealistically high ullage/margin/residual numbers with unrealistically high stage dry masses, it’s unsurprising that the estimated performance to high altitude, high inclination orbits (ISS orbits) was substantially lower than what ULA (and the Air Force) estimate the performance at.
But just in case this travesty weren’t humorous enough, take a look at the bottom of page 35, where they give the numbers for the “new upper stage” that they are always claiming the EELVs need in order to be man-rated. Their 4x RL-10 stage is estimated to weigh 15,115lbm! And that in spite of carrying only about 30% more propellant. I’m not even sure how they can make the stage performance that lousy. We’re talking nearly three times the dry mass of the DEC stage, for only 30% more propellant, and 2 extra engines (that each weigh about 400-500lb each). Even assuming that everything was only designed to a FOS of 1.25 instead of the magical 1.4 (which by the way the Shuttle ET doesn’t meet), that wouldn’t account for much of the weight.
I know that Mike Griffin was claiming that part of the reason for doing Ares-I was to teach NASA how to design launch vehicles again, and I guess we have documented proof of the need.
Seriously though, this is a common rookie mistake. You don’t go basing decisions worth tens of billions of dollars on an unvalidated design tool. Now, this isn’t saying that INTROS is a useless tool, just that it obviously doesn’t capture all the state of the art in stage design, and until it does, it’s results ought to be taken with the appropriate sized (apparently multi-ton) grain of salt.
Anyhow, that’s all for now, but I figured that my claims from my last post deserved some actual documentation and discussion.