Before I get into pointing out the flaws I saw on Mark Whittington’s blog the other day, I need to lay a little groundwork. A few weeks ago, I got an email from one of the coauthors of an interesting new book, The Rocket Company. Some of the chapters from the book were initially released on Hobbyspace.com about a year ago as a serial. I read some of them at the time, and it definitely had some interesting ideas. Anyhow, one of the coauthors wrote me asking if I’d like to review a copy of the finished book. Twisting my own arm, I agreed, and have been reading it with Tiffany ever since (whenever we have time we’ll sneak a chapter in). Tiffany is taking it like a trooper (her nerdiness only goes so far), but I’ve been rather enjoying it. I’ll do a further review once we actually get done.
Anyhow, one of the profound pieces in the book had to do with the title of this post. Basically, as one of the characters was explaining about the history of aerospace projects, one of the biggest hurdles most engineering companies (and government agencies) faced was figuring out how to keep Congress or NASA or other government agencies from constantly second guessing them in all their decisions. In order to deal with this challenge a set of psuedoscientific tools was crafted to give the veneer of absolute certainty to what is in reality a rather uncertain field–product design. One of those tools was the trade study (emphasis and spelling errors mine):
It worked like this: the systems engineers, working with the design engineers, came up with an approach that, based on their experience and investigations, they thought was most likely to work. They then engaged in an imaginative game of listing every possible design approach they could think of, no matter how far-fetched. The preferred approach and all of the alternatives were then analyzed according to a complicated equation involving many parametric constants and weighting factors for the requirements. This analysis generated so-called figures of merit for each option. The constants and the equation were then tinkered with until the chosen design approach came out as clearly superior.
When the design concept was presented to the top brass or congressional committees, it was presented as the best solution out of all possible solutions, arrived at by a purely scientific process. This made it essentially impossible for the design decision to be questioned. If there was any questioning of the design, the program managers would launch into interminable explanations and arguments concerning all of the many different design concepts, the figures of merit, and the trade table, until the questioner was exhasuted. Thus any attempt to interfere with the decision that the engineers had already made could be blocked.
Having worked in engineering, I can vouch for the fact that at some level this is pretty much true for all trade studies. Even when you haven’t completely made up your mind in advance, when your favorite approach(es) don’t do as well, it must be because you didn’t assign the right weighting factors, or didn’t take the right factors into consideration. After all, you’re a really smart person, and your initial hunch has got to be right for some reason!
As an example, about two-and-a-half years ago or so, I went to watch a static test firing of a 2000lbf Nitrous/Rubber hybrid that BYU and USU had been working on. Due to igniter problems, they had several misfires, and it ended up taking all day to get two or three good short firings (which were tres cool when they actually did go however). Having just read about some of John Carmack’s work with catalysts, and how flowing hydrogen over a catalyst in the presence of air caused catalytic ignition, I got the idea that maybe a catalyst igniter would solve all their problems. I started a project with the BYU Space Development Club trying to investigate that (finding along the way that there had actually been significant previous work by other smart people like myself–geniuses, geniuses I tell you!). When I started doing some work with MSS, I was still pushing this idea. It seemed so simple. Just open GOX, open GH2, and hot flamey stuff must surely come out the thruster thingo!
After getting more involved with the project, and finding some cleverly hidden challenges of this sort of gas-gas igniter, I decided it was time to document my work, so the others could all be on the same page as I was. As part of the documentation, I decided to put in a little trade study. You know, to show why this was such a good idea that we should keep throwing money at the problem. After all, we were getting pretty close. So, I thought up a couple different styles of igniters: GOX/IPA spark, GOX/IPA resonance, TEA/TAB, a few others far-out ones to make my idea look better, and I took all those and crammed them into a table. I then tried to figure out what the key figures of merit should be (particularly making sure I chose ones that catalyst ignition did well, as well as a few that it didn’t do quite as well in just to be honest), and assigned what I figured were reasonable scores for each design in each category. I then figured out what reasonable weighting factors would be and…..stupid trade table! You’re not supposed to say that resonance and spark ignition are better than catalyst work! I’m smart. I know what’s the best way to do things! I’ll show you! So, I start tweaking with the numbers. “You know, resonance ignition is probably really, really tough to develop, so I’ll drop its ‘development cost score a bit'”….”Hrm…the simplicity of not having a spark isn’t enough to counteract the fact that spark igniters are proven….better add an ‘low EMI category'”….
Anyhow, after several minutes of rationalization, I finally realized I was being a schmuck. The ammount of torture I was having to put the data to (because after all, while data never lies, if you torture it, it will confess) in order to justify my idea was just absurd. That evening I suggested to MSS that we shelve the catalyst igniter project, and ramp up the spark igniter project (which Pierce had fortunately been stubborn enough to insist on starting in parallel). The result is that we now have a pretty darn reliable spark igniter. The resonance igniter came out a little ahead in most categories, so maybe I’ll try dabbling with that in the future, but I learned a valuable lesson. Trade studies can be dangerous tools of rationalization and self-justification if you aren’t careful.
[Warning, this post is close captioned for the snark impaired. Excessive sarcasm and lampoonage coming up. Stop reading now if you’re drinking soda through a straw–that could be rather messy!]
And that’s one of the big issues I see with the whole ESAS report. Here we get blinded by the science. The fact that this “scientific” trade study was carried out by tons of really smart guys at NASA over the course of months and months, that painstakingly analyzed all the possible booster combinations in every way possible means that what they came up with must be the absolutely 100% super best uber-Rocket design that is physically possible! Yay! Ve can now shut off our brains and vatch as our nation builds very very big rockets und our brave astronauts return to the moon! After all, if these NASA super geniuses did a trade study (pay no attention to the fact that almost none of those engineers have actually designed anything that’s been built and succesfully flown to space in oh, say 30 years, unlike the engineers at Boeing and Lockmart and several of the other contractors who’ve actually at least designed, built, and flown stuff within the past 10)…..(oh, and also pay no attention to the fact that in spite of being the obviously best possible approach that is humanly possible for thus sayeth the trade study, Amen! The trade studies done by smart people at Boeing, Lockheed, t/Space, and other groups came up with substantially different approaches like–*gasp* dry launch, reusable lunar landers, etc. Those engineers must not be as wise as them NASA engineers. After all, what have they done over the past several years other than design, build, and fly space hardware! Heck, they haven’t even published any major acedemic papers on hypersonic scramjets or SSTO VTHL RLVs! The noive, the noive of those people)
[We now return you to your regularly scheduled, desnarkified blogging. In all seriousness.]
Seriously though, before anyone accuses me of conspiracy theories (Mark?), I’m just pointing out a real phenomenon common in engineering design projects. I’ve seen it even when the engineers had no malicious intent whatsoever, no larceny, no conspiracy. Just ego. And come on, if you don’t realize that engineers are some of the most egotistical people on the planet, you must not know too many.
Even more seriously though, it is also important to realize that even if your engineers are perfectly self-honest, perfectly unbiased, and perfectly humble, trade studies are still a Garbage-in-garbage-out process. If you start with the wrong requirements, it will lead you to the wrong approaches. If one of your key goals is how best to “keep the team together”, and how best to “use current capabilities”, you will end up picking stuff that looks pretty darned similar to what you’ve been doing already. James Womack in his excellent intro to Lean Manufacturing called “Lean Thinking” made a very important point. Value lies in meeting people’s needs at a price they want and where and when they want it. Value doesn’t lie in maximizing the utilization of existing resources. A customer doesn’t give a darn about the fact that you built your new product keeping most of your old team together and facilities used if that product doesn’t meet their needs, costs too much, or is delivered too-late. Getting the maximum utility out of previously sunk costs is not the way to deliver real value.
Anyhow, that’s enough of a rant for this morning. I think I’ve said enough to lay the groundwork for the points I want to make about Mark’s posts tomorrow.