Lies, Darn Lies, and Trade Studies

Before I get into pointing out the flaws I saw on Mark Whittington’s blog the other day, I need to lay a little groundwork. A few weeks ago, I got an email from one of the coauthors of an interesting new book, The Rocket Company. Some of the chapters from the book were initially released on about a year ago as a serial. I read some of them at the time, and it definitely had some interesting ideas. Anyhow, one of the coauthors wrote me asking if I’d like to review a copy of the finished book. Twisting my own arm, I agreed, and have been reading it with Tiffany ever since (whenever we have time we’ll sneak a chapter in). Tiffany is taking it like a trooper (her nerdiness only goes so far), but I’ve been rather enjoying it. I’ll do a further review once we actually get done.

Anyhow, one of the profound pieces in the book had to do with the title of this post. Basically, as one of the characters was explaining about the history of aerospace projects, one of the biggest hurdles most engineering companies (and government agencies) faced was figuring out how to keep Congress or NASA or other government agencies from constantly second guessing them in all their decisions. In order to deal with this challenge a set of psuedoscientific tools was crafted to give the veneer of absolute certainty to what is in reality a rather uncertain field–product design. One of those tools was the trade study (emphasis and spelling errors mine):

It worked like this: the systems engineers, working with the design engineers, came up with an approach that, based on their experience and investigations, they thought was most likely to work. They then engaged in an imaginative game of listing every possible design approach they could think of, no matter how far-fetched. The preferred approach and all of the alternatives were then analyzed according to a complicated equation involving many parametric constants and weighting factors for the requirements. This analysis generated so-called figures of merit for each option. The constants and the equation were then tinkered with until the chosen design approach came out as clearly superior.

When the design concept was presented to the top brass or congressional committees, it was presented as the best solution out of all possible solutions, arrived at by a purely scientific process. This made it essentially impossible for the design decision to be questioned. If there was any questioning of the design, the program managers would launch into interminable explanations and arguments concerning all of the many different design concepts, the figures of merit, and the trade table, until the questioner was exhasuted. Thus any attempt to interfere with the decision that the engineers had already made could be blocked.

Having worked in engineering, I can vouch for the fact that at some level this is pretty much true for all trade studies. Even when you haven’t completely made up your mind in advance, when your favorite approach(es) don’t do as well, it must be because you didn’t assign the right weighting factors, or didn’t take the right factors into consideration. After all, you’re a really smart person, and your initial hunch has got to be right for some reason!

As an example, about two-and-a-half years ago or so, I went to watch a static test firing of a 2000lbf Nitrous/Rubber hybrid that BYU and USU had been working on. Due to igniter problems, they had several misfires, and it ended up taking all day to get two or three good short firings (which were tres cool when they actually did go however). Having just read about some of John Carmack’s work with catalysts, and how flowing hydrogen over a catalyst in the presence of air caused catalytic ignition, I got the idea that maybe a catalyst igniter would solve all their problems. I started a project with the BYU Space Development Club trying to investigate that (finding along the way that there had actually been significant previous work by other smart people like myself–geniuses, geniuses I tell you!). When I started doing some work with MSS, I was still pushing this idea. It seemed so simple. Just open GOX, open GH2, and hot flamey stuff must surely come out the thruster thingo!

After getting more involved with the project, and finding some cleverly hidden challenges of this sort of gas-gas igniter, I decided it was time to document my work, so the others could all be on the same page as I was. As part of the documentation, I decided to put in a little trade study. You know, to show why this was such a good idea that we should keep throwing money at the problem. After all, we were getting pretty close. So, I thought up a couple different styles of igniters: GOX/IPA spark, GOX/IPA resonance, TEA/TAB, a few others far-out ones to make my idea look better, and I took all those and crammed them into a table. I then tried to figure out what the key figures of merit should be (particularly making sure I chose ones that catalyst ignition did well, as well as a few that it didn’t do quite as well in just to be honest), and assigned what I figured were reasonable scores for each design in each category. I then figured out what reasonable weighting factors would be and…..stupid trade table! You’re not supposed to say that resonance and spark ignition are better than catalyst work! I’m smart. I know what’s the best way to do things! I’ll show you! So, I start tweaking with the numbers. “You know, resonance ignition is probably really, really tough to develop, so I’ll drop its ‘development cost score a bit'”….”Hrm…the simplicity of not having a spark isn’t enough to counteract the fact that spark igniters are proven….better add an ‘low EMI category'”….

Anyhow, after several minutes of rationalization, I finally realized I was being a schmuck. The ammount of torture I was having to put the data to (because after all, while data never lies, if you torture it, it will confess) in order to justify my idea was just absurd. That evening I suggested to MSS that we shelve the catalyst igniter project, and ramp up the spark igniter project (which Pierce had fortunately been stubborn enough to insist on starting in parallel). The result is that we now have a pretty darn reliable spark igniter. The resonance igniter came out a little ahead in most categories, so maybe I’ll try dabbling with that in the future, but I learned a valuable lesson. Trade studies can be dangerous tools of rationalization and self-justification if you aren’t careful.

[Warning, this post is close captioned for the snark impaired. Excessive sarcasm and lampoonage coming up. Stop reading now if you’re drinking soda through a straw–that could be rather messy!]

And that’s one of the big issues I see with the whole ESAS report. Here we get blinded by the science. The fact that this “scientific” trade study was carried out by tons of really smart guys at NASA over the course of months and months, that painstakingly analyzed all the possible booster combinations in every way possible means that what they came up with must be the absolutely 100% super best uber-Rocket design that is physically possible! Yay! Ve can now shut off our brains and vatch as our nation builds very very big rockets und our brave astronauts return to the moon! After all, if these NASA super geniuses did a trade study (pay no attention to the fact that almost none of those engineers have actually designed anything that’s been built and succesfully flown to space in oh, say 30 years, unlike the engineers at Boeing and Lockmart and several of the other contractors who’ve actually at least designed, built, and flown stuff within the past 10)…..(oh, and also pay no attention to the fact that in spite of being the obviously best possible approach that is humanly possible for thus sayeth the trade study, Amen! The trade studies done by smart people at Boeing, Lockheed, t/Space, and other groups came up with substantially different approaches like–*gasp* dry launch, reusable lunar landers, etc. Those engineers must not be as wise as them NASA engineers. After all, what have they done over the past several years other than design, build, and fly space hardware! Heck, they haven’t even published any major acedemic papers on hypersonic scramjets or SSTO VTHL RLVs! The noive, the noive of those people)

[We now return you to your regularly scheduled, desnarkified blogging. In all seriousness.]

Seriously though, before anyone accuses me of conspiracy theories (Mark?), I’m just pointing out a real phenomenon common in engineering design projects. I’ve seen it even when the engineers had no malicious intent whatsoever, no larceny, no conspiracy. Just ego. And come on, if you don’t realize that engineers are some of the most egotistical people on the planet, you must not know too many.

Even more seriously though, it is also important to realize that even if your engineers are perfectly self-honest, perfectly unbiased, and perfectly humble, trade studies are still a Garbage-in-garbage-out process. If you start with the wrong requirements, it will lead you to the wrong approaches. If one of your key goals is how best to “keep the team together”, and how best to “use current capabilities”, you will end up picking stuff that looks pretty darned similar to what you’ve been doing already. James Womack in his excellent intro to Lean Manufacturing called “Lean Thinking” made a very important point. Value lies in meeting people’s needs at a price they want and where and when they want it. Value doesn’t lie in maximizing the utilization of existing resources. A customer doesn’t give a darn about the fact that you built your new product keeping most of your old team together and facilities used if that product doesn’t meet their needs, costs too much, or is delivered too-late. Getting the maximum utility out of previously sunk costs is not the way to deliver real value.

Anyhow, that’s enough of a rant for this morning. I think I’ve said enough to lay the groundwork for the points I want to make about Mark’s posts tomorrow.

The following two tabs change content below.
Jonathan Goff

Jonathan Goff

President/CEO at Altius Space Machines
Jonathan Goff is a space technologist, inventor, and serial space entrepreneur who created the Selenian Boondocks blog. Jon was a co-founder of Masten Space Systems, and the founder and CEO of Altius Space Machines, a space robotics startup that he sold to Voyager Space in 2019. Jonathan is currently the Product Strategy Lead for the space station startup Gravitics. His family includes his wife, Tiffany, and five boys: Jarom (deceased), Jonathan, James, Peter, and Andrew. Jon has a BS in Manufacturing Engineering (1999) and an MS in Mechanical Engineering (2007) from Brigham Young University, and served an LDS proselytizing mission in Olongapo, Philippines from 2000-2002.
Jonathan Goff

About Jonathan Goff

Jonathan Goff is a space technologist, inventor, and serial space entrepreneur who created the Selenian Boondocks blog. Jon was a co-founder of Masten Space Systems, and the founder and CEO of Altius Space Machines, a space robotics startup that he sold to Voyager Space in 2019. Jonathan is currently the Product Strategy Lead for the space station startup Gravitics. His family includes his wife, Tiffany, and five boys: Jarom (deceased), Jonathan, James, Peter, and Andrew. Jon has a BS in Manufacturing Engineering (1999) and an MS in Mechanical Engineering (2007) from Brigham Young University, and served an LDS proselytizing mission in Olongapo, Philippines from 2000-2002.
This entry was posted in Uncategorized. Bookmark the permalink.

11 Responses to Lies, Darn Lies, and Trade Studies

  1. Mark says:

    I await your onslaught with keen anticipation. Mind, because of Rita, it may be a few days before I can read it, not to mention respond. But I suspect I will have to.

  2. Iain McClatchie says:

    My initial thought was that the ATK camp were just trying to save their own jobs. I looked around for costs, and found an estimate that each Shuttle SRB costs $40M per launch.

    That’s $12.12/pound thrust, which is quite a bit less than the $26.47/pound thrust of the Merlins. I was pretty surprised by that result. SpaceX is supposed to be cheap!

    Maybe that $40M number is wrong. If it’s not, that suggests that the ATK folks have a pretty good first stage right now. Nevermind that solids are dangerous and not sexy and not really reusable. It would appear that the primary problem with the Stick is the cryogenic upper stage.

    So… ATK might do better to gang up with a Kero/LOX upper stage for Stick and Magnum alternatives. A good design might beat SpaceX pricing!

  3. Jon Goff says:

    I think you’re vastly overestimating the actual per unit cost of the Falcons and Merlins. I’ve been meaning to write that up too, but I have a backlog. Hint: you’re assuming a lot about the difference between price and cost for SpaceX

  4. Iain McClatchie says:

    What matters is the price to NASA. My comparison is bogus because that $40M number is detritus I found on the web somewhere. Also, the price from SpaceX is for a complete flight, and the $40M price is for a booster, no ops, no range fees, no insurance, some assembly required.

    I’ll stand behind my claim that the primary problem with the Stick is the upper stage, though. Mostly, the choice of LH2.

    Is there a web page with an analysis of all the designs they considered but dismissed? Surely they considered an air-started RD-180 for the second stage of the Magnum?

  5. murphydyne says:

    Hey Jon, I don’t mind taking a few lobs. I know Mark’s a pretty stout guy from over at the old RttM Board.

    Mark Whittington notes over in his Curmudgeon’s Corner –

    “But I’m not certain that it’s the government’s place to build a commercial space transportation infrastructure.”

    Thank goodness we don’t think the same way with regards to commercial ground transportation infrastructure. We’d be a much poorer people without the extensive highway system we’ve developed in this country.

    It wasn’t all put in place at once; it was developed, growing organically into the massive ribbons of concrete that carry goods and persons around the nation today.

    I’m not saying that NASA should build a ribbon of concrete to the Moon, but I do wish that they would take a moment and think about how we can build an infrastructure that EVERYONE uses. Spending $100Bn to build a transport system that ONLY NASA can use does not strike me as a particularly appropriate disbursement of our nation’s capital.

    What if the Eisenhower interstate highway system had been restricted to solely military use? It certainly conjures in my mind a much different system and outcome than we have now.

    That’s the beauty of launching stuff 20 mt at a time. You can adapt quickly to changes. The cargoes you’re launching five years from now might not be the ones you expected to launch five years back. Trying to lock down a 300 mt (presumably our rocket scientists are smart enough to use metric) in LEO going to the Moon payload thirteen years out is kind of silly. Besides, if they want 300 mt in LEO you can get that with 5 EELVs/yr in 3 years. (let’s see, assuming $500Mn/launch x 15 launches = $7.5Bn). Well now that doesn’t seem so bad.

    I’m not against space elevators, and do think that they are THE permanent solution to space access (besides, Dr. Edwards’ company is here in Dallas), but right here, right now I think we need to go with the flexibility of NASA sharing the same basic launch infrastructure as everyone else.

    We can pioneer how to live and work in space, and make that knowledge available to the world. How to put bigger things together, inspect our assets post-launch, stay for long periods outside the Van Allen belts, master the gravity curves, and so on.

    Which also brings me to one of my curmudgeonly topics, the political shenanigans many years ago that killed a TGV-type rail infrastructure in Texas, linking the triangle of Dallas-Austin-Houston. If we had that kind of passenger rail infrastructure we could have carried huge numbers of people out of Houston instead of having the parking lot that is I-45. Personally, I’d use it to go down to Austin on the Friday night party train (flashbacks to the Brussels-Amsterdam party train rides) and drag my sorry butt back to Dallas at some point during the weekend.

    Instead, I have to brave IH-35 and the gazillion 18-wheelers on it. Oh that we had a TGV in Texas. In 20/20 hindsight, had one been built it probably would have been extended from Houston to Nawlins, though I’ll bet the last part would have been tricky. Still, that would have been a nice asset to have right about now.

    With regards to ambivalent’s question about discards in the launcher trades, I kind of doubt it because a lot of the private sector data is going to be classified as commercially sensitve. (One see’s that quite a bit in 10-Ks over on Edgar).

    Besides, I had the misfortune of spending my birthday vacation in Houston at the Moon Exploration Architectures Study group or something like that, and the groundwork for what we’re seeing now was being laid even back then. I came away with an ill and sinking feeling in my stomach that the optimizations in the way the NASA guys were looking at the problem in their trades was going to lead them right back to a disposable Son of Saturn.

    I didn’t realize that it would take the form of shuttle-derived Frankenstein’s monsters. I should have been clued in from the initial CEV submissions, as on the picture page of getting to orbit included a Shaft CEV as one of the possibilities. I knew the jig was up when the earliest parts of it were leaked, and then I go to the Paris Air Show, find a poster of the Shaft CEV, as well as an ATK CD-ROM laying out the architecture dated August 2004.

    And since I know you’re going to ask, no it has no really usefully information on it, just a 10 min. pretty picture movie.

  6. Iain McClatchie says:


    discards in the launcher trades

    What’s that? Do you mean, finding out what ATK charges NASA for each SRB?

  7. Mark says:

    Murphydyne I think has the wrong analogy, though one can also argue that government involvement in highway building has led to hundreds of billions spent on pork barrel projects over the years.

    The analogy that fits better is air travel. The government did not try to build a national air line nor did it try to build a national air liner. It did put into place policies that fostered thr growth of commercial air lines, however.

  8. gbaikie says:

    “Murphydyne I think has the wrong analogy, though one can also argue that government involvement in highway building has led to hundreds of billions spent on pork barrel projects over the years.”

    You could also argue that roads don’t actually go anywhere. Roads are similar to launch pads. And govt didn’t built many roads that only govt employees used. And govt roads were intended to be used by the public. Govt didn’t build cars or the vehicles that actually travelled down the road. So if govt want to build launch pads and allow public to use these launch pad for free or low cost, then you have something similar.

  9. Kelly Starks says:

    As to the book, I strongly disagreed with their chapter on systems engineering and trade studies. Oh they CAN be used to just justify what you liked in the first place, but any form of analysis can be used to do that. Requirements written to suit the design you guessed at and got fond of, Design based on your favorite concept – not on what works best, fuel trade offs based on everyone knowing LH/non-SSTO’s/high mass fraction/whatever designs are best etc. But I’ve done a lot of trade studies on several big programs, and they usually are a real listing of the pluses and minuses of various options. Often the assumptions your team came with dominate the analysis (which is why trade studies should not be done with folks already vested in a idea), but this blanket assumption that that’s all they are is ridiculous. Very certainly programs that made no attempt to do them, or systems engineering, were disasters!! (I’ve literally seen execs sys systems engineering was just a paperwork project to keep the gov happy, and “we weren’t to get in the way of real engineers, doing real work.” They had finished the designs, before getting a requirements list.)

    If your going to reject analysis because it can be misused, what do you do – engineer by off the cuff guess? I’ve seen places do it it isn’t pretty.

  10. Jon Goff says:

    The book didn’t totally reject the utility of trade studies, just said that they weren’t some magic way to make infallibly perfect decisions. In the book, they actually mention doing trade studies on various TPS systems.

    It’s sometimes helpful to do them, but it’s important to always remember that they are a summation of tons of subjective opinions, and not some hard scientific fact or gospel truth.

    If used wisely they can be useful, but all too often it gets abused.


  11. Kelly Starks says:

    They did list using trade studies – though systems engineering didn’t get even that. Both are pretty critical — but yes also can be abused badly.

    As a contractor I’ve seen many creative ways folks trashed their programs with these. 😉

Leave a Reply

Your email address will not be published. Required fields are marked *