SpaceX Falcon 1.1 Post Mortem

Ok, I just spent the past two or three hours fighting with a model that was actually working just fine. I feel retarded with a capital R, so I think it’s time for a break.

How many of you have had a chance to read the SpaceX post mortem on their second Falcon 1 launch attempt? It’s rather interesting reading.

The sad thing is that it looks like the chain of events leading up to the launch vehicle failure may have started with something as simple as improper configuration control on the code for the Merlin engine computer.

Here’s the basic sequence of failures as I can piece together from the post mortem:

1. Someone uploads some code to the Merlin 1 with some outdated tables for propellant mixture ratio control.

2. Merlin 1 runs leaner than planned at the start and richer than planned at the end, resulting in staging occuring at a much lower altitude than typically happens.

3. Staging occurs in a regime with much higher aerodynamic forces then planned for, which when combined with an extra torque caused by the Merlin 1 shutdown causes the Kestrel engine to impact with the first stage during separation.

4. The perturbation to the tank caused by the impact is larger than any of the perturbations modeled by SpaceX during upper stage tank design, and since the design doesn’t have baffles, and the control program isn’t designed to handle slosh on that scale, the slosh grows.

5. Slosh problems grow, eventually causing a roll to start.

6. The roll overcomes the roll-control thrusters.

7. Upper stage starts corkscrewing.

8. At some point the LOX inlet is oncover, causing Kestrel to shut down.

9. Demosat2 placed in fishing orbit.

It’s a really sobering lesson, particularly for rockets with any sort of computer control. Configuration and revision controls are critical for stuff like this. We’ve learned some of those lessons the not-quite-as-hard-but-harder-then-we-would-like way ourselves. Just kind of amazing that a $6M rocket didn’t make orbit, possibly because a few lines of code.

The following two tabs change content below.
Jonathan Goff

Jonathan Goff

President/CEO at Altius Space Machines
Jonathan Goff is a space technologist, inventor, and serial space entrepreneur who created the Selenian Boondocks blog. Jon was a co-founder of Masten Space Systems, and is the founder and CEO of Altius Space Machines, a space robotics startup in Broomfield, CO. His family includes his wife, Tiffany, and five boys: Jarom (deceased), Jonathan, James, Peter, and Andrew. Jon has a BS in Manufacturing Engineering (1999) and an MS in Mechanical Engineering (2007) from Brigham Young University, and served an LDS proselytizing mission in Olongapo, Philippines from 2000-2002.
This entry was posted in Uncategorized. Bookmark the permalink.

22 Responses to SpaceX Falcon 1.1 Post Mortem

  1. Mike Puckett says:

    Jon, OT but though this might interest you:

    http://spaceflightnow.com/atlas/av009/status.html

    Looks like a possible Centaur underperformance.

  2. Jon Goff says:

    Mike,
    Yeah, I saw that link earlier. It’s a pity. LM had a pretty solid streak going for a while there. Makes you wish we had a more robust space transportation infrastructure, where underperformance like this could be made up for by stuff like orbital refueling or something…

    …anyhow, back to that thesis.

    blech.

    ~Jon

  3. Mike Puckett says:

    “where underperformance like this could be made up for by stuff like orbital refueling or something…

    Like one of Mr. Wingo’s tugs?

  4. Daniel Rosły says:

    I think SpaceX definitely needs something invented some years ago: “Configuration Management”… as a professional in this area, I offer them my service 🙂

  5. Derwik R. Bains says:

    Given that the slosh only followed from a long sequence of other anomalies, which will now be dealt with, why is Elon adding baffles to the 2nd stage? Their analyses were perfectly correct, but assumed that staging would occur at the proper altitude and various other things Jon mentions. As the other fixes will guarantee proper staging, etc., why add baffles? Personally, I find it baffling.

  6. Karl says:

    Given that the slosh only followed from a long sequence of other anomalies, which will now be dealt with, why is Elon adding baffles to the 2nd stage? Their analyses were perfectly correct, but assumed that staging would occur at the proper altitude and various other things Jon mentions. As the other fixes will guarantee proper staging, etc., why add baffles? Personally, I find it baffling.

    Why assume there won’t be anomalies? There’s a good chance something will throw things off. And baffles might have saved that mission despite the chain of things that went wrong.

  7. Habitat Hermit says:

    Also they can easily remove the second stage slosh baffles later on when the issues have been proven solved. Better safe than sorry etc.

  8. Mike Puckett says:

    “Derwik R. Bains said…
    Given that the slosh only followed from a long sequence of other anomalies, which will now be dealt with, why is Elon adding baffles to the 2nd stage? Their analyses were perfectly correct, but assumed that staging would occur at the proper altitude and various other things Jon mentions. As the other fixes will guarantee proper staging, etc., why add baffles? Personally, I find it baffling.

    3:03 AM

    Belt, Suspenders, Duct Tape……

  9. Monte says:

    Once again, reality bites the airy belief that the high costs of “mainstream” space are all (or mostly) unnecessary paperwork and review meetings and “overhead” such as rigorous configuration management… and therefore that lean, mean alt.space teams can pare all that away and save big bucks.

    Once again, I’ll recommend Stephen B. Johnson’s The Secret of Apollo: Systems Management in American and European Space Programs. That and his earlier, overlapping USAF and the Culture of Innovation (on 1950s ICBM development) are the best short accounts I know of how much of that “overhead” was forced on rocketeers by the combination of complexity (lots of fault trees), tight integration, and a very unforgiving operating regime. I knew some of those people, and believe me, they loved Skunk Works simplicity as much as anyone today; they just found that it let them down too often.

    Or for the short version, just take seriously what Musk himself said after the first Falcon 1 launch:

    “Nobody said rockets would be easy, but it’s harder than that.”

  10. Rand Simberg says:

    It should be noted that configuration management isn’t just important for software–it’s just as important for drawings and the hardware associated with them. If you don’t know what you flew, it makes it hard to troubleshoot problems when it makes a smoking hole in the desert.

  11. Anonymous says:

    It also appears that the FC was inadequate WRT inertial cross-coupling.

  12. Matt says:

    I’m surprised that their solution to the control problem on the second stage was not better control software instead of baffles. From what I know of control theory, eliminating oscillations is difficult but not impossible. Is there something about fuel sloshing that cannot be modeled well enough to test control software on?

  13. Jon Goff says:

    Matt,
    Yeah, I do believe it is possible to handle it entirely in code. Centaur for instance has no slosh baffles and it does fine. But for SpaceX it’s probably better to do as the others have said, and go belt-suspenders-duct tape on the problem. Do the software and the hardware fix. Then as you get more real data, at some point you can make the decision to remove the hardware fix if it appears to be no longer necessary.

    ~Jon

  14. Jon Goff says:

    Mike,
    “Like one of Mr. Wingo’s tugs?”

    It probably depends a lot on how underperforming the launch was. If it’s something like this where they were just trying to go into a high LEO, doing the burn with the satellites, then retopping off the satellite using an Orbital Express like platform would likely make sense. If the satellite was bound for GEO though, Mr Wingo’s tug might make more sense, if it got stranded in LEO. If it got stranded in a messed up GTO, it might be trickier, but there probably are solutions. So long as there’s a way to retank satellites, or to attach a ion-tug propulsion package, you can probably eliminate at least a decent fraction of the current “partial failures”…

    …anyhow, back to that thesis…

    ~Jon

  15. gliderguider says:

    The initial slosh perturbation wasn’t caused by the minor engine bell impact. It was caused a bit later on, after engine start, when the computer gimballed the engine hard over to get the rocket pointed back the right way.

    Note that the engine gimballed *towards* the same side that got the impact, thus yawing the second stage further in the same direction as the impact had. That’s because both were caused by the pre-existing rotation.

    Also, for those who haven’t read the report, the first stage underperformed because of the mixture ratio problem, but they still would have made the correct orbit if they’d been able to keep propellent going into the pump intake.

  16. Anonymous says:

    >Also, for those who haven’t read the report, the first >stage underperformed because of the mixture ratio >problem, but they still would have made the correct >orbit if they’d been able to keep propellent going >into the pump intake.

    For those who watched the extra video that was on Youtube you will see that the nozzle went hard over to its stop and the stage spun around 180 degress while it was still firing.

  17. Jesse says:

    >For those who watched
    >the extra video that was
    >on Youtube you will see
    >that the nozzle went hard
    >over to its stop and the
    >stage spun around 180
    >degress while it was
    >still firing.

    actually, a hundred seconds later, that spin looked like a dryer cycle, according to Wired Magazine’s article. See http://www.spacex.com/media.php?page=45

  18. Anonymous says:

    >actually, a hundred seconds later, that spin looked like a dryer cycle, according to Wired Magazine’s article. See http://www.spacex.com/media.php?page=45

    So why the bs statement that if the fuel had not sloshed away from the inlet that it would have made orbit? The thing was in a spin and had no chance of recovery as the nozzle went hard over to its stop. You can see the increasing travel of the nozzle as it attempts to compensate for the pitch/yaw motion. It did not couple to a roll until very late in the mission. The motion does not even look like a slosh mode, rather a vortice mode as the fuel/oxidizer is drawn down in the tank

    If they don’t understand what was happening, how are they going to fix it?

  19. mz says:

    STS ET had initially lots of slosh baffles and they’ve removed em as they’ve gained more experience.

    Prudent that way.

    (It was in the STS program lessons learned presentation)

    They kept improving baffles in the Apollo LEMs too (had too little at first), although of course it’s a very different environment.

  20. reader says:

    again. Is nobody running a few simple lowpass filters on their servo outputs, just to at least detect oscillations ?

    i have worked with robotics extensively enough and done lots of different control loops, only the simplest control algorithms look at only a few current data samples.
    normally you record both your input and output samples, and run couple of obvious analysis filter over them, not every loop but often enough to detect anomalys.
    oscillations like these are easy enough to detect this way and at least flag an alarm, if not take corrective action.

    sort of a watchdog over a primary control loop.

  21. reader says:

    just to clarify : the idea is to monitor your primary control algorithm performance.
    simulate your nominal mission, and run filters on the whole path and all significant control algorithm outputs, also significant primary sensor inputs like IMU. record filter outputs in different stages of flight.
    now when given filter output in actual flight, at given stage, significantly differs from what was simulated, you flag an alarm and possibly take corrective action.

    for oscillations, run lowpass filters, for vibrations, do highpass. there are some others that are useful.

    in the SpaceX test flight, the stage behaved like it didnt know it was messing up, while it was immediately obvious for human eye.

  22. Anonymous says:

    Dear Fellow Scientist,

    NASA’s rocket technology not for real space exploration but here is one.

    Sir, don’t be dismayed to see how little information there is on the internet. Despite that, I hope you totally understand my need for anonymity. Assuming that the technology is as effective as I say it is, releasing it to the public in all its splendor could make the world think that a) I am off my rocker, b) that I’m completely wrong or c) just some sci-fi aficionado who’s gone a bit too far.

    Sad state of affairs, but hey, that’s the price of true innovation right?

    http://nlspropulsion.net

    Regards,

    The Inventor

Leave a Reply

Your email address will not be published. Required fields are marked *