--- /dev/null
+[[!tag tags/rockets]]
+Keith and I have discovered a change in the behavior of the protection
+circuits integrated in the LiPo batteries we sell for use with
+[Altus Metrum](http://altusmetrum.org) products that poses a risk for
+our customers. This post is meant to document what we now know, communicate
+changes we're planning as a result, and explain what we think flyers of our
+existing electronics and batteries may want to do to maximize their chances
+of successful flights.
+
+Background
+==========
+
+Choosing batteries and designing pyro circuits for high power rocketry avionics
+involves a variety of trade-offs. Reliability is the highest concern, both
+because nobody wants to lose an airframe due to a failed pyro event, and
+also because airframes recovering anomalously have safety implications. But
+we also care about other factors including cost and weight, and usually
+want to minimize the complexity of both the electronics and the overall
+installation.
+
+The objective of a pyro circuit is to dump enough energy rapidly into an
+electric match to cause it to catch fire. We need batteries both to power
+the electronics that decide when to fire the charge, and to provide the
+energy that actually ignites the match.
+
+The two most common battery types seen in the rocketry hobby are alkaline
+cells, often the nominal 9V rectangular variety, and rechargeable batteries
+based on Lithium Polymer (LiPo) chemistry. LiPo cells are 3.7 volts per
+cell nominal voltage, are very light, and have a high energy density.
+
+LiPo capacity is measured in units of current times time, so an 850mAh
+cell should be able to deliver 850 milliamps for an hour. The battery
+industry also uses something called a "C rate" to describe how fast the
+battery can be usefully discharged, wich is a multiplier relative to the
+battery capacity. So a battery with 850mAh capacity and a "2C" rating can
+deliver current at a sustained rate of 1700mA until discharged, while the
+same capacity at a "5C" rating can deliver 4250mA.
+
+By comparison, most 9V alkaline batteries are actually composed of 6
+individual 1.5V cells enclosed in a wrapper. It's hard to get hard numbers
+for capacity and discharge rate, since in an alkaline cell the two are not
+independent, and the discharge rate is related to the volume of each
+individual cell. The
+[data sheet for an Energizer 522](http://data.energizer.com/PDFs/522.pdf)
+shows just over 600mAh at a 25mA discharge rate, dropping to about 300mAh
+at a 500mA discharge rate.
+
+Importantly for use in pyro circuits, LiPo cells have a *very* low source
+impedance, which means they can source immense amounts of current. It's not
+unusual for cells in the 1000mAh range to have ratings in excess of 30C!
+Because this rapid discharge ability can pose a risk of fire, it's common
+for LiPo cells to come with a "protection board" integrated into the battery
+assembly that is designed to limit the current to some rate such as 2C
+continuous duty.
+
+In large airframes, or projects that involve staging, air-starts, or other
+complex pyro event sequences, the most reliable approach will always be
+to use separate batteries for the control electronics and the source of
+pyro firing power. In the limit, having separate pyro batteries for each
+pyro charge with the control electronics only providing the switching to
+connect the batteries to the charges could even make sense. But for most
+airframes, this is overkill, and the increases in mass, volume, and wiring
+complexity just don't make sense.
+
+The challenge, then, is how to design electronics that will robustly initiate
+pyro events without negatively affecting the rest of the electronics when
+operating from a single shared battery.
+
+Altus Metrum Pyro Circuits
+==========================
+
+The very first prototypes of [TeleMetrum](http://altusmetrum.org/TeleMetrum/)
+were designed to use a single-cell LiPo battery, and had an on-board 100mA
+charging circuit. Because we needed 5 volts to power the accelerometer, we
+had a small switching regulator that up-converted the LiPo voltage, and we
+used some of that regulator's output to charge a 1000uF capacitor. The pyro
+circuit used Fairchild FDN335N N-channel MOSFET switches in a low-side
+switching configuration to dump the energy stored in the capacitor through
+an attached ematch. Those FETs had an on resistance of under a tenth of an
+ohm in our operating conditions. The circuit worked very reliably, but the
+1000uF electrolytic cap was huge and we struggled with the mechanics of such
+a large part hanging off the board...
+
+It turns out that 3.7 volts is way more than enough to fire a typical
+low-current e-match or equivalents like the
+[Quest Q2G2](http://www.questaerospace.com/cgi-bin/commerce.exe?search=action&category=OJ)
+igniters. In fact,
+bench testing with a good digital oscilloscope showed that a typical e-match
+with resistance of 1.3-1.8 ohms will fire in approximately 13 microseconds
+when hit with the nominal 3.7 volts from a LiPo.
+
+So, starting with our v0.2 boards, we switched to using the LiPo battery
+voltage directly to fire the pyro charges, eliminating the clunky electrolytic
+capacitor entirely. We also switched to the Fairchild FDS9926A dual N-channel
+MOSFET whose specs are better in all regards for our application. The on
+resistance is down around 40 milli-ohms in our circuit, such that the current
+ratings are much higher (FET current limits are primarily driven by how much
+heat is generated due to current flowing through the channel's on resistance).
+
+Because using the LiPo voltage directly means we're effectively temporarily
+putting a very low resistance across the battery during the pyro events, the
+input voltage to the voltage regulator gets pulled down. To ensure the
+processor could "ride through" these events, we added a 100uF surface mount
+bulk capacitor on the 3.3 volt regulated voltage rail, which bench testing
+demonstrated was more than sufficient to maintain processor operation through
+pyro events. And that is essentially the same pyro circuit on all the boards,
+both [TeleMetrum](http://altusmetrum.org/TeleMetrum) and
+[TeleMini](http://altusmetrum.org/TeleMini), that we have shipped to date.
+
+What's Changed
+==============
+
+The LiPo batteries we source and sell with our electronics come with a
+protection board and cable terminated in a 2-pin, 2mm "JST" connector. The
+specs on the protection board have always been "2C continuous", but we
+observed the
+ability to source much higher currents for short periods such as the 13
+micro-seconds or so required to fire an e-match. Thus these protection
+circuits seemed just fine .. we could fire e-matches with a burst of current
+but were protected against short circuits in the wiring or our boards by
+the 2C continuous limit.
+
+Unfortunately, the most recent batch of batteries we sourced seem to have a
+much "twitchier" protection circuit. We can draw more than 2C for short
+bursts, but not as much as with prior boards, and not for as long an interval.
+With a 1 ohm power resistor on the pyro terminals of one of our boards, we
+get about 9 milliseconds of power before the protection circuit cuts in and
+shuts the battery down. The power stays down until *all* load is removed,
+which at least means the board is turned off and back on again, and in some
+cases could even mean the battery is unplugged and re-plugged since we draw
+trace current to keep the GPS memory alive even when the power is turned off,
+and at least some of the new batteries see that as enough to keep the power
+turned off after an over-current event.
+
+For many e-matches, this isn't an issue, since 9 milliseconds is *way* longer
+than the 13 microseconds needed to fire the charge. Unfortunately, we've
+discovered that many of the e-matches bought and used in the rocketry hobby
+are actually made for use in the fireworks industry, where it is desireable
+to retain continuity
+after firing so that series connections of e-matches all can fire even if
+some fire faster than others! This means that while the resistance goes
+up some after firing, *sometimes* the drain on the battery remains sufficient
+to cause the protection circuit to kick in even after the pyro charge has
+fired.
+
+What We're Doing About It
+=========================
+
+If we remove the protection circuit from the LiPo (or jumper around it), all
+existing Altus Metrum products will operate successfully with pyro charges
+thave have an effective resistance of as low as 1 ohm. That's lower than
+any e-match or Q2G2 we've ever seen, so in effect what this means is that
+if you have an existing Altus Metrum flight computer, and you remove the LiPo
+protection circuit, you're good to go. This does not really make things any
+more "dangerous", since our battery chargers are all current limited and our
+discharge patterns will never cause heating of the battery. Frankly, in a
+rocket, the most likely way to cause a problem with a LiPo is by smashing or
+puncturing the actual battery during bench work or during a crash... and
+those cause the same problems with or without a protection board present.
+
+In the future, we will ship batteries that have either a much higher C rating
+on the protection circuit, or have no protection circuit at all.
+
+The number of problems reported by actual customers that we think should be
+attributed to LiPo protection circuit boards is *very* low, and we
+suspect most of our customers who are happily flying their boards can continue
+to do so. Ground testing where you fire pyro charges (or at least e-matches)
+using RF to issue the commands (not USB, since the LiPo charger is running
+any time USB is connected!) will confirm whether there's a problem. If the
+board resets (does startup beeps) after a pyro event, or shuts down completely
+(no LED activity), then you have a problem. If the matches light and the
+board keeps running, you're good to go.
+
+However, *any* Altus Metrum customer with LiPo batteries sourced from us or
+our distributors who is worried about this problem (even if you haven't seen
+problems in ground testing or previous flights), and who doesn't want to try
+soldering on the battery circuit board yourself, is welcome to contact us
+about removing the protection circuit for you. We won't charge anything
+other than shipping.
+
+To take advantage of this offer, just send email to
+fixmybattery@altusmetrum.org telling us how many of which capacity batteries
+you have that you'd like updated, and we'll respond with an RMA number and
+shipping details.
+
+ [ make sure Keith is ok with working it this way ]
+
+Going Even Further
+==================
+
+As previously indicated, with the LiPo protection circuits removed, all of
+our current products will work reliably with at least 1 ohm across the pyro
+terminals. That should cover all real-world flying conditions just fine,
+but we're not satisfied yet.
+
+We've designed a new pyro control circuit that transfers the maximum possible
+energy to the load regardless of battery voltage without ever allowing the
+voltage to the processor to droop at all. We're testing this new circuit in
+various prototypes now, and if it pans out it will probably show up first in
+[MegaMetrum](http://altusmetrum.org/MegaMetrum) and then trickle down to
+[TeleMetrum](http://altusmetrum.org/TeleMetrum) and
+[TeleMini](http://altusmetrum.org/TeleMini) as those products
+are updated later this year. The new pyro circuit tolerates 0 ohms (dead
+shorts) on the pyro terminals for as long as the battery can provide
+current, which is as good as it gets. We think the circuit is clever enough
+that we'll probably write more about it once we're finished validating it.
+