what is the optimal false alarm rate for tornado warnings?

Last weekend, I patiently sat in my tiny downstairs bathroom with my three daughters and a laptop during two sequential tornado alarms. We followed the storm on twitter until it was safe to emerge. My husband didn’t fit in the bathroom, so he kept his eyes and ears ont he storm outside, occasionally complaining about high tornado warning false alarm rates.

My husband and I both have roots in the Midwest and are used to frequent tornado warnings. In Illinois, where sirens sounded during every tornado warning. Before I had kids, I usually ignored the alarms unless it looked or sounded ominous outside. I take shelter these days, at least when I know about the alarms.  We don’t have tornado sirens in Virginia, so we often miss them. This makes me uneasy, since there are a lot of tornadoes in Virginia, even though Virginia isn’t in “tornado row.” But TV warnings and social media are usually sufficient for alerting me during warnings.

Tradeoffs.Tornado warning are supposed to be conservative, meaning that the false alarm rate is high in order to ensure a low false negative rate. The false negatives–the alarm not signaling before a tornado strikes–can be deadly. I’m OK with a lot of false alarms so long as I can keep my kids safe. There are social costs to have too many false alarms in order to drive the false negatives to zero. Namely, we will all have in our storm shelter during a perpetual tornado warning. There are social costs with having a reasonable number of tornado warnings. The main issue here is that false alarms are akin to “crying wolf”–tornado warnings are eventually ignored, leading to people being in danger when there really is a tornado.

It turns out that there is some research on this topic.

Error rates. First, I would like to note that there is about a 7% false negative rate. It’s not as low as I would like, but it’s certainly quite low considering the rarity of tornadoes. No system has perfect sensitivity and specificity. Now let’s discuss false alarms. The National Weather Service defines a false alarm ratio (FAR):

FAR = unverified warnings / (verified warnings + unverified warnings)

A paper by Kevin Simmons and Daniel Sutter in the Weather, Climate, and Society (an American Meteorological Society journal) studied areas with lower and higher false alarm rates. They found:

A statistically significant and large false-alarm effect is found: tornadoes that occur in an area with a higher false-alarm ratio kill and injure more people, everything else being constant. The effect is consistent across false-alarm ratios defined over different geographies and time intervals. A one-standard-deviation increase in the false-alarm ratio increases expected fatalities by between 12% and 29% and increases expected injuries by between 14% and 32%. The reduction in the national tornado false-alarm ratio over the period reduced fatalities by 4%–11% and injuries by 4%–13%. The casualty effects of false alarms and warning lead times are approximately equal in magnitude,  suggesting that the National Weather Service could not reduce casualties by trading off a higher probability of detection for a higher false-alarm ratio, or vice versa.

The April 27, 2011 tornado outbreak in Alabama cost many lives and resulted in some discourse on the false alarm rate and what people are willing to live with. A meteorologist from Alabama argues for fewer tornado alarms:

I firmly believe apathy and complacency due to a high false alarm ratio over the years led to inaction in many cases that could have cost lives.

The FAR (false alarm ratio) for many NWS offices when it comes to tornado warnings is in the 80-90 percent category. I say this is simply not acceptable. Sure, the POD is excellent (probability of detection), but if most of the warnings are bad, then what good is a high POD?

A blog post in the Washington Post further discusses the perils of high false alarm rate. The problem, however, has to do with how people react in response to false alarms, not with the high false alarm rates. I do wonder if we should be optimizing people instead of trying to optimize the false alarm rate. Public awareness campaigns could be helpful here. I don’t know how worthwhile this would be, since I have endured hundreds of tornado alarms and ignored many of them in my youth despite the many public service announcements in the Midwest. Perhaps TV stations could frequently air the movie Twister instead.

Discussing false alarms is silly if no one actually knows about the warnings. I have accidentally ignored tornado alarms when I am unplugged due to a lack of sirens in Virginia. My university doesn’t have tornado sirens. After the 2007 Virginia Tech shootings, a campus alert system was installed. A siren was installed to warn of campus dangers. The alarm sounds exactly like a tornado siren. It is even tested once a month, just like the tornado sirens in the Midwest (first Tuesday of the month at 10am in Champaign-Urbana, IL). I wondered why the siren couldn’t be dual purpose, since a back of the envelope calculation suggested that my risk of dying in a tornado on campus was much higher than my risk of during in a mass shooting event on campus. Someone else must have drawn the same conclusion, because the campus siren was used to signal a tornado warning on campus last week (yet another tornado alarm last week). The only problem then was that Virginians didn’t know what to do or where to go during a tornado warning (review what to do here). In this day and age, tornado warnings can be easily sent via text message or via social networking, meaning that the perceived false alarm rate could actually be higher than what it used to be.
Do you take shelter during tornado warnings? How would you balance tornado warning true and false alarms?

8 responses to “what is the optimal false alarm rate for tornado warnings?

  • prubin73

    Post 2007, MSU installed an SMS warning system that covers incidents of violence, tornadoes and probably other calamities — so I now get SMS tornado warnings. (It’s the midwest, so we also have sirens, TV news etc.) I don’t react to tornado “watches”; if it’s a “warning”, I check for sightings, likely paths etc. and stay indoors (but don’t automatically head for the bathtub). I’m not sure that better education (or endless reruns of Twister) will offset the “cried wolf” effect, though. Perhaps more alert levels, and more specific geographic targeting, could help? (“If you live in Hooterville, [stay close to home | stay indoors | head for the basement or bathroom | DUCK!!!].”)

  • Dr Nic

    This is really interesting and a great example of applied probability to use in teaching. We don’t get tornadoes here, but we do currently get earthquakes. These come without warning, which has the advantage of no false alarms, but the disadvantage that we have an entire city on edge waiting for the next one.

  • Jeffrey W. Herrmann

    We have a weather radio that sounds a scary alarm when a tornado warning is issued for our area (the radio is programmed with a specific code for our county). Thus, we are aware of any tornado warning for our county. But in 16 years we’ve never had a tornado go through our small town (for which I am very thankful)! So our conditional tornado risk (given a tornado warning) seems to be very low.

    So, when we get a tornado warning, we turn on the radio and the TV and try to find out precisely where the worst weather is and where it is going. It is almost always going somewhere else.

    Still, I want that weather radio to wake me up just in case.

    I did hear during a recent spate of tornado warnings that newer weather radar can identify even smaller tornados (or tornadic activity), so we may have more warnings in the future.

  • Tornado False Alarms | Centives

    […] To read more including how McLay’s compliance changed with time, the role of sirens, the importance of educating people about what to do when they hear a siren, the chance of dying in a tornado versus being shot in a mass shooting event on one college campus, how social networks can change things, and other interesting details about the issue, click here. […]

  • Another linkdump « God plays dice

    […] Laura McLay discusses the optimal false alarm rate for tornado warnings. […]

  • Which type of error do you prefer? | Learn and Teach Statistics and Operations Research

    […] McLay wrote a very interesting post entitled. “what is the optimal false alarm rate for tornado warnings?” . A high rate of false alarms is likened to the “boy who cried wolf”, to whom nobody […]

  • w9lw

    Your blog leads this reader to wonder if you are aware that a weather alert radio will inform you of warnings even when you are “unplugged” and that sirens are not designed to alert people who are indoors.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: