Tag Archives: weather

Type II errors are the ones that get you fired: the Atlanta edition

A couple weeks ago, a comment on twitter reminded me about Type I and Type II errors, which in turn reminded me of my first introduction to Type II errors in a probability and statistics course as an undergraduate student.

A type I error is the incorrect rejection of a true null hypothesis.

A type II error is the failure to reject a false null hypothesis.

Type I and Type II errors are a little confusing when you are first introduced to them. To make things easier, my professor gave us some practical information about Type II errors to help us put them in perspective:

And that brings us to the mess that was Atlanta this past week. If you recall, about 3″ of snowfall iced over, leading to mayhem. Schools and government did not shut down before the storm. Instead, they all closed at the same time, leading to an incredible amount of congestion that overwhelmed the impaired transportation network. Cars were abandoned on the highway and students camped out at school and in grocery stores for the night (presumably, everyone was stopping for bread and milk on the way home). I recommend these two articles from the Atlantic Cities to see just how bad it was in Atlanta [Link and Link]. Here is a time-lapse of the highways in Atlanta. Traffic went from fine to a disaster in an hour:

In this case, the weather itself was not a disaster. Poor management of the situation led to a disaster. (That almost sounds like it should be a bumper sticker: Weather doesn’t make disasters, people make disasters! At other times, the weather really is a disaster) David Levinson, a civil engineering professor at the University of Minnesota (a former Atlanta inhabitant, find him at @trnsprttnst) wrote an excellent piece on CNN about his perspective [Link]. I don’t have a whole lot to add except that managing the effects of severe weather has been and will continue to be a big issue in operations research (and civil engineering too).

  • Should you instead try to mitigate the ice by investing in salt and trucks to prepare the roads? This is not very practical in the South where it rarely snows.
  • Do you always play it safe and close schools? I lived that way in Virginia, and while it is safer, the Type I errors aren’t ideal. One year, school was canceled 5 days for a sum total of 1″ of snow across all of the days of canceled school.
  • If you decide not to close schools and later change your mind, should you stagger the closures? Yes. This is critical in congested cities like Atlanta and DC.

Related posts:

Many of my readers are from or have lived in Atlanta. What is your take?



the conditional probability of being struck by lightning (Part 4)

The National Weather Service released a report on being struck by lightning [Story here] that may be of interest to those of you read my previous posts on lightning (see below). The study confirms that men account for 82% of those struck by lightning, which I’ve blogged about before. More people are struck by lightning in July than in any other month.

Lightning fatalities by month

Lightning fatalities by month

NOAA states that June, July and August have the most lightning in addition to having the most lightning fatalities, In fact, I was caught in a severe storm that rolled in quickly when I was running in a forest preserve in June. I suspect that the conditional probability of being struck by lightning doesn’t depend on seasonality as much as the picture above suggests, but I don’t have any proof.

The Washington Post Magazine ran a nice story about the ‘Spark Ranger’ Roy Sullivan who was struck by lightning seven times [Link]. A man who worked outside, Roy Sullivan was at an increased risk of getting struck by lightning. I’m not sure how that happened seven times.

For more reading:

what is the optimal false alarm rate for tornado warnings?

Last weekend, I patiently sat in my tiny downstairs bathroom with my three daughters and a laptop during two sequential tornado alarms. We followed the storm on twitter until it was safe to emerge. My husband didn’t fit in the bathroom, so he kept his eyes and ears ont he storm outside, occasionally complaining about high tornado warning false alarm rates.

My husband and I both have roots in the Midwest and are used to frequent tornado warnings. In Illinois, where sirens sounded during every tornado warning. Before I had kids, I usually ignored the alarms unless it looked or sounded ominous outside. I take shelter these days, at least when I know about the alarms.  We don’t have tornado sirens in Virginia, so we often miss them. This makes me uneasy, since there are a lot of tornadoes in Virginia, even though Virginia isn’t in “tornado row.” But TV warnings and social media are usually sufficient for alerting me during warnings.

Tradeoffs.Tornado warning are supposed to be conservative, meaning that the false alarm rate is high in order to ensure a low false negative rate. The false negatives–the alarm not signaling before a tornado strikes–can be deadly. I’m OK with a lot of false alarms so long as I can keep my kids safe. There are social costs to have too many false alarms in order to drive the false negatives to zero. Namely, we will all have in our storm shelter during a perpetual tornado warning. There are social costs with having a reasonable number of tornado warnings. The main issue here is that false alarms are akin to “crying wolf”–tornado warnings are eventually ignored, leading to people being in danger when there really is a tornado.

It turns out that there is some research on this topic.

Error rates. First, I would like to note that there is about a 7% false negative rate. It’s not as low as I would like, but it’s certainly quite low considering the rarity of tornadoes. No system has perfect sensitivity and specificity. Now let’s discuss false alarms. The National Weather Service defines a false alarm ratio (FAR):

FAR = unverified warnings / (verified warnings + unverified warnings)

A paper by Kevin Simmons and Daniel Sutter in the Weather, Climate, and Society (an American Meteorological Society journal) studied areas with lower and higher false alarm rates. They found:

A statistically significant and large false-alarm effect is found: tornadoes that occur in an area with a higher false-alarm ratio kill and injure more people, everything else being constant. The effect is consistent across false-alarm ratios defined over different geographies and time intervals. A one-standard-deviation increase in the false-alarm ratio increases expected fatalities by between 12% and 29% and increases expected injuries by between 14% and 32%. The reduction in the national tornado false-alarm ratio over the period reduced fatalities by 4%–11% and injuries by 4%–13%. The casualty effects of false alarms and warning lead times are approximately equal in magnitude,  suggesting that the National Weather Service could not reduce casualties by trading off a higher probability of detection for a higher false-alarm ratio, or vice versa.

The April 27, 2011 tornado outbreak in Alabama cost many lives and resulted in some discourse on the false alarm rate and what people are willing to live with. A meteorologist from Alabama argues for fewer tornado alarms:

I firmly believe apathy and complacency due to a high false alarm ratio over the years led to inaction in many cases that could have cost lives.

The FAR (false alarm ratio) for many NWS offices when it comes to tornado warnings is in the 80-90 percent category. I say this is simply not acceptable. Sure, the POD is excellent (probability of detection), but if most of the warnings are bad, then what good is a high POD?

A blog post in the Washington Post further discusses the perils of high false alarm rate. The problem, however, has to do with how people react in response to false alarms, not with the high false alarm rates. I do wonder if we should be optimizing people instead of trying to optimize the false alarm rate. Public awareness campaigns could be helpful here. I don’t know how worthwhile this would be, since I have endured hundreds of tornado alarms and ignored many of them in my youth despite the many public service announcements in the Midwest. Perhaps TV stations could frequently air the movie Twister instead.

Discussing false alarms is silly if no one actually knows about the warnings. I have accidentally ignored tornado alarms when I am unplugged due to a lack of sirens in Virginia. My university doesn’t have tornado sirens. After the 2007 Virginia Tech shootings, a campus alert system was installed. A siren was installed to warn of campus dangers. The alarm sounds exactly like a tornado siren. It is even tested once a month, just like the tornado sirens in the Midwest (first Tuesday of the month at 10am in Champaign-Urbana, IL). I wondered why the siren couldn’t be dual purpose, since a back of the envelope calculation suggested that my risk of dying in a tornado on campus was much higher than my risk of during in a mass shooting event on campus. Someone else must have drawn the same conclusion, because the campus siren was used to signal a tornado warning on campus last week (yet another tornado alarm last week). The only problem then was that Virginians didn’t know what to do or where to go during a tornado warning (review what to do here). In this day and age, tornado warnings can be easily sent via text message or via social networking, meaning that the perceived false alarm rate could actually be higher than what it used to be.
Do you take shelter during tornado warnings? How would you balance tornado warning true and false alarms?

what is the conditional probability of being struck by lightning? Part 3

This is my third post on the conditional probability of being struck by lightning in response so an NPR article (HT to Tim Hopper). The National Weather Service tweeted:

More than 80% of lightning victims are male. Be a force of nature by knowing your risk, taking action and being an example #ImAForce

This suggests that the conditional probability of being struck by lightning depends on your gender. You might think that the conditional probability of being struck by lightning for a man is four times higher than four a woman. Not so fast.

In the NPR article, Susan Buchanan, a spokeswoman from the National Oceanic and Atmospheric Administration offered four explanations for why. They are quoted below with some minor changes from me. The first two explanations would affect the prior probability of being exposed to a thunderstorm:

1.  Men are more likely to be outside.

2.  More are more likely to have jobs that require them to work outside.

The second two would affect the conditional probability of being struck by lightning given one’s gender and that he/she is in a thunderstorm.

3. Men take more risks than women. “If you look at the percentage of men who take part in high risk sports that might give you an idea,” said Buchanan. Therefore, a man would be less likely to go inside during a thunderstorm.

4.  Men don’t want to be seen as “wimps.” This theory, she said, was backed up by talking to the Boy Scouts who said no one wants to be the one to say it’s time to go inside.

Putting this together:

1 < [Conditional probability of being struck by lightning given that one is a man and is in a thunderstorm] / [Conditional probability of being struck by lightning given that one is a women and is in a thunderstorm] < 4.

For more reading:

what is the conditional probability of being struck by lightning? Part 2

My brother-in-law is a meteorologist and was happy to chat with me about the conditional probability of being struck by lightning (see my previous post here):

However, I can say right off the bat that there isn’t going to be one right answer.  There is substantial variation in both the microphysics and dynamics of moist convective systems.  As a result, there is wide variation in the lightning frequency from different storms.  Take, for example, so-called MCS’s (Mesoscale Convective Systems) that originate on the Front Range of the Rockies and propagate eastward in the nighttime hours, eventually affecting places like MSP, MSN, or ORD in the morning hours.  These are profligate lightning generators, often producing essentially continuous lightning.  Meanwhile, a completely different animal from MCS’s are baroclinic frontal convective systems.  Often these will pass through with only occasional lightning (occasional lightning is officially defined by the weather service as no more than once every 2 minutes).  Then, perhaps most interesting of all, there are some storms that generate copious cloud-to-cloud or cloud-to-air lightning but very little cloud-to-ground lightning.  The specific reasons for the wide variation in lightning frequency and type are still the subject of research (we have generally poor observations of microphysical processes, especially those processes in violent convective storms).

So, the type of storm will affect your conditional probability of being struck by lightning.

My brother-in-law also helped me to compute the probability of being struck by lightning conditioned on whether there is a storm.

P(L|S) = P(S|L) P(L) / P(S)
L = event of getting struck by lightning
S = event that there is a storm
We need to estimate values for P(S|L), P(L), and P(S).
P(S|L) is easy; It’s 1 (you can’t have lightning without a storm).
Estimates for P(L) and P(S) can be obtained from the public sphere.
A little searching reveals that ~ 400 people per year are struck by lightning.  If we assume the strike incidents are independent and uniformly distributed across the 365 days of the year, then the number of people struck on any given day is roughly 400/365. An estimate for P(L) for any given day is then the number of people struck on any given day divided by the total US population.  Using 310 million as the US population we get P(L) for any given day is (400/365) / (310×10^6) = ~ 3.5 x 10^(-9).

An estimate of P(S) can be obtained from the so-called “thunderstorm-day” statistic, which is the number of days in some prescribed time interval in which thunder is heard at a given location (and hence a storm must have been within striking distance of that location).  A little searching reveals that the thunderstorm-days per year of Washington D.C. is 32.8, based upon the last 30 years of data.  If we assume the stormy days are independent and uniformly distributed across the 365 days of the year, then P(S) for any given day is 32.8 / 365 = ~.09.  However, this is not a reasonable assumption: the estimate of P(S) for Washington, D.C., for a given day in July is ~.25

Given the above:

P(L|S) = 1 * 3.5×10^(-9) / .09 = 3.9×10^(-8)

I tried to extend this analysis to compute the conditional probability of interest:

P(L | O, S)

where O = event that you are outside. When applying Bayes Rule and rearranging the probabilities, I eventually get to expressions that depend on P(L), such as this one:

P(L | O, S) = P(S | L, O) P(O | L) P(L) / (P(A | S) P(S) ) = P(L) / (P(A | S) P(S) )

(assuming that P(S | L, O) = P(O | L) = 1 for our purposes). However, this is problematic, since estimates for P(L) (like the one that my brother-in-law reports above) implicitly capture the likelihood that people are outside, P(L | O, S). This is circular reasoning.

If you can shed additional light on this problem, please leave a comment.

what is the conditional probability of being struck by lightning?

There is not nearly enough information available on the Internet on the conditional probability of being struck by lightning.

I think of the odds of being struck by lightning and conditional probability every time I am in a thunderstorm. The conditional probability of being struck by lightning is probably not 1 in 6. NOAA estimates that the (unconditional) probability of being struck by lightning is ~1 in 1,000,000 per year and ~1 in 10,000 over a lifetime. That’s of little solace when I’m out running a few miles from home when a thunderstorm rolls in.

Jeffrey Seth Rosenthal’s book Struck by Lightning does not exactly answer this question. He does, however, report the annual probability of being struck by lightning conditioned on where on lives. Below are the figures from his book on the most dangerous states, the safest states, and the most dangerous countries (at least in terms of being struck by lightning). The “Annual Rates” report the expected number of people struck by lightning per year per 100,000. This could be due to differences in the prior probabilities (some areas have more thunderstorms than others) or in attitudes toward risk in thunderstorms.

Any idea of what the conditional probability of being struck by lightning would be given that one is outside in a thunderstorm? I need to know how much to panic the next time a thunderstorm rolls in when I am running.

There is a part II, part III, and part IV to this series.

is there bias in predicting snowfall?

Given this week’s record blizzard in the Midwest—so far it’s the third largest snowfall in Chicago on record–I was curious about whether there is bias in predicting the amount of snowfall.  In other words, do forecasts tend to overpredict the amount of snowfall?  And if so, do we catch on and underprepare for snowstorms?

I found a paper by Bruce Rose from the Weather Channel (with Joseph Koval and Eric Floehr) that answers this question.  They report that for a small set of snowstorms from 2007-08, there is an inverse relationship between the intensity of a snowstorm and the reported level of over-prediction, meaning that lighter storms are more exaggerated while heavier storms are less exaggerated.  While they could not confirm the findings for a larger data set from the Weather Channel’s database, they write that there is a “marked tendency to over‐predict light events, but this trend is reversed with moderate and heavy snowfalls.”  This seems to be consistent with what I have observed in Virginia, where there is much ado about virtually no snowfall.

As for whether people over-prepare or under-prepare for storms, I’ll have to answer that another day.

Related posts:

Lake Shore Drive in the 2011 Chicago blizzard

Lake Shore Drive in the 2011 Chicago blizzard: A bad trip on LSD for these people

weather forecasts and low temperatures

One of my pet peeves is a weather forecast that doesn’t contain the current temperature. It has been colder than usual for the past 3+ weeks, and almost every day, the morning temperature outside is 5-10 degrees below the low temperature in the forecast. As I am checking the weather on TV, for example, the current temperature will be 16 with a low of 23 (See the picture below for an example of this). They can’t add an if statement to update the high and low temperatures for the day? Seriously.

Example of a broken weather forecast that doesnt contain the current temperature

Example of a broken weather forecast that doesn't contain the current temperature

I ran in a race this weekend. The forecast predicted a low of 28 degrees outside. But I ran 15K in 19 degree weather. As I was running and trying to distract myself from the fact that I couldn’t feel my face, I speculated that some aspect of large-scale meteorology algorithms resulted in bad low temperature forecasts. After all, weather forecasts seem to do a pretty good job predicting hot temperatures.

Amazingly, there is evidence to suggest that what I have been observing is in fact true! This blog post from ForecastAdvisor provides data showing that weather forecasts get worse as the temperature deviates more from normal. The key figure is illustrated below. Most interesting is that there is a bias toward accuracy in predicting hotter temperatures–forecasts for colder-than-normal weather are worse than hotter-than-normal weather!

Forecasted temperature error

Forecasted temperature error

Meteorologists: please explain this phenomenon in more detail! Read an old post about OR models for meteorology.