Monthly Archives: December 2009

a quantitative analysis of a teaching experiment

I decided to try an experiment in my class this semester. Since some of my students aced both midterms and since others bombed one of their midterms (not all of whom deserved their low grades) I decided to offer a choice to my students. They could either:

  1. get a “do over” on their lowest midterm score and replace it with their final exam score, thus increasing the weight of the final to 50% of their grade (even if they do worse on the final),
  2. not take the final exam and receive a final grade with certainty based on their other grades,
  3. take the final as is (with the final accounting for 30% of their grade).

The goal was twofold. First, I wanted to give each student the grade that they deserved. Some students had a low midterm score (there were many legitimate medical emergencies this semester, partly due to swine flu). Second, I wanted the students to actually use the material we learned this semester to make a tough decision under uncertainty.

I’m not sure I achieved my goals, but here are a few observations:

  • 28/44 decided not to take the final exam.
  • Of course, all of the A students opted out of the final, but a few others opted out as well.  8/11 B students, 3/6 C students, and 1/4 D students also opted out of the final. I didn’t like this, but these students took a systems perspective and opted for more time to study for their other finals (these letter grades are based on the grades going into the final exam).
  • 14/16 students taking the final replaced a midterm score (option #1 above). One asked about replacing both midterms with the final (with the final then counting for 70% of the final grade), which I allowed and offered to the rest of the class.  In the end, the others were deterred by the possible risk of this choice.
  • Two students took the final as originally planned. These students were typically close to the a grade cutoff and did not have much to gain from the first two policies, so they chose the status quo.
  • Two students asked me for “favors” to change their grade. Of course, this would not be fair to the other students. However, when I made changes to the grading policy, it somehow gave the students permission to come up with their own grading schemes that, of course, benefitted themselves to a great degree.
  • All the time I saved from grading finals was more than made up for with extra discussions with students and extra bookkeeping.  But I enjoyed how open students became about talking about their final grade with me before the final.  I started a dialog where students were encouraged to talk about their grade concerns.  I often have trouble getting the students at the bottom of the curve to come in and talk to me before the semester ends.  This really did the trick.
  • Did this experiment actually work?  Of course, I can never know how the students who opted out of the final would have done.  I computed the grades of those who took the final and compared them to their grades if they chose another one of the options.  Of the 16 students taking the final, five ended up with a better letter grade than if they had opted out of the final (one of whom improved by two letter grades).  Of the fourteen students that replaced a midterm score, four ended up with a better letter grade had they opted to choose the status quo (and none did worse).  One of the two students who selected the status quo, however, would have improved by a letter grade if they had opted to replace a midterm score.
  • The most amazing thing is how responsible the students became. I have never seen such conscientiousness in a large group of undergraduates.  By offering a choice to the students, it put the ball in their court.  They rose to the challenge and accepted the amount of control they exerted over their own grades.  I heard a lot more accountability for grades in the language that they used (“If I score at least an 82 on the final, I can get a B…” rather than “Would you give me a B if I scored an 82…?”).

If I do something like this again, I will likely make some changes in the offer that I make to the students.  And although I wish all of my C and D students took the final, I was happy with the vast majority of the student decisions as well as my interactions with them.  The students who needed to take the final really thought about making a tough decision under uncertainty.  That was what I was trying to teach them in the first place.


How many people get arrested?

I recently wrote about how OR can be used to determine how employer criminal background checks could be conducted. This is a follow up post.

An excellent article by Carl Bialik in the WSJ summarizes arrest patterns in the United States. It reports that according to a report from the President’s Commission on Law Enforcement and Administration of Justice, 52% of American men will be arrested in their lifetime. This is consistent with the number that Al Blumstein reported in the 1960s.

Both then and now, the researchers who came up with this number were surprised that so many men get arrested. For a law-abiding citizen, it is hard to understand how so many people are arrested. The image from Bialik’s article (displayed below) helps in that regard. Note that the proportion of men that have been arrested (approximately 40%) is lower, since some men have not yet been arrested for the first time yet. The proportion of men who will be arrested in four times higher than the proportion of women that will be arrested.

Bialik explores the 52% claim in greater detail in his blog.

Let’s replace the BCS system…using OR

I did not follow college football this year like I should have, mainly because my team had a very disappointing season.  I checked the college football rankings yesterday and was surprised to see that there were five undefeated teams this year.  Two of these teams (Alabama and Texas) will get to play for the national championship and the other three (Cincinnati, TCU, and Boise State) basically get a consolation prize.

The reason for this is that the Bowl Championship Series (BCS) was designed for picking out the best two teams.  It uses a complex algorithm based in part on national rankings, difficulty of schedule, and win-loss records.  It is tweaked periodically.  Almost everyone likes to beat up on the BCS system.  The BCS algorithm seems to work fine.  It ends up with rankings almost identical to the consensus of experts (with a few notable exceptions).  The problem is more in how it is used.  It is used to pick the top two teams, assuming that the rest should not have a chance at the national championship.  This year, it seems like at least three other teams deserve a shot at the national championship.

The announcement of the bowl games were announced last night signaled the start of the complaining-about-the-BCS system.  The most common proposal for reforming the BCS is starting an eight team playoff (even our President endorses this idea).  Can OR play a role in reforming the BCS?

Wayne Winston’s Huffington Post article analyzes the BCS issue from an OR point of view.  He weighs the pros and cons of the current BCS system (debunking the pros along the way).  The best part is when he describes what an eight-team playoff would look like. This is important, since we all know what the problem is, but we need to know what the new system would look like if we were to endorse it.

Using power ratings for each year we took the two BCS selected teams and the next 6 ranked teams and used the Palisade Excel Monte Carlo simulation add-in @RISK to play out an 8 team tournament 5000 times. We seeded the BCS selected teams 1 and 2 in the tournament, so they received the easiest possible road to the title game. We found that one of the BCS’ top two teams won only 50% of the time. Even in 2005 when nobody doubted that USC and Texas should have played for the title, there was a 23% chance that both these teams would have been knocked off in an 8 team tournament. So how can the BCS claim they are crowning a legitimate champion?

Link:  Read Wayne Winston’s article.

one more link

Yesterday, I posted a list of links that I shared with my class this semester.  I left one off.  I have quite a few pre-med students, so I like to talk about medical applications when necessary.  I shared a brief history of evidence-based medicine with my students, as conveniently summarized in a Business Week article.  The article is about David Eddy, the medical doctor turned operations researcher who transformed how medicine is practiced by using more math.  The article begs for more OR to be applied to health care applications:

The human brain, Eddy explains, needs help to make sense of patients who have combinations of diseases, and of the complex probabilities involved in each.

The article describes many of the challenges in the medical domain as well as some of the benefits of using advanced analytical methods for approaching medicine.

[Eddy’s] PhD thesis made front-page news in 1980 by overturning the guidelines of the time. It showed that annual chest X-rays and yearly Pap smears for women at low risk of cervical cancer were a waste of resources, and it won the most prestigious award in the field of operations research, the Frederick W. Lanchester prize. Based on his results, the American Cancer Society changed its guidelines.

The recent changes in how we screen for prostate cancer and breast cancer are part of Eddy’s legacy.  They are controversial, but few medical treatments have been proven in clinical trials (the article estimates this could be as low as 20%).


Land O Links

I have blogged a few times about the course that I am teaching this semester (intro prob and stats for engineers).  It’s a lot of fun to teach.  I enjoy sharing some stories about statistics in the real world.  I am posting a short list of the links I shared with my students this semester:

  • When teaching Bayes rule, I took a tangent to discuss how statistics is important for understanding the health insurance debate.  I tried to make the point that preventative health care never saves money (as some were claiming earlier in the year), since the basic premise is that many people are screened to treat just a few for disease.  I referred my students a Shields & Brooks interview on the PBS New Hour.  David Brooks explains this concept better than I can. provides a nice analysis on their Truth-O-Meter.
  • Before Election Day, we discussed how many people are driven by their civic duty to vote, particularly in years when there is a Presidential Election.  With the additional drivers on the road, there are additional risks.  A JAMA article indicates that these risks are statistically significant, when comparing election traffic to a set of equivalent control days.  See the image below.
  • Statistics is aimed at using small samples to make inferences about an entire population.  Sometimes, samples are too small (the so-called Law of Small Numbers).  Wayne Winston writes about the significance of small samples in analyzing NBA data.  A great defense of small samples.
  • A colleague shared an interesting link with me about how the max heartrate formula was devised.  You have seen this formula that appears everywhere (Max heartrate = 220 – your age, a linear model).  Sadly, this formula is not based on a rigorous statistical analysis.  A doctor literally eyeballed it, and drew a line through a few points. Sadly, this poor implementation of linear regression is widely used, which on some level undermines what I was teaching in class.
  • Richard Florida of The Atlantic wrote an article about the geography of obesity using linear regression, using state-level obesity data.  His article has many nifty images, one of which is reproduced below.
Individuals in Fatal Crashes During Presidential Elections

Individuals in Fatal Crashes During Presidential Elections. Data are counts of individuals in the crashes (in which not all persons necessarily died) during polling hours (8:00 AM to 7:59 PM, except where noted with alternative hours). Because 2 control days are available for each election day, expected deaths were calculated as total control deaths divided by 2. CI indicates confidence interval.