I taught multiobjective decision analysis (MODA) this semester. It is a lot of fun to teach. I always learn a lot when I teach it. One of the most enjoyable parts of the class (for me at least!) is to run a class project that we chip away at during class over the course of the semester. Our project is to find the best restaurant for us to celebrate at the end of the semester. “Best” here is relative to the people in the class and the .
The project is a great way to teach about the MODA process. The process not only includes the modeling, but also the craft of working with decision makers and iteratively improving the model. It’s useful for students to be exposed to the entire analysis process. I don’t do this in my other classes.
On the first day of class, we came up with our objectives hierarchy. I did this by passing out about five Post It notes to each student. They each wrote one criteria for selecting a restaurant on each Post It note. They stuck their Post It notes to the wall. Together, we regrouped and organized our criteria into an objectives hierarchy. Some of the objectives because “weed out criteria,” such as making sure that the restaurant could accommodate all of us and comply with dietary restrictions.
Our initial criteria were:
- Distance
- Quality of food
- Variety of food
- Service: Fast service
- Service: Waiting time for a table
- Service: Friendly service
- Atmosphere: Noise level
- Atmosphere: Cleanliness
- Cost
Our final criteria were as follows (from most to least important):
- Quality of food
- Cost (tie with #3)
- Distance
- Fast service (tie with #5)
- Noise level
- Cleanliness
We removed variety of food, waiting time, and friendly service because classroom discussions indicated that they weren’t important compared to the other criteria. Variety, for example, was less important if we were eating delicious food at an ethnic restaurant that had less “variety” (variety in quotes here, because it depends on you you measure it).
In the next few weeks, we worked on identifying how we would actually measure our criteria. Then, we came up with a list of our favorite restaurants. During this process, we removed objectives that no longer made sense.
We collaboratively scored each of the restaurants in each of the six categories by using a google docs spreadsheet.
- Quality of food = average score (1-5 scale)
- Cost (tie with #3) = cost of an entree, drink, tax, and tip
- Distance = distance from the class (in minutes walk/drive)
- Fast service (tie with #5) = three point scale based on fast service, OK service, or very slow service
- Noise level = four point scale based on yelp.com ratings
- Cleanliness: based on the last inspection. Score = # minor violations + 4*# major violations.
A real challenge was to come up with:
- the single dimensional value functions that translated each restaurant score for an objective into a value between 0 and 1.
- the weights that balanced our preferences across objectives using swing weight thinking. FYI, we used an additive model.
I won’t elaborate on these parts of the process further. Ask me about these if you are interested.
When we finished our model, the “best” decision was to forego a restaurant and do a potluck instead. No one was happy with this. We examined why this happened. This was great: ending up with a bad solution was a great opportunity for learning. We concluded that we didn’t account for the hidden costs associated with a potluck. Namely, it would entail either making a trip to the grocery store or cooking, approximately a 30 minute penalty. We decided that this was equivalent to driving to a distant restaurant, a 26 minute drive in our model. It was also hard to evaluate cleanliness since the state do not inspect classrooms like they do restaurants. But since cleanliness didn’t account for much of our decision, we decided not to make adjustments there.
The final model is in a google docs spreadsheet.
We performed a sensitivity analysis on all of the weights. Regardless of what they were, most of the restaurants were dominated, meaning that they would not be optimal no matter what the weights were. The sensitivity was not in google docs, since we downloaded the document and performed sensitivity on our own. I show the sensitivity wrt to the weight for quality below. The base weight for quality is 0.3617. When the weight is zero and quality is not important, Chipotle would have been our most preferred restaurant. The Local would be preferred only across a tiny range.
We celebrated in Ipanema, a semi-vegetarian restaurant in Richmond. I think our model came up with a great restaurant. We all enjoyed a nice meal together. Interestingly, Mamma Zu scored almost identically to Ipanema (see the figure below).
I cannot claim credit for this fun class project. I shamelessly stole this idea from Dr. Don Buckshaw, who uses it in MODA short courses. We use the Craig Kirkwood’s Strategic Decision Making as the textbook for the course. I also recommend Ralph Keeney’s Value Focused Thinking and John Hammond’s Smart Choices.
How do you choose a restaurant?
- Sensitivity with respect to the weight for quality (0.3617 in the base case).
Punk Rock Operations Research 
May 29th, 2012 at 9:41 am
I know very little about MODA because it seems like a solution technique in search of a problem. Your classroom example does nothing to diminish my concern: I suspect that a talented group facilitator with no training whatsoever in any OR skills, or even without any applied math background whatsoever, could have determined the “solution” your classroom came up with, and with as much or more “fun” as you all enjoyed. (Indeed, it seems that being a group facilitator is essential to the successful implementation of MODA.)
Can you, or anyone, give an example of a MODA problem in which a) the aforesaid facilitator couldn’t give as satisfying an answer, through as satisfying a process, and b) MODA gives a satisfying answer?
Else, why should MODA be part of an OR analyst’s tool-kit?
Or should group facilitation be part of an analyst’s training (as, say, part of client management?)
May 29th, 2012 at 2:19 pm
Rick, you bring up some good points. I’ll try to answer your questions.
(1) The classroom activity is indeed silly. We don’t need to build a model for that. But in most classes, we don’t really teach students how to solve problems from beginning to end. We teach them about a methodology in depth without discussing the entire lifecycle of how to use each method (although teaching facilitation wouldn’t make sense in most classes). Our students have few opportunities to work with a decision-maker on how to model the problem, how to adjust the model when assumptions are wrong, how to iterate to find a good solution, etc. They get a taste of it in this class. In other classes, I like students to present their research findings to get used to communicating quantitative results to other people.
(2) I’m not sure that MODA is “a solution technique in search of a problem.” It seems to work quite well in big decisions that effect many people (think of the class decision context of where to locate an answer). I’ve seen it used in government decision-making quite nicely. The students discussed MODA papers in class, but since they were written by the OR crowd, I have no idea how satisfying the process or solution was for the customers. Some papers did discuss how customers felt about the process, but I do not remember what they are off the top of my head. This paper seemed to give a satisfying answer:
http://www.jstor.org/stable/10.2307/2631285
(3). I do not do research in MODA, so I’m probably not the best person to answer your questions. I know almost nothing about group facilitation. I have gotten the impression that MODA has its place alongside other OR methodologies. In addition, MODA is valuable in the classroom, since it has introduced me to utility theory and risk analysis, which I would otherwise not have learned about.
May 30th, 2012 at 12:32 am
I use Saaty’s AHP all the time; as a group facilitation technique. My work (a state governmental entity) uses “quality points” to scale procurement bids to allow high-cost, high-quality suppliers the ability to compete. Under this process, suppliers are asked questions and their answers are scored. Suppliers are given a total quality point value which is used to scale their cost. Prior to my involvement in the process, Quality Points were generated with little thought or scrutiny. The resulting scores were averaged between all scorers and an irrational compromised score was generated. We then did all sort of unhealthy data manipulations on the compromised score and only by luck did the final result support the strategy for the initiative.
I use AHP to apply concepts like swing weights, naturally existing scales, independent dimensions, and a repeatable process. I reduced the time to quality point development from months to days and actually end-up with results that are palatable by my teams; even after the initiative is complete and all information is known. My teams couldn’t care less whether I use a power iteration to calculate eigenvalues or pull them randomly from my hat. But they take solace in the psudo-math nature and readily accept the final weightings. Ultimately, the process of discussion and facilitation drives value, but the results of a finalized weighting inline with the “general sense” of the team expects hammers home the acceptance.
I think you are both right… A super facilitator could probably get similar results. But MODA and AHP is a facilitation technique that I have found effective in getting the result I want from the teams I lead; namely integration and rigor.
June 4th, 2012 at 11:27 am
Thanks for this great example – it is not only a useful example of MODA but also a great example of how to teach. I plan to try this (or something similar) when I teach my decision-making course next spring.
June 16th, 2012 at 6:24 am
It’s so key to career preparation to have an exercise in running the analytical process all the way through. The kind of isolated problems that students are given pre-defined don’t relate well to real world situations. The only way that I’d expand this to display reality is to gather the objectives and weights from students outside of the class, and then explain the reasoning for the results to them. Communication of an analytical technique to non-analysts is an important skill to have.